Unit Tests

From Open Babel

Open Babel now uses the "Test Anything Protocol" for test output. The TAP format is relatively user-friendly in its normal form, but can also be easily summarized by the Perl tool "prove" as indicated below:

aromatic......ok
atom..........ok
bond..........ok
cansmi........ok
conversion....ok
data..........ok
format........ok
formula.......ok
formalcharge..ok
inchi.........ok
inchi2........ok
internalcoord.ok
iterators.....ok
invalidsmarts.ok
invalidsmiles.ok
logp_psa......ok
math..........ok
mol...........ok
residue.......ok
rings.........ok
smarts........ok
smilesmatch...ok
unitcell......ok
cml...........ok
test-set......skipped
        all skipped: - roundtrip test set not found.
All tests successful, 1 test skipped.
Files=25, Tests=107586, 76 wallclock secs (36.76 cusr +  9.17 csys = 45.93 CPU)

Note that the "prove" command divides each individual test program (file) into a summary line, with "ok" or "not ok" depending on the success of the test. It also gives a list of any failed tests:

inchi.....FAILED test 1                                                      
        Failed 1/2 tests, 50.00% okay
inchi2....ok                                                                 
Failed Test Stat Wstat Total Fail  Failed  List of Failed
-------------------------------------------------------------------------------
inchi.pl                   2    1  50.00%  1
Failed 1/2 test scripts, 50.00% okay. 1/22 subtests failed, 95.45% okay.

Contents 1 Using the TAP Output Format 2 Adding Tests to Open Babel 2.1 Wrapper.sh 2.2 Generate.sh 3 More Details from Prove

Using the TAP Output Format

The TAP manual provides detailed instructions in what TAP test programs should output.

The main details are as follows:

• Either the first line or the last line should include "1..N" where N is the expected number of tests.
• Lines should start with "ok" or "not ok" depending on the result of a subtest (i.e., success or failure)
• Lines should ideally include a subtest number, e.g. "ok 4" which makes it easier to find failing tests in the output
• Anything after a "#" character is considered a comment and is essentially ignored

Considering the amount of text generated by tests and sent to std::cout, several C++ features should be considered:

1. Use std::ios::sync_with_stdio(false);. This minimizes synching between C++ and C-style output (like printf) which should be avoided.
2. Minimize use of std::endl and use "\n" instead. A call to std::endl includes a mandatory flush of output buffering, slowing the test.

The resulting tests will run 20-30% faster. Most tests currently in Open Babel follow these guidelines.

Adding Tests to Open Babel

To add tests to the suite, you must add the source code to the test/ directory and edit the Makefile.am to compile it. Follow the examples of tests like atom or smilesmatch or ask for further help.

There are two general "helper" scripts for running the Open Babel test suite:

• wrapper.sh
• generate.sh

Wrapper.sh

The wrapper script makes sure to set appropriate environment variables to be sure to use the correct version of the Open Babel library and data.

It then runs the entire test suite through the prove command, e.g.,:

prove atoms mols bonds residues conversion

One note is that some test programs expect command-line arguments (e.g., the file to test). In this case, additional shell or Perl scripts are used to provide single commands for the tests, e.g.:

File: aromatic.pl

#!/usr/bin/perl
use Env qw(TESTDATADIR);
# the TESTDATADIR is usually set by the Makefile, pointing to the files/ subdirectory containing test files

# Call the "aromatic" command with the test file
if (defined $TESTDATADIR) {
    system("./aromatic $TESTDATADIR/aromatics.smi");
} else {
    system("./aromatic files/aromatics.smi");
}

# If the program failed to execute, ignore the test
if ($? == -1) {
    print "1..0 skip because program would not run";
}

# Otherwise exit, returning the same value as the system() command
exit($? >>8);

Generate.sh

Much like the wrapper script, the generate script sets environment variables to use the correct version of Open Babel library and data.

It then runs any programs which generate a validation set. For example, the SMARTS and ring detection tests compare values calculated by the library to a set of expected values. This primarily ensures changes to the library do not break these features.

However, bug fixes to SMARTS matching or ring detection require updated validation files. Be sure to re-run generate.sh after such fixes.

More Details from Prove

The prove tool has a variety of useful testing features. The following flags can be passed along to prove via wrapper.sh

• --shuffle - Run the tests (i.e., each test program) in random order, in case hidden dependencies exist
• --debug - Provide information on what "prove" is doing
• --verbose - Show full output, including every "ok" or "not ok" line.

./wrapper.sh --verbose --shuffle