Adding a new test#

Tests allow us to maintain code quality, ensure that code is working, prevent regressions, and facilitate refactoring. Personally, I find that there is no better motivation for writing tests than knowing that that bug I fixed will stay fixed, and that feature I implemented will not be broken by others. As an open source developer, I never have enough time; tests ensure that what time I have is not wasted.

We can divide the existing tests into three classes, based on how they test the Open Babel codebase:

  1. Tests written in C++ that test the public API

  2. Tests written in Python that use the SWIG bindings to test the public API

  3. Tests written in Python that use the command-line executables for testing

Which type of test should you write? It doesn’t really matter - it’s more important that you write some type of test. Personally, I can more quickly test more if I write the test in Python, so generally I write and check-in tests of type (2) above; when I need to run a testcase in a debugger, I write a short test of type (1) so that I can step through and see what’s happening.

Running tests#

To begin with, we need to configure CMake to enable tests: -DENABLE_TESTS=ON. This adds the make test target and builds the C++ tests. For tests of type 3 (above), you will also need to enable the Python bindings: -DPYTHON_BINDINGS=ON -DRUN_SWIG=ON. Some tests are dependent on optional dependencies; if you don’t build with support for these, then the corresponding tests will not be run.

To actually run the tests, you can run the entire test suite in one go or run individual tests. To run the entire suite, use make test or ctest (note that you can use the -j option to speed up ctest). The ctest command also allows a single test or a list of tests to be specified, and in combination with -VV (verbose) may be useful to run an individual test. However, I find it more useful to run individual tests directly. Here is an example of how to run an individual test for each of the three types discussed earlier:

  1. test_runner regressionstest 1

    This will run test number 1 in regressionstest.cpp. Nothing will happen…unless the test fails. (test_runner is a testing harness generated by CMake.)

  2. python test\testbindings.py TestSuite.testAsterisk

    This will run the testAsterisk test in testbindings.py. This will write out a single dot, and some summary information.

  3. python test\testbabel.py testOBabel.testSMItoInChI

    This will run the testSMItoInChI test in testbabel.py.

The next few sections describe adding a new test of types 1 to 3. The same test will be added, a test to ensure that the molecular weight of ethanol is reported as 46.07.

Test using C++#

The easiest place to add new tests is into test/regressionstest.cpp. Look at the switch statement at the end of the file and pick a number for the test. Let’s say 260. Add the following:

case 260:
  test_Ethanol_MolWt();
  break;

Now add the value of 260 to test/CMakeLists.txt so that it will be run as part of the testsuite.:

set (regressions_parts 1 221 222 223 224 225 226 227 260)

Now let’s add the actual test somewhere near the top of the file:

void test_Ethanol_MolWt()
{
  OBMol mol;
  OBConversion conv;
  OB_REQUIRE(conv.SetInFormat("smi"));
  conv.ReadString(&mol, "CCO");
  double molwt = mol.GetMolWt();
  OB_ASSERT(molwt - 46.07 < 0.01);
}

The various assert methods are listed in obtest.h and are as follows:

  • OB_REQUIRE(exp) - This must evaluate to true or else the test will be marked as failing and will terminate. This is useful for conditions that must be true or else the remaining tests cannot be run (e.g. was the necessary OBFormat found?).

  • OB_ASSERT(exp) - This must evaluate to true or else the test will be marked as failing. In contrast to OB_REQUIRE, the test does not terminate in this case, but continues to run. This feature can be useful because it lets you know (based on the output) how many and which OB_ASSERT statements failed.

  • OB_COMPARE(expA, expB) - Expressions A and B must be equal or else the test fails (but does not terminate).

It is often useful to write a test that uses a checked-in testfile. Let’s do this for our example testcase. If you place a file ethanol.smi into test/files, then the following will read it using a convenience function provided by obtest.h.:

void test_Ethanol_MolWt()
{
  OBMolPtr mol = OBTestUtil::ReadFile("ethanol.smi")
  double molwt = mol.GetMolWt();
  OB_ASSERT(molwt - 46.07 < 0.01);
}

As well as ReadFile (which is convenient for single molecules), the OBTestUtil struct provides GetFilename which will return the full path to the testfile, if you wish to open it yourself.

Test using a command-line executable#

At the command-line we can calculate the molecular weight of ethanol as shown below. We are going to do something similar using the Python test framework:

> obabel -:CCO --append MW -otxt
46.0684

Open test/testbabel.py in an editor. I have grouped tests related to the obabel executable into a class testOBabel, so let’s add a new test there. Somewhere in that class (for example, at the end), add a function such as the following (note: it must begin with the word “test”):

def testMolWtEthanol(self):
    """Check the molecular weight of ethanol"""
    self.canFindExecutable("obabel")
    answers = [
            ("CCO", 46.07),
            ("[H]", 1.01),
            ("[2H]", 2.01),
            ]
    for smi, molwt in answers:
        output, error = run_exec('obabel -:%s --append mw -otxt' % smi)
        my_molwt = round(float(output), 2)
        self.assertEqual(my_molwt, molwt)

We provide a few convenience functions to help write these tests. The most important of these is run_exec(command) which runs the commandline executable returns a tuple of stdout and stderr. Behind the scenes, it adds the full path to the named executable. In the example above, run_exec(stdin, command) took a single argument; the next example will show its use with two arguments - the additional argument is a string which is treated as stdin, and piped through the executable.

In the previous example, each SMILES string was passed in one-at-a-time. However, it is more efficient to do them all in one go as in the following example:

def testMolWtEthanol(self):
    """Check the molecular weight of ethanol"""
    self.canFindExecutable("obabel")
    smifile = """CCO
    [H]
    [2H]
    """
    answers = [46.07, 1.01, 2.01]
    output, error = run_exec(smifile, 'obabel -ismi --append mw -otxt')
    for ans, my_ans in zip(answers, output.split("\n")):
        self.assertEqual(ans, round(float(my_ans), 2))

To use a testfile placed in test/files, the getTestFile() member function is provided:

def testMolWtEthanol(self):
    """Check the molecular weight of ethanol"""
    self.canFindExecutable("obabel")
    answers = [46.07, 1.01, 2.01]
    smifile = self.getTestFile("ethanol.smi")
    output, error = run_exec('obabel %s --append mw -otxt' % smifile)
    for ans, my_ans in zip(answers, output.split("\n")):
        self.assertEqual(ans, round(float(my_ans), 2))

The full list of provided convenience functions is:

  • run_exec(command), run_exec(stdin, command) - see above

  • BaseTest.getTestFile(filename) - returns the full path to a testfile

  • BaseTest.canFindExecutable(executable) - checks whether the executable exists in the expected location

  • BaseTest.assertConverted(stderr, N) - An assert statement that takes the stderr from run_exec and will check whether the number of molecules reported as converted matches N

Test the API using Python#

The easiest place to add new tests is into test/testbindings.py. Classes are used to organise the tests, but for a single ‘miscellaneous’ test a good place is the TestSuite class. Somewhere in that class add the following function:

def testMolWtEthanol(self):
    """Check the molecular weight of ethanol"""
    answers = [
            ("CCO", 46.07),
            ("[H]", 1.01),
            ("[2H]", 2.01),
            ]
    for smi, molwt in answers:
        my_molwt = round(pybel.readstring("smi", smi).molwt, 2)
        self.assertEqual(my_molwt, molwt)

The variable here is defined in testbindings.py and may be used find the path to testfiles. For example, given the test/ethanol.smi, the following may be used to read it:

def testMolWtEthanol(self):
    """Check the molecular weight of ethanol"""
    answers = [46.07, 1.01, 2.01]
    testfile = os.path.join(here, "test", "ethanol.smi")
    for mol, answer in zip(pybel.readfile("smi", testfile), answers):
        my_molwt = round(mol.molwt, 2)
        self.assertEqual(my_molwt, molwt)

The tests use the standard unittest framework. One thing to note, which is not obvious, is how to test for exceptions. A typical case is checking that a dodgy SMILES is rejected on reading; in this instance, Pybel.readstring() will raise an IOError. To assert that this is the case, rather than use try/except, the following syntax is required:

self.assertRaises(IOError, pybel.readstring, "smi", "~*&*($")

If you have multiple tests to add on a single ‘topic’, you will probably want to add your own class either into testbindings.py or a new Python file. Note that if you create a new Python file, it should start with the word test and you will need to add the rest of the name to the pybindtest list in test/CMakeLists.txt.

Some final comments#

Some thoughts on the topic of the perfect test:

  • When adding a regression test for a bug fix, the test should fail without the fix, but pass afterwards.

  • When adding a test for a new feature, the test should provide complete coverage for all code paths.

  • Test not just perfect input, but all sorts of dodgy input like molecules with no atoms, empty strings, and so forth.

  • Don’t be afraid to add tests for things which you already (think you) know will pass; such tests may surprise you, and even if they don’t they will prevent regressions.

Potential problems/gotchas:

  • Why isn’t your Python test being run? Test functions name must begin with the word test.

  • If your new test passes first time, check that it is actually running correctly, by changing your asserts so that they should fail.

  • The C++ tests will be marked as failing if the test writes any of the following to stdout: ERROR, FAIL, Test failed. This is actually how the assert methods work.

  • It’s best to avoid writing to disk, and instead write to a variable or stdout and capture it (as in the examples above).