Dear cctbx developers and users,

 

For a side-project of mine I experimented with native python unit tests (some information available on https://docs.python.org/2/library/unittest.html if you are not familiar with it).

The python unittest framework has a couple of advantages over the libtbx testing world. I found that the two most immediate advantages are:

 

Automated test discovery.

There is no need to keep a complete list of test files in another python file, which is what we currently do in run_tests.py.

 

Multiple test functions per python test file.

At the moment we can run multiple tests via command line parameters to the python test files. This of course means that you need to keep a list of ‘(test file, parameter)’ tuples in run_tests.py, and you need some boilerplate dispatcher in your test file. This can lead to two situations: you may have a number of tests with numeric parameters, and no one knows what the parameters stand for. One example of this can be found in DIALS, where tst_index.py is run with numeric parameters 1 to 4 and 7 to 15, but not 5 and 6. The alternative situation is that you end up with a massive test file, that tests the entire class under test, but figuring out what specifically went wrong is probably more time consuming than necessary.

In contrast in Python unittests any function in the file that is named test_* will be identified and run as a separate test. I find that this automatically leads to smaller and thus more readable tests.

 

I understand that libtbx has been developed before Python unittest was around, that there is a substantial amount of code around that relies on current libtbx behaviour, and that there is no pressing need to change anything.

However I would like to be able to take advantage of the tools that are out there, so I wrote a function that uses Python unittest for test discovery, and can be placed in run_tests.py. With this function you can then use regular Python unittests and run them with either through libtbx, with the python unittest tools, or directly. In each case you end up with the test results reported in the way you would expect.

 

What I would like to do is integrate this logic in libtbx directly, so that we can use it for all cctbx modules. Since it does not modify libtbx default behaviour and only becomes active after a change in the run_tests.py of the relevant module it should be fully backwards compatible with virtually no downsides. What do you think?

 

-Markus

 

 

 

run_tests.py previously:

(..)

tst_list = (

  "$D/test/algorithms/profile_model/nave/tst_model.py",

)

(..)

 

run_tests.py new:

(..)

tst_list = (

  "$D/test/algorithms/profile_model/nave/tst_model.py",

) + discover_unittests("dlstbx")

(..)

 

discover_unittests function:

def discover_unittests(module, pattern='tst_*.py'):

  try:

    import inspect

    import os

    import sys

    import unittest

  except:

    return tuple([])

 

  dist_dir = libtbx.env.dist_path(module)

  found_tests = unittest.defaultTestLoader.discover(dist_dir, pattern=pattern)

 

  def recursive_TestSuite_to_list(suite):

    list = []

    for t in suite:

      if isinstance(t, unittest.TestSuite):

        list.extend(recursive_TestSuite_to_list(t))

      elif isinstance(t, unittest.TestCase):

        module = t.__class__.__module__

        if module == 'unittest.loader':

          # This indicates a loading error.

          # Regenerate file name and try to run file directly.

          path = t._testMethodName.replace('.', os.path.sep)

          list.append("$D/%s.py" % path)

        else:

          module = inspect.getsourcefile(sys.modules[module])

          function = "%s.%s" % (t.__class__.__name__, t._testMethodName)

          list.append([module, function])

      else:

        raise Exception("Unknown test object (%s)" % t)

    return list

  test_list = recursive_TestSuite_to_list(found_tests)

  return tuple(test_list)