[cctbxbb] Native python unittests for the libtbx testing framework

markus.gerstel at diamond.ac.uk markus.gerstel at diamond.ac.uk
Fri May 15 08:00:51 PDT 2015


Dear cctbx developers and users,

For a side-project of mine I experimented with native python unit tests (some information available on https://docs.python.org/2/library/unittest.html if you are not familiar with it).
The python unittest framework has a couple of advantages over the libtbx testing world. I found that the two most immediate advantages are:

Automated test discovery.
There is no need to keep a complete list of test files in another python file, which is what we currently do in run_tests.py.

Multiple test functions per python test file.
At the moment we can run multiple tests via command line parameters to the python test files. This of course means that you need to keep a list of '(test file, parameter)' tuples in run_tests.py, and you need some boilerplate dispatcher in your test file. This can lead to two situations: you may have a number of tests with numeric parameters, and no one knows what the parameters stand for. One example of this can be found in DIALS, where tst_index.py is run with numeric parameters 1 to 4 and 7 to 15, but not 5 and 6. The alternative situation is that you end up with a massive test file, that tests the entire class under test, but figuring out what specifically went wrong is probably more time consuming than necessary.
In contrast in Python unittests any function in the file that is named test_* will be identified and run as a separate test. I find that this automatically leads to smaller and thus more readable tests.

I understand that libtbx has been developed before Python unittest was around, that there is a substantial amount of code around that relies on current libtbx behaviour, and that there is no pressing need to change anything.
However I would like to be able to take advantage of the tools that are out there, so I wrote a function that uses Python unittest for test discovery, and can be placed in run_tests.py. With this function you can then use regular Python unittests and run them with either through libtbx, with the python unittest tools, or directly. In each case you end up with the test results reported in the way you would expect.

What I would like to do is integrate this logic in libtbx directly, so that we can use it for all cctbx modules. Since it does not modify libtbx default behaviour and only becomes active after a change in the run_tests.py of the relevant module it should be fully backwards compatible with virtually no downsides. What do you think?

-Markus




run_tests.py previously:
(..)
tst_list = (
  "$D/test/algorithms/profile_model/nave/tst_model.py",
)
(..)

run_tests.py new:
(..)
tst_list = (
  "$D/test/algorithms/profile_model/nave/tst_model.py",
) + discover_unittests("dlstbx")
(..)

discover_unittests function:
def discover_unittests(module, pattern='tst_*.py'):
  try:
    import inspect
    import os
    import sys
    import unittest
  except:
    return tuple([])

  dist_dir = libtbx.env.dist_path(module)
  found_tests = unittest.defaultTestLoader.discover(dist_dir, pattern=pattern)

  def recursive_TestSuite_to_list(suite):
    list = []
    for t in suite:
      if isinstance(t, unittest.TestSuite):
        list.extend(recursive_TestSuite_to_list(t))
      elif isinstance(t, unittest.TestCase):
        module = t.__class__.__module__
        if module == 'unittest.loader':
          # This indicates a loading error.
          # Regenerate file name and try to run file directly.
          path = t._testMethodName.replace('.', os.path.sep)
          list.append("$D/%s.py" % path)
        else:
          module = inspect.getsourcefile(sys.modules[module])
          function = "%s.%s" % (t.__class__.__name__, t._testMethodName)
          list.append([module, function])
      else:
        raise Exception("Unknown test object (%s)" % t)
    return list
  test_list = recursive_TestSuite_to_list(found_tests)
  return tuple(test_list)


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20150515/7939c7fe/attachment-0001.htm>


More information about the cctbxbb mailing list