Native python unittests for the libtbx testing framework
![](https://secure.gravatar.com/avatar/2139a5f865957ce885e88203493d7ae1.jpg?s=120&d=mm&r=g)
Dear cctbx developers and users, For a side-project of mine I experimented with native python unit tests (some information available on https://docs.python.org/2/library/unittest.html if you are not familiar with it). The python unittest framework has a couple of advantages over the libtbx testing world. I found that the two most immediate advantages are: Automated test discovery. There is no need to keep a complete list of test files in another python file, which is what we currently do in run_tests.py. Multiple test functions per python test file. At the moment we can run multiple tests via command line parameters to the python test files. This of course means that you need to keep a list of '(test file, parameter)' tuples in run_tests.py, and you need some boilerplate dispatcher in your test file. This can lead to two situations: you may have a number of tests with numeric parameters, and no one knows what the parameters stand for. One example of this can be found in DIALS, where tst_index.py is run with numeric parameters 1 to 4 and 7 to 15, but not 5 and 6. The alternative situation is that you end up with a massive test file, that tests the entire class under test, but figuring out what specifically went wrong is probably more time consuming than necessary. In contrast in Python unittests any function in the file that is named test_* will be identified and run as a separate test. I find that this automatically leads to smaller and thus more readable tests. I understand that libtbx has been developed before Python unittest was around, that there is a substantial amount of code around that relies on current libtbx behaviour, and that there is no pressing need to change anything. However I would like to be able to take advantage of the tools that are out there, so I wrote a function that uses Python unittest for test discovery, and can be placed in run_tests.py. With this function you can then use regular Python unittests and run them with either through libtbx, with the python unittest tools, or directly. In each case you end up with the test results reported in the way you would expect. What I would like to do is integrate this logic in libtbx directly, so that we can use it for all cctbx modules. Since it does not modify libtbx default behaviour and only becomes active after a change in the run_tests.py of the relevant module it should be fully backwards compatible with virtually no downsides. What do you think? -Markus run_tests.py previously: (..) tst_list = ( "$D/test/algorithms/profile_model/nave/tst_model.py", ) (..) run_tests.py new: (..) tst_list = ( "$D/test/algorithms/profile_model/nave/tst_model.py", ) + discover_unittests("dlstbx") (..) discover_unittests function: def discover_unittests(module, pattern='tst_*.py'): try: import inspect import os import sys import unittest except: return tuple([]) dist_dir = libtbx.env.dist_path(module) found_tests = unittest.defaultTestLoader.discover(dist_dir, pattern=pattern) def recursive_TestSuite_to_list(suite): list = [] for t in suite: if isinstance(t, unittest.TestSuite): list.extend(recursive_TestSuite_to_list(t)) elif isinstance(t, unittest.TestCase): module = t.__class__.__module__ if module == 'unittest.loader': # This indicates a loading error. # Regenerate file name and try to run file directly. path = t._testMethodName.replace('.', os.path.sep) list.append("$D/%s.py" % path) else: module = inspect.getsourcefile(sys.modules[module]) function = "%s.%s" % (t.__class__.__name__, t._testMethodName) list.append([module, function]) else: raise Exception("Unknown test object (%s)" % t) return list test_list = recursive_TestSuite_to_list(found_tests) return tuple(test_list) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
![](https://secure.gravatar.com/avatar/7984547b53fca6ae837ba286f9c92e29.jpg?s=120&d=mm&r=g)
Hi Markus, lucky you, I wish I have time for side projects! I've been working on cctbx (and writing tests!) for the past 12 years and can't name any problem using "libtbx testing world". Most tests are written this way. Why I am not so keen to use an alternative? - why introduce inconsistency for no obvious clear pressing reasons; - I have no spare time to learn and get used to an alternative framework (again, for no obvious clear pressing reason); - some rare developers (mostly postdocs that come and go) used Python test framework in the past on the code that I work too and that I have to maintain when people are gone. I find it is an irritating overhead for me dealing with these tests, so typically I bother to re-write them to use libtbx tools. All the best, Pavel On 5/15/15 8:00 AM, [email protected] wrote:
Dear cctbx developers and users,
For a side-project of mine I experimented with native python unit tests (some information available on https://docs.python.org/2/library/unittest.html if you are not familiar with it).
The python unittest framework has a couple of advantages over the libtbx testing world. I found that the two most immediate advantages are:
/Automated test discovery./
There is no need to keep a complete list of test files in another python file, which is what we currently do in run_tests.py.
/Multiple test functions per python test file./
At the moment we can run multiple tests via command line parameters to the python test files. This of course means that you need to keep a list of ‘(test file, parameter)’ tuples in run_tests.py, and you need some boilerplate dispatcher in your test file. This can lead to two situations: you may have a number of tests with numeric parameters, and no one knows what the parameters stand for. One example of this can be found in DIALS, where tst_index.py is run with numeric parameters 1 to 4 and 7 to 15, but not 5 and 6. The alternative situation is that you end up with a massive test file, that tests the entire class under test, but figuring out what specifically went wrong is probably more time consuming than necessary.
In contrast in Python unittests any function in the file that is named test_* will be identified and run as a separate test. I find that this automatically leads to smaller and thus more readable tests.
I understand that libtbx has been developed before Python unittest was around, that there is a substantial amount of code around that relies on current libtbx behaviour, and that there is no pressing need to change anything.
However I would like to be able to take advantage of the tools that are out there, so I wrote a function that uses Python unittest for test discovery, and can be placed in run_tests.py. With this function you can then use regular Python unittests and run them with either through libtbx, with the python unittest tools, or directly. In each case you end up with the test results reported in the way you would expect.
What I would like to do is integrate this logic in libtbx directly, so that we can use it for all cctbx modules. Since it does not modify libtbx default behaviour and only becomes active after a change in the run_tests.py of the relevant module it should be fully backwards compatible with virtually no downsides. What do you think?
-Markus
*run_tests.py previously:*
/(..)/
tst_list = (
"$D/test/algorithms/profile_model/nave/tst_model.py",
)
/(..)/
*run_tests.py new:*
/(..)/
tst_list = (
"$D/test/algorithms/profile_model/nave/tst_model.py",
) + discover_unittests("dlstbx")
/(..)/
*discover_unittests function:*
def discover_unittests(module, pattern='tst_*.py'):
try:
import inspect
import os
import sys
import unittest
except:
return tuple([])
dist_dir = libtbx.env.dist_path(module)
found_tests = unittest.defaultTestLoader.discover(dist_dir, pattern=pattern)
def recursive_TestSuite_to_list(suite):
list = []
for t in suite:
if isinstance(t, unittest.TestSuite):
list.extend(recursive_TestSuite_to_list(t))
elif isinstance(t, unittest.TestCase):
module = t.__class__.__module__
if module == 'unittest.loader':
# This indicates a loading error.
# Regenerate file name and try to run file directly.
path = t._testMethodName.replace('.', os.path.sep)
list.append("$D/%s.py" % path)
else:
module = inspect.getsourcefile(sys.modules[module])
function = "%s.%s" % (t.__class__.__name__, t._testMethodName)
list.append([module, function])
else:
raise Exception("Unknown test object (%s)" % t)
return list
test_list = recursive_TestSuite_to_list(found_tests)
return tuple(test_list)
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb
![](https://secure.gravatar.com/avatar/2cf13f99074a055ce708406c52f233c9.jpg?s=120&d=mm&r=g)
Hi Markus,
The python unittest framework has a couple of advantages over the libtbx testing world. I found that the two most immediate advantages are:
my 2p: First create a parallel testing bed which you trigger from the existing testing framework. That way, you don’t disturb anybody. Then, please, use pick py.test instead. It is a far superior test framework, which can make intelligent use of plain asserts by magically providing some nice printouts when they fail. The amount of boiler plate required by unittest is too high imho. And then, I am pretty sure you could write a test runner which could find all the existing tests, if we eventually decide to move in that direction. Which brings me to Pavel’s comments. I am definitively on his side. Yes, the cctbx does reinvent the wheel for its test system and in an ideal world we would be using more standard tools (I mean we reinvented the wheel for the build system too as we don’t use SCons as it is intended to). But the train has left the platform a long time ago: there is no way we will ever convert the existing tests to unittest classes. Hence my proposition to use py.test as this could make the transition painless. Best wishes, Luc
participants (3)
-
Luc Bourhis
-
markus.gerstel@diamond.ac.uk
-
Pavel Afonine