[cctbxbb] Native python unittests for the libtbx testing framework

Pavel Afonine pafonine at lbl.gov
Fri May 15 09:07:33 PDT 2015


Hi Markus,

lucky you, I wish I have time for side projects!

I've been working on cctbx (and writing tests!) for the past 12 years 
and can't name any problem using "libtbx testing world". Most tests are 
written this way. Why I am not so keen to use an alternative?
    - why introduce inconsistency for no obvious clear pressing reasons;
    - I have no spare time to learn and get used to an alternative 
framework (again, for no obvious clear pressing reason);
    - some rare developers (mostly postdocs that come and go) used 
Python test framework in the past on the code that I work too and that I 
have to maintain when people are gone. I find it is an irritating 
overhead for me dealing with these tests, so typically I bother to 
re-write them to use libtbx tools.

All the best,
Pavel


On 5/15/15 8:00 AM, markus.gerstel at diamond.ac.uk wrote:
>
> Dear cctbx developers and users,
>
> For a side-project of mine I experimented with native python unit 
> tests (some information available on 
> https://docs.python.org/2/library/unittest.html if you are not 
> familiar with it).
>
> The python unittest framework has a couple of advantages over the 
> libtbx testing world. I found that the two most immediate advantages are:
>
> /Automated test discovery./
>
> There is no need to keep a complete list of test files in another 
> python file, which is what we currently do in run_tests.py.
>
> /Multiple test functions per python test file./
>
> At the moment we can run multiple tests via command line parameters to 
> the python test files. This of course means that you need to keep a 
> list of ‘(test file, parameter)’ tuples in run_tests.py, and you need 
> some boilerplate dispatcher in your test file. This can lead to two 
> situations: you may have a number of tests with numeric parameters, 
> and no one knows what the parameters stand for. One example of this 
> can be found in DIALS, where tst_index.py is run with numeric 
> parameters 1 to 4 and 7 to 15, but not 5 and 6. The alternative 
> situation is that you end up with a massive test file, that tests the 
> entire class under test, but figuring out what specifically went wrong 
> is probably more time consuming than necessary.
>
> In contrast in Python unittests any function in the file that is named 
> test_* will be identified and run as a separate test. I find that this 
> automatically leads to smaller and thus more readable tests.
>
> I understand that libtbx has been developed before Python unittest was 
> around, that there is a substantial amount of code around that relies 
> on current libtbx behaviour, and that there is no pressing need to 
> change anything.
>
> However I would like to be able to take advantage of the tools that 
> are out there, so I wrote a function that uses Python unittest for 
> test discovery, and can be placed in run_tests.py. With this function 
> you can then use regular Python unittests and run them with either 
> through libtbx, with the python unittest tools, or directly. In each 
> case you end up with the test results reported in the way you would 
> expect.
>
> What I would like to do is integrate this logic in libtbx directly, so 
> that we can use it for all cctbx modules. Since it does not modify 
> libtbx default behaviour and only becomes active after a change in the 
> run_tests.py of the relevant module it should be fully backwards 
> compatible with virtually no downsides. What do you think?
>
> -Markus
>
> *run_tests.py previously:*
>
> /(..)/
>
> tst_list = (
>
> "$D/test/algorithms/profile_model/nave/tst_model.py",
>
> )
>
> /(..)/
>
> *run_tests.py new:*
>
> /(..)/
>
> tst_list = (
>
> "$D/test/algorithms/profile_model/nave/tst_model.py",
>
> ) + discover_unittests("dlstbx")
>
> /(..)/
>
> *discover_unittests function:*
>
> def discover_unittests(module, pattern='tst_*.py'):
>
>   try:
>
>     import inspect
>
>     import os
>
>     import sys
>
>     import unittest
>
>   except:
>
>     return tuple([])
>
>   dist_dir = libtbx.env.dist_path(module)
>
>   found_tests = unittest.defaultTestLoader.discover(dist_dir, 
> pattern=pattern)
>
>   def recursive_TestSuite_to_list(suite):
>
>     list = []
>
>     for t in suite:
>
>       if isinstance(t, unittest.TestSuite):
>
> list.extend(recursive_TestSuite_to_list(t))
>
>       elif isinstance(t, unittest.TestCase):
>
>         module = t.__class__.__module__
>
>         if module == 'unittest.loader':
>
>           # This indicates a loading error.
>
>           # Regenerate file name and try to run file directly.
>
>           path = t._testMethodName.replace('.', os.path.sep)
>
>           list.append("$D/%s.py" % path)
>
>         else:
>
>           module = inspect.getsourcefile(sys.modules[module])
>
>           function = "%s.%s" % (t.__class__.__name__, t._testMethodName)
>
>           list.append([module, function])
>
>       else:
>
>         raise Exception("Unknown test object (%s)" % t)
>
>     return list
>
>   test_list = recursive_TestSuite_to_list(found_tests)
>
>   return tuple(test_list)
>
> -- 
>
> This e-mail and any attachments may contain confidential, copyright 
> and or privileged material, and are for the use of the intended 
> addressee only. If you are not the intended addressee or an authorised 
> recipient of the addressee please notify us of receipt by returning 
> the e-mail and do not use, copy, retain, distribute or disclose the 
> information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual 
> and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
> attachments are free from viruses and we cannot accept liability for 
> any damage which you may sustain as a result of software viruses which 
> may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in 
> England and Wales with its registered office at Diamond House, Harwell 
> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United 
> Kingdom
>
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20150515/9efe7892/attachment.htm>


More information about the cctbxbb mailing list