<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Markus,<br>
<br>
lucky you, I wish I have time for side projects!<br>
<br>
I've been working on cctbx (and writing tests!) for the past 12
years and can't name any problem using "libtbx testing world". Most
tests are written this way. Why I am not so keen to use an
alternative?<br>
�� - why introduce inconsistency for no obvious clear pressing
reasons;<br>
�� - I have no spare time to learn and get used to an alternative
framework (again, for no obvious clear pressing reason);<br>
�� - some rare developers (mostly postdocs that come and go) used
Python test framework in the past on the code that I work too and
that I have to maintain when people are gone. I find it is an
irritating overhead for me dealing with these tests, so typically I
bother to re-write them to use libtbx tools.<br>
<br>
All the best,<br>
Pavel<br>
<br>
<br>
<div class="moz-cite-prefix">On 5/15/15 8:00 AM,
<a class="moz-txt-link-abbreviated" href="mailto:markus.gerstel@diamond.ac.uk">markus.gerstel@diamond.ac.uk</a> wrote:<br>
</div>
<blockquote
cite="mid:415F7F573406BB498256C68158D45674536CF0@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Dear cctbx developers and users,<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal">For a side-project of mine I experimented
with native python unit tests (some information available on
<a moz-do-not-send="true"
href="https://docs.python.org/2/library/unittest.html">https://docs.python.org/2/library/unittest.html</a>
if you are not familiar with it).<o:p></o:p></p>
<p class="MsoNormal">The python unittest framework has a couple
of advantages over the libtbx testing world. I found that the
two most immediate advantages are:<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal"><i>Automated test discovery.<o:p></o:p></i></p>
<p class="MsoNormal">There is no need to keep a complete list of
test files in another python file, which is what we currently
do in run_tests.py.<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal"><i>Multiple test functions per python test
file.<o:p></o:p></i></p>
<p class="MsoNormal">At the moment we can run multiple tests via
command line parameters to the python test files. This of
course means that you need to keep a list of �(test file,
parameter)� tuples in run_tests.py, and you need some
boilerplate dispatcher in your test file. This can lead to two
situations: you may have a number of tests with numeric
parameters, and no one knows what the parameters stand for.
One example of this can be found in DIALS, where tst_index.py
is run with numeric parameters 1 to 4 and 7 to 15, but not 5
and 6. The alternative situation is that you end up with a
massive test file, that tests the entire class under test, but
figuring out what specifically went wrong is probably more
time consuming than necessary.<o:p></o:p></p>
<p class="MsoNormal">In contrast in Python unittests any
function in the file that is named test_* will be identified
and run as a separate test. I find that this automatically
leads to smaller and thus more readable tests.<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal">I understand that libtbx has been developed
before Python unittest was around, that there is a substantial
amount of code around that relies on current libtbx behaviour,
and that there is no pressing need to change anything.<o:p></o:p></p>
<p class="MsoNormal">However I would like to be able to take
advantage of the tools that are out there, so I wrote a
function that uses Python unittest for test discovery, and can
be placed in run_tests.py. With this function you can then use
regular Python unittests and run them with either through
libtbx, with the python unittest tools, or directly. In each
case you end up with the test results reported in the way you
would expect.<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal">What I would like to do is integrate this
logic in libtbx directly, so that we can use it for all cctbx
modules. Since it does not modify libtbx default behaviour and
only becomes active after a change in the run_tests.py of the
relevant module it should be fully backwards compatible with
virtually no downsides. What do you think?<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal">-Markus<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<div
style="mso-element:para-border-div;border:none;border-bottom:solid
windowtext 1.0pt;padding:0cm 0cm 1.0pt 0cm">
<p class="MsoNormal" style="border:none;padding:0cm"><o:p>�</o:p></p>
</div>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal"><b>run_tests.py previously:<o:p></o:p></b></p>
<p class="MsoNormal"><i>(..)<o:p></o:p></i></p>
<p class="MsoNormal">tst_list = (<o:p></o:p></p>
<p class="MsoNormal">�
"$D/test/algorithms/profile_model/nave/tst_model.py",<o:p></o:p></p>
<p class="MsoNormal">)<o:p></o:p></p>
<p class="MsoNormal"><i>(..)<o:p></o:p></i></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal"><b>run_tests.py new:<o:p></o:p></b></p>
<p class="MsoNormal"><i>(..)<o:p></o:p></i></p>
<p class="MsoNormal">tst_list = (<o:p></o:p></p>
<p class="MsoNormal">�
"$D/test/algorithms/profile_model/nave/tst_model.py",<o:p></o:p></p>
<p class="MsoNormal">) + discover_unittests("dlstbx")<o:p></o:p></p>
<p class="MsoNormal"><i>(..)<o:p></o:p></i></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal"><b>discover_unittests function:<o:p></o:p></b></p>
<p class="MsoNormal">def discover_unittests(module,
pattern='tst_*.py'):<o:p></o:p></p>
<p class="MsoNormal">� try:<o:p></o:p></p>
<p class="MsoNormal">��� import inspect<o:p></o:p></p>
<p class="MsoNormal">��� import os<o:p></o:p></p>
<p class="MsoNormal">��� import sys<o:p></o:p></p>
<p class="MsoNormal">��� import unittest<o:p></o:p></p>
<p class="MsoNormal">� except:<o:p></o:p></p>
<p class="MsoNormal">��� return tuple([])<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal">� dist_dir = libtbx.env.dist_path(module)<o:p></o:p></p>
<p class="MsoNormal">� found_tests =
unittest.defaultTestLoader.discover(dist_dir, pattern=pattern)<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
<p class="MsoNormal">� def recursive_TestSuite_to_list(suite):<o:p></o:p></p>
<p class="MsoNormal">��� list = []<o:p></o:p></p>
<p class="MsoNormal">��� for t in suite:<o:p></o:p></p>
<p class="MsoNormal">����� if isinstance(t, unittest.TestSuite):<o:p></o:p></p>
<p class="MsoNormal">�������
list.extend(recursive_TestSuite_to_list(t))<o:p></o:p></p>
<p class="MsoNormal">����� elif isinstance(t,
unittest.TestCase):<o:p></o:p></p>
<p class="MsoNormal">������� module = t.__class__.__module__<o:p></o:p></p>
<p class="MsoNormal">������� if module == 'unittest.loader':<o:p></o:p></p>
<p class="MsoNormal">��������� # This indicates a loading error.<o:p></o:p></p>
<p class="MsoNormal">��������� # Regenerate file name and try to
run file directly.<o:p></o:p></p>
<p class="MsoNormal">��������� path =
t._testMethodName.replace('.', os.path.sep)<o:p></o:p></p>
<p class="MsoNormal">��������� list.append("$D/%s.py" % path)<o:p></o:p></p>
<p class="MsoNormal">������� else:<o:p></o:p></p>
<p class="MsoNormal">��������� module =
inspect.getsourcefile(sys.modules[module])<o:p></o:p></p>
<p class="MsoNormal">��������� function = "%s.%s" %
(t.__class__.__name__, t._testMethodName)<o:p></o:p></p>
<p class="MsoNormal">��������� list.append([module, function])<o:p></o:p></p>
<p class="MsoNormal">����� else:<o:p></o:p></p>
<p class="MsoNormal">������� raise Exception("Unknown test
object (%s)" % t)<o:p></o:p></p>
<p class="MsoNormal">��� return list<o:p></o:p></p>
<p class="MsoNormal">� test_list =
recursive_TestSuite_to_list(found_tests)<o:p></o:p></p>
<p class="MsoNormal">� return tuple(test_list)<o:p></o:p></p>
<p class="MsoNormal"><o:p>�</o:p></p>
</div>
<p align="justify">�</p>
<p align="justify">--�</p>
<p align="justify">This e-mail and any attachments may contain
confidential, copyright and or privileged material, and are for
the use of the intended addressee only. If you are not the
intended addressee or an authorised recipient of the addressee
please notify us of receipt by returning the e-mail and do not
use, copy, retain, distribute or disclose the information in or
attached to the e-mail.<br>
Any opinions expressed within this e-mail are those of the
individual and not necessarily of Diamond Light Source Ltd. <br>
Diamond Light Source Ltd. cannot guarantee that this e-mail or
any attachments are free from viruses and we cannot accept
liability for any damage which you may sustain as a result of
software viruses which may be transmitted in or with the
message.<br>
Diamond Light Source Limited (company no. 4375679). Registered
in England and Wales with its registered office at Diamond
House, Harwell Science and Innovation Campus, Didcot,
Oxfordshire, OX11 0DE, United Kingdom<br>
�</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
cctbxbb mailing list
<a class="moz-txt-link-abbreviated" href="mailto:cctbxbb@phenix-online.org">cctbxbb@phenix-online.org</a>
<a class="moz-txt-link-freetext" href="http://phenix-online.org/mailman/listinfo/cctbxbb">http://phenix-online.org/mailman/listinfo/cctbxbb</a>
</pre>
</blockquote>
<br>
</body>
</html>