[cctbxbb] [git/cctbx] master: rename test files, remove them from run_tests (23a4a6fe4)

markus.gerstel at diamond.ac.uk markus.gerstel at diamond.ac.uk
Sun Mar 4 08:34:19 PST 2018

Hi Nigel,

if you run it via libtbx.run_tests_parallel then nothing changes.
If you run it via pytest directly then the exit code will be 0 for a successful run and >0 otherwise. The full list of possible exit codes is given here: https://docs.pytest.org/en/latest/usage.html#possible-exit-codes


From: cctbxbb-bounces at phenix-online.org [cctbxbb-bounces at phenix-online.org] on behalf of Nigel Moriarty [nwmoriarty at lbl.gov]
Sent: Saturday, March 03, 2018 22:30
To: cctbx mailing list
Subject: Re: [cctbxbb] [git/cctbx] master: rename test files, remove them from run_tests (23a4a6fe4)

Does a failed test return non-zero to the calling program? This is needed for automatic reporting in BuildBot.



Nigel W. Moriarty
Building 33R0349, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
Berkeley, CA 94720-8235
Phone : 510-486-5709     Email : NWMoriarty at LBL.gov
Fax   : 510-486-5909       Web  : CCI.LBL.gov<http://CCI.LBL.gov>

On Sat, Mar 3, 2018 at 3:03 AM, <markus.gerstel at diamond.ac.uk<mailto:markus.gerstel at diamond.ac.uk>> wrote:
Hi Pavel,

pytest was suggested* by a couple of people in 2015 when I asked about unittest. Your recommendation at that time was to go ahead, so thats what I did - with the result that in April 2016 the pytest compatibility layer was introduced to cctbx and pytest included in the base build, so we can use pytests alongside libtbx tests. In our packages we have been using pytests for well over 2 years now, and just one month ago converted xia2 to pytest completely.

As to "why pytest" I would like to point you to our wiki page "Why pytest?" at https://github.com/dials/dials/wiki/Why-pytest%3F which does make the case in more detail and includes instructions to enable pytest for cctbx modules. However I'm more than happy to demonstrate part of it here using your example.

First of all, the example you gave is not quite correct. You are missing the import for approx_equal and missed the crucial line in run_tests.py that ensures the test is actually run. This is one of the drawbacks of libtbx testing: there is no automatic test discovery. And this tends to happen quite a lot, cf. https://github.com/dials/dials/issues/506

In pytest your example would look like this:

import pytest
def test_that_2_times_2_is_4():
  x = 2
  result = x*x
  assert result == pytest.approx(4)

which I would argue has the exact same documentation/example value.

Let's change the expectation value from 4 to 5, forget everything we know about the code, run it and compare the output:

$ python tst_multiply.py
approx_equal eps: 1e-06
approx_equal multiplier: 10000000000.0
4.0 approx_equal ERROR
5.0 approx_equal ERROR

Traceback (most recent call last):
  File "tst_multiply.py", line 10, in <module>
  File "tst_multiply.py", line 7, in exercise
    assert approx_equal(result, 5., 1.e-6)

*Something* went wrong, it has to do with 1e-06, a very large number (I still don't understand what that means), two errors for 4.0 and 5.0, and a function called exercise(), which tells me exactly nothing. I have to look into the code to understand what went wrong and why. From the outset I don't know what the code is supposed to do.

$ pytest
=================================== test session starts ===================================
platform linux2 -- Python 2.7.13, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: /dls/tmp/wra62962/directories/lh0UFeF6, inifile:
plugins: xdist-1.20.1, timeout-1.2.0, forked-0.2, cov-2.5.1
collected 2 items

test_multiplication.py .F

======================================== FAILURES =========================================
___________________________ test_that_two_times_two_equals_five ___________________________

    def test_that_two_times_two_equals_five():
      x = 2
      result = x*x
>     assert result == pytest.approx(5)
E     AssertionError: assert 4 == 5 +- 5.0e-06
E      +  where 5 +- 5.0e-06 = <class '_pytest.python.approx'>(5)
E      +    where <class '_pytest.python.approx'> = pytest.approx

test_multiplication.py:11: AssertionError
=========================== 1 failed, 1 passed in 0.04 seconds ============================

Oh look, a test to ensure two times two equals five failed. Apparently it compared 4 to 5 +- 5e-06.
You also see that the other test in the file that I left in (which compares 2*2 to 4) worked. You didn't know that from the libtbx-style test output.

Now I can fix the test, and pytest allows me to just rerun the failed tests, not everything.
If you want to know how, have a look at https://github.com/dials/dials/wiki/pytest which contains a lot more information about running pytest and converting tests.

If you don't want to use it that is fine, too. Thanks to the compatibility layer you can still use libtbx.run_tests_parallel, and you will still get the more useful assertion messages. I would encourage you to try it though. You might find it useful.


* And rightly so; pytest requires much less boilerplate, produces cleaner code, and is overall just better - Thank you, Luc. Thank you, Jamasi.
PS: On fable specifically: thanks to the conversion I already found and fixed one broken test which didn't fail, and another test with race conditions.

From: cctbxbb-bounces at phenix-online.org<mailto:cctbxbb-bounces at phenix-online.org> [cctbxbb-bounces at phenix-online.org<mailto:cctbxbb-bounces at phenix-online.org>] on behalf of Pavel Afonine [pafonine at lbl.gov<mailto:pafonine at lbl.gov>]
Sent: Saturday, March 03, 2018 07:13
To: cctbx mailing list; Winter, Graeme (DLSLtd,RAL,LSCI)
Subject: Re: [cctbxbb] [git/cctbx] master: rename test files, remove them from run_tests (23a4a6fe4)

I'd say at least because:

- the first 10+ years of CCTBX did not use pytest. AFAIK, the first
attempt was by our postdoc Youval Dar back in 2015 (correct me if I'm
wrong). I feel adding different testing styles are only to make the
code-base inconsistent (very much like mixing flex and np arrays isn't
cool, in my opinion!).

- originally tests were considered as simple usage examples for
functionalities they are testing; this is because writing and (most
importantly!) maintaining the proper documentation was not provisioned.
A simple test like

def exercise():
   """ Make sure 2*2 is 4. """
   assert approx_equal(result, 4., 1.e-6)

if(__name__ == "__main__"):
   print "OK"

is much easier to grasp rather than the same cluttered with the stuff
(that, to add to the trouble, one needs to learn in the first place!).

All the best,

On 3/3/18 14:36, Graeme.Winter at diamond.ac.uk<mailto:Graeme.Winter at diamond.ac.uk> wrote:
> What’s bad about pytest?
>> On 3 Mar 2018, at 02:26, Pavel Afonine <pafonine at lbl.gov<mailto:pafonine at lbl.gov>> wrote:
>> Just to make sure: you are converting to use pytest this particular codes (fable), correct?
>> Pavel
>> P.S.: I'm allergic to pytest.
>> On 3/3/18 07:46, CCTBX commit wrote:
>>> This in preparation for pytestification.
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> http://phenix-online.org/mailman/listinfo/cctbxbb

cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>

More information about the cctbxbb mailing list