[cctbxbb] [git/cctbx] master: rename test files, remove them from run_tests (23a4a6fe4)

Nigel Moriarty nwmoriarty at lbl.gov
Sat Mar 3 14:30:47 PST 2018 

Does a failed test return non-zero to the calling program? This is needed
for automatic reporting in BuildBot.

Cheers

Nigel

---
Nigel W. Moriarty
Building 33R0349, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
Berkeley, CA 94720-8235
Phone : 510-486-5709     Email : NWMoriarty at LBL.gov
Fax   : 510-486-5909       Web  : CCI.LBL.gov

On Sat, Mar 3, 2018 at 3:03 AM, <markus.gerstel at diamond.ac.uk> wrote:

> Hi Pavel,
>
> pytest was suggested* by a couple of people in 2015 when I asked about
> unittest. Your recommendation at that time was to go ahead, so thats what I
> did - with the result that in April 2016 the pytest compatibility layer was
> introduced to cctbx and pytest included in the base build, so we can use
> pytests alongside libtbx tests. In our packages we have been using pytests
> for well over 2 years now, and just one month ago converted xia2 to pytest
> completely.
>
> As to "why pytest" I would like to point you to our wiki page "Why
> pytest?" at https://github.com/dials/dials/wiki/Why-pytest%3F which does
> make the case in more detail and includes instructions to enable pytest for
> cctbx modules. However I'm more than happy to demonstrate part of it here
> using your example.
>
> First of all, the example you gave is not quite correct. You are missing
> the import for approx_equal and missed the crucial line in run_tests.py
> that ensures the test is actually run. This is one of the drawbacks of
> libtbx testing: there is no automatic test discovery. And this tends to
> happen quite a lot, cf. https://github.com/dials/dials/issues/506
>
> In pytest your example would look like this:
>
> import pytest
> def test_that_2_times_2_is_4():
>   x = 2
>   result = x*x
>   assert result == pytest.approx(4)
>
> which I would argue has the exact same documentation/example value.
>
>
> Let's change the expectation value from 4 to 5, forget everything we know
> about the code, run it and compare the output:
>
>
> $ python tst_multiply.py
> approx_equal eps: 1e-06
> approx_equal multiplier: 10000000000.0
> 4.0 approx_equal ERROR
> 5.0 approx_equal ERROR
>
> Traceback (most recent call last):
>   File "tst_multiply.py", line 10, in <module>
>     exercise()
>   File "tst_multiply.py", line 7, in exercise
>     assert approx_equal(result, 5., 1.e-6)
> AssertionError
>
> *Something* went wrong, it has to do with 1e-06, a very large number (I
> still don't understand what that means), two errors for 4.0 and 5.0, and a
> function called exercise(), which tells me exactly nothing. I have to look
> into the code to understand what went wrong and why. From the outset I
> don't know what the code is supposed to do.
>
>
> $ pytest
> =================================== test session starts
> ===================================
> platform linux2 -- Python 2.7.13, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
> rootdir: /dls/tmp/wra62962/directories/lh0UFeF6, inifile:
> plugins: xdist-1.20.1, timeout-1.2.0, forked-0.2, cov-2.5.1
> collected 2 items
>
> test_multiplication.py .F
>
> ======================================== FAILURES
> =========================================
> ___________________________ test_that_two_times_two_equals_five
> ___________________________
>
>     def test_that_two_times_two_equals_five():
>       x = 2
>       result = x*x
> >     assert result == pytest.approx(5)
> E     AssertionError: assert 4 == 5 +- 5.0e-06
> E      +  where 5 +- 5.0e-06 = <class '_pytest.python.approx'>(5)
> E      +    where <class '_pytest.python.approx'> = pytest.approx
>
> test_multiplication.py:11: AssertionError
> =========================== 1 failed, 1 passed in 0.04 seconds
> ============================
>
>
> Oh look, a test to ensure two times two equals five failed. Apparently it
> compared 4 to 5 +- 5e-06.
> You also see that the other test in the file that I left in (which
> compares 2*2 to 4) worked. You didn't know that from the libtbx-style test
> output.
>
> Now I can fix the test, and pytest allows me to just rerun the failed
> tests, not everything.
> If you want to know how, have a look at https://github.com/dials/
> dials/wiki/pytest which contains a lot more information about running
> pytest and converting tests.
>
> If you don't want to use it that is fine, too. Thanks to the compatibility
> layer you can still use libtbx.run_tests_parallel, and you will still get
> the more useful assertion messages. I would encourage you to try it though.
> You might find it useful.
>
> -Markus
>
>
> * And rightly so; pytest requires much less boilerplate, produces cleaner
> code, and is overall just better - Thank you, Luc. Thank you, Jamasi.
> PS: On fable specifically: thanks to the conversion I already found and
> fixed one broken test which didn't fail, and another test with race
> conditions.
>
> ________________________________________
> From: cctbxbb-bounces at phenix-online.org [cctbxbb-bounces at phenix-online.org]
> on behalf of Pavel Afonine [pafonine at lbl.gov]
> Sent: Saturday, March 03, 2018 07:13
> To: cctbx mailing list; Winter, Graeme (DLSLtd,RAL,LSCI)
> Subject: Re: [cctbxbb] [git/cctbx] master: rename test files, remove them
> from run_tests (23a4a6fe4)
>
> I'd say at least because:
>
> - the first 10+ years of CCTBX did not use pytest. AFAIK, the first
> attempt was by our postdoc Youval Dar back in 2015 (correct me if I'm
> wrong). I feel adding different testing styles are only to make the
> code-base inconsistent (very much like mixing flex and np arrays isn't
> cool, in my opinion!).
>
> - originally tests were considered as simple usage examples for
> functionalities they are testing; this is because writing and (most
> importantly!) maintaining the proper documentation was not provisioned.
> A simple test like
>
> def exercise():
>    """ Make sure 2*2 is 4. """
>    x=2.
>    result=x*x
>    assert approx_equal(result, 4., 1.e-6)
>
> if(__name__ == "__main__"):
>    exercise()
>    print "OK"
>
> is much easier to grasp rather than the same cluttered with the stuff
> (that, to add to the trouble, one needs to learn in the first place!).
>
> All the best,
> Pavel
>
> On 3/3/18 14:36, Graeme.Winter at diamond.ac.uk wrote:
> > What’s bad about pytest?
> >
> >
> >
> >> On 3 Mar 2018, at 02:26, Pavel Afonine <pafonine at lbl.gov> wrote:
> >>
> >> Just to make sure: you are converting to use pytest this particular
> codes (fable), correct?
> >> Pavel
> >> P.S.: I'm allergic to pytest.
> >>
> >>
> >> On 3/3/18 07:46, CCTBX commit wrote:
> >>> This in preparation for pytestification.
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb at phenix-online.org
> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20180303/a8231fa3/attachment.htm>

More information about the cctbxbb mailing list