[cctbxbb] [Dials-support] DIALS source tarball

Tue Apr 7 14:36:49 PDT 2015

Hi Richard,

At the moment, buildbot for dials runs these three tests:
libtbx.import_all_ext
cctbx_regression.test_nightly
libtbx.run_tests_parallel module=dials nproc=auto verbosity=1

Which doesn't line up exactly with what you described.  If you run
libtbx.run_tests_parallel
module=cctbx module=dxtbx, then you are right that the fix
I committed wouldn't be relevant because mmtbx wouldn't be tested.
However, since the LBL build bot runs cctbx_regression.test_nightly, my
change is needed.

After some more discussion, I see that Markus's changes to
tst_geometry_restraints_2.py are fine.  It looks like there are already
sub-tests in there that check for the presence of mmtbx and chem data
before running.  I think tst_geometry_restraints_2.py was the only test
outside of mmtbx that was looking for the monomer libraries.

I'm re-running the buildbot dials build now to see if my change to
cctbx_regression.test_nightly along with Markus's changes to
tst_geometry_restraints_2.py clean up the build.

Finally, and perhaps on a different note, should we add dxtbx to
cctbx_regression.test_nightly?  Right now the list of modules it tests
includes libtbx, boost_adaptbx, scitbx, cctbx, iotbx, smtbx, and if
chem_data is present, mmtbx.  I don't think the build bot is running the
dxtbx tests at present.

-Aaron

On Tue, Apr 7, 2015 at 1:36 PM, <richard.gildea at diamond.ac.uk> wrote:

> Hi Aaron,
>
> I'm not sure your fix as I understand it actually helps. The test that was
> failing was in the cctbx module, not mmtbx, and as far as I am aware we are
> running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our
> nightly builds (and that is what we as developers are running locally on
> our own development machines). If individual tests (particularly those in
> the more general modules such as cctbx, iotbx, etc.) rely on external
> dependencies (such as chem_data) being available, then the individual tests
> should test for presence of said dependencies and skip gracefully if not
> available. I have a feeling that we're not even including mmtbx in our
> nightly build tests (too many fail due to missing dependencies such as
> chem_data).
>
> Cheers,
>
> Richard
>
> ________________________________
> From: Aaron Brewster [asbrewster at lbl.gov]
> Sent: 07 April 2015 20:29
> To: cctbx mailing list
> Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI);
> Johan Hattne; <dials-support at lists.sourceforge.net>
> Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball
>
> Hi Markus, two points.
>
> First, you are correct that the dials builds are not passing tests related
> to the chem_data folder which includes monomer libraries.  This is because
> chem_data is required by mmtbx but isn't included in the dials build.
> Partly because we're unsure if pieces of it are proprietary.  In the
> meantime, I've worked with Nigel to change the
> cctbx_regression.test_nightly to look for the chem_data repository and skip
> the mmtbx tests if it's not present.  mmtbx will still be tested nightly as
> part of the standalone cctbx and phenix packages.  Because of this change,
> perhaps you could revert your recent changes to
> cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check
> for chem_data in every sub test as it is tested for at the top level now.
>
> Second, the log you provided is not the one on buildbot.  Your log
> indicates you have a CCP4 installation that includes a monomer library that
> is being picked up by the tests.  Is one of the MMTBX_CCP4_MONOMER_LIB, or
> CLIBD_MON environment variables set on the system in question?
>
> Thanks,
> -Aaron
>
>
>
> On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel at diamond.ac.uk<mailto:
> markus.gerstel at diamond.ac.uk>> wrote:
> Cc'ing cctbx folks
>
> David Waterman wrote:
> > So maybe once we have a source tarball up at dials.diamond.ac.uk<
> http://dials.diamond.ac.uk> (..)
>
> Speaking of which: our DIALS builds are currently failing at the installer
> testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py
> failing deep in mmtbx territory.
> So far I could not figure out how to get this running properly.
>
>
> Call:
>
> libtbx.python tst_geometry_restraints_2.py
>
> Standard Output
>
> Skipping exercise_with_zeolite(): input file not available
> Skipping exercise_with_pdb(): chem_data directory not available
>
> Standard Error
>
> Traceback (most recent call last):
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py",
> line 866, in <module>
>     exercise_all(sys.argv[1:])
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py",
> line 862, in exercise_all
>     exercise_na_restraints_output_to_geo(verbose=verbose)
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py",
> line 805, in exercise_na_restraints_output_to_geo
>     log=out1)
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py",
> line 5270, in run
>     log=log)
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py",
> line 4876, in __init__
>     restraints_loading_flags=restraints_loading_flags,
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py",
> line 2902, in __init__
>     log=log,
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py",
> line 2287, in __init__
>     m_j=mm)
>   File
> "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py",
> line 1513, in get_lib_link
>     return mon_lib_srv.link_link_id_dict["rna3p"]
> KeyError: 'rna3p'
>
>
> (On a side note: We really should call multiple tests within the same
> python file separately, instead of running them together within a signle
> run() method. This file contains 4 tests, but these will result in only a
> single return value for any upstream processing of the test results, such
> as Jenkins.)
>
> -Markus
>
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20150407/c4afe491/attachment-0001.htm>