- cctbxbb - phenix-online.org

Re: [cctbxbb] [Dials-support] DIALS source tarball
by Graeme.Winter＠diamond.ac.uk 08 Apr '15

08 Apr '15

Sadly this also does not fix anything :o( not sure why this was not flagged when I configured chem_data... libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py" [FAIL] Time: 6.44 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 818, in exercise_na_restraints_output_to_geo log=out2) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5274, in run nonbonded_distance_threshold=nonbonded_distance_threshold) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4952, in geometry_restraints_manager ss_manager.initialize(log=self.log) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 280, in initialize self.find_automatically(log=log) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 313, in find_automatically self.sec_str_from_pdb_file = self.find_sec_str(log=log) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 369, in find_sec_str (records, stderr) = run_ksdssp_direct(pdb_str) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 728, in run_ksdssp_direct exe_path = get_ksdssp_exe_path() File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 709, in get_ksdssp_exe_path raise RuntimeError("ksdssp module is not configured") RuntimeError: ksdssp module is not configured On 8 Apr 2015, at 11:02, <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> wrote: $ du -h --max-depth=0 chem_data 1.6G chem_data Not sure we really want to include this in the dials builds, just to make one test happy, especially when we have no use for any of the monomer library functionality in dials. Cheers, Richard ________________________________ From: Graeme Winter [graeme.winter(a)gmail.com<mailto:[email protected]>] Sent: 08 April 2015 10:58 To: Gildea, Richard (DLSLtd,RAL,LSCI); Gerstel, Markus (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]> Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball can we add this to the dials build through some kind of tarball download which does not need a cci account? Thanks graeme On Wed, Apr 8, 2015 at 10:52 AM <richard.gildea(a)diamond.ac.uk<mailto:[email protected]><mailto:[email protected]>> wrote: Hi Markus, I agree that it would be better if the tests would either work gracefully with CCP4 monomer library, or (possibly better?) only run if chem_data is available. In the latter case, a simple libtbx.env.has_module("chem_data") could be used in the test to check whether chem_data is available. As a workaround, we developers should be able to add the chem_data repository to our sources after which the tests should pick up the correct version of the monomer library: $ svn co svn+ssh://[email protected]/chem_data/trunk<http://[email protected]/chem_data/trunk> chem_data $ libtbx.configure chem_data Cheers, Richard ________________________________________ From: Gerstel, Markus (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:33 To: Gildea, Richard (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Richard, I think you've found it. I have unset CLIBD_MON, and now the test skips cleanly. Of course this also means that users should not run cctbx tests on their machines... Is there no central 'mmtbx_skip_test_if_chem_data_db_missing' function, that just goes 'prints "Skip this test"; exit' if the library isn't present? Or a boolean _is_available() function if you prefer. Either would resolve this problem as well as the current untestability of mmtbx. -Markus -----Original Message----- From: Gildea, Richard (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:15 To: Gerstel, Markus (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Markus, Presumably the test is looking for a monomer that is present in chem_data but not in the ccp4 monomer library? Could we unset the CLIBD_MON variable for the purposes of testing the dials installers? Cheers, Richard ________________________________________ From: Gerstel, Markus (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:07 To: Gildea, Richard (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Richard, Aaron, This is correct, we do not currently run mmtbx tests at all, as up to 107 of them do not fail gracefully, ie. skip properly. The test in question works fine (ie: is skipped because CCP4 is not made available) on our cctbx test system, but breaks in our final dials installer tests (where CCP4 is made available). CLIBD_MON is set to the /lib/data/monomers folder in the shared DLS CCP4 6.5 update 1 installation. Best wishes, Markus -----Original Message----- From: richard.gildea(a)diamond.ac.uk<mailto:[email protected]><mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] Sent: 07 April 2015 21:36 To: asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball Hi Aaron, I'm not sure your fix as I understand it actually helps. The test that was failing was in the cctbx module, not mmtbx, and as far as I am aware we are running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our nightly builds (and that is what we as developers are running locally on our own development machines). If individual tests (particularly those in the more general modules such as cctbx, iotbx, etc.) rely on external dependencies (such as chem_data) being available, then the individual tests should test for presence of said dependencies and skip gracefully if not available. I have a feeling that we're not even including mmtbx in our nightly build tests (too many fail due to missing dependencies such as chem_data). Cheers, Richard ________________________________ From: Aaron Brewster [asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>] Sent: 07 April 2015 20:29 To: cctbx mailing list Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); Johan Hattne; <dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]>> Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball Hi Markus, two points. First, you are correct that the dials builds are not passing tests related to the chem_data folder which includes monomer libraries. This is because chem_data is required by mmtbx but isn't included in the dials build. Partly because we're unsure if pieces of it are proprietary. In the meantime, I've worked with Nigel to change the cctbx_regression.test_nightly to look for the chem_data repository and skip the mmtbx tests if it's not present. mmtbx will still be tested nightly as part of the standalone cctbx and phenix packages. Because of this change, perhaps you could revert your recent changes to cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check for chem_data in every sub test as it is tested for at the top level now. Second, the log you provided is not the one on buildbot. Your log indicates you have a CCP4 installation that includes a monomer library that is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or CLIBD_MON environment variables set on the system in question? Thanks, -Aaron On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Cc'ing cctbx folks David Waterman wrote: So maybe once we have a source tarball up at dials.diamond.ac.uk<http://dials.diamond.ac.uk><http://dials.diamond.ac.uk><http://dials.diamond.ac.uk> (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------------------------------------------------------ BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF _______________________________________________ DIALS-support mailing list DIALS-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/dials-support

1 0

Re: [cctbxbb] [Dials-support] DIALS source tarball
by markus.gerstel＠diamond.ac.uk 08 Apr '15

08 Apr '15

Hi Richard, I think you've found it. I have unset CLIBD_MON, and now the test skips cleanly. Of course this also means that users should not run cctbx tests on their machines... Is there no central 'mmtbx_skip_test_if_chem_data_db_missing' function, that just goes 'prints "Skip this test"; exit' if the library isn't present? Or a boolean _is_available() function if you prefer. Either would resolve this problem as well as the current untestability of mmtbx. -Markus -----Original Message----- From: Gildea, Richard (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:15 To: Gerstel, Markus (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Markus, Presumably the test is looking for a monomer that is present in chem_data but not in the ccp4 monomer library? Could we unset the CLIBD_MON variable for the purposes of testing the dials installers? Cheers, Richard ________________________________________ From: Gerstel, Markus (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:07 To: Gildea, Richard (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Richard, Aaron, This is correct, we do not currently run mmtbx tests at all, as up to 107 of them do not fail gracefully, ie. skip properly. The test in question works fine (ie: is skipped because CCP4 is not made available) on our cctbx test system, but breaks in our final dials installer tests (where CCP4 is made available). CLIBD_MON is set to the /lib/data/monomers folder in the shared DLS CCP4 6.5 update 1 installation. Best wishes, Markus -----Original Message----- From: richard.gildea(a)diamond.ac.uk [mailto:[email protected]] Sent: 07 April 2015 21:36 To: asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball Hi Aaron, I'm not sure your fix as I understand it actually helps. The test that was failing was in the cctbx module, not mmtbx, and as far as I am aware we are running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our nightly builds (and that is what we as developers are running locally on our own development machines). If individual tests (particularly those in the more general modules such as cctbx, iotbx, etc.) rely on external dependencies (such as chem_data) being available, then the individual tests should test for presence of said dependencies and skip gracefully if not available. I have a feeling that we're not even including mmtbx in our nightly build tests (too many fail due to missing dependencies such as chem_data). Cheers, Richard ________________________________ From: Aaron Brewster [asbrewster(a)lbl.gov] Sent: 07 April 2015 20:29 To: cctbx mailing list Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); Johan Hattne; <dials-support(a)lists.sourceforge.net> Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball Hi Markus, two points. First, you are correct that the dials builds are not passing tests related to the chem_data folder which includes monomer libraries. This is because chem_data is required by mmtbx but isn't included in the dials build. Partly because we're unsure if pieces of it are proprietary. In the meantime, I've worked with Nigel to change the cctbx_regression.test_nightly to look for the chem_data repository and skip the mmtbx tests if it's not present. mmtbx will still be tested nightly as part of the standalone cctbx and phenix packages. Because of this change, perhaps you could revert your recent changes to cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check for chem_data in every sub test as it is tested for at the top level now. Second, the log you provided is not the one on buildbot. Your log indicates you have a CCP4 installation that includes a monomer library that is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or CLIBD_MON environment variables set on the system in question? Thanks, -Aaron On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> wrote: Cc'ing cctbx folks David Waterman wrote: > So maybe once we have a source tarball up at > dials.diamond.ac.uk<http://dials.diamond.ac.uk> (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] [Dials-support] DIALS source tarball
by markus.gerstel＠diamond.ac.uk 08 Apr '15

08 Apr '15

Hi Richard, Aaron, This is correct, we do not currently run mmtbx tests at all, as up to 107 of them do not fail gracefully, ie. skip properly. The test in question works fine (ie: is skipped because CCP4 is not made available) on our cctbx test system, but breaks in our final dials installer tests (where CCP4 is made available). CLIBD_MON is set to the /lib/data/monomers folder in the shared DLS CCP4 6.5 update 1 installation. Best wishes, Markus -----Original Message----- From: richard.gildea(a)diamond.ac.uk [mailto:[email protected]] Sent: 07 April 2015 21:36 To: asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball Hi Aaron, I'm not sure your fix as I understand it actually helps. The test that was failing was in the cctbx module, not mmtbx, and as far as I am aware we are running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our nightly builds (and that is what we as developers are running locally on our own development machines). If individual tests (particularly those in the more general modules such as cctbx, iotbx, etc.) rely on external dependencies (such as chem_data) being available, then the individual tests should test for presence of said dependencies and skip gracefully if not available. I have a feeling that we're not even including mmtbx in our nightly build tests (too many fail due to missing dependencies such as chem_data). Cheers, Richard ________________________________ From: Aaron Brewster [asbrewster(a)lbl.gov] Sent: 07 April 2015 20:29 To: cctbx mailing list Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); Johan Hattne; <dials-support(a)lists.sourceforge.net> Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball Hi Markus, two points. First, you are correct that the dials builds are not passing tests related to the chem_data folder which includes monomer libraries. This is because chem_data is required by mmtbx but isn't included in the dials build. Partly because we're unsure if pieces of it are proprietary. In the meantime, I've worked with Nigel to change the cctbx_regression.test_nightly to look for the chem_data repository and skip the mmtbx tests if it's not present. mmtbx will still be tested nightly as part of the standalone cctbx and phenix packages. Because of this change, perhaps you could revert your recent changes to cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check for chem_data in every sub test as it is tested for at the top level now. Second, the log you provided is not the one on buildbot. Your log indicates you have a CCP4 installation that includes a monomer library that is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or CLIBD_MON environment variables set on the system in question? Thanks, -Aaron On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> wrote: Cc'ing cctbx folks David Waterman wrote: > So maybe once we have a source tarball up at > dials.diamond.ac.uk<http://dials.diamond.ac.uk> (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] [Dials-support] DIALS source tarball
by Aaron Brewster 07 Apr '15

07 Apr '15

Hi Richard, At the moment, buildbot for dials runs these three tests: libtbx.import_all_ext cctbx_regression.test_nightly libtbx.run_tests_parallel module=dials nproc=auto verbosity=1 Which doesn't line up exactly with what you described. If you run libtbx.run_tests_parallel module=cctbx module=dxtbx, then you are right that the fix I committed wouldn't be relevant because mmtbx wouldn't be tested. However, since the LBL build bot runs cctbx_regression.test_nightly, my change is needed. After some more discussion, I see that Markus's changes to tst_geometry_restraints_2.py are fine. It looks like there are already sub-tests in there that check for the presence of mmtbx and chem data before running. I think tst_geometry_restraints_2.py was the only test outside of mmtbx that was looking for the monomer libraries. I'm re-running the buildbot dials build now to see if my change to cctbx_regression.test_nightly along with Markus's changes to tst_geometry_restraints_2.py clean up the build. Finally, and perhaps on a different note, should we add dxtbx to cctbx_regression.test_nightly? Right now the list of modules it tests includes libtbx, boost_adaptbx, scitbx, cctbx, iotbx, smtbx, and if chem_data is present, mmtbx. I don't think the build bot is running the dxtbx tests at present. -Aaron On Tue, Apr 7, 2015 at 1:36 PM, <richard.gildea(a)diamond.ac.uk> wrote: > Hi Aaron, > > I'm not sure your fix as I understand it actually helps. The test that was > failing was in the cctbx module, not mmtbx, and as far as I am aware we are > running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our > nightly builds (and that is what we as developers are running locally on > our own development machines). If individual tests (particularly those in > the more general modules such as cctbx, iotbx, etc.) rely on external > dependencies (such as chem_data) being available, then the individual tests > should test for presence of said dependencies and skip gracefully if not > available. I have a feeling that we're not even including mmtbx in our > nightly build tests (too many fail due to missing dependencies such as > chem_data). > > Cheers, > > Richard > > ________________________________ > From: Aaron Brewster [asbrewster(a)lbl.gov] > Sent: 07 April 2015 20:29 > To: cctbx mailing list > Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); > Johan Hattne; <dials-support(a)lists.sourceforge.net> > Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball > > Hi Markus, two points. > > First, you are correct that the dials builds are not passing tests related > to the chem_data folder which includes monomer libraries. This is because > chem_data is required by mmtbx but isn't included in the dials build. > Partly because we're unsure if pieces of it are proprietary. In the > meantime, I've worked with Nigel to change the > cctbx_regression.test_nightly to look for the chem_data repository and skip > the mmtbx tests if it's not present. mmtbx will still be tested nightly as > part of the standalone cctbx and phenix packages. Because of this change, > perhaps you could revert your recent changes to > cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check > for chem_data in every sub test as it is tested for at the top level now. > > Second, the log you provided is not the one on buildbot. Your log > indicates you have a CCP4 installation that includes a monomer library that > is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or > CLIBD_MON environment variables set on the system in question? > > Thanks, > -Aaron > > > > On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto: > markus.gerstel(a)diamond.ac.uk>> wrote: > Cc'ing cctbx folks > > David Waterman wrote: > > So maybe once we have a source tarball up at dials.diamond.ac.uk< > http://dials.diamond.ac.uk> (..) > > Speaking of which: our DIALS builds are currently failing at the installer > testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py > failing deep in mmtbx territory. > So far I could not figure out how to get this running properly. > > > Call: > > libtbx.python tst_geometry_restraints_2.py > > Standard Output > > Skipping exercise_with_zeolite(): input file not available > Skipping exercise_with_pdb(): chem_data directory not available > > Standard Error > > Traceback (most recent call last): > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", > line 866, in <module> > exercise_all(sys.argv[1:]) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", > line 862, in exercise_all > exercise_na_restraints_output_to_geo(verbose=verbose) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", > line 805, in exercise_na_restraints_output_to_geo > log=out1) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 5270, in run > log=log) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 4876, in __init__ > restraints_loading_flags=restraints_loading_flags, > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 2902, in __init__ > log=log, > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 2287, in __init__ > m_j=mm) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 1513, in get_lib_link > return mon_lib_srv.link_link_id_dict["rna3p"] > KeyError: 'rna3p' > > > (On a side note: We really should call multiple tests within the same > python file separately, instead of running them together within a signle > run() method. This file contains 4 tests, but these will result in only a > single return value for any upstream processing of the test results, such > as Jenkins.) > > -Markus > > -- > This e-mail and any attachments may contain confidential, copyright and or > privileged material, and are for the use of the intended addressee only. If > you are not the intended addressee or an authorised recipient of the > addressee please notify us of receipt by returning the e-mail and do not > use, copy, retain, distribute or disclose the information in or attached to > the e-mail. > Any opinions expressed within this e-mail are those of the individual and > not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for any > damage which you may sustain as a result of software viruses which may be > transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England > and Wales with its registered office at Diamond House, Harwell Science and > Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > >

1 0

Generating crystal copies
by Nicholas Pearce 07 Apr '15

07 Apr '15

Hi Everyone, I'm currently trying to generate crystal contacts/copies. What I want out at the end is a list of symmetry operations to generate the crystallographic copies that form contacts with my input model. I've been using the asu_mappings object, but I'm not sure I'm using the output correctly or if it's the right thing to use. Code snippet used to calculate the mappings: # Extract the xray structure from the reference hierarchy ref_struc = ref_hierarchy.extract_xray_structure(crystal_symmetry=crystal_symmetry) # Extract the mappings that will tell us the adjacent symmetry copies asu_mappings = ref_struc.asu_mappings(buffer_thickness=5) # Symmetry operations for each atom mappings = asu_mappings.mappings() I printed out all of the mappings, as I presumed the vast majority of the them would be the identity, meaning that any non-identity operations would be from atoms in the buffer area I have defined. These non-identity operations would presumably be the ones that would generate the crystal contacts. However, a large number of the atoms only had one mapping, which was not the identity - e.g. asu_mappings.get_rt_mx(x).as_xyz() might give '-x+1/2,-y+1/2,z-1/2' for an atom with only one mapping. The effect of this is that the operations given by this generate symmetry copies that do not contact my input model in real space. I presume this means that asu_mappings is mapping the atoms in my model to some other arbitrary asu, as well as the atoms in the buffer zone around it? Therefore the symmetry operations given are for these mapped atoms? Can anyone help, or suggest an alternative way to generate the crystal contacts? Thanks, Nick

4 3

Re: [cctbxbb] [Dials-support] DIALS source tarball
by markus.gerstel＠diamond.ac.uk 07 Apr '15

07 Apr '15

Cc'ing cctbx folks David Waterman wrote: > So maybe once we have a source tarball up at dials.diamond.ac.uk (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

programmatic access to PDB through cctbx?
by Kaushik Hatti 01 Apr '15

01 Apr '15

Hello, Sorry if this is not relevant to this mailing list. I was wondering if there are any modules available under cctbx to query PDB database programmatically. There was an option to SQL-query PDBe (from here : http://www.ebi.ac.uk/pdbe-as/pdbedatabase/ ). But, this service has been retired since Mar 2015. How else could complex query on PDB possible? An example query could be: list of PDBid with SCOPid="1hba" and (Resolution < 2 and R/Rfree < 15/20) or (Resolution < 3 and R/Rfree < 20/25 ) and Completeness < 95% and structureFactorAvailability="Y" Such a query was possible via above mentioned SQL link. However, I am unable to find any other means now. Thanks in advance, Kaushik Hatti, Molecular Biophysics Unit, Indian Institute of Science, Bangalore, India -- People living deeply have no fear of death - Anais Nin Caution: I am still the dumbest person I have ever known :-)

2 2

dxtbx test failure
by markus.gerstel＠diamond.ac.uk 25 Mar '15

25 Mar '15

Hello everyone, Our last nightly build on RHEL linux x64 failed because the test dxtbx/tests/tst_dxtbx.py , which normally takes roughly 3 seconds to run on the build machine, got stuck for over 5 minutes. All the other tests (cctbx/scitbx/libtbx/iotbx/rstbx and remaining dxtbx) completed successfully. Tests were run via libtbx.run_tests_parallel with nproc=4. As far as I can tell there is no debug output. The libtbx testing suite does not support timeouts like some timeout decorators on python unittest? As far as I can tell tst_dxtbx.py at the moment runs 5 separate tests, so for future debugging it may make sense to split these up. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

compilation issues
by Dave Lodowski 16 Mar '15

16 Mar '15

Hi, sorry if this is a repeat, but an email sent to the cctbx(a)cci.lbl.gov bounced back to me. We are attempting to install the DIALS package and CCTBX is required for compilation. During compilation of CCTBX, Xrender fails to find the X windows server. OS is Ubuntu LTS 12.04.5 text from console upon failure: Installing Xrender... log file is /home/dave/cctbx/build/base_tmp/Xrender_install_log getting package xrender-0.8.3.tar.gz... downloading from http://cci.lbl.gov/cctbx_dependencies : local copy is current using ./xrender-0.8.3.tar.gz (cached) installing xrender-0.8.3.tar.gz... Traceback (most recent call last): File "../auto_build/install", line 9, in <module> install_base_packages.installer(args=sys.argv, log=sys.stdout) File "../auto_build/install_base_packages.py", line 161, in __init__ self.build_dependencies(packages=packages) File "../auto_build/install_base_packages.py", line 505, in build_dependencies getattr(self, 'build_%s'%i)() File "../auto_build/install_base_packages.py", line 781, in build_render self.build_compiled_package_simple(pkg_name=pkg, pkg_name_label=name) File "../auto_build/install_base_packages.py", line 407, in build_compiled_package_simple log=pkg_log) File "../auto_build/install_base_packages.py", line 392, in configure_and_build self.call("./configure %s" % " ".join(list(config_args)), log=log) File "../auto_build/install_base_packages.py", line 243, in call return call(args, log=log, verbose=self.verbose, **kwargs) File "../auto_build/installer_utils.py", line 71, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to './configure --prefix="/home/dave/cctbx/build/base"' failed with exit code 1 Log file indicates everything fine till X is required then : checking for X... no configure: error: X is required, but it was either disabled or not found. any suggestions as to how to proceed would be appreciated thanks, --David Lodowski

2 1

test 2
by Nicholas Sauter 05 Mar '15

05 Mar '15

-- Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Physical Biosciences Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 64R0121 Berkeley, CA 94720-8118 (510) 486-5713

1 0