May 2016 - cctbxbb - phenix-online.org

Boost Python 1.56
by Luc Bourhis 14 Apr '17

14 Apr '17

Hi, I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick’s request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56. So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling. Best wishes, Luc

14 36

Git
by Graeme.Winter＠diamond.ac.uk 01 Aug '16

01 Aug '16

Folks, Can we revisit the idea of moving to git for cctbx? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 18

bootstrap.py build on Ubuntu
by David Waterman 14 Jun '16

14 Jun '16

Hi folks I recently tried building cctbx+dials on Ubuntu 15.04 following the instructions here: http://dials.github.io/documentation/installation_developer.html This failed during installation of pango-1.16.1. Looking at pango_install_log, I see the command that failed was as follows: gcc -DHAVE_CONFIG_H -I. -I. -I../.. -DSYSCONFDIR=\"/home/fcx32934/sw/dials_bootstrap_test/base/etc\" -DLIBDIR=\"/home/fcx32934/sw/dials_bootstrap_test/base/lib\" -DG_DISABLE_CAST_CHECKS -I../.. -DG_DISABLE_DEPRECATED -I/home/fcx32934/sw/dials_bootstrap_test/base/include -I/home/fcx32934/sw/dials_bootstrap_test/base/include/freetype2 -g -O2 -Wall -MT fribidi.lo -MD -MP -MF .deps/fribidi.Tpo -c fribidi.c -fPIC -DPIC -o .libs/fribidi.o In file included from fribidi.h:31:0, from fribidi.c:28: fribidi_config.h:1:18: fatal error: glib.h: No such file or directory The file glib.h appears to be in base/include/glib-2.0/, however this directory was not explicitly included in the command above, only its parent. This suggests a configuration failure in pango to me. Taking a look at base_tmp/pango-1.16.1/config.log, I see what look like the relevant lines: configure:22227: checking for GLIB configure:22235: $PKG_CONFIG --exists --print-errors "$GLIB_MODULES" configure:22238: $? = 0 configure:22253: $PKG_CONFIG --exists --print-errors "$GLIB_MODULES" configure:22256: $? = 0 configure:22304: result: yes but this doesn't tell me very much. Does anyone have any suggestions as to how I might proceed? Many thanks -- David

3 10

The curious incident of the test at install time
by R. D. Oeffner 13 May '16

13 May '16

Hi Markus, Those changes were made by me and Oleg. I just tested a fresh installation of phenix-dev-2407-intel-linux-2.6-centos5 which is built with cctbx_project revision 24442. The test of cctbx_project/cctbx/regression/tst_grm_pickling.py passed without problems. I'm not aware of why this test is failing on dials after installation. Since the test is to do with the phenix restraints manager perhaps this test is not critical to dials. Perhaps the phenix restraints manager code together with this test should be moved into the phenix repository but that is not my call. Rob -----Original Message----- From: cctbxbb-request(a)phenix-online.org Sent: Friday, May 13, 2016 20:00 To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 86, Issue 7 Message: 2 Date: Fri, 13 May 2016 09:47:18 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: [cctbxbb] The curious incident of the test at install time Message-ID: <415F7F573406BB498256C68158D45674612998A2(a)EXCHMBX01.fed.cclrc.ac.uk> Content-Type: text/plain; charset=iso-8859-1 Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------ Message: 3 Date: Fri, 13 May 2016 09:50:03 +0000 From: <Graeme.Winter(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] The curious incident of the test at install time Message-ID: <13999B6FC038EB44B067771E3CFBD091BC2E3E7A(a)EXCHMBX03.fed.cclrc.ac.uk> Content-Type: text/plain; charset="us-ascii" Markus I got this error this morning in the regular update / compile shuffle and it's on my to-nag list, also feel like I have seen this before... Cheerio G -----Original Message----- From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of markus.gerstel(a)diamond.ac.uk Sent: 13 May 2016 10:47 To: cctbxbb(a)phenix-online.org Subject: [cctbxbb] The curious incident of the test at install time Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb ------------------------------ _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb End of cctbxbb Digest, Vol 86, Issue 7 **************************************

1 0

The curious incident of the test at install time
by markus.gerstel＠diamond.ac.uk 13 May '16

13 May '16

Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

easy_mp stack traces
by Graeme.Winter＠diamond.ac.uk 13 May '16

13 May '16

Dear Gabor, some time back Markus added the capability to retain stack traces through easy_mp execution. These were reverted which now means that by default the stack traces are lost :( I don't have a clear understanding what the exact design of the newly built scheduling/stacktrace system is, perhaps you could explain this since we’re in the dark here. We are however pretty certain that the corner cases that lead to you modifying the code do not apply to us with DIALS or xia2. Could you please advise how we can now enable the stacktrace to be preserved globally, in all cases, and for all dials.*/xia2.* programs? Just today this has come up twice, and is a support issue for us, see e.g. https://github.com/dials/dials/issues/139 where we are simply flying blind. Thanks in advance and best wishes, Graeme & xia2 / DIALS team -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Installation with distutils?
by Luc Bourhis 09 May '16

09 May '16

Hi fellow cctbx developers, unless I am missing something, nobody has written a “setup.py” to install the cctbx with distutils in Python standard site-packages? If so, I am planning to contribute one as this would make it easier for us at Bruker. I am just asking in case I would duplicate existing work. Best wishes, Luc

3 4

problem when using Python to convert from numpy array to flex.int array
by Jun Aishima 04 May '16

04 May '16

I have used the bootstrap method to build cctbx with my own Python (in a virtual environment). I setup my virtual environment, then source $CCTBX/build/setpaths.sh (bash). I am currently trying to turn a detector image (uint16) read in with fabio, which results in a numpy ndarray, and then turn that into a scitbx flex array. Since I've had problems doing this in my system, I've written a little script that reproduces the problem: ********** import fabio file=fabio.open('testcrystal_1_001.img') # adsc detector image file.data[0] from scitbx.array_family import flex flex.int(file.data[3071]) # this is a 3072x3072 image. For some reason, row 0 causes no problems. ********** At the end of this email is the stack trace I get. Has anybody had similar issues? I first saw this when I was trying to convert the entire image into a flex array, but now I've seen it with only one row [3071], while still working for [0:1500], so it doesn't seem like a simple memory issue. Any help would be appreciated. Jun Aishima ********** show_stack(1): <stdin>(1) <module> libc backtrace (23 frames, most recent call last): /home/aishimaj/spotter/bin/python() [0x414199] /lib64/libc.so.6(__libc_start_main+0xfd) [0x366d21ed5d] /home/aishimaj/spotter/bin/python(Py_Main+0xb22) [0x4150b2] /home/aishimaj/spotter/bin/python(PyRun_AnyFileExFlags+0x6b) [0x4ce82b] /home/aishimaj/spotter/bin/python(PyRun_InteractiveLoopFlags+0x4e) [0x4ce01e] /home/aishimaj/spotter/bin/python(PyRun_InteractiveOneFlags+0x1ff) [0x4cdd9f] /home/aishimaj/spotter/bin/python(PyEval_EvalCode+0x32) [0x4ad362] /home/aishimaj/spotter/bin/python(PyEval_EvalCodeEx+0x877) [0x4ad257] /home/aishimaj/spotter/bin/python(PyEval_EvalFrameEx+0x6a0b) [0x4aa8bb] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /home/aishimaj/spotter/bin/python() [0x46df9f] /home/aishimaj/spotter/bin/python() [0x47623f] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /home/aishimaj/spotter/bin/python() [0x42296f] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2d088) [0x7f5561101088] /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x43) [0x7f5561112013] /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2fe08) [0x7f5561103e08] /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0xef) [0x7f55611031ff] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(boost::python::objects::signature_py_function_impl<boost::python::detail::caller<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >* (*)(boost::python::numeric::array const&), boost::python::detail::constructor_policy<boost::python::default_call_policies>, boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&> >, boost::mpl::v_item<void, boost::mpl::v_item<boost::python::api::object, boost::mpl::v_mask<boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&>, 1>, 1>, 1> >::operator()(_object*, _object*)+0x52) [0x7f555ed68452] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::boost_python::flex_int_from_numpy_array(boost::python::numeric::array const&)+0x29) [0x7f555f039f79] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > > scitbx::af::boost_python::versa_flex_from_numpy_array<int>(boost::python::numeric::array const&)+0x1c87) [0x7f555f0415c7] /lib64/libc.so.6() [0x366d2326a0] Segmentation fault (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Jun Aishima | Post Doc - MX | Australian Synchrotron p: (03) 8540 4114 | f: (03) 8540 4200 jun.aishima(a)synchrotron.org.au<mailto:[email protected]> | www.synchrotron.org.au 800 Blackburn Road, Clayton, Victoria 3168

3 6