- cctbxbb - phenix-online.org

Re: [cctbxbb] dials install error
by markus.gerstel＠diamond.ac.uk 12 Jan '16

12 Jan '16

Dear Sid, Unfortunately I don't know what the proper solution to your problem is. We had issues with pango/glibc reported just a few days ago, but I don't know what the outcome to that was. Dear David, Nick, Billy, Following up on the discussion in the thread 'bootstrap.py build on Ubuntu', here is an installer failure on CentOS5 that comes down to pango/libfontconfig/libglib missing some libraries, trace attached. -Markus From: Sidhu, Khushwant (Dr.) [mailto:[email protected]] Sent: 12 January 2016 11:26 To: Gerstel, Markus (DLSLtd,RAL,LSCI) Cc: Sidhu, Khushwant (Dr.) Subject: Re: dials install error Dear Markus, using the --verbose flag, I get the following .... ./install --verbose (snip) : Regenerating module files in /usr/local/dials-dev20160112/base : generating pangorc file at /usr/local/dials-dev20160112/base/etc/pango/pangorc : generating pango.modules file at /usr/local/dials-dev20160112/base/etc/pango/pango.modules : /usr/local/dials-dev20160112/base/bin/pango-querymodules: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /usr/local/dials-dev20160112/base/lib/libfontconfig.so.1) : /usr/local/dials-dev20160112/base/bin/pango-querymodules: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /usr/local/dials-dev20160112/base/lib/libglib-2.0.so.0) : Traceback (most recent call last): : File "/usr/local/dials-dev20160112/modules/cctbx_project/libtbx/configure.py", line 22, in <module> : if not run(): : File "/usr/local/dials-dev20160112/modules/cctbx_project/libtbx/configure.py", line 17, in run : libtbx.env_config.cold_start(sys.argv) : File "/usr/local/dials-dev20160112/modules/cctbx_project/libtbx/env_config.py", line 2267, in cold_start : env.refresh() : File "/usr/local/dials-dev20160112/modules/cctbx_project/libtbx/env_config.py", line 1544, in refresh : regenerate_module_files.run(libtbx.env.under_base('.'), only_if_needed=True) : File "/usr/local/dials-dev20160112/modules/cctbx_project/libtbx/auto_build/regenerate_module_files.py", line 153, in run : fix_pango(base_dir, out) : File "/usr/local/dials-dev20160112/modules/cctbx_project/libtbx/auto_build/regenerate_module_files.py", line 59, in fix_pango : call(("%s/bin/pango-querymodules > %s") % (base_dir, pangomodules), log=out) : File "/usr/local/dials-dev20160112/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call : raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) : RuntimeError: Call to '/usr/local/dials-dev20160112/base/bin/pango-querymodules > /usr/local/dials-dev20160112/base/etc/pango/pango.modules' failed with exit code 1 Traceback (most recent call last): File "/usr/local/dials-installer-dev/bin/install.py", line 46, in ? installer(sys.argv[1:]).install() File "/usr/local/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 197, in install self.install_from_binary() File "/usr/local/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 318, in install_from_binary self.reconfigure(log=log) File "/usr/local/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 484, in reconfigure call(args=args, log=log, verbose=self.options.verbose) File "/usr/local/dials-installer-dev/lib/libtbx/auto_build/installer_utils.py", line 81, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to '/usr/local/dials-dev20160112/base/bin/python /usr/local/dials-dev20160112/modules/cctbx_project/libtbx/configure.py --current_working_directory /usr/local/dials-dev20160112/build dials xia2' failed with exit code 1 regards Sid -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

status of mmtbx/phenix.molprobity
by Marcin Wojdyr 04 Jan '16

04 Jan '16

Dear All, we'd like to use mmtbx validation code. Is all the code in mmtbx meant to be open-source, even if the dispatchers have names starting with phenix? In particular I'm thinking about phenix.molprobity. (the names are not a problem, I just want to double check that it's ok if we use it) All the best in 2016, Marcin

2 1

Failure in phenix.map_box when working with map files....
by Ezra Peisach 11 Dec '15

11 Dec '15

Has anyone observed a failure of phenix.map_box when working with map files? When working with a structure factor file with map coefficients, it works fine. When a map file is used - the following occurs: Traceback (most recent call last): File "/local/scratch/CCTBX/cctbx_build/../cctbx_sources/mmtbx/command_line/map_box.py", line 194, in <module> run(args=sys.argv[1:]) File "/local/scratch/CCTBX/cctbx_build/../cctbx_sources/mmtbx/command_line/map_box.py", line 158, in run selection = selection) File "/local/scratch/CCTBX/cctbx_sources/mmtbx/utils/__init__.py", line 2938, in __init__ self.map_box = maptbx.copy(map_data, gridding_first, gridding_last) Boost.Python.ArgumentError: Python argument types in cctbx_maptbx_ext.copy(double, list, list) did not match C++ signature: copy(scitbx::af::const_ref<double, cctbx::maptbx::c_grid_padded_p1<3ul> > map_unit_cell, scitbx::af::tiny<int, 3ul> first, scitbx::af::tiny<int, 3ul> last) copy(scitbx::af::const_ref<float, cctbx::maptbx::c_grid_padded_p1<3ul> > map_unit_cell, scitbx::af::tiny<int, 3ul> first, scitbx::af::tiny<int, 3ul> last) copy(scitbx::af::const_ref<double, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > > map, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > result_grid) copy(scitbx::af::const_ref<float, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > > map, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > result_grid) Any suggestions? The map_data is the same format in the call....

4 5

crystallography Q&A site proposal on StackExchange
by Marcin Wojdyr 06 Nov '15

06 Nov '15

Hello, I'm writing about it first to developer mailing lists assuming that developers know StackOverflow. SO has now a number of sister sites on StackExchange and some Q&A sites in the same format are self-hosted, e.g. biostars.org. Here is a list of sites in the science category on SE: http://stackexchange.com/sites#science-traffic I though that such a site for crystallography would be useful, mostly because it has lower psychological and technical barrier for asking questions. I'm not sure how it's on phenixbb and other mailing lists, but on ccp4bb maybe a few percent of all 6000+ subscribers have asked anything at all. I assume people have questions in mind, they are just not eager to ask it by mailing 6000 other people. I don't know if a Q&A site would help, but just to try here is proposal for Crystallography site on SE: http://area51.stackexchange.com/proposals/91841/crystallography Using SE is simpler than hosting it, and even if the proposal doesn't make it at least we'll have an idea what was the interest. Technicalities. New sites are created basically when enough people is interested. Currently the proposal is in the "definition" stage, which means coming up with example questions - to show what kind of questions could be asked when the site exists. To complete this stage we need 40 question with 10+ votes. Everyone has 5 up-votes for selecting good questions. Votes can be changed at any time. So at least 80 voting people would be needed for this stage. Having more example questions would also help. Marcin

1 0

(no subject)
by Nigel Moriarty 06 Nov '15

06 Nov '15

Folks Calling for articles and short communications of interest to structural biologists. The deadline for submission in the January 2016 issue is 18 December 2015 (with some leeway if you are still working on the text, especially as it can be polished over the break). The Computational Crystallography Newsletter (CCN) is an electronic newsletter for structural biologists, and is published online every 6 months. Feature articles, meeting announcements and reports can be submitted to me at any time for consideration. Submission of text by email or word-processing files using the CCN templates is requested. Past newsletters and a Microsoft Word template are available at www.phenix-online.org/newsletter. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

2 2

Unused/old modules on the top-level of cctbx
by Oleg Sobolev 18 Oct '15

18 Oct '15

Hi, I went through top-level directories of cctbx and discovered the following: compcomm - two papers from IUCr computational crystallography authored by Ralf with some scripts. These scripts are referenced in these papers. One paper in LaTeX by Luc Bourhis "Small-molecule toolbox". dox - probably was meant as documentation, actually contains some documentation in RST, Ralf's talks from Siena, 2005, some tutorial in LaTeX, the content for the cctbx website. fftw3tbx - probably is not used anywhere ipctbx - most likely is not used anywhere, wrapper for sys/ipc.h - C library for interprocess communication access structure. rpmbuild - not sure whether it is still needed/working, last change to the only file was on 7-29-2009. setup - not clear whether the only .sh script in this directory is still needed/working. I suggest to move compcomm to dox despite the reference in 2009 paper and remove fftw3tbx, ipctbx, rpmbuild, setup if nobody has an idea why they should stay where they are/be moved somewhere. Best regards, Oleg Sobolev.

3 3

cctbx documentation
by Paul Adams 13 Oct '15

13 Oct '15

Dear Colleagues, we are looking at the cctbx documentation. Currently the world is directed to the documentation on the Sourceforge site, which hasn’t been updated for a long time and is looking somewhat old. Last year, Oliver and Ian (and others) worked on Sphinx-based documentation for cctbx (the last published version of which can be found here: http://cci.lbl.gov/cctbx_docs/). The question for the cctbx developer community therefore is whether we should move over to the Sphinx system for cctbx documentation. The benefits are: a more consistent and modern interface, and automated documentation generation from source code header file information. As part of making this move we would need to integrate some of the additional information from the original cctbx site into the Sphinx documentation. We would also move the hosting of the cctbx documentation to the CCI web server in Berkeley (and off the Sourceforge site). Another email related to that topic soon. I’d appreciate any feedback on this topic. Cheers, Paul -- Paul Adams Interim Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 80, Room 247 Building 978, Room 4126 Tel: 1-510-486-4225, Fax: 1-510-486-5909 http://cci.lbl.gov/paul Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ LBenvenue(a)lbl.gov ][ 1-510-495-2506 ] --

2 1

cctbx and Github
by Paul Adams 11 Oct '15

11 Oct '15

Dear Colleagues, there has been some discussion in this bulletin board about moving the cctbx repository from Sourceforge to Github. We are seriously considering this option, especially in light of the Sourgeforge outage earlier in the year. One concern is what impact this will have on the cctbx developer community - which goes beyond Phenix and DIALS. We’d appreciate any feedback about this proposed move. Note that this will take some time to orchestrate and will involve contacting any known users of the cctbx package to inform them of the change. Cheers, Paul -- Paul Adams Interim Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 80, Room 247 Building 978, Room 4126 Tel: 1-510-486-4225, Fax: 1-510-486-5909 http://cci.lbl.gov/paul Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ LBenvenue(a)lbl.gov ][ 1-510-495-2506 ] --

1 0

CentOS7 - fixed
by markus.gerstel＠diamond.ac.uk 30 Sep '15

30 Sep '15

Thanks for your suggestions. They did not do the trick though. Some more detective work got me down to ../base/bin/python ../modules/cctbx_project/libtbx/configure.py dials failing and both libtbx.python ../modules/cctbx_project/libtbx/configure.py dials and LD_LIBRARY_PATH=lib ../base/bin/python ../modules/cctbx_project/libtbx/configure.py dials succeeding. A run with LD_DEBUG=files identified the problem: /lib64/libboost_python.so: error: symbol lookup error: undefined symbol: PyUnicodeUCS4_FromEncodedObject (fatal) CentOS7 comes with its own set of incompatible boost libraries. I've put a fix in: http://sourceforge.net/p/cctbx/code/22986/ Installation on CentOS7 now works for me. As far as I could tell this does not break other things. -Markus ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Nigel Moriarty [nwmoriarty(a)lbl.gov] Sent: Tuesday, September 29, 2015 18:36 To: cctbx mailing list Subject: Re: [cctbxbb] CentOS7 bugs (cont.) libtbx.configure --scan-boost phenix Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> On Tue, Sep 29, 2015 at 10:20 AM, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]>> wrote: Nigel, Do remember how to force libtbx.scons to scan the boost directory? Nick On Tue, Sep 29, 2015 at 9:25 AM, Nigel Moriarty <nwmoriarty(a)lbl.gov<mailto:[email protected]>> wrote: Markus The default usage of libtbx.scons in the build directory does not scan the boost directory so maybe that has something to do with the .so not behaving on Centos7 - because it was built on Centos5/6? Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709<tel:510-486-5709> Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909<tel:510-486-5909> Web : CCI.LBL.gov<http://CCI.LBL.gov> On Tue, Sep 29, 2015 at 8:29 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> wrote: Hello cctbx folks, We are having problems running, building and/or installing DIALS on CentOS7. Running the linux installer unceremoniously ends in a segfault. The installer is even broken when it is build on the same system using bootstrapped cctbx/dials. The broken installer can be traced to the DIALS extension loader, which uses python’s pkgutil loader, which triggers some library scanning routine within python, which in turn falls over boost_python_meta_ext.so and dies with the strace shown in the email below. The installer can be successfully “fixed” by deleting build/lib/boost_python_meta_ext.so during the installation. So I see two issues here: 1. What is broken with build/lib/boost_python_meta_ext.so that it causes segfaults on CentOS7? 2. Why is this not covered by any test? Neither of the following caught anything: a. python bootstrap.py builder=dials tests b. libtbx.run_tests_parallel module=cctbx module=dials module=boost_adaptbx [NB: The previously reported 3 broken tests on CentOS7 are red herrings and are all fixed by running ‘yum update’] I don’t know how to proceed from here. -Markus From: markus.gerstel(a)diamond.ac.uk<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] Sent: 29 September 2015 13:29 To: dials-support(a)lists.sourceforge.net<mailto:[email protected]> Subject: Re: [Dials-support] CentOS7 bugs (cont.) Hello everyone, (..) Sorry, I hit ctrl+enter by accident… In addition to these failed tests the installer dies a boost-related death according to strace: (..) poll([{fd=13, events=POLLIN|POLLPRI}, {fd=15, events=POLLIN|POLLPRI}], 2, 4294967295) = 1 ([{fd=13, revents=POLLIN}]) read(13, "\tlinux-vdso.so.1 => (0x00007fff"..., 4096) = 486 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7648, si_status=0, si_utime=0, si_stime=0} --- poll([{fd=13, events=POLLIN|POLLPRI}, {fd=15, events=POLLIN|POLLPRI}], 2, 4294967295) = 2 ([{fd=13, revents=POLLHUP}, {fd=15, revents=POLLHUP}]) close(13) = 0 close(15) = 0 wait4(7648, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 7648 stat("$prefix/dials-dev20150928/build/lib/boost_python_meta_ext", 0x7fff6421d3f0) = -1 ENOENT (No such file or directory) open("$prefix/dials-dev20150928/build/lib/boost_python_meta_ext.so", O_RDONLY) = 11 fstat(11, {st_mode=S_IFREG|0775, st_size=200624, ...}) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x18a86} --- +++ killed by SIGSEGV (core dumped) +++ Segmentation fault (core dumped) Not sure if relevant, but CentOS7 compiler is GCC 4.8.3 20140911 (Red Hat 4.8.3-9) compared to RHEL6 GCC 4.4.7 20120313 (Red Hat 4.4.7-11) Anybody have any ideas? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Physical Biosciences Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 64R0121 Berkeley, CA 94720-8118 (510) 486-5713<tel:%28510%29%20486-5713> _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb

1 0

Re: [cctbxbb] CentOS7 bugs (cont.)
by markus.gerstel＠diamond.ac.uk 29 Sep '15

29 Sep '15

Hello cctbx folks, We are having problems running, building and/or installing DIALS on CentOS7. Running the linux installer unceremoniously ends in a segfault. The installer is even broken when it is build on the same system using bootstrapped cctbx/dials. The broken installer can be traced to the DIALS extension loader, which uses python's pkgutil loader, which triggers some library scanning routine within python, which in turn falls over boost_python_meta_ext.so and dies with the strace shown in the email below. The installer can be successfully "fixed" by deleting build/lib/boost_python_meta_ext.so during the installation. So I see two issues here: 1. What is broken with build/lib/boost_python_meta_ext.so that it causes segfaults on CentOS7? 2. Why is this not covered by any test? Neither of the following caught anything: a. python bootstrap.py builder=dials tests b. libtbx.run_tests_parallel module=cctbx module=dials module=boost_adaptbx [NB: The previously reported 3 broken tests on CentOS7 are red herrings and are all fixed by running 'yum update'] I don't know how to proceed from here. -Markus From: markus.gerstel(a)diamond.ac.uk [mailto:[email protected]] Sent: 29 September 2015 13:29 To: dials-support(a)lists.sourceforge.net Subject: Re: [Dials-support] CentOS7 bugs (cont.) Hello everyone, (..) Sorry, I hit ctrl+enter by accident... In addition to these failed tests the installer dies a boost-related death according to strace: (..) poll([{fd=13, events=POLLIN|POLLPRI}, {fd=15, events=POLLIN|POLLPRI}], 2, 4294967295) = 1 ([{fd=13, revents=POLLIN}]) read(13, "\tlinux-vdso.so.1 => (0x00007fff"..., 4096) = 486 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7648, si_status=0, si_utime=0, si_stime=0} --- poll([{fd=13, events=POLLIN|POLLPRI}, {fd=15, events=POLLIN|POLLPRI}], 2, 4294967295) = 2 ([{fd=13, revents=POLLHUP}, {fd=15, revents=POLLHUP}]) close(13) = 0 close(15) = 0 wait4(7648, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 7648 stat("$prefix/dials-dev20150928/build/lib/boost_python_meta_ext", 0x7fff6421d3f0) = -1 ENOENT (No such file or directory) open("$prefix/dials-dev20150928/build/lib/boost_python_meta_ext.so", O_RDONLY) = 11 fstat(11, {st_mode=S_IFREG|0775, st_size=200624, ...}) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x18a86} --- +++ killed by SIGSEGV (core dumped) +++ Segmentation fault (core dumped) Not sure if relevant, but CentOS7 compiler is GCC 4.8.3 20140911 (Red Hat 4.8.3-9) compared to RHEL6 GCC 4.4.7 20120313 (Red Hat 4.4.7-11) Anybody have any ideas? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

Phi-Psi Angles
by Nicholas Pearce 21 Sep '15

21 Sep '15

Hi Everyone, Phenomenally stupid question, but is there an easy way to calculate phi-psi angles for a protein using cctbx? And, following on from that, Chi-1, Chi-2 etc angles? I’ve had a poke around, but couldn’t find any obvious (at least to me) way that worked… I presume this must be implemented rigorously somewhere, and I didn’t want to waste time reimplementing the wheel. Thanks, Nick

2 2

Move to github
by Graeme.Winter＠diamond.ac.uk 15 Sep '15

15 Sep '15

Hi Folks A while back we asked about moving cctbx (along with DIALS) to github; postponement was asked until Phenix release is out. Phenix release is out. How do people feel about this? Enthusiastic endorsement of this project from DIALS-East & xia2 developer camps. Latter has already migrated… Go? No go? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Moving to github
by Graeme.Winter＠diamond.ac.uk 08 Sep '15

08 Sep '15

Hi Folks Looks like phenix release has happened… should we begin the conversation about moving cctbx and DIALS over to the happy world of github? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 3

tst_miller failing
by markus.gerstel＠diamond.ac.uk 26 Aug '15

26 Aug '15

Hi everyone, particularly Pavel I guess, I had a look at one intermittently failing test: cctbx/regression/tst_miller.py, run with parameter P31 The relevant piece of code is: def exercise_randomize_amplitude_and_phase(): xrs = random_structure.xray_structure( space_group_info=sgtbx.space_group_info(number=1), elements=["C"]*300) fc = xrs.structure_factors(d_min=1).f_calc() def run(a,p): fc_ = fc.randomize_amplitude_and_phase(amplitude_error=a, phase_error_deg=p, random_seed=1312425) d1, d2 = flex.abs(fc.data()), flex.abs(fc_.data()) r = flex.sum(flex.abs(d1-d2))/flex.sum(flex.abs(d1+d2))*2 return (r, fc.mean_phase_error(phase_source=fc_)) # assert approx_equal(run(0,0), (0,0)) for v in list(xrange(0,91, 10)): r = run(v/100.,v) assert approx_equal(r[0], v/100., 0.01) assert approx_equal(r[1], v, 1) The test fails in the last line for varying values of v. I saw that the values of r[] increase proportionally to v, so I ran the test 2000 times for v=90, plotting the values of r[0] and r[1]; results attached. With the current statistics the r[1] test is expected to fail in 1 run out of every 80. Could you please have a look at this, and either increase the tolerance on the test or identify whether these values are true unexpected outliers indicating a problem in the code? Don't know what the r[1] distribution should look like, but that's clearly not normal. :) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Structure factor calculation for Si
by Allan Lyckegaard 26 Aug '15

26 Aug '15

Dear CCTBX-list I am trying to reproduce the (scaled) intensities for Si reported here: http://rruff.geo.arizona.edu/AMS/xtal_data/DIFfiles/15285.txt based on this information: http://rruff.geo.arizona.edu/AMS/CIF_text_files/15285_cif.txt I am using CCTBX in Python: from cctbx.array_family import flex from cctbx import sgtbx, crystal, miller, xray sgnum = 227 sg = sgtbx.space_group_info(sgnum).group() latticepar = [5.43070, 5.43070, 5.43070, 90, 90, 90] dmin = 1.0 atompos = [['Si',(0.0, 0.0, 0.0)]] scatterers = flex.xray_scatterer() scatterers.append(xray.scatterer(label=atompos[0][0], site=atompos[0][1])) pg = sg.build_derived_point_group() sym = crystal.symmetry(space_group=sg,unit_cell=latticepar) sps = crystal.special_position_settings(crystal_symmetry=sym) structure = xray.structure(special_position_settings=sps,scatterers=scatterers) F = structure.structure_factors(d_min=dmin,algorithm='direct').f_calc() list(F.amplitudes()) This gives: [((0, 2, 2), 0.0), ((0, 4, 0), 120.21765812638594), ((1, 1, 1), 84.18849550806966), ((1, 3, 1), 65.48371727205928), ((1, 3, 3), 57.45438308324155), ((1, 5, 1), 51.48759686950681), ((2, 2, 2), 128.54540698861973), ((2, 4, 2), 0.0), ((3, 3, 3), 51.48759686950681)] How come CCTBX give an amplitude of 0.0 for (0,2,2) and (2,4,2)? The measured intensities in the DIFfile above does not show this. Am I missing something? Cheers, /Allan

4 3

Computational Crystallography Newsletter - Volume 6, Number 2
by Nigel Moriarty 25 Aug '15

25 Aug '15

I am pleased to announce the publication of the latest issue of the Computational Crystallography Newsletter: http://www.phenix-online.org/newsletter/ A listing of the articles and short communications is given below. Please note that the newsletter accepts articles of a general nature of interest to all crystallographers. Please send any articles to me at NWMoriarty(a)lbl.gov noting that there is a Word Template on the website to streamline production. July 2015 <http://www.phenix-online.org/newsletter/CCN_2015_07.pdf> Articles - A context-sensitive guide to RNA & DNA base-pair & base-stack geometry <http://www.phenix-online.org/newsletter/CCN_2015_07.pdf#page=22> Short Communications - New Phaser-MR search panel in Phenix <http://www.phenix-online.org/newsletter/CCN_2015_07.pdf#page=7> - Quantum chemical techniques for minimising ligand geometries in the active site <http://www.phenix-online.org/newsletter/CCN_2015_07.pdf#page=10> - 13 typical occupancy refinement scenarios and available options in *phenix.refine* <http://www.phenix-online.org/newsletter/CCN_2015_07.pdf#page=12> Fitting tips #10 – How do your base pairs touch and twist? <http://www.phenix-online.org/newsletter/CCN_2015_07.pdf#page=3> Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Shelx HKL file with original indices from unmerged MTZ
by Graeme.Winter＠diamond.ac.uk 06 Aug '15

06 Aug '15

Hi Folks Using reader = any_reflection_file(hklin) intensities = [ma for ma in reader.as_miller_arrays(merge_equivalents=False) if ma.info().labels == ['I', 'SIGI']][0] help(intensities) with open('%s.hkl' % prefix, 'wb') as f: intensities.export_as_shelx_hklf(f, normalise_if_format_overflow=True) to get a file for input to shelx and it appears to have only indices in the ASU not the Fridel pairs. What do I need to do to output with original indices (ie. P1 indices?) I assume that this has been solved before... Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Tonights build/testing failures
by markus.gerstel＠diamond.ac.uk 05 Aug '15

05 Aug '15

Hi everyone, tonights test (cctbx) or build (dials) failures were caused by changes to the CBFlib in r568. http://sourceforge.net/p/cbflib/code-0/commit_browser Error messages may include ImportError: (..)/build/lib/_pycbf.so: undefined symbol: cbf_airy_disk_volume Do we have a red phone to cbflib folks? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

cctbx developers: sf outage has lead to local corruption of your cctbx_project
by Aaron Brewster 27 Jul '15

27 Jul '15

Hi folks, because of the recent parallel commits to the cctbx repository due to the sourceforge outage, you will likely need to move your old cctbx_project directory aside and check it out fresh again. The 'too long didn't read' version of this email is you need to do this or several files will remain out of sync. This email only applies to developers or users who use svn directly. How to proceed: $ cd <your sources/modules directory> $ mv cctbx_project cctbx_project.old $ svn co https://svn.code.sf.net/p/cctbx/code/trunk cctbx_project If you have local edits in your cctbx_project directory, you'll need to copy them over to your new folder. More detail for those interested. The following example illustrates the problem: [phenix-src/cctbx_project/scitbx/lstbx]$ svn update [phenix-src/cctbx_project/scitbx/lstbx]$ svn status According to svn, this directory is up to date [phenix-src/cctbx_project/scitbx/lstbx]$ svn info normal_equations.h Path: normal_equations.h Name: normal_equations.h URL: https://svn.code.sf.net/p/cctbx/code/trunk/scitbx/lstbx/normal_equations.h Repository Root: https://svn.code.sf.net/p/cctbx/code Repository UUID: 3c1a6239-ac2c-0410-bc49-92500e444112 Revision: 22630 Node Kind: file Schedule: normal Last Changed Author: luc_j_bourhis Last Changed Rev: 15039 Last Changed Date: 2012-03-12 10:03:05 -0700 (Mon, 12 Mar 2012) Text Last Updated: 2015-07-27 10:24:16 -0700 (Mon, 27 Jul 2015) Checksum: ec2e555a9ad1d2553bb37bc04f677ef5 Note the revision number is 22630, and it lists the last revision for this file as 15039. After I move my directory aside and re-check out the directory: [phenix-src/cctbx_project/scitbx/lstbx]$ svn info normal_equations.h Path: normal_equations.h Name: normal_equations.h URL: https://svn.code.sf.net/p/cctbx/code/trunk/scitbx/lstbx/normal_equations.h Repository Root: https://svn.code.sf.net/p/cctbx/code Repository UUID: 3c1a6239-ac2c-0410-bc49-92500e444112 Revision: 22630 Node Kind: file Schedule: normal Last Changed Author: nksauter Last Changed Rev: 22624 Last Changed Date: 2015-07-25 17:16:55 -0700 (Sat, 25 Jul 2015) Text Last Updated: 2015-07-27 10:46:36 -0700 (Mon, 27 Jul 2015) Checksum: e5e5e623ecdde362afadf321d37b2c5e The correct revision is now retrieved. Thanks, -Aaron

1 0

Re: [cctbxbb] git on BuildBot machines
by markus.gerstel＠diamond.ac.uk 26 Jul '15

26 Jul '15

Hi Aaron, Is there some documentation about the buildbot objects I could use to figure out a sensible way to determine whether git is available? I could of course just do try-run-catch... -Markus On 26 Jul 2015 16:35, Aaron Brewster <asbrewster(a)lbl.gov> wrote: Moving the cctbx_project directory to cctbx_project.old forced svn to get the latest version from scratch so buildbot at LBL has no versioning issues like Jenkins at Diamond does. Further, svn update on my local sources got the latest version, so I'm inclined to agree with Markus that that's a Jenkins issue. I installed git on the build nodes at LBL to accommodate the changes to bootstrap.py, however reconfig.sh still fails. This new line from r22623 and r22626 is the culprit: git_available = self.shell(command=['git', '--version'], haltOnFailure=False, quiet=True, silent=True).run() == 0 While this clearly works on Jenkins, when ran by buildbot, self.shell() is returning a buildbot shell object instead of a Builder shell object, so the .run call fails (not implemented). I'm out of time until this afternoon to look deeper. I've left the build bot server running using r22622 of bootstrap.py for now. -Aaron On Sun, Jul 26, 2015 at 7:00 AM, Nigel Moriarty <nwmoriarty(a)lbl.gov<mailto:[email protected]>> wrote: Aaron Recent changes to include git in bootstrap.py have broken the buildbot master (won't start) because git is not available on the VM. I don't have time before I fly to ACA, so please install git when you can. One reason I was hoping to see the buildbot website was because it has an interface to see all the commits to cctbx. This is likely on the buildbot master in some other format. It may be useful that I just moved the old cctbx_project to cctbx_project.old on all the builds. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

lost cctbx commits
by Aaron Brewster 26 Jul '15

26 Jul '15

Hi folks, after more investigation I now agree we have parallel commits. The following commits have been lost, all done on July 15th: nwmoriatry: r22622 phyy-nx: r22623-4 tmmclark: r22625 olegsobolev, r22626-7, r22629 alyubimov: r22628 These revisions number have been reused by svn and are no longer valid. The authors should recommit their changes. -Aaron

2 2

git on BuildBot machines
by Nigel Moriarty 26 Jul '15

26 Jul '15

Aaron Recent changes to include git in bootstrap.py have broken the buildbot master (won't start) because git is not available on the VM. I don't have time before I fly to ACA, so please install git when you can. One reason I was hoping to see the buildbot website was because it has an interface to see all the commits to cctbx. This is likely on the buildbot master in some other format. It may be useful that I just moved the old cctbx_project to cctbx_project.old on all the builds. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

2 1

SVN is back on SourceForge
by Nigel Moriarty 26 Jul '15

26 Jul '15

Folks The return of SVN on SF has caused a few problems on the BuildBot but I moved the cctbx_project directory and forced some builds. I'm hoping things will progress but will check later tonight. There have been some strange error messages causing some emails to be send but I'm hoping things will settle soon. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

7 9

console
by Nigel Moriarty 26 Jul '15

26 Jul '15

I reverted the version of bootstrap.py on cci-vm-6 and got the attached console GUI. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Re: [cctbxbb] after SourceForge awakens
by Graeme.Winter＠diamond.ac.uk 22 Jul '15

22 Jul '15

Hi Nigel We're thinking along the same lines moving xia2 across first then maybe dials, depending on how people feel. Moving all of it across in the longer term would work well. Thanks Graeme On 20 Jul 2015 20:42, Nigel Moriarty <nwmoriarty(a)lbl.gov> wrote: Folks I have not read the emails regarding the current SourceForge situation. However, we had been planning to move to github in the near future. SF has been very reliable so I don't think it was too much of pain given the long use we've had from them. It does, however, come at a time when we are trying to get out a major release of Phenix. To that end, it would be helpful if any commits were bug fixes only to minimise the risk of breaking to the codebase. We will be actively evaluating the options regarding to github after Phenix 1.10 is safely away... Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

after SourceForge awakens
by Nigel Moriarty 20 Jul '15

20 Jul '15

Folks I have not read the emails regarding the current SourceForge situation. However, we had been planning to move to github in the near future. SF has been very reliable so I don't think it was too much of pain given the long use we've had from them. It does, however, come at a time when we are trying to get out a major release of Phenix. To that end, it would be helpful if any commits were bug fixes only to minimise the risk of breaking to the codebase. We will be actively evaluating the options regarding to github after Phenix 1.10 is safely away... Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Re: [cctbxbb] List usage
by Graeme.Winter＠diamond.ac.uk 20 Jul '15

20 Jul '15

Hi Nigel Sensible however unknowable in advance that this is the behaviour & also who is on list! Sorry Graeme On 20 Jul 2015 18:13, Nigel Moriarty <nwmoriarty(a)lbl.gov> wrote: Folks To those of you on the cctbx list (and even those not), as moderator I have received over a dozen requests from the server. One cause of the problem is non-members posting to a members-only list. This is usually caused by people including the ccbx list in a long list of recipients. This leads to another problem. The list complains about too many recipients. I'm sure that some of the discussion regarding the current SourceForge situation has been garbled as a result. There is an easy solution. When sending to the list, include only the list in the recipients. Anybody who is interested should subscribe to the list. Non members such as David Waterman should be encouraged to join the list. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

List usage
by Nigel Moriarty 20 Jul '15

20 Jul '15

Folks To those of you on the cctbx list (and even those not), as moderator I have received over a dozen requests from the server. One cause of the problem is non-members posting to a members-only list. This is usually caused by people including the ccbx list in a long list of recipients. This leads to another problem. The list complains about too many recipients. I'm sure that some of the discussion regarding the current SourceForge situation has been garbled as a result. There is an easy solution. When sending to the list, include only the list in the recipients. Anybody who is interested should subscribe to the list. Non members such as David Waterman should be encouraged to join the list. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Building cctbx bundles without phenix.* dispatcher scripts
by Graeme.Winter＠diamond.ac.uk 02 Jul '15

02 Jul '15

Hi Folks Following on from various conversations going on here about programs in the default dials bundle which necessarily includes cctbx & lots of useful utilities in there, the question was asked why it includes bits of Phenix. In real life it does not include bits of phenix - they are bits of e.g. mmtbx which have dispatcher scripts which ask to be called phenix.xyz for example, as they are *also* used in phenix. Is there a way we can set the dispatcher prefix at the time of running libtbx.configure? That way you could have phenix.xyz or cctbx.xyz in other situations… then in the dials bundles we could make it clear that there are no bits of Phenix in there. Many thanks in advance, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

bootstrap external dependencies
by markus.gerstel＠diamond.ac.uk 16 Jun '15

16 Jun '15

Good morning, last night we ran into a problem with some of the remaining external dependencies in the bootstrap script base install step. The sphinx package will currently download 11 files during its installation (list below). As with all other downloads I would like to be able to download these files beforehand and then pass them to our build systems directly, so that the subsequent builds are no longer dependant on network connectivity or pypi server reliability. I would like to propose a python-extra package for the bootstrap script, that checks the existence of a base_tmp/python_extra directory, and, if found, would install all python packages in that folder. This would be an easy facility to automatically build extra python packages (at DLS we already add a few extra packages, such as junit-xml, so this would actually simplify our build process), and would not affect any other bootstrap users. Alternatively, since the dependencies are fixed as they come from the Sphinx archive on the cci servers, we could package them up as well and put static build instructions into build_sphinx() in install_base_packages.py. Thoughts? -Markus The packages are: https://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.7.3.tar.gz#md5=b9… https://pypi.python.org/packages/source/d/docutils/docutils-0.12.tar.gz#md5… https://pypi.python.org/packages/source/P/Pygments/Pygments-2.0.2.tar.gz#md… https://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.23.tar.gz… https://pypi.python.org/packages/source/n/nose/nose-1.3.7.tar.gz#md5=4d3ad0… https://pypi.python.org/packages/source/p/pyparsing/pyparsing-2.0.3.tar.gz#… https://pypi.python.org/packages/source/t/tornado/tornado-4.2.tar.gz#md5=c5… https://pypi.python.org/packages/source/p/python-dateutil/python-dateutil-2… https://pypi.python.org/packages/source/c/certifi/certifi-2015.04.28.tar.gz… https://pypi.python.org/packages/source/b/backports.ssl_match_hostname/back… https://pypi.python.org/packages/source/s/six/six-1.9.0.tar.gz#md5=476881ef… -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Race condition in bootstrap python installation
by markus.gerstel＠diamond.ac.uk 27 May '15

27 May '15

We have recently seen intermittent build failures with the bootstrap base installation at Diamond. These all occurred during the compilation of python and were usually of the form : gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wl,-rpath=\$ORIGIN/../lib Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -lpthread -ldl -lutil -o Parser/pgen : Parser/node.o: file not recognized: File truncated : collect2: ld returned 1 exit status : make[1]: *** [Parser/pgen] Error 1 : make[1]: Leaving directory `/scratch/jenkins_slave/workspace/dials_bootstrap_platforms/compilationtarget/native/label/dials-ws154/build_dials/base_tmp/Python-2.7.8_cci' : make: *** [Include/graminit.h] Error 2 : make: *** Waiting for unfinished jobs.... Traceback (most recent call last): File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1194, in <module> installer(args=sys.argv, log=sys.stdout) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 184, in __init__ self.build_dependencies(packages=packages) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 571, in build_dependencies getattr(self, 'build_%s'%i)() File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 622, in build_python self.call('make -j %s install'%(self.nproc), log=log, cwd=python_dir) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 264, in call return call(args, log=log, verbose=self.verbose, **kwargs) File "/scratch/jenkins_slave/workspace/dials_bootstrap_platforms/compilationtarget/native/label/dials-ws154/build_dials/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to 'make -j 4 install' failed with exit code 2 Other indicated errors are /usr/bin/ld: final link failed: File truncated or /usr/bin/ld: can not read symbols: File truncated I suspect that these errors are manifestations of a race condition in the python build process, which the bootstrap script now by default runs with a -j 4 parallel make. To test this we are now using --nproc=1 for the base installation step. We haven't seen these build failures since. I suggest we set the python compilation to never run with parallel make. Any other ideas? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Native python unittests for the libtbx testing framework
by markus.gerstel＠diamond.ac.uk 19 May '15

19 May '15

Dear cctbx developers and users, For a side-project of mine I experimented with native python unit tests (some information available on https://docs.python.org/2/library/unittest.html if you are not familiar with it). The python unittest framework has a couple of advantages over the libtbx testing world. I found that the two most immediate advantages are: Automated test discovery. There is no need to keep a complete list of test files in another python file, which is what we currently do in run_tests.py. Multiple test functions per python test file. At the moment we can run multiple tests via command line parameters to the python test files. This of course means that you need to keep a list of '(test file, parameter)' tuples in run_tests.py, and you need some boilerplate dispatcher in your test file. This can lead to two situations: you may have a number of tests with numeric parameters, and no one knows what the parameters stand for. One example of this can be found in DIALS, where tst_index.py is run with numeric parameters 1 to 4 and 7 to 15, but not 5 and 6. The alternative situation is that you end up with a massive test file, that tests the entire class under test, but figuring out what specifically went wrong is probably more time consuming than necessary. In contrast in Python unittests any function in the file that is named test_* will be identified and run as a separate test. I find that this automatically leads to smaller and thus more readable tests. I understand that libtbx has been developed before Python unittest was around, that there is a substantial amount of code around that relies on current libtbx behaviour, and that there is no pressing need to change anything. However I would like to be able to take advantage of the tools that are out there, so I wrote a function that uses Python unittest for test discovery, and can be placed in run_tests.py. With this function you can then use regular Python unittests and run them with either through libtbx, with the python unittest tools, or directly. In each case you end up with the test results reported in the way you would expect. What I would like to do is integrate this logic in libtbx directly, so that we can use it for all cctbx modules. Since it does not modify libtbx default behaviour and only becomes active after a change in the run_tests.py of the relevant module it should be fully backwards compatible with virtually no downsides. What do you think? -Markus run_tests.py previously: (..) tst_list = ( "$D/test/algorithms/profile_model/nave/tst_model.py", ) (..) run_tests.py new: (..) tst_list = ( "$D/test/algorithms/profile_model/nave/tst_model.py", ) + discover_unittests("dlstbx") (..) discover_unittests function: def discover_unittests(module, pattern='tst_*.py'): try: import inspect import os import sys import unittest except: return tuple([]) dist_dir = libtbx.env.dist_path(module) found_tests = unittest.defaultTestLoader.discover(dist_dir, pattern=pattern) def recursive_TestSuite_to_list(suite): list = [] for t in suite: if isinstance(t, unittest.TestSuite): list.extend(recursive_TestSuite_to_list(t)) elif isinstance(t, unittest.TestCase): module = t.__class__.__module__ if module == 'unittest.loader': # This indicates a loading error. # Regenerate file name and try to run file directly. path = t._testMethodName.replace('.', os.path.sep) list.append("$D/%s.py" % path) else: module = inspect.getsourcefile(sys.modules[module]) function = "%s.%s" % (t.__class__.__name__, t._testMethodName) list.append([module, function]) else: raise Exception("Unknown test object (%s)" % t) return list test_list = recursive_TestSuite_to_list(found_tests) return tuple(test_list) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

Re: [cctbxbb] Native python unittests for the libtbx testing framework
by Jan Marten Simons 19 May '15

19 May '15

Am Freitag 15 Mai 2015 18:07:33 schrieb Pavel Afonine: > Hi Markus, > > lucky you, I wish I have time for side projects! > > I've been working on cctbx (and writing tests!) for the past 12 years > and can't name any problem using "libtbx testing world". Most tests are > written this way. Why I am not so keen to use an alternative? > - why introduce inconsistency for no obvious clear pressing reasons; > - I have no spare time to learn and get used to an alternative > framework (again, for no obvious clear pressing reason); > - some rare developers (mostly postdocs that come and go) used > Python test framework in the past on the code that I work too and that I > have to maintain when people are gone. I find it is an irritating > overhead for me dealing with these tests, so typically I bother to > re-write them to use libtbx tools. Hi Pavel, those are sound and valid reasons from your point of view. But please also consider this alternative view: Coming from other python projects people are likely to have experience with the python unit test framework. As cctbx is using its own testing functionality people might be put off from writing tests or at least have to overcome this additional barrier on entry. I'm a strong supporter of sticking/moving to standards like PEP0008 [1] or pip/whl* based installs as conforming to those helps people to quickly find their way into the code. * wouldn't it be great if one could simply tell people this: "To install cctbx simply type 'pip install cctbx' in your terminal."? [1] https://www.python.org/dev/peps/pep-0008/ With regards, Dipl. Phys. Jan M. Simons Institute of Crystallography RWTH Aachen University

4 3

Re: [cctbxbb] Fw: Build constraints in SConscript?
by Graeme.Winter＠diamond.ac.uk 15 May '15

15 May '15

Nigel If you export the environment variable I sent before before kicking off a build you should make fully portable binaries. We've been doing this in dials for a while. Best wishes Graeme On 15 May 2015 18:00, Nigel Moriarty <nwmoriarty(a)lbl.gov> wrote: Greame Did you try replacing 10.8 in the SConscript file with 10.7? Please do so and let me know if this works. I was working on getting the mechanism to be OS specific on the buildbot and bootstrap when Rob solved the problem independently. If you don't tell me if changing 10.8 to 10.7 works I will do something to get it working here and we can worry about your 2011 vintage OS next week. Cheers Nigel --- Nigel W. Moriarty Building 64R0246B, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> On Fri, May 15, 2015 at 2:31 AM, <Graeme.Winter(a)diamond.ac.uk<mailto:[email protected]>> wrote: Dear Rob Given that the tests are currently not working right anyway, and there are no build / deployment tests for testing builds on an older version of OS X than they are compiled on, this is an easy problem to get in to. It's also a collaborative project where we're all working hard on making things work. This will happen from time to time. I had thought the same environment variable based mechanism for the builds was already in place for PHENIX - am surprised it is not! So, no worries. Perhaps I came across more grumpy than I meant to below. Occurs to me it would be very handy to have one email address for all cctbx related developers where (someone) could ask (something like this) and get an answer perhaps suggesting a more optimum solution? Thoughts anyone? (lots of mail lists in send: list!) Thanks & best wishes Graeme -----Original Message----- From: R.D. Oeffner [mailto:[email protected]<mailto:[email protected]>] Sent: 15 May 2015 10:26 To: Winter, Graeme (DLSLtd,RAL,LSCI); cctbxbb(a)phenix-online.org<mailto:[email protected]>; dials-support(a)lists.sourceforge.net<mailto:[email protected]>; dials-dev(a)cci.lbl.gov<mailto:[email protected]>; Nigel W Moriarty Subject: Re: Fw: Build constraints in SConscript? Dear Graeme, I'm sorry for the trouble my change caused and the time taken to identify it. Ironically we had the same issue as you with Phenix not installing on anything lower than OSX 10.9. As nothing appeared to be specified in our build system to that effect my change was a quick fix to address the problem. It was genuinely well intended. Nigel, I defer to you for implementing the minimum OSX platform for Phenix builds. Rob > FROM: Graeme.Winter(a)Diamond.ac.uk<mailto:[email protected]> > SENT: Friday, May 15, 2015 9:09 AM > TO: Graeme.Winter(a)Diamond.ac.uk<mailto:[email protected]> ; cctbxbb(a)phenix-online.org<mailto:[email protected]> ; > robert(a)oeffner.net<mailto:[email protected]> > CC: dials-support(a)lists.sourceforge.net<mailto:[email protected]> ; dials-dev(a)cci.lbl.gov<mailto:[email protected]> > SUBJECT: RE: Build constraints in SConscript? > > [gw56@ws133 libtbx]$ svn commit -m "Reverting with extreme prejudice > hard coded 10.8 requirement, build no longer works on 10.7 and > earlier; please use ENV VAR method to achieve this kind of constraint; > reverted revision=22202" SConscript > > Sending SConscript > > Transmitting file data . > > Committed revision 22223. > > Please use MACOSX_DEPLOYMENT_TARGET=(version) locally for your > personal requirement, export this before kicking off bootstrap.py > > Best wishes Graeme > > ==================== DIFFS FOLLOW ==================== > > Revision: 22223 > http://sourceforge.net/p/cctbx/code/22223 [1] > Author: graeme_winter > Date: 2015-05-15 08:06:10 +0000 (Fri, 15 May 2015) > Log Message: > ----------- > Reverting with extreme prejudice hard coded 10.8 requirement, build no > longer works on 10.7 and earlier; please use ENV VAR method to achieve > this kind of constraint; reverted revision=22202 > > Modified Paths: > -------------- > trunk/libtbx/SConscript > > Modified: trunk/libtbx/SConscript > =================================================================== > --- trunk/libtbx/SConscript 2015-05-15 02:13:42 UTC (rev 22222) > +++ trunk/libtbx/SConscript 2015-05-15 08:06:10 UTC (rev 22223) > @@ -803,7 +803,7 @@ > env_etc.static_libraries = 1 > env_etc.static_bpl = 0 > > - link_flags = ["-w", "-mmacosx-version-min=10.8"] # suppress > "source/lib does not exist" warning > + link_flags = ["-w"] # suppress "source/lib does not exist" warning > if libtbx.env.build_options.force_32bit: > if (env_etc.gcc_version and env_etc.gcc_version >= 40201) or > env_etc.clang_version: > link_flags.extend(["-arch", "i386"]) > @@ -811,8 +811,7 @@ > env_etc.c_link = "cc" > env_etc.ccflags_base = [ > "-fPIC", > - "-fno-strict-aliasing", > - "-mmacosx-version-min=10.8"] > + "-fno-strict-aliasing"] > if env_etc.clang_version: > setup_clang_warning(env_etc, libtbx.env.build_options.warning_level) > else: > @@ -828,7 +827,7 @@ > env_etc.ccflags_base.extend(["-no-cpp-precomp"]) > if (env_etc.gcc_version < 40201): > env_etc.ccflags_base.append("-Wno-long-double") > - env_etc.cxxflags_base = ["-mmacosx-version-min=10.8"] > + env_etc.cxxflags_base = [] > if env_etc.clang_version is None and env_etc.gcc_version < 40000: > env_etc.cxxflags_base.append("-fcoalesce-templates") > env_etc.cxxflags_base.append("-DBOOST_ALL_NO_LIB") > @@ -861,11 +860,10 @@ > if env_etc.apple_gcc_builds.gcc < 5646: > # fixed on 10.6 (Snow Leopard) whose Dev Tools start with that build > opts.insert(0, "-gstabs+") > - env_etc.shlinkflags = ["-mmacosx-version-min=10.8"] > + env_etc.shlinkflags = [] > env_etc.shlinkflags_bpl = [ > "-w", # suppress "source/lib does not exist" warning > - "-bundle", "-mmacosx-version-min=10.8", > - "-undefined", "dynamic_lookup"] > + "-bundle", "-undefined", "dynamic_lookup"] > if libtbx.env.build_options.force_32bit: > if (env_etc.gcc_version and env_etc.gcc_version >= 40201) or > env_etc.clang_version: > env_etc.ccflags_base.extend(["-arch", "i386"]) > > FROM: Graeme.Winter(a)diamond.ac.uk<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] > > SENT: 15 May 2015 09:01 > TO: cctbxbb(a)phenix-online.org<mailto:[email protected]>; robert(a)oeffner.net<mailto:[email protected]> > CC: dials-support(a)lists.sourceforge.net<mailto:[email protected]>; dials-dev(a)cci.lbl.gov<mailto:[email protected]> > SUBJECT: Build constraints in SConscript? > > Hi Folks > > A while back we worked on making DIALS work right built on 10.10 but > executed on 10.6 or later – this is very useful as lots of people > have 10.6/7 macs & are not keen to update. This is done by the very > flexible mechanism of exporting an environment variable prior to > compilation > > Then this commit appears > > env_etc.static_bpl = 0 > @@ -802,7 +803,7 @@ > env_etc.static_libraries = 1 > env_etc.static_bpl = 0 > > - link_flags = ["-w"] # suppress "source/lib does not exist" warning > + link_flags = ["-w", "-mmacosx-version-min=10.8"] # suppress > "source/lib does not exist" warning > if libtbx.env.build_options.force_32bit: > if (env_etc.gcc_version and env_etc.gcc_version >= 40201) or > env_etc.clang_version: > link_flags.extend(["-arch", "i386"]) > @@ -810,7 +811,8 @@ > env_etc.c_link = "cc" > env_etc.ccflags_base = [ > "-fPIC", > - "-fno-strict-aliasing"] > + "-fno-strict-aliasing", > + "-mmacosx-version-min=10.8"] > if env_etc.clang_version: > setup_clang_warning(env_etc, libtbx.env.build_options.warning_level) > else: > @@ -826,7 +828,7 @@ > env_etc.ccflags_base.extend(["-no-cpp-precomp"]) > if (env_etc.gcc_version < 40201): > env_etc.ccflags_base.append("-Wno-long-double") > - env_etc.cxxflags_base = [] > + env_etc.cxxflags_base = ["-mmacosx-version-min=10.8"] > if env_etc.clang_version is None and env_etc.gcc_version < 40000: > env_etc.cxxflags_base.append("-fcoalesce-templates") > env_etc.cxxflags_base.append("-DBOOST_ALL_NO_LIB") > @@ -859,10 +861,10 @@ > > to > > Modified Paths: > -------------- > trunk/libtbx/SConscript > > and now our build no longer works on 10.7. This is *REALLY* annoying. > The head of the commit message follows below. > > Please could someone fix this? The environment variable method is > *FAR* better, works for the dependencies and also allows dials to work > on older machines. You broke our build. Before this it worked fine. > > (we can’t automate this test as we are not supposed to deploy out of > support machines on lab network, so this was missed) > > Thanks Graeme > > EFFNER > > to cctbx-cvs > > May 13 > > Revision: 22202 > http://sourceforge.net/p/cctbx/code/22202 [2] > Author: oeffner > Date: 2015-05-13 15:48:24 +0000 (Wed, 13 May 2015) > Log Message: > ----------- > ensure installers can run on MacOSX 10.8 or higher > > -- > > This e-mail and any attachments may contain confidential, copyright > and or privileged material, and are for the use of the intended > addressee only. If you are not the intended addressee or an authorised > recipient of the addressee please notify us of receipt by returning > the e-mail and do not use, copy, retain, distribute or disclose the > information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual > and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for > any damage which you may sustain as a result of software viruses which > may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in > England and Wales with its registered office at Diamond House, Harwell > Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United > Kingdom > > No virus found in this message. > Checked by AVG - www.avg.com<http://www.avg.com> [3] > Version: 2015.0.5941 / Virus Database: 4342/9780 - Release Date: > 05/15/15 > > Links: > ------ > [1] http://sourceforge.net/p/cctbx/code/22223 > [2] http://sourceforge.net/p/cctbx/code/22202 > [3] http://www.avg.com -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] Fw: Build constraints in SConscript?
by Graeme.Winter＠diamond.ac.uk 15 May '15

15 May '15

Dear Rob Given that the tests are currently not working right anyway, and there are no build / deployment tests for testing builds on an older version of OS X than they are compiled on, this is an easy problem to get in to. It's also a collaborative project where we're all working hard on making things work. This will happen from time to time. I had thought the same environment variable based mechanism for the builds was already in place for PHENIX - am surprised it is not! So, no worries. Perhaps I came across more grumpy than I meant to below. Occurs to me it would be very handy to have one email address for all cctbx related developers where (someone) could ask (something like this) and get an answer perhaps suggesting a more optimum solution? Thoughts anyone? (lots of mail lists in send: list!) Thanks & best wishes Graeme -----Original Message----- From: R.D. Oeffner [mailto:[email protected]] Sent: 15 May 2015 10:26 To: Winter, Graeme (DLSLtd,RAL,LSCI); cctbxbb(a)phenix-online.org; dials-support(a)lists.sourceforge.net; dials-dev(a)cci.lbl.gov; Nigel W Moriarty Subject: Re: Fw: Build constraints in SConscript? Dear Graeme, I'm sorry for the trouble my change caused and the time taken to identify it. Ironically we had the same issue as you with Phenix not installing on anything lower than OSX 10.9. As nothing appeared to be specified in our build system to that effect my change was a quick fix to address the problem. It was genuinely well intended. Nigel, I defer to you for implementing the minimum OSX platform for Phenix builds. Rob > FROM: Graeme.Winter(a)Diamond.ac.uk > SENT: Friday, May 15, 2015 9:09 AM > TO: Graeme.Winter(a)Diamond.ac.uk ; cctbxbb(a)phenix-online.org ; > robert(a)oeffner.net > CC: dials-support(a)lists.sourceforge.net ; dials-dev(a)cci.lbl.gov > SUBJECT: RE: Build constraints in SConscript? > > [gw56@ws133 libtbx]$ svn commit -m "Reverting with extreme prejudice > hard coded 10.8 requirement, build no longer works on 10.7 and > earlier; please use ENV VAR method to achieve this kind of constraint; > reverted revision=22202" SConscript > > Sending SConscript > > Transmitting file data . > > Committed revision 22223. > > Please use MACOSX_DEPLOYMENT_TARGET=(version) locally for your > personal requirement, export this before kicking off bootstrap.py > > Best wishes Graeme > > ==================== DIFFS FOLLOW ==================== > > Revision: 22223 > http://sourceforge.net/p/cctbx/code/22223 [1] > Author: graeme_winter > Date: 2015-05-15 08:06:10 +0000 (Fri, 15 May 2015) > Log Message: > ----------- > Reverting with extreme prejudice hard coded 10.8 requirement, build no > longer works on 10.7 and earlier; please use ENV VAR method to achieve > this kind of constraint; reverted revision=22202 > > Modified Paths: > -------------- > trunk/libtbx/SConscript > > Modified: trunk/libtbx/SConscript > =================================================================== > --- trunk/libtbx/SConscript 2015-05-15 02:13:42 UTC (rev 22222) > +++ trunk/libtbx/SConscript 2015-05-15 08:06:10 UTC (rev 22223) > @@ -803,7 +803,7 @@ > env_etc.static_libraries = 1 > env_etc.static_bpl = 0 > > - link_flags = ["-w", "-mmacosx-version-min=10.8"] # suppress > "source/lib does not exist" warning > + link_flags = ["-w"] # suppress "source/lib does not exist" warning > if libtbx.env.build_options.force_32bit: > if (env_etc.gcc_version and env_etc.gcc_version >= 40201) or > env_etc.clang_version: > link_flags.extend(["-arch", "i386"]) > @@ -811,8 +811,7 @@ > env_etc.c_link = "cc" > env_etc.ccflags_base = [ > "-fPIC", > - "-fno-strict-aliasing", > - "-mmacosx-version-min=10.8"] > + "-fno-strict-aliasing"] > if env_etc.clang_version: > setup_clang_warning(env_etc, libtbx.env.build_options.warning_level) > else: > @@ -828,7 +827,7 @@ > env_etc.ccflags_base.extend(["-no-cpp-precomp"]) > if (env_etc.gcc_version < 40201): > env_etc.ccflags_base.append("-Wno-long-double") > - env_etc.cxxflags_base = ["-mmacosx-version-min=10.8"] > + env_etc.cxxflags_base = [] > if env_etc.clang_version is None and env_etc.gcc_version < 40000: > env_etc.cxxflags_base.append("-fcoalesce-templates") > env_etc.cxxflags_base.append("-DBOOST_ALL_NO_LIB") > @@ -861,11 +860,10 @@ > if env_etc.apple_gcc_builds.gcc < 5646: > # fixed on 10.6 (Snow Leopard) whose Dev Tools start with that build > opts.insert(0, "-gstabs+") > - env_etc.shlinkflags = ["-mmacosx-version-min=10.8"] > + env_etc.shlinkflags = [] > env_etc.shlinkflags_bpl = [ > "-w", # suppress "source/lib does not exist" warning > - "-bundle", "-mmacosx-version-min=10.8", > - "-undefined", "dynamic_lookup"] > + "-bundle", "-undefined", "dynamic_lookup"] > if libtbx.env.build_options.force_32bit: > if (env_etc.gcc_version and env_etc.gcc_version >= 40201) or > env_etc.clang_version: > env_etc.ccflags_base.extend(["-arch", "i386"]) > > FROM: Graeme.Winter(a)diamond.ac.uk [mailto:[email protected]] > > SENT: 15 May 2015 09:01 > TO: cctbxbb(a)phenix-online.org; robert(a)oeffner.net > CC: dials-support(a)lists.sourceforge.net; dials-dev(a)cci.lbl.gov > SUBJECT: Build constraints in SConscript? > > Hi Folks > > A while back we worked on making DIALS work right built on 10.10 but > executed on 10.6 or later – this is very useful as lots of people > have 10.6/7 macs & are not keen to update. This is done by the very > flexible mechanism of exporting an environment variable prior to > compilation > > Then this commit appears > > env_etc.static_bpl = 0 > @@ -802,7 +803,7 @@ > env_etc.static_libraries = 1 > env_etc.static_bpl = 0 > > - link_flags = ["-w"] # suppress "source/lib does not exist" warning > + link_flags = ["-w", "-mmacosx-version-min=10.8"] # suppress > "source/lib does not exist" warning > if libtbx.env.build_options.force_32bit: > if (env_etc.gcc_version and env_etc.gcc_version >= 40201) or > env_etc.clang_version: > link_flags.extend(["-arch", "i386"]) > @@ -810,7 +811,8 @@ > env_etc.c_link = "cc" > env_etc.ccflags_base = [ > "-fPIC", > - "-fno-strict-aliasing"] > + "-fno-strict-aliasing", > + "-mmacosx-version-min=10.8"] > if env_etc.clang_version: > setup_clang_warning(env_etc, libtbx.env.build_options.warning_level) > else: > @@ -826,7 +828,7 @@ > env_etc.ccflags_base.extend(["-no-cpp-precomp"]) > if (env_etc.gcc_version < 40201): > env_etc.ccflags_base.append("-Wno-long-double") > - env_etc.cxxflags_base = [] > + env_etc.cxxflags_base = ["-mmacosx-version-min=10.8"] > if env_etc.clang_version is None and env_etc.gcc_version < 40000: > env_etc.cxxflags_base.append("-fcoalesce-templates") > env_etc.cxxflags_base.append("-DBOOST_ALL_NO_LIB") > @@ -859,10 +861,10 @@ > > to > > Modified Paths: > -------------- > trunk/libtbx/SConscript > > and now our build no longer works on 10.7. This is *REALLY* annoying. > The head of the commit message follows below. > > Please could someone fix this? The environment variable method is > *FAR* better, works for the dependencies and also allows dials to work > on older machines. You broke our build. Before this it worked fine. > > (we can’t automate this test as we are not supposed to deploy out of > support machines on lab network, so this was missed) > > Thanks Graeme > > EFFNER > > to cctbx-cvs > > May 13 > > Revision: 22202 > http://sourceforge.net/p/cctbx/code/22202 [2] > Author: oeffner > Date: 2015-05-13 15:48:24 +0000 (Wed, 13 May 2015) > Log Message: > ----------- > ensure installers can run on MacOSX 10.8 or higher > > -- > > This e-mail and any attachments may contain confidential, copyright > and or privileged material, and are for the use of the intended > addressee only. If you are not the intended addressee or an authorised > recipient of the addressee please notify us of receipt by returning > the e-mail and do not use, copy, retain, distribute or disclose the > information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual > and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for > any damage which you may sustain as a result of software viruses which > may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in > England and Wales with its registered office at Diamond House, Harwell > Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United > Kingdom > > No virus found in this message. > Checked by AVG - www.avg.com [3] > Version: 2015.0.5941 / Virus Database: 4342/9780 - Release Date: > 05/15/15 > > Links: > ------ > [1] http://sourceforge.net/p/cctbx/code/22223 > [2] http://sourceforge.net/p/cctbx/code/22202 > [3] http://www.avg.com -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

Build constraints in SConscript?
by Graeme.Winter＠diamond.ac.uk 15 May '15

15 May '15

Hi Folks A while back we worked on making DIALS work right built on 10.10 but executed on 10.6 or later - this is very useful as lots of people have 10.6/7 macs & are not keen to update. This is done by the very flexible mechanism of exporting an environment variable prior to compilation Then this commit appears env_etc.static_bpl = 0 @@ -802,7 +803,7 @@ env_etc.static_libraries = 1 env_etc.static_bpl = 0 - link_flags = ["-w"] # suppress "source/lib does not exist" warning + link_flags = ["-w", "-mmacosx-version-min=10.8"] # suppress "source/lib does not exist" warning if libtbx.env.build_options.force_32bit: if (env_etc.gcc_version and env_etc.gcc_version >= 40201) or env_etc.clang_version: link_flags.extend(["-arch", "i386"]) @@ -810,7 +811,8 @@ env_etc.c_link = "cc" env_etc.ccflags_base = [ "-fPIC", - "-fno-strict-aliasing"] + "-fno-strict-aliasing", + "-mmacosx-version-min=10.8"] if env_etc.clang_version: setup_clang_warning(env_etc, libtbx.env.build_options.warning_level) else: @@ -826,7 +828,7 @@ env_etc.ccflags_base.extend(["-no-cpp-precomp"]) if (env_etc.gcc_version < 40201): env_etc.ccflags_base.append("-Wno-long-double") - env_etc.cxxflags_base = [] + env_etc.cxxflags_base = ["-mmacosx-version-min=10.8"] if env_etc.clang_version is None and env_etc.gcc_version < 40000: env_etc.cxxflags_base.append("-fcoalesce-templates") env_etc.cxxflags_base.append("-DBOOST_ALL_NO_LIB") @@ -859,10 +861,10 @@ to Modified Paths: -------------- trunk/libtbx/SConscript and now our build no longer works on 10.7. This is *really* annoying. The head of the commit message follows below. Please could someone fix this? The environment variable method is *far* better, works for the dependencies and also allows dials to work on older machines. You broke our build. Before this it worked fine. (we can't automate this test as we are not supposed to deploy out of support machines on lab network, so this was missed) Thanks Graeme effner to cctbx-cvs [https://ssl.gstatic.com/bt/C3341AA7A1A076756462EE2E5CD71C11/2x/ic_expand_sm…] May 13 Revision: 22202 http://sourceforge.net/p/cctbx/code/22202 Author: oeffner Date: 2015-05-13 15:48:24 +0000 (Wed, 13 May 2015) Log Message: ----------- ensure installers can run on MacOSX 10.8 or higher -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 1

Does cctbx have an easy to access spherical harmonic table?
by Graeme.Winter＠diamond.ac.uk 09 May '15

09 May '15

Hi Folks, Looked and something is used from boost but they don’t appear to be clearly wrapped to Python. in scitbx.math.nss_spherical_harmonics found nearly what I was looking for but it seems to be for some other purpose & not really do what I want (and needs mystery things in constructor….) >>> _sh = sh(2, 1) Traceback (most recent call last): File "<stdin>", line 1, in <module> Boost.Python.ArgumentError: Python argument types in nss_spherical_harmonics.__init__(nss_spherical_harmonics, int, int) did not match C++ signature: __init__(_object*, int l_max, int mangle, scitbx::math::zernike::log_factorial_generator<double> lgf) Any clues would be welcome, else wise most sensible thing seems to be to wrap up to Python the boost spherical harmonic code… Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

bin data by a Miller index
by Mooers, Blaine H.M. (HSC) 24 Apr '15

24 Apr '15

Dear cctbxbb: I have a simple question. I have a data set with a strong off origin Patterson peak (0,0,0.11). I want to plot the average I/sig per zone l = n. I expect to see enhanced values at l=9n. How do you bin data by the L Miller index in iotbx? Best regards, Blaine Blaine Mooers, Ph.D. Assistant Professor Director of the Laboratory of Biomolecular Structure and Function Department of Biochemistry and Molecular Biology University of Oklahoma Health Sciences Center S.L. Young Biomedical Research Center Rm. 466 Shipping address: 975 NE 10th Street, BRC 466 Oklahoma City, OK 73104-5419 Letter address: P.O. Box 26901, BRC 466 Oklahoma City, OK 73190 office: (405) 271-8300 lab: (405) 271-8313 fax: (405) 271-3910 e-mail: blaine-mooers(a)ouhsc.edu Faculty webpage: http://www.oumedicine.com/department-of-biochemistry-and-molecular-biology/… Small Angle Scattering webpage: http://www.oumedicine.com/docs/default-source/ad-biochemistry-workfiles/sma… X-ray lab webpage: http://www.oumedicine.com/department-of-biochemistry-and-molecular-biology/…

2 1

scitbx test crashing on mac
by markus.gerstel＠diamond.ac.uk 14 Apr '15

14 Apr '15

Good morning, The test scitbx.rigid_body.proto.tst_singular has been failing consistently on our MacOS 10.10 system for a day now. Last known good svn rev. is 21989. It passes on the linux systems. Apologies if this is not the right place to report scitbx problems to. -Markus Stacktrace 11 Python 0x0000000100010d03 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101a0f893 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x0000000101a147f1 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x0000000101a0fe5a boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x0000000101a0db6a boost::python::objects::function::call(_object*, _object*) const + 826 6 scitbx_linalg_ext.so 0x000000010518b521 boost::python::objects::caller_py_function_impl<boost::python::detail::caller<scitbx::af::shared<double> (scitbx::matrix::eigensystem::real_symmetric<double>::*)() const, boost::python::default_call_policies, boost::mpl::vector2<scitbx::af::shared<double>, scitbx::matrix::eigensystem::real_symmetric<double>&> > >::operator()(_object*, _object*) + 65 5 scitbx_linalg_ext.so 0x000000010518b6be _object* boost::python::detail::invoke<boost::python::to_python_value<scitbx::af::shared<double> const&>, scitbx::af::shared<double> (scitbx::matrix::eigensystem::real_symmetric<double>::*)() const, boost::python::arg_from_python<scitbx::matrix::eigensystem::real_symmetric<double>&> >(boost::python::detail::invoke_tag_<false, true>, boost::python::to_python_value<scitbx::af::shared<double> const&> const&, scitbx::af::shared<double> (scitbx::matrix::eigensystem::real_symmetric<double>::*&)() const, boost::python::arg_from_python<scitbx::matrix::eigensystem::real_symmetric<double>&>&) + 46 4 ??? 0x000000010183fc20 0x0 + 4320394272 3 libsystem_platform.dylib 0x00007fff86f16f1a _sigtramp + 26 2 boost_python_meta_ext.so 0x00000001005bdf80 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Standard Error show_stack(1): /Users/dlshudson/jenkins_slave/workspace/cctbx_tests_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/scitbx/rigid_body/proto/featherstone.py(38) generalized_inverse show_stack(2): /Users/dlshudson/jenkins_slave/workspace/cctbx_tests_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/scitbx/rigid_body/proto/featherstone.py(373) FDab show_stack(3): /Users/dlshudson/jenkins_slave/workspace/cctbx_tests_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/scitbx/rigid_body/proto/test_simulation.py(28) energies_and_accelerations_update show_stack(4): /Users/dlshudson/jenkins_slave/workspace/cctbx_tests_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/scitbx/rigid_body/proto/test_simulation.py(16) __init__ show_stack(5): /Users/dlshudson/jenkins_slave/workspace/cctbx_tests_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/scitbx/rigid_body/proto/tst_singular.py(72) exercise_six_dof show_stack(6): /Users/dlshudson/jenkins_slave/workspace/cctbx_tests_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/scitbx/rigid_body/proto/tst_singular.py(146) run show_stack(7): /Users/dlshudson/jenkins_slave/workspace/cctbx_tests_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/scitbx/rigid_body/proto/tst_singular.py(162) <module> libc backtrace (43 frames, most recent call last): 44 ??? 0x0000000000000003 0x0 + 3 43 Python 0x0000000100000d71 Python + 3441 42 Python 0x0000000100000e58 Python + 3672 41 Python 0x00000001000edd44 Py_Main + 3076 40 Python 0x00000001000d64a7 PyRun_SimpleFileExFlags + 711 39 Python 0x00000001000d6984 PyRun_FileExFlags + 164 38 Python 0x00000001000ad596 PyEval_EvalCode + 54 37 Python 0x00000001000adc22 PyEval_EvalCodeEx + 1666 36 Python 0x00000001000b179f PyEval_EvalFrameEx + 14111 35 Python 0x00000001000b5556 fast_function + 118 34 Python 0x00000001000adc22 PyEval_EvalCodeEx + 1666 33 Python 0x00000001000b179f PyEval_EvalFrameEx + 14111 32 Python 0x00000001000b5556 fast_function + 118 31 Python 0x00000001000adc22 PyEval_EvalCodeEx + 1666 30 Python 0x00000001000b295b PyEval_EvalFrameEx + 18651 29 Python 0x0000000100010d03 PyObject_Call + 99 28 Python 0x0000000100069e23 type_call + 339 27 Python 0x000000010006fccd slot_tp_init + 141 26 Python 0x0000000100010d03 PyObject_Call + 99 25 Python 0x000000010001e207 instancemethod_call + 247 24 Python 0x0000000100010d03 PyObject_Call + 99 23 Python 0x0000000100036a6c function_call + 364 22 Python 0x00000001000adc22 PyEval_EvalCodeEx + 1666 21 Python 0x00000001000b179f PyEval_EvalFrameEx + 14111 20 Python 0x00000001000b5556 fast_function + 118 19 Python 0x00000001000adc22 PyEval_EvalCodeEx + 1666 18 Python 0x00000001000b179f PyEval_EvalFrameEx + 14111 17 Python 0x00000001000b5556 fast_function + 118 16 Python 0x00000001000adc22 PyEval_EvalCodeEx + 1666 15 Python 0x00000001000b179f PyEval_EvalFrameEx + 14111 14 Python 0x00000001000b5556 fast_function + 118 13 Python 0x00000001000adc22 PyEval_EvalCodeEx + 1666 12 Python 0x00000001000b295b PyEval_EvalFrameEx + 18651 11 Python 0x0000000100010d03 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101a0f893 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x0000000101a147f1 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x0000000101a0fe5a boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x0000000101a0db6a boost::python::objects::function::call(_object*, _object*) const + 826 6 scitbx_linalg_ext.so 0x000000010518b521 boost::python::objects::caller_py_function_impl<boost::python::detail::caller<scitbx::af::shared<double> (scitbx::matrix::eigensystem::real_symmetric<double>::*)() const, boost::python::default_call_policies, boost::mpl::vector2<scitbx::af::shared<double>, scitbx::matrix::eigensystem::real_symmetric<double>&> > >::operator()(_object*, _object*) + 65 5 scitbx_linalg_ext.so 0x000000010518b6be _object* boost::python::detail::invoke<boost::python::to_python_value<scitbx::af::shared<double> const&>, scitbx::af::shared<double> (scitbx::matrix::eigensystem::real_symmetric<double>::*)() const, boost::python::arg_from_python<scitbx::matrix::eigensystem::real_symmetric<double>&> >(boost::python::detail::invoke_tag_<false, true>, boost::python::to_python_value<scitbx::af::shared<double> const&> const&, scitbx::af::shared<double> (scitbx::matrix::eigensystem::real_symmetric<double>::*&)() const, boost::python::arg_from_python<scitbx::matrix::eigensystem::real_symmetric<double>&>&) + 46 4 ??? 0x000000010183fc20 0x0 + 4320394272 3 libsystem_platform.dylib 0x00007fff86f16f1a _sigtramp + 26 2 boost_python_meta_ext.so 0x00000001005bdf80 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

5 9

bootstrap script broken - more rotamer goodness
by markus.gerstel＠diamond.ac.uk 08 Apr '15

08 Apr '15

Hi, We received an error report from a dials user that he could not install dials from source. This appears to be caused by the mmtbx.rebuild_rotarama_cache command run after the compilation, which has been introduced in http://sourceforge.net/p/cctbx/code/21921/ I guess this needs either to be run conditional on the library being installed or be changed to fail gracefully. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

Re: [cctbxbb] [Dials-support] DIALS source tarball
by Graeme.Winter＠diamond.ac.uk 08 Apr '15

08 Apr '15

Sadly this also does not fix anything :o( not sure why this was not flagged when I configured chem_data... libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py" [FAIL] Time: 6.44 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 818, in exercise_na_restraints_output_to_geo log=out2) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5274, in run nonbonded_distance_threshold=nonbonded_distance_threshold) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4952, in geometry_restraints_manager ss_manager.initialize(log=self.log) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 280, in initialize self.find_automatically(log=log) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 313, in find_automatically self.sec_str_from_pdb_file = self.find_sec_str(log=log) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 369, in find_sec_str (records, stderr) = run_ksdssp_direct(pdb_str) File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 728, in run_ksdssp_direct exe_path = get_ksdssp_exe_path() File "/Users/graeme/svn/cctbx/modules/cctbx_project/mmtbx/secondary_structure/__init__.py", line 709, in get_ksdssp_exe_path raise RuntimeError("ksdssp module is not configured") RuntimeError: ksdssp module is not configured On 8 Apr 2015, at 11:02, <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> wrote: $ du -h --max-depth=0 chem_data 1.6G chem_data Not sure we really want to include this in the dials builds, just to make one test happy, especially when we have no use for any of the monomer library functionality in dials. Cheers, Richard ________________________________ From: Graeme Winter [graeme.winter(a)gmail.com<mailto:[email protected]>] Sent: 08 April 2015 10:58 To: Gildea, Richard (DLSLtd,RAL,LSCI); Gerstel, Markus (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]> Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball can we add this to the dials build through some kind of tarball download which does not need a cci account? Thanks graeme On Wed, Apr 8, 2015 at 10:52 AM <richard.gildea(a)diamond.ac.uk<mailto:[email protected]><mailto:[email protected]>> wrote: Hi Markus, I agree that it would be better if the tests would either work gracefully with CCP4 monomer library, or (possibly better?) only run if chem_data is available. In the latter case, a simple libtbx.env.has_module("chem_data") could be used in the test to check whether chem_data is available. As a workaround, we developers should be able to add the chem_data repository to our sources after which the tests should pick up the correct version of the monomer library: $ svn co svn+ssh://[email protected]/chem_data/trunk<http://[email protected]/chem_data/trunk> chem_data $ libtbx.configure chem_data Cheers, Richard ________________________________________ From: Gerstel, Markus (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:33 To: Gildea, Richard (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Richard, I think you've found it. I have unset CLIBD_MON, and now the test skips cleanly. Of course this also means that users should not run cctbx tests on their machines... Is there no central 'mmtbx_skip_test_if_chem_data_db_missing' function, that just goes 'prints "Skip this test"; exit' if the library isn't present? Or a boolean _is_available() function if you prefer. Either would resolve this problem as well as the current untestability of mmtbx. -Markus -----Original Message----- From: Gildea, Richard (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:15 To: Gerstel, Markus (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Markus, Presumably the test is looking for a monomer that is present in chem_data but not in the ccp4 monomer library? Could we unset the CLIBD_MON variable for the purposes of testing the dials installers? Cheers, Richard ________________________________________ From: Gerstel, Markus (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:07 To: Gildea, Richard (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Richard, Aaron, This is correct, we do not currently run mmtbx tests at all, as up to 107 of them do not fail gracefully, ie. skip properly. The test in question works fine (ie: is skipped because CCP4 is not made available) on our cctbx test system, but breaks in our final dials installer tests (where CCP4 is made available). CLIBD_MON is set to the /lib/data/monomers folder in the shared DLS CCP4 6.5 update 1 installation. Best wishes, Markus -----Original Message----- From: richard.gildea(a)diamond.ac.uk<mailto:[email protected]><mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] Sent: 07 April 2015 21:36 To: asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>; cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]> Cc: dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball Hi Aaron, I'm not sure your fix as I understand it actually helps. The test that was failing was in the cctbx module, not mmtbx, and as far as I am aware we are running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our nightly builds (and that is what we as developers are running locally on our own development machines). If individual tests (particularly those in the more general modules such as cctbx, iotbx, etc.) rely on external dependencies (such as chem_data) being available, then the individual tests should test for presence of said dependencies and skip gracefully if not available. I have a feeling that we're not even including mmtbx in our nightly build tests (too many fail due to missing dependencies such as chem_data). Cheers, Richard ________________________________ From: Aaron Brewster [asbrewster(a)lbl.gov<mailto:[email protected]><mailto:[email protected]>] Sent: 07 April 2015 20:29 To: cctbx mailing list Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); Johan Hattne; <dials-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]>> Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball Hi Markus, two points. First, you are correct that the dials builds are not passing tests related to the chem_data folder which includes monomer libraries. This is because chem_data is required by mmtbx but isn't included in the dials build. Partly because we're unsure if pieces of it are proprietary. In the meantime, I've worked with Nigel to change the cctbx_regression.test_nightly to look for the chem_data repository and skip the mmtbx tests if it's not present. mmtbx will still be tested nightly as part of the standalone cctbx and phenix packages. Because of this change, perhaps you could revert your recent changes to cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check for chem_data in every sub test as it is tested for at the top level now. Second, the log you provided is not the one on buildbot. Your log indicates you have a CCP4 installation that includes a monomer library that is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or CLIBD_MON environment variables set on the system in question? Thanks, -Aaron On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Cc'ing cctbx folks David Waterman wrote: So maybe once we have a source tarball up at dials.diamond.ac.uk<http://dials.diamond.ac.uk><http://dials.diamond.ac.uk><http://dials.diamond.ac.uk> (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------------------------------------------------------ BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF _______________________________________________ DIALS-support mailing list DIALS-support(a)lists.sourceforge.net<mailto:[email protected]><mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/dials-support

1 0

Re: [cctbxbb] [Dials-support] DIALS source tarball
by markus.gerstel＠diamond.ac.uk 08 Apr '15

08 Apr '15

Hi Richard, I think you've found it. I have unset CLIBD_MON, and now the test skips cleanly. Of course this also means that users should not run cctbx tests on their machines... Is there no central 'mmtbx_skip_test_if_chem_data_db_missing' function, that just goes 'prints "Skip this test"; exit' if the library isn't present? Or a boolean _is_available() function if you prefer. Either would resolve this problem as well as the current untestability of mmtbx. -Markus -----Original Message----- From: Gildea, Richard (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:15 To: Gerstel, Markus (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Markus, Presumably the test is looking for a monomer that is present in chem_data but not in the ccp4 monomer library? Could we unset the CLIBD_MON variable for the purposes of testing the dials installers? Cheers, Richard ________________________________________ From: Gerstel, Markus (DLSLtd,RAL,LSCI) Sent: 08 April 2015 09:07 To: Gildea, Richard (DLSLtd,RAL,LSCI); asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: RE: [Dials-support] [cctbxbb] DIALS source tarball Hi Richard, Aaron, This is correct, we do not currently run mmtbx tests at all, as up to 107 of them do not fail gracefully, ie. skip properly. The test in question works fine (ie: is skipped because CCP4 is not made available) on our cctbx test system, but breaks in our final dials installer tests (where CCP4 is made available). CLIBD_MON is set to the /lib/data/monomers folder in the shared DLS CCP4 6.5 update 1 installation. Best wishes, Markus -----Original Message----- From: richard.gildea(a)diamond.ac.uk [mailto:[email protected]] Sent: 07 April 2015 21:36 To: asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball Hi Aaron, I'm not sure your fix as I understand it actually helps. The test that was failing was in the cctbx module, not mmtbx, and as far as I am aware we are running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our nightly builds (and that is what we as developers are running locally on our own development machines). If individual tests (particularly those in the more general modules such as cctbx, iotbx, etc.) rely on external dependencies (such as chem_data) being available, then the individual tests should test for presence of said dependencies and skip gracefully if not available. I have a feeling that we're not even including mmtbx in our nightly build tests (too many fail due to missing dependencies such as chem_data). Cheers, Richard ________________________________ From: Aaron Brewster [asbrewster(a)lbl.gov] Sent: 07 April 2015 20:29 To: cctbx mailing list Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); Johan Hattne; <dials-support(a)lists.sourceforge.net> Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball Hi Markus, two points. First, you are correct that the dials builds are not passing tests related to the chem_data folder which includes monomer libraries. This is because chem_data is required by mmtbx but isn't included in the dials build. Partly because we're unsure if pieces of it are proprietary. In the meantime, I've worked with Nigel to change the cctbx_regression.test_nightly to look for the chem_data repository and skip the mmtbx tests if it's not present. mmtbx will still be tested nightly as part of the standalone cctbx and phenix packages. Because of this change, perhaps you could revert your recent changes to cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check for chem_data in every sub test as it is tested for at the top level now. Second, the log you provided is not the one on buildbot. Your log indicates you have a CCP4 installation that includes a monomer library that is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or CLIBD_MON environment variables set on the system in question? Thanks, -Aaron On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> wrote: Cc'ing cctbx folks David Waterman wrote: > So maybe once we have a source tarball up at > dials.diamond.ac.uk<http://dials.diamond.ac.uk> (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] [Dials-support] DIALS source tarball
by markus.gerstel＠diamond.ac.uk 08 Apr '15

08 Apr '15

Hi Richard, Aaron, This is correct, we do not currently run mmtbx tests at all, as up to 107 of them do not fail gracefully, ie. skip properly. The test in question works fine (ie: is skipped because CCP4 is not made available) on our cctbx test system, but breaks in our final dials installer tests (where CCP4 is made available). CLIBD_MON is set to the /lib/data/monomers folder in the shared DLS CCP4 6.5 update 1 installation. Best wishes, Markus -----Original Message----- From: richard.gildea(a)diamond.ac.uk [mailto:[email protected]] Sent: 07 April 2015 21:36 To: asbrewster(a)lbl.gov; cctbxbb(a)phenix-online.org Cc: dials-support(a)lists.sourceforge.net Subject: Re: [Dials-support] [cctbxbb] DIALS source tarball Hi Aaron, I'm not sure your fix as I understand it actually helps. The test that was failing was in the cctbx module, not mmtbx, and as far as I am aware we are running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our nightly builds (and that is what we as developers are running locally on our own development machines). If individual tests (particularly those in the more general modules such as cctbx, iotbx, etc.) rely on external dependencies (such as chem_data) being available, then the individual tests should test for presence of said dependencies and skip gracefully if not available. I have a feeling that we're not even including mmtbx in our nightly build tests (too many fail due to missing dependencies such as chem_data). Cheers, Richard ________________________________ From: Aaron Brewster [asbrewster(a)lbl.gov] Sent: 07 April 2015 20:29 To: cctbx mailing list Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); Johan Hattne; <dials-support(a)lists.sourceforge.net> Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball Hi Markus, two points. First, you are correct that the dials builds are not passing tests related to the chem_data folder which includes monomer libraries. This is because chem_data is required by mmtbx but isn't included in the dials build. Partly because we're unsure if pieces of it are proprietary. In the meantime, I've worked with Nigel to change the cctbx_regression.test_nightly to look for the chem_data repository and skip the mmtbx tests if it's not present. mmtbx will still be tested nightly as part of the standalone cctbx and phenix packages. Because of this change, perhaps you could revert your recent changes to cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check for chem_data in every sub test as it is tested for at the top level now. Second, the log you provided is not the one on buildbot. Your log indicates you have a CCP4 installation that includes a monomer library that is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or CLIBD_MON environment variables set on the system in question? Thanks, -Aaron On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> wrote: Cc'ing cctbx folks David Waterman wrote: > So maybe once we have a source tarball up at > dials.diamond.ac.uk<http://dials.diamond.ac.uk> (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] [Dials-support] DIALS source tarball
by Aaron Brewster 07 Apr '15

07 Apr '15

Hi Richard, At the moment, buildbot for dials runs these three tests: libtbx.import_all_ext cctbx_regression.test_nightly libtbx.run_tests_parallel module=dials nproc=auto verbosity=1 Which doesn't line up exactly with what you described. If you run libtbx.run_tests_parallel module=cctbx module=dxtbx, then you are right that the fix I committed wouldn't be relevant because mmtbx wouldn't be tested. However, since the LBL build bot runs cctbx_regression.test_nightly, my change is needed. After some more discussion, I see that Markus's changes to tst_geometry_restraints_2.py are fine. It looks like there are already sub-tests in there that check for the presence of mmtbx and chem data before running. I think tst_geometry_restraints_2.py was the only test outside of mmtbx that was looking for the monomer libraries. I'm re-running the buildbot dials build now to see if my change to cctbx_regression.test_nightly along with Markus's changes to tst_geometry_restraints_2.py clean up the build. Finally, and perhaps on a different note, should we add dxtbx to cctbx_regression.test_nightly? Right now the list of modules it tests includes libtbx, boost_adaptbx, scitbx, cctbx, iotbx, smtbx, and if chem_data is present, mmtbx. I don't think the build bot is running the dxtbx tests at present. -Aaron On Tue, Apr 7, 2015 at 1:36 PM, <richard.gildea(a)diamond.ac.uk> wrote: > Hi Aaron, > > I'm not sure your fix as I understand it actually helps. The test that was > failing was in the cctbx module, not mmtbx, and as far as I am aware we are > running libtbx.run_tests_parallel module=cctbx module=dxtbx etc in our > nightly builds (and that is what we as developers are running locally on > our own development machines). If individual tests (particularly those in > the more general modules such as cctbx, iotbx, etc.) rely on external > dependencies (such as chem_data) being available, then the individual tests > should test for presence of said dependencies and skip gracefully if not > available. I have a feeling that we're not even including mmtbx in our > nightly build tests (too many fail due to missing dependencies such as > chem_data). > > Cheers, > > Richard > > ________________________________ > From: Aaron Brewster [asbrewster(a)lbl.gov] > Sent: 07 April 2015 20:29 > To: cctbx mailing list > Cc: Waterman, David (STFC,RAL,SC); Gildea, Richard (DLSLtd,RAL,LSCI); > Johan Hattne; <dials-support(a)lists.sourceforge.net> > Subject: Re: [cctbxbb] [Dials-support] DIALS source tarball > > Hi Markus, two points. > > First, you are correct that the dials builds are not passing tests related > to the chem_data folder which includes monomer libraries. This is because > chem_data is required by mmtbx but isn't included in the dials build. > Partly because we're unsure if pieces of it are proprietary. In the > meantime, I've worked with Nigel to change the > cctbx_regression.test_nightly to look for the chem_data repository and skip > the mmtbx tests if it's not present. mmtbx will still be tested nightly as > part of the standalone cctbx and phenix packages. Because of this change, > perhaps you could revert your recent changes to > cctbx/regression/tst_geometry_restraints_2.py? It isn't necessary to check > for chem_data in every sub test as it is tested for at the top level now. > > Second, the log you provided is not the one on buildbot. Your log > indicates you have a CCP4 installation that includes a monomer library that > is being picked up by the tests. Is one of the MMTBX_CCP4_MONOMER_LIB, or > CLIBD_MON environment variables set on the system in question? > > Thanks, > -Aaron > > > > On Tue, Apr 7, 2015 at 6:41 AM, <markus.gerstel(a)diamond.ac.uk<mailto: > markus.gerstel(a)diamond.ac.uk>> wrote: > Cc'ing cctbx folks > > David Waterman wrote: > > So maybe once we have a source tarball up at dials.diamond.ac.uk< > http://dials.diamond.ac.uk> (..) > > Speaking of which: our DIALS builds are currently failing at the installer > testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py > failing deep in mmtbx territory. > So far I could not figure out how to get this running properly. > > > Call: > > libtbx.python tst_geometry_restraints_2.py > > Standard Output > > Skipping exercise_with_zeolite(): input file not available > Skipping exercise_with_pdb(): chem_data directory not available > > Standard Error > > Traceback (most recent call last): > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", > line 866, in <module> > exercise_all(sys.argv[1:]) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", > line 862, in exercise_all > exercise_na_restraints_output_to_geo(verbose=verbose) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", > line 805, in exercise_na_restraints_output_to_geo > log=out1) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 5270, in run > log=log) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 4876, in __init__ > restraints_loading_flags=restraints_loading_flags, > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 2902, in __init__ > log=log, > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 2287, in __init__ > m_j=mm) > File > "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", > line 1513, in get_lib_link > return mon_lib_srv.link_link_id_dict["rna3p"] > KeyError: 'rna3p' > > > (On a side note: We really should call multiple tests within the same > python file separately, instead of running them together within a signle > run() method. This file contains 4 tests, but these will result in only a > single return value for any upstream processing of the test results, such > as Jenkins.) > > -Markus > > -- > This e-mail and any attachments may contain confidential, copyright and or > privileged material, and are for the use of the intended addressee only. If > you are not the intended addressee or an authorised recipient of the > addressee please notify us of receipt by returning the e-mail and do not > use, copy, retain, distribute or disclose the information in or attached to > the e-mail. > Any opinions expressed within this e-mail are those of the individual and > not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for any > damage which you may sustain as a result of software viruses which may be > transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England > and Wales with its registered office at Diamond House, Harwell Science and > Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > >

1 0

Generating crystal copies
by Nicholas Pearce 07 Apr '15

07 Apr '15

Hi Everyone, I'm currently trying to generate crystal contacts/copies. What I want out at the end is a list of symmetry operations to generate the crystallographic copies that form contacts with my input model. I've been using the asu_mappings object, but I'm not sure I'm using the output correctly or if it's the right thing to use. Code snippet used to calculate the mappings: # Extract the xray structure from the reference hierarchy ref_struc = ref_hierarchy.extract_xray_structure(crystal_symmetry=crystal_symmetry) # Extract the mappings that will tell us the adjacent symmetry copies asu_mappings = ref_struc.asu_mappings(buffer_thickness=5) # Symmetry operations for each atom mappings = asu_mappings.mappings() I printed out all of the mappings, as I presumed the vast majority of the them would be the identity, meaning that any non-identity operations would be from atoms in the buffer area I have defined. These non-identity operations would presumably be the ones that would generate the crystal contacts. However, a large number of the atoms only had one mapping, which was not the identity - e.g. asu_mappings.get_rt_mx(x).as_xyz() might give '-x+1/2,-y+1/2,z-1/2' for an atom with only one mapping. The effect of this is that the operations given by this generate symmetry copies that do not contact my input model in real space. I presume this means that asu_mappings is mapping the atoms in my model to some other arbitrary asu, as well as the atoms in the buffer zone around it? Therefore the symmetry operations given are for these mapped atoms? Can anyone help, or suggest an alternative way to generate the crystal contacts? Thanks, Nick

4 3

Re: [cctbxbb] [Dials-support] DIALS source tarball
by markus.gerstel＠diamond.ac.uk 07 Apr '15

07 Apr '15

Cc'ing cctbx folks David Waterman wrote: > So maybe once we have a source tarball up at dials.diamond.ac.uk (..) Speaking of which: our DIALS builds are currently failing at the installer testing stage. This is due to cctbx/regression/tst_geometry_restraints_2.py failing deep in mmtbx territory. So far I could not figure out how to get this running properly. Call: libtbx.python tst_geometry_restraints_2.py Standard Output Skipping exercise_with_zeolite(): input file not available Skipping exercise_with_pdb(): chem_data directory not available Standard Error Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 866, in <module> exercise_all(sys.argv[1:]) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 862, in exercise_all exercise_na_restraints_output_to_geo(verbose=verbose) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/cctbx/regression/tst_geometry_restraints_2.py", line 805, in exercise_na_restraints_output_to_geo log=out1) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5270, in run log=log) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 4876, in __init__ restraints_loading_flags=restraints_loading_flags, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2902, in __init__ log=log, File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2287, in __init__ m_j=mm) File "/scratch/jenkins_slave/workspace/dials_test_installer/dials-dev/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1513, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' (On a side note: We really should call multiple tests within the same python file separately, instead of running them together within a signle run() method. This file contains 4 tests, but these will result in only a single return value for any upstream processing of the test results, such as Jenkins.) -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

programmatic access to PDB through cctbx?
by Kaushik Hatti 01 Apr '15

01 Apr '15

Hello, Sorry if this is not relevant to this mailing list. I was wondering if there are any modules available under cctbx to query PDB database programmatically. There was an option to SQL-query PDBe (from here : http://www.ebi.ac.uk/pdbe-as/pdbedatabase/ ). But, this service has been retired since Mar 2015. How else could complex query on PDB possible? An example query could be: list of PDBid with SCOPid="1hba" and (Resolution < 2 and R/Rfree < 15/20) or (Resolution < 3 and R/Rfree < 20/25 ) and Completeness < 95% and structureFactorAvailability="Y" Such a query was possible via above mentioned SQL link. However, I am unable to find any other means now. Thanks in advance, Kaushik Hatti, Molecular Biophysics Unit, Indian Institute of Science, Bangalore, India -- People living deeply have no fear of death - Anais Nin Caution: I am still the dumbest person I have ever known :-)

2 2

dxtbx test failure
by markus.gerstel＠diamond.ac.uk 25 Mar '15

25 Mar '15

Hello everyone, Our last nightly build on RHEL linux x64 failed because the test dxtbx/tests/tst_dxtbx.py , which normally takes roughly 3 seconds to run on the build machine, got stuck for over 5 minutes. All the other tests (cctbx/scitbx/libtbx/iotbx/rstbx and remaining dxtbx) completed successfully. Tests were run via libtbx.run_tests_parallel with nproc=4. As far as I can tell there is no debug output. The libtbx testing suite does not support timeouts like some timeout decorators on python unittest? As far as I can tell tst_dxtbx.py at the moment runs 5 separate tests, so for future debugging it may make sense to split these up. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

compilation issues
by Dave Lodowski 16 Mar '15

16 Mar '15

Hi, sorry if this is a repeat, but an email sent to the cctbx(a)cci.lbl.gov bounced back to me. We are attempting to install the DIALS package and CCTBX is required for compilation. During compilation of CCTBX, Xrender fails to find the X windows server. OS is Ubuntu LTS 12.04.5 text from console upon failure: Installing Xrender... log file is /home/dave/cctbx/build/base_tmp/Xrender_install_log getting package xrender-0.8.3.tar.gz... downloading from http://cci.lbl.gov/cctbx_dependencies : local copy is current using ./xrender-0.8.3.tar.gz (cached) installing xrender-0.8.3.tar.gz... Traceback (most recent call last): File "../auto_build/install", line 9, in <module> install_base_packages.installer(args=sys.argv, log=sys.stdout) File "../auto_build/install_base_packages.py", line 161, in __init__ self.build_dependencies(packages=packages) File "../auto_build/install_base_packages.py", line 505, in build_dependencies getattr(self, 'build_%s'%i)() File "../auto_build/install_base_packages.py", line 781, in build_render self.build_compiled_package_simple(pkg_name=pkg, pkg_name_label=name) File "../auto_build/install_base_packages.py", line 407, in build_compiled_package_simple log=pkg_log) File "../auto_build/install_base_packages.py", line 392, in configure_and_build self.call("./configure %s" % " ".join(list(config_args)), log=log) File "../auto_build/install_base_packages.py", line 243, in call return call(args, log=log, verbose=self.verbose, **kwargs) File "../auto_build/installer_utils.py", line 71, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to './configure --prefix="/home/dave/cctbx/build/base"' failed with exit code 1 Log file indicates everything fine till X is required then : checking for X... no configure: error: X is required, but it was either disabled or not found. any suggestions as to how to proceed would be appreciated thanks, --David Lodowski

2 1

test 2
by Nicholas Sauter 05 Mar '15

05 Mar '15

-- Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Physical Biosciences Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 64R0121 Berkeley, CA 94720-8118 (510) 486-5713

1 0

Atom selections for distance (embarrassingly simple)
by Morten Grøftehauge 18 Feb '15

18 Feb '15

Hi everyone, I'm trying to do some distance extraction of hetero atoms in pdbs. I've got for model in pdb_obj.hierarchy.models(): for chain in model.chains(): for rg in chain.residue_groups(): for ag in rg.atom_groups(): for atom in ag.atoms(): if atom.hetero == True: print atom.distance(atom) Which prints out 0.0 as expected. But I can't figure out how to, say, measure the distance to an atom in the previous residue (if there is a previous residue). I also noticed something weird in this tutorial: http://cctbx.sourceforge.net/sbgrid2008/tutorial.html They truncate a non-standard amino acid to the beta-carbon but this only works because there's an alternate conformation with a standard amino acid (Cys). So it wouldn't work with a seleno-met for instance (I know there's a convert-to-ala function in mmtbx - I said it was weird, not terrible). Anyway, I hope someone can explain the distances thing for me. I'm sure there are multiple ways of doing it (should I just do a selection of "hetatm" and iterate over them and try to get some distanes?). Cheers, Morten

2 1

Partitioning of Grid/Unit Cell
by Nicholas Pearce 15 Feb '15

15 Feb '15

Hi Everyone, I’m trying to partition a grid/space based on smallest distance to atomic sites - as in a voronoi diagram (http://en.wikipedia.org/wiki/Voronoi_diagram <http://en.wikipedia.org/wiki/Voronoi_diagram>). Is there any code in cctbx that could be used to accomplish this? Thanks, Nick Nicholas Pearce, DPhil Student, Structural Genomics Consortium, University of Oxford

3 2

iotbx xray.reconstructed_amplitude and changed SIGF
by Huw Jenkins 11 Feb '15

11 Feb '15

Hi, This is my first post to this list and I hope I can explain my confusion properly. I have been using iotbx to manipulate data stored in mtz files and found that values in columns that should not have been altered by my code are different in the output mtz file. This seems only to occur when the input mtz file contains DANO, SIGDANO columns. When I run the code below F and DANO in input.mtz and output.mtz are the same but SIGF and SIGDANO are different (shown for 5 reflections listed). Is this expected behaviour? Given that in my understanding that code should just read a mtz file, convert the data to a miller array then write the array back out to a new mtz I would expect the values in the input and output to be identical (within rounding error). Thanks, Huw from __future__ import division import sys from iotbx import mtz from iotbx import reflection_file_reader from iotbx import reflection_file_utils def run(args): mtz_file = reflection_file_reader.any_reflection_file("input.mtz") mtz_as_miller_array = mtz_file.as_miller_arrays() f_obs = mtz_as_miller_array[0] mtz_out = f_obs.as_mtz_dataset(column_root_label="F_iotbx") mtz_out.mtz_object().write("output.mtz") if (__name__ == "__main__"): run(sys.argv[1:]) some reflections from input.mtz are: H K L F_ccp4 SIGF_ccp4 DANO_ccp4 SIGDANO_ccp4 ISYM_ccp4 1 2 2 4.90461 2.31536 0 3.27441 0 1 2 3 186.172 1.86381 119.745 3.84862 0 1 2 4 289.31 3.06709 -5.42197 6.46531 0 1 2 5 378.928 2.7118 87.3381 5.56933 0 1 2 6 540.866 4.38513 46.4155 8.8053 0 The same reflections in output.mtz are: H K L F_iotbx SIGF_iotbx DANOF_iotbx SIGDANOF_iotbx ISYMF_iotbx 1 2 2 4.90461 2.00516 0 4.01032 0 1 2 3 186.172 1.8943 119.745 3.7886 0 1 2 4 289.31 3.15096 -5.42197 6.30192 0 1 2 5 378.928 2.74848 87.3381 5.49695 0 1 2 6 540.866 4.3939 46.4155 8.7878 0

2 2

Jenkins spam imminent (but not now)
by markus.gerstel＠diamond.ac.uk 03 Feb '15

03 Feb '15

Hello everyone, We are currently building a continuous integration setup for cctbx using Jenkins at Diamond. This means Jenkins will run all tests for every source code change. If any test (or the build itself) fails, it will send a notification email to this list. This could result in a number of mails. However they follow a pretty obvious format, so should be easy to filter (and we can fine-tune the spamminess if needed). Mails will start soonishly. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Best place to get ccp4io, boost, etc?
by Luc Bourhis 29 Jan '15

29 Jan '15

Hi, I noticed that the nightly build get their dependencies from http://cci.lbl.gov/repositories/ccp4io.gz and the like. Is this the preferred way to make sure one is in synch with trunk then? Best wishes, Luc

2 1

directory of cctbx tutorials
by David Waterman 14 Jan '15

14 Jan '15

Hi all, I've just spent a little time with Google collecting together tutorials for CCTBX and related open-source projects. I'm sure this is not a complete list, so I thought I'd share in case anyone knows of some other sources to add to it. I'm collecting this because the DIALS project is involved in workshops and teaching. My main interest is in diffraction data processing, but all tutorials that use core modules and types from cctbx are of interest, for potential updating/cannibalisation. General/developer - Tour of the cctbx http://cctbx.sourceforge.net/current/tour.html - Making compiled extensions. cctbx_introduction_3 at https://sourceforge.net/p/dials/code/HEAD/tree/trunk/doc/developer/ Data processing - Various tutorials on free-electron laser data processing (cctbx.xfel) at http://viper.lbl.gov/cctbx.xfel/index.php/Tutorials - CCTBX for diffraction calculations. cctbx_introduction_{1,2} at https://sourceforge.net/p/dials/code/HEAD/tree/trunk/doc/developer/ - Powder diffraction unit cell refinement ( <http://cctbx.sourceforge.net/siena2005/>Siena 2005) http://cctbx.sourceforge.net/siena2005/unit_cell_refinement.html - Data processing with DIALS. 4 tutorials at http://dials.sourceforge.net/doc/index.html Coordinate handling - PDB truncate to Ala (SBGrid 2008) (iotbx.pdb) at http://cctbx.sourceforge.net/sbgrid2008/tutorial.html Phasing - Direct methods (Siena 2005) http://cctbx.sourceforge.net/siena2005/direct_methods_light.html Model refinement - Rigid body refinement (IUCr 2008) at http://cctbx.sourceforge.net/iucr2008/rigid_body_refinement_core.html -- David

2 1

Boost Import Error
by Garrett Thompson 12 Jan '15

12 Jan '15

Hello, I recently looking at the cctbxbb mailing archive and I came across your email response a few months ago : http://phenix-online.org/pipermail/cctbxbb/2014-June/000933.html I was wondering if you could elaborate a bit on how to use the wrappers cctbx.python? I consider myself to be quite a beginner in this area and can't seem to import the same module, import scitbx.array_family.flex *ImportError: No module named boost.optional* When I do I receive the same error about having boost.optional being an import dependency. If you are able to, please be as explicit as possible, I am pretty bad with common conventions and terminology, as I have never studied computer science or any related area formally. Any help would be much appreciated! Sincerely, Garrett Thompson

2 1

e density from atomic model?
by TJ Lane 06 Jan '15

06 Jan '15

Crystallography pros, Field-outsider asking for advice here. Is there a standard way to estimate the electron density on a grid from an atomic model (e.g. PDB coordinates of atoms)? One thing that occurred to me was to simply FT the Cromer-Mann atom-centered Gaussians commonly used to model atomic form factors. Perhaps there is a more standard way, though? And possible code to mooch...? Thanks much! TJ

2 1

building on Mac OSX
by Roger Martin 05 Jan '15

05 Jan '15

Hi, I'm attempting to build on Mac. Via jenkins. The scons finds /usr/bin/c++ but my builds are based on gcc located at /opt/gcc/current/bin. A which c++ returns /opt/gcc/current/bin/c++. I tried to tell it by making a site_init.py according to http://www.scons.org/doc/production/HTML/scons-user.html#idm12919064 --------------- cd build mkdir -p site_scons cat <<EOF >./site_scons/site_init.py env = DefaultEnvironment(tools = ['gcc', 'gnulink'], CC = '/opt/gcc/current/bin/gcc', CXX = '/opt/gcc/currentbin/g++') EOF -------------- Do you have a way to tell it which compiler version to find that is compatible with the source code level? Regards, Roger

2 3

building from cctbx SVN repository
by Roger Martin 11 Dec '14

11 Dec '14

Hi, http://cctbx.sourceforge.net/current/installation.html shows svn export svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/development/cctbx_svn_getti… ./cctbx_svn_getting_started.csh where cctbx_svn_getting_started.csh starts with a curl: curl http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.selfx -o cctbx_bundle.selfx yet currently http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.selfx doesn't exist. http://cci.lbl.gov/cctbx_build/results/current/ and has the Nov 13 bundle. Has the build changed or this url http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.selfx really missing? Thank you, I'm wanting to be at svn HEAD for development in case I find something to add or patch. Roger

1 0

Correlation coefficient for symmetry-related reflections
by Vanbenschoten, Andrew 08 Dec '14

08 Dec '14

Hi everyone, I'm analyzing a diffraction dataset with very weak intensities and I'm getting extremely high R-linear values. Though this may simply be due to poor data, I'd like to calculate the correlation coefficient between sets of symmetry-related reflections to make sure the low intensity values aren't artificially inflating the observed R-linears. Is there a way to do this in Phenix or cctbx? Thanks! -Andrew

1 0

mean weighted phase error by resolution bin
by Mooers, Blaine H.M. (HSC) 05 Dec '14

05 Dec '14

Thanks to Pavel's suggestion, I was able to calculate the overall mean weighted phase error between to sets of phases in miller_arrays 2 and 5. I am stumped on how to retrieve the mean weighted phase error by resolution bin. #! /usr/bin/env cctbx.python import sys import iotbx mtz_filename = sys.argv[1] mtz_file = mtz.object(mtz_filename) miller_arrays = mtz_file.as_miller_arrays() binner=miller_array.setup_binner_d_star_sq_step(d_star_sq_step=0.01) mwpe = miller_arrays[2].mean_weighted_phase_error(miller_arrays[5]) Best regards, Blaine Blaine Mooers, Ph.D. Assistant Professor Department of Biochemistry and Molecular Biology University of Oklahoma Health Sciences Center S.L. Young Biomedical Research Center Rm. 466 Shipping address: 975 NE 10th Street, BRC 466 Oklahoma City, OK 73104-5419 Letter address: P.O. Box 26901, BRC 466 Oklahoma City, OK 73190 office: (405) 271-8300 lab: (405) 271-8313 fax: (405) 271-3910 e-mail: blaine-mooers(a)ouhsc.edu Faculty webpage: http://www.oumedicine.com/department-of-biochemistry-and-molecular-biology/… Small Angle Scattering webpage: http://www.oumedicine.com/docs/default-source/ad-biochemistry-workfiles/sma… X-ray lab webpage: http://www.oumedicine.com/department-of-biochemistry-and-molecular-biology/… ________________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of cctbxbb-request(a)phenix-online.org [cctbxbb-request(a)phenix-online.org] Sent: Wednesday, December 03, 2014 2:00 PM To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 69, Issue 3 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. Re: computing mean phase error between final refined model and experimental phases (Pavel Afonine) ---------------------------------------------------------------------- Message: 1 Date: Tue, 02 Dec 2014 12:06:08 -0800 From: Pavel Afonine <pafonine(a)lbl.gov> To: cctbx mailing list <cctbxbb(a)phenix-online.org>, Blaine-Mooers(a)ouhsc.edu Subject: Re: [cctbxbb] computing mean phase error between final refined model and experimental phases Message-ID: <547E1BB0.3030103(a)lbl.gov> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi Blaine, key functionality for this is the method of Miller array class called mean_phase_error, see: cctbx/miller/__init__.py . Unit and regression tests are best usage examples most of the time. In this case see: cctbx/regression/tst_miller.py . Let me know if you have any questions. I don't have a good grasp of phase error value as a quality measure. For example, I do not know what number is good, what is poor, etc. I think to some extent what you are trying to do can be converted to Fourier map comparison: just compute two maps, each one using identical amplitudes and two different sets of phases, and then compare maps. To me at least the map CC is a more telling metric. However, map comparison is tricky too (usual map CC may be misleading!) and you need to use proper metrics, as explained in details here: Acta Cryst. (2014). D70, 2593-2606. Metrics for comparison of crystallographic maps A. Urzhumtsev, P. V. Afonine, V. Y. Lunin, T. C. Terwilliger and P. D. Adams Pavel On 12/2/14 9:20 AM, Mooers, Blaine H.M. (HSC) wrote: > Dear CCTBXBB, > > I want to compare the phases from the final refined model and a set of > experimental phases > to calculate the weighted mean phase error. > > How do I this using python? > > Best regards, > > Blaine > > Blaine Mooers, Ph.D. > Assistant Professor > Department of Biochemistry and Molecular Biology > University of Oklahoma Health Sciences Center > S.L. Young Biomedical Research Center Rm. 466 > > Shipping address: > > > office: (405) 271-8300 lab: (405) 271-8313 fax: (405) 271-3910 > e-mail: blaine-mooers(a)ouhsc.edu > > Faculty webpage: > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.oumedicine.com_depa… > > Small Angle Scattering webpage: > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.oumedicine.com_docs… > > X-ray lab webpage: > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.oumedicine.com_depa… > > > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…

2 1

computing mean phase error between final refined model and experimental phases
by Mooers, Blaine H.M. (HSC) 02 Dec '14

02 Dec '14

Dear CCTBXBB, I want to compare the phases from the final refined model and a set of experimental phases to calculate the weighted mean phase error. How do I this using python? Best regards, Blaine Blaine Mooers, Ph.D. Assistant Professor Department of Biochemistry and Molecular Biology University of Oklahoma Health Sciences Center S.L. Young Biomedical Research Center Rm. 466 Shipping address: office: (405) 271-8300 lab: (405) 271-8313 fax: (405) 271-3910 e-mail: blaine-mooers(a)ouhsc.edu Faculty webpage: http://www.oumedicine.com/department-of-biochemistry-and-molecular-biology/… Small Angle Scattering webpage: http://www.oumedicine.com/docs/default-source/ad-biochemistry-workfiles/sma… X-ray lab webpage: http://www.oumedicine.com/department-of-biochemistry-and-molecular-biology/…

2 1

structure factors from map
by castro＠biochem.mpg.de 01 Dec '14

01 Dec '14

Hi, Could somebody please help to find the function to transform electron density map into structures factors? I found a function "from_map()" within cctbx/maptbx/structure_factor.h, But it don't know how to transform the map into a "complex_map" format. Thanks a lot in advance, Marcelino

3 3

Problem with executing Pymol with Tk (I build it with cctbx)
by Tam＠em.uni-frankfurt.de 17 Nov '14

17 Nov '14

Dear all, I have a problem to execute Pymol that it is without external GUI and just only the Viewer window. This is the error message I got after executing the pymol. Traceback (most recent call last): File "/home/mauricetam/software/cctbx/cctbx_plus_build/base/lib/python2.7/site-packages/pymol/__init__.py", line 401, in launch_gui __import__(self.invocation.options.gui) File "/home/mauricetam/software/cctbx/cctbx_plus_build/base/lib/python2.7/site-packages/pmg_tk/__init__.py", line 22, in <module> from PMGApp import * File "/home/mauricetam/software/cctbx/cctbx_plus_build/base/lib/python2.7/site-packages/pmg_tk/PMGApp.py", line 28, in <module> from Tkinter import * File "/home/mauricetam/software/cctbx/cctbx_plus_build/base/lib/python2.7/lib-tk/Tkinter.py", line 39, in <module> import _tkinter # If this fails your Python may not be configured for Tk ImportError: No module named _tkinter Thanks for the help. Regards HK

1 0

using the c++ fft_to_real_map_unpadded to make maps
by Roger Martin 14 Nov '14

14 Nov '14

Hi, I posted to the sourceforge help forum: http://sourceforge.net/p/cctbx/discussion/76274/thread/16122dcc/ about using fft_to_real_map_unpadded. Have had great success with it making maps for visualizing results. What I'm not understanding is the shape and properties of maps for 'P 32 2 1' and other space groups that have translations of 1/3, 1/4 or 1/6 in symmetry operators. Is there something special or extra that needs done for the fourier coefficients, n_real dimensions or the returned versa map? cctbx::sgtbx::space_group spaceGroup(cctbx::sgtbx::space_group_symbols(154)); const scitbx::af::int3 n_real(n, m, l); scitbx::af::versa< double, scitbx::af::c_grid<3>> pDelta=cctbx::maptbx::fft_to_real_map_unpadded(spaceGroup, n_real, miller_indices.const_ref(), structure_factors.const_ref()); Looking inside I see it does tha padding for me. So running out of things to try. Any ideas would be appreciated --Roger

1 0

Electron density hitogram from MTZ file
by castro＠biochem.mpg.de 28 Oct '14

28 Oct '14

Hi everybody, I am new in crystallogrphy and in the use of cctbx. I am interested in building a electron density histogram from a mtz file. Could somebody give a hint on how to accomplish this task using cctbx libraries (I am scripting in python)? Let's say that I already have the columns of interest as "miller arrays". Then (I think), a grid (on the real space) needs to be created based on unit cell parameters, to finally make the fft. I think that most of my doubts are about the grid. How small the grid should be with respect to the low-resolution limit? Does the space group information needed or it is enough with the the cell parameters? Is there all ready a routine doing this? Thanks a lot for your help. Marcelino

3 4

which cctbx version is recommended now?
by Marcin Wojdyr 15 Oct '14

15 Oct '14

Hi, we need to pick a version of cctbx for CCP4 for use in phaser and DIALS. The latest logs at cci.lbl.gov/cctbx_build/show_error_summary.cgi?build_tag=2014_10_05_2205 show in all builds: [Errno 2] No such file or directory Traceback (most recent call last): ImportError: __import__("boost_python_meta_ext"): No module named boost_python_meta_ext Is it better to pick older version? Marcin

2 3

Miller arrays from lists of indices
by Oliver Zeldin 06 Oct '14

06 Oct '14

Dear cctbx devs, Does anyone know how to make a miller set from a list of (h, k, l) indices? e.g. [(1,2,3), (34, 1, 45), .... ] Cheers, Oliver -- Dr. Oliver B. Zeldin Brunger Group Stanford University

2 2

new cctbx-plus installers
by Nathaniel Echols 23 Sep '14

23 Sep '14

Hi all-- We now have improved installers for CCTBX with all optional third-party dependencies, including numpy, wxPython, and matplotlib; these are basically just a simplified Phenix installer (minus anything that isn't open-source, obviously). For anyone doing visualization these are probably preferable to the default packages. More useful for doing active development is that since these include the pre-build Python, if you are working directly with the CCTBX sources you can bootstrap from these packages. Yesterday's build is here: http://cci.lbl.gov/cctbx_build/plus_bundles/2014_09_08_0005/ and in general the latest version can be obtained here: http://cci.lbl.gov/cctbx_build/plus_bundles/current although today's build isn't finished yet. The next build will also incorporate IPython. I still need to fix the download page so these are listed alongside the old bundles. Once this is done we'll probably declare an official "release". -Nat

3 5

Tests which randomly pass or fail (nag)
by Graeme.Winter＠diamond.ac.uk 16 Sep '14

16 Sep '14

Hi Folks Quick nag if I may... if people have tests which employ random inputs (e.g. random unit cell, random amplitudes and phases) please could they set a seed or select from some seeds which should give random values which work, or fix their tests so that all random values work (ideally!) That way when tests fail people know something is wrong, rather than everything is fine but the random values drawn are troublesome... Sorry for sounding grumpy! Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

order bias in flex.random_permutation
by Keitaro Yamashita 11 Sep '14

11 Sep '14

Dear cctbx developers, I happened to notice that the implementation of flex.random_permutation (scitbx/random.h) results in biased order because each element is always swapped with the element randomly selected from entire range of array. For details, please see the second paragraph and the right figure of http://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle#Implementation_er… I reproduced the plot using flex.random_permutation. Actually this seems to have less impact because flex.random_permutation is rarely used in cctbx or phenix as far as I know. If needed, I would commit the corrected version (using the algorithm in the wikipedia article above). Best regards, Keitaro

2 1

IPython with cctbx
by Oliver Zeldin 08 Sep '14

08 Sep '14

Dear Devs, Has anyone tried to get IPython working with cctbx? I'd really like to be able to use IPython notebooks for examples/tutorials/documented experiments, and was wondering if anyone else had thought the same thing. Cheers, Oli

3 10

Re: [cctbxbb] Sphinx docs for CCTBX
by Nigel Moriarty 19 Aug '14

19 Aug '14

These links don't work for me. On Mon, Aug 18, 2014 at 3:16 PM, Ian Rees <irees(a)lbl.gov> wrote: > I'm using a separate build configuration for the docs. You can view > the nightly results here: > > http://cci-vm-6.lbl.gov:8010/builders/cctbx%2Bdocs-linux-64 > > If you click the first build, you can view the log files for each > step, and view the Sphinx log file by clicking "docs:cctbx stdio", > e.g. > > > http://cci-vm-6.lbl.gov:8010/builders/cctbx%2Bdocs-linux-64/builds/6/steps/… > > I'm working through the warnings now. Mostly missing references and > import errors. > > Ian > > On Fri, Aug 15, 2014 at 8:37 AM, Nathaniel Echols <nechols(a)lbl.gov> wrote: > > On Fri, Aug 15, 2014 at 8:06 AM, Richard Gildea <rgildea(a)gmail.com> > wrote: > >> > >> I had a go at starting to document cctbx.uctbx.unit_cell, which as you > >> know is mainly a Boost.Python extension > >> (https://sourceforge.net/p/cctbx/code/20467/). Before I travel too much > >> further down this route, is this an appropriate way of documenting > >> Boost.Python extensions or is there a better way we can do this? > > > > > > I haven't had enough time to experiment to decide on the best approach. > I > > see three choices - the first is what you've done: > > > > class my_class (ext.my_class) : > > """doc""" > > > > def boosted_function (self, some_argument) : > > """function doc""" > > super(ext.my_class, self).boosted_function(some_argument) > > > > OR (2): > > > > ext.my_class.__doc__ = """doc""" > > ext.my_class.boosted_function.__func__.__doc__ = """function doc""" > > > > OR (3): > > > > put the docstrings in C++ code, for instance: > > > > .def("boosted_function", &my_class::boosted_function, "function doc") > > > > (not sure what the equivalent is for classes but it's probably pretty > > similar) > > > > My opinion is that (1) is much too prone to confusion and API conflicts > > unless we're already using this code structure (and FYI, you broke the > call > > signatures for a couple of methods). (3) is just gross. (2) is also > gross > > but has the virtue of being the least likely to break, and doesn't > require > > modifying C++ files. So I'm inclined to take that approach > > > > (For what it's worth, when modifying a class using the boost.python > > injector, we appear to be able to set __doc__ inside there; see > > iotbx.pdb.hierarchy for an example.) > > > >> It would be good to get the documentation for the core cctbx classes up > >> and running, however several of the core classes (e.g. > >> cctbx.sgtbx.space_group, cctbx.uctbx.unit_cell, flex arrays) are mostly > >> Boost.Python extensions, so it would be good to arrive at a sensible > way to > >> document these extensions. > > > > > > Agreed. I think for flex arrays we will be stuck writing the docstrings > in > > C++ - however since these are relatively stable APIs it should be > reasonably > > straightforward. > > > > -Nat > -- Nigel W. Moriarty Building 64R0246B, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

2 1

real-space symmetry operators
by Nathaniel Echols 18 Aug '14

18 Aug '14

Hi all-- I'm trying to figure out how to orthogonalize symmetry operators given a unit cell, which doesn't seem to be something we have code for. These appear in PDB files in the REMARK 290 section but I haven't been able to find any other software that generates them, and the relationship between the symops and the real-space equivalents is unclear. I assume this is conceptually similar to orthogonalizing fractional coordinates, but unit_cell.orthogonalize() does not quite do what I wanted. In the PDB, for P63 it has this (excerpted): REMARK 290 2555 -Y,X-Y,Z ... REMARK 290 SMTRY1 2 -0.500000 -0.866025 0.000000 0.00000 REMARK 290 SMTRY2 2 0.866025 -0.500000 0.000000 0.00000 REMARK 290 SMTRY3 2 0.000000 0.000000 1.000000 0.00000 which looks like it should be relatively straightforward, but I'm not very good with matrices. Does anyone know the correct math for this? The corresponding rt_mx object from sgtbx looks like this: r=(0.0, -1.0, 0.0, 1.0, -1.0, 0.0, 0.0, 0.0, 1.0) t=(0.0, 0.0, 0.0) (Yes, I realize that there are much easier ways to apply symmetry operators to coordinates, but I want to limit the number of matrix multiplications involved when starting from Cartesian coordinates.) thanks, Nat

3 3

Sphinx docs for CCTBX
by Nathaniel Echols 15 Aug '14

15 Aug '14

Thanks to some superb work by Nader Morshed, Oliver Zeldin, and Ian Rees, we now have automatically updated Sphinx docs for CCTBX: http://cci.lbl.gov/cctbx_docs/ Obviously these are still pretty sparse - right now most of the formal documentation is clustered here: libtbx libtbx.utils libtbx.phil (mostly re-using Ralf's old documentation) cctbx.miller iotbx.pdb (built on Ralf's newsletter article) mmtbx.command_line These are a combination of inline docstrings and separate restructured text files in cctbx_project/sphinx. We have automatically generated rst files for some modules, but others will need to be added manually. In the end I think we'll want to curate every file, but the current layout will get us started. To facilitate additional contributions, I've modifed the installation script for base packages to include a --sphinx option. After setting up CCTBX you can then do this: cd $BUILD libtbx.configure sphinx cd $SRC/cctbx_project/sphinx make html and you will get the full documentation built locally. I am still figuring out how to use Sphinx effectively; it is definitely possible to document Boost.Python extensions but it's not quite as obvious. I don't expect us to retroactively document several hundred thousand lines of code written over a decade, but if we can get the core parts thoroughly covered that will greatly reduce the barrier for new developers. Suggestions for particular modules to focus on are appreciated; I am going to try to cover the basic PDB+reflection file functionality. I encourage anyone doing CCTBX-related development to contribute as you go. (They don't need to be your own APIs, just modules you're reasonably familiar with.) Those of you who already have a ton of undocumented code can expect me to harass you periodically about fixing this. (Note that I've cannibalized at least one newsletter article for this, and continue to do so; we should re-use existing prose descriptions as much as possible without violating copyrights.) -Nat

4 5

Re: [cctbxbb] resend
by Mustyakimov, Marat 24 Jul '14

24 Jul '14

Loop through symops: apply each symop to the original asu, then apply unit translations to minimize its distance to the original asu, test if new asu intersects with the target 27-bricks-large box, use it, if it does. Keep in mind that asu is not always a brick, it is always a convex polyhedron in cctbx. I do not see any good reason for doing all that. I do not read cctbxbb. On Jul 24, 2014, at 2:43 PM, Alastair Fyfe <afyfe(a)soe.ucsc.edu> wrote: >> Hi Marat, my apologies if this is a duplicate. From Pavel's comment it occurred to me you might not follow posts to the cctxbb >> Alastair >> ------ >> >> Many thanks for your replies. >> >> Pavel : by "slow" I meant to (a) apply sym ops to asu and refer results >> back to unit cell thus obtaining a "full" unit cell (I expect >> "expand_to_p1()" does this?) (b) translate the cell and (c) find overlap >> with the 26-brick neighborhood of the starting brick. This should work >> but seems inefficient. >> >> Marat: applying SymOps to the basis vectors seems a great idea. >> However, it wasn't clear to me how to combine sg SymOps to yield the >> 26 adjacent bricks. Does the map expansion code do this? >> >> >

1 0

Re: [cctbxbb] 26-fold symm expansion of asu?
by Alastair Fyfe 24 Jul '14

24 Jul '14

Many thanks for your replies. Pavel : by "slow" I meant to (a) apply sym ops to asu and refer results back to unit cell thus obtaining a "full" unit cell (I expect "expand_to_p1()" does this?) (b) translate the cell and (c) find overlap with the 26-brick neighborhood of the starting brick. This should work but seems inefficient. Marat: applying SymOps to the basis vectors seems a great idea. However, it wasn't clear to me how to combine sg SymOps to yield the 26 adjacent bricks. Does the map expansion code do this? On 07/23/2014 12:00 PM, cctbxbb-request(a)phenix-online.org wrote: > Send cctbxbb mailing list submissions to > cctbxbb(a)phenix-online.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://phenix-online.org/mailman/listinfo/cctbxbb > or, via email, send a message with subject or body 'help' to > cctbxbb-request(a)phenix-online.org > > You can reach the person managing the list at > cctbxbb-owner(a)phenix-online.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cctbxbb digest..." > > > Today's Topics: > > 1. Re: 26-fold symm expansion of asu? (Pavel Afonine) > 2. Re: 26-fold symm expansion of asu? (Mustyakimov, Marat) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 22 Jul 2014 18:58:32 -0700 > From: Pavel Afonine <pafonine(a)lbl.gov> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Cc: "Mustyakimov, Marat" <mustyakimovm(a)ornl.gov> > Subject: Re: [cctbxbb] 26-fold symm expansion of asu? > Message-ID: <53CF16C8.5040504(a)lbl.gov> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi Alastair, > > I think I can write a piece of code using cctbx that would do what you > want and probably that would be what you call "slow". Could you please > explain some more what it is for and why, and may be give some context? > Perhaps that helps me to see if we have something similar already > available.. > > For instance we have very efficient code that computes maps in ASU, and > also that expands map in ASU into the whole P1 box. This is something > Marat wrote recently and is very efficient. May be you can reuse that or > at least harvest ideas? > > I'm copying to Marat in case he has comments. > > Pavel > > On 7/21/14, 2:14 PM, Alastair Fyfe wrote: >> Hi, >> I'd be grateful for any pointers to clipper or cctbx code that >> addresses the following: given a "brick" of map grid coordinates that >> enclose the asu, obtain the 26 symmetry-related "bricks" which contact >> it. Calculating this by successively generating symmetry mates and >> testing seems slow; I was wondering whether there was a better way. >> thank you, >> Alastair Fyfe >> UCSC >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb > > > ------------------------------ > > Message: 2 > Date: Wed, 23 Jul 2014 14:01:13 +0000 > From: "Mustyakimov, Marat" <mustyakimovm(a)ornl.gov> > To: Pavel Afonine <pafonine(a)lbl.gov> > Cc: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] 26-fold symm expansion of asu? > Message-ID: <9457BEEC-4821-4BBB-8D91-E903AAFAE7F0(a)ornl.gov> > Content-Type: text/plain; charset="iso-8859-1" > > The fast way to generate symmetry equivalent ASU: > Apply symmetry operation to unit vectors: ex = SymOp*(1,0,0), ey=SymOp*(0,1,0) and ez=SymOp*(0,0,1); > Then every map point of the symmetry related asu is obtained from the original one: > new_ijk = i*ex + j*ey + k*ez where (i,j,k) is the grid point in the original ASU. > > Marat Mustyakimov > P.O. Box 2008, Bldg. 8600, MS-6475 > Oak Ridge National Lab > Oak Ridge TN 37831 USA > e-mail: mustyakimovm(a)ornl.gov<mailto:[email protected]> > e-mail: marat.mustyakimov(a)gmail.com<mailto:[email protected]> > Tel: +1-865-253-3924 > > On Jul 22, 2014, at 9:58 PM, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]>> wrote: > > Hi Alastair, > > I think I can write a piece of code using cctbx that would do what you want and probably that would be what you call "slow". Could you please explain some more what it is for and why, and may be give some context? Perhaps that helps me to see if we have something similar already available.. > > For instance we have very efficient code that computes maps in ASU, and also that expands map in ASU into the whole P1 box. This is something Marat wrote recently and is very efficient. May be you can reuse that or at least harvest ideas? > > I'm copying to Marat in case he has comments. > > Pavel > > On 7/21/14, 2:14 PM, Alastair Fyfe wrote: > Hi, > I'd be grateful for any pointers to clipper or cctbx code that addresses the following: given a "brick" of map grid coordinates that enclose the asu, obtain the 26 symmetry-related "bricks" which contact it. Calculating this by successively generating symmetry mates and testing seems slow; I was wondering whether there was a better way. > thank you, > Alastair Fyfe > UCSC > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > > > ------------------------------ > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb > > > End of cctbxbb Digest, Vol 64, Issue 2 > **************************************

1 0

26-fold symm expansion of asu?
by Alastair Fyfe 23 Jul '14

23 Jul '14

Hi, I'd be grateful for any pointers to clipper or cctbx code that addresses the following: given a "brick" of map grid coordinates that enclose the asu, obtain the 26 symmetry-related "bricks" which contact it. Calculating this by successively generating symmetry mates and testing seems slow; I was wondering whether there was a better way. thank you, Alastair Fyfe UCSC

3 2

Phenix dependencies in CCTBX code
by Graeme.Winter＠diamond.ac.uk 22 Jun '14

22 Jun '14

Dear All, In cctbx_project/iotbx/pdb/multimer_reconstruction.py there is a dependency on Phenix from phenix.command_line.simple_ncs_from_pdb import simple_ncs_from_pdb from phenix.command_line.simple_ncs_from_pdb import ncs_master_params Please could someone remove this? I just had a look at cleaning things up so that this was encapsulated but was not obvious and figured risk of breaking was more substantial. This is breaking our nightly build tests at Diamond as we don’t check out phenix :o( Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Phenix dependencies in CCTBX code
by Graeme.Winter＠diamond.ac.uk 22 Jun '14

22 Jun '14

Dear All, In cctbx_project/iotbx/pdb/multimer_reconstruction.py there is a dependency on Phenix from phenix.command_line.simple_ncs_from_pdb import simple_ncs_from_pdb from phenix.command_line.simple_ncs_from_pdb import ncs_master_params Please could someone remove this? I just had a look at cleaning things up so that this was encapsulated but was not obvious and figured risk of breaking was more substantial. This is breaking our nightly build tests at Diamond as we don’t check out phenix :o( Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] Boost Import Error
by Magnus Andersson 19 Jun '14

19 Jun '14

Hi all, In case anyone else runs into the same problems (which is not unlikely), the following worked for me: > svn co https://pymol.svn.sourceforge.net/svnroot/pymol/trunk/pymol > cd pymol > cctbx.python setup.py build > cctbx.python setup.py install > Thanks to Luc & Nat for being helpful and persistent! Cheers, Magnus > > On 17 Jun 2014, at 21:34, Luc Bourhis <luc_j_bourhis(a)mac.com> wrote: > >> Hi, >> >> it's a bit late here in Paris but I can at least fix one point: >> >>> port select --set python27 >>> Error: The 'set' command expects two arguments: <group>, <version> >> >>> I tried to figure this out but without success... sorry to bug you with syntax issues. >> >> I'm the one to be sorry here as I pulled it from the top of my head and got it wrong. It's >> >> port select --set python python27 >> >> As for your original problem, I'll try to regroup and think through our former exchange to see what I may have missed. >> >> Best wishes, >> >> Luc >> >

1 0

Boost Import Error
by Magnus Andersson 17 Jun '14

17 Jun '14

Hi, I've installed the cctbx package from the selfextracting file found in http://cci.lbl.gov/cctbx_build/ but I run into problems when importing cctbx in my python script: import iotbx.pdb File "/Users/magnus/Downloads/cctbx_sources/iotbx/pdb/__init__.py", line 2, in <module> from cctbx.array_family import flex File "/Users/magnus/Downloads/cctbx_sources/cctbx/array_family/flex.py", line 2, in <module> import scitbx.array_family.flex File "/Users/magnus/Downloads/cctbx_sources/scitbx/array_family/flex.py", line 2, in <module> import boost.optional # import dependency ImportError: No module named boost.optional It is flex.py that wants to import boost.optional and does not find it. However, I find these three files: optional.hpp optional_fwd.hpp optional_io.hpp in the following directory: /Users/magnus/Downloads/cctbx_sources/boost/boost/optional which suggests that it's present. What am I missing here? I'm using Mac OS X Version 10.9.3 and Python2.7. Cheers, Magnus

4 26

sparse matrix
by David Waterman 10 Jun '14

10 Jun '14

Hi, I have a problem where the use of a sparse matrix is appropriate, so I was pretty happy to grep cctbx for "sparse" and discover the code in scitbx/sparse. I'm just browsing through the code and tests to get a feel for how to use these types. In my existing Python code I set the values for some columns using flex.double vectors, then ultimately fill a flex.double(flex.grid(n,m)) by these columns using matrix_paste_column_in_place. It seems to me I can use a very similar pattern by replacing the flex.doubles with sparse.matrix_columns and setting columns in the sparse.matrix using slice notation: mat[:,0] = column. So far so good. The issue is that I'm not sure how best to go about setting the values in the matrix_column in the first place. At the moment I'm making a lot of use of flex.size_t selections and the set_selected method of flex.double to efficiently (from Python) set elements of one flex.double from another. It appears that there is no similar functionality for sparse.matrix_column. Am I right? Possibly the best I can do is construct a Python dictionary consisting of the indices from my size_t selection and the values from my flex.double array and use these to construct the matrix_column, as done in tst_sparse, e.g. sparse.matrix_column(m, {1:0.1, 2:0.2}). Without having tried this, I sense it might be a burden. I have a lot of values to set and don't really want to convert from flex.double via a dictionary. Are there any philosophical/technical/other reasons for not writing a set_selected method for sparse.vector? If not, I might try to do this. Thanks -- David

2 5

where is the mmtbx.mp_geo command
by Sun, Miao 13 Mar '14

13 Mar '14

Hi, I have installed cctbx software, however, I cant find the command mmtbx.mp_geo. I wanna to validate the backbone bond and angles. Is there any other command could do that? Thanks. -Miao

3 2

Re: [cctbxbb] Adding columns to new *unmerged* MTZ object
by Graeme.Winter＠diamond.ac.uk 13 Feb '14

13 Feb '14

In related news…. Should void object::adjust_column_array_sizes(int new_nref) { CMtz::MTZ* p = ptr(); if (!p->refs_in_memory) return; if (new_nref > p->nref) { reserve(new_nref); for(int i=0;i<p->nxtal;i++) { for(int j=0;j<p->xtal[i]->nset;j++) { for(int k=0;k<p->xtal[i]->set[j]->ncol;k++) { CMtz::MTZCOL* col_k = p->xtal[i]->set[j]->col[k]; int old_size = column_array_size(col_k); if (new_nref > old_size) { ccp4array_resize(col_k->ref, new_nref); for(int iref=old_size;iref<new_nref;iref++) { *(reinterpret_cast<union float_uint_uchar*>( &col_k->ref[iref])) = not_a_number_value_; } } } } } } } assign nref? otherwise repeated calling of this will keep on re-allocating the arrays Thanks Graeme On 12 Feb 2014, at 10:40, Graeme Winter <graeme.winter(a)diamond.ac.uk> wrote: > All, > > By adding > > m_out.adjust_column_array_sizes(len(mi)) > > to the bpl wrapper and defining > > m_out.set_n_reflections(len(mi)) > > in the object.h and bpl wrapper I can do what I want… > > I will add some tests and some warnings that this may make nonsense of the values… however before committing any comments on the interface? or behaviour? > > Thanks Graeme > > > > On 12 Feb 2014, at 09:11, <Graeme.Winter(a)diamond.ac.uk> <Graeme.Winter(a)diamond.ac.uk> wrote: > >> Hi Folks, >> >> I am trying to create a new MTZ file containing unmerged and it seems to be less simple than I would think - set_reals() on a column requires a set of Miller indices and values and does a lookup internally (which I don’t want for unmerged values) and set_values() requires that the array is already the right size - when it starts off as size 0 and I have no idea how to make enough room in there from the Python API… >> >> Any clues? I have done quite a chunk of grepping around to try and figure this one out and have run out of ideas… >> >> Thanks Graeme_______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >

3 11

Adding columns to new *unmerged* MTZ object
by Graeme.Winter＠diamond.ac.uk 12 Feb '14

12 Feb '14

Hi Folks, I am trying to create a new MTZ file containing unmerged and it seems to be less simple than I would think - set_reals() on a column requires a set of Miller indices and values and does a lookup internally (which I don’t want for unmerged values) and set_values() requires that the array is already the right size - when it starts off as size 0 and I have no idea how to make enough room in there from the Python API… Any clues? I have done quite a chunk of grepping around to try and figure this one out and have run out of ideas… Thanks Graeme

1 1

building with external boost
by Marcin Wojdyr 30 Jan '14

30 Jan '14

Hi all, would it be ok if I added an option to use installed boost libraries? Debian and gentoo patches add boolean --use_system_libs flag. I could use the same, but with a list of arguments (at first only --use_system_libs=boost). Marcin

2 2

boost python and metaclass inheritance
by David Waterman 10 Jan '14

10 Jan '14

Hi, Does anyone know how to use __metaclass__ specification alongside inheritance from boost python classes? This code illustrates the problem: import abc > from scitbx.lstbx import normal_eqns > class Refinery(object): > __metaclass__ = abc.ABCMeta > pass > class AdaptLstbx(Refinery, normal_eqns.non_linear_ls): > pass The result is Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/usr/lib/python2.7/abc.py", line 87, in __new__ > cls = super(ABCMeta, mcls).__new__(mcls, name, bases, namespace) > TypeError: Error when calling the metaclass bases > metaclass conflict: the metaclass of a derived class must be a > (non-strict) subclass of the metaclasses of all its bases The real Refinery is a class that has one method that must be supplied by a concrete subclass. I like the idea of making it impossible to instantiate a base Refinery rather than relying on e.g. raise NotImplementedError() in that method definition. This is what ABCMeta would give me, if only I could get it to work here. Google found me some answers that require me to know the __metaclass__ of the bases (as suggested by the error message), but normal_eqns.non_linear_ls.__metaclass__ does not exist. I don't terribly mind the raise NotImplementedError() alternative, but I thought that if I can easily use ABCMeta, I'd like to. Cheers -- David

2 5

cctbx.python unnecessarily unfriendly
by James Stroud 10 Jan '14

10 Jan '14

Hello CCTBX developers, I noticed in the cctbx.python script the following lines that call the python interpreter if [ -n "$LIBTBX__VALGRIND_FLAG__" ]; then exec $LIBTBX_VALGRIND "$LIBTBX_PYEXE" -Qnew "$@" elif [ $# -eq 0 ]; then exec "$LIBTBX_PYEXE" -Qnew else exec "$LIBTBX_PYEXE" -Qnew "$@" fi Here, the “-Qnew” flag forces the python interpreter to work with new division (where “/“ does not strictly mean integer division). This flag basically overrides the expected python behavior to which python programmers have been writing code for over a decade, wrecks some dependencies, and stifles new code that itself depends on cctbx. It is highly frustrating to work around this highly unnecessary flag!! Fortunately, python has a better way to handle this problem by just having “from __future__ import division” at the top of .py files. In fact, I have spot-checked several files in the cctbx source tree and noticed that the ALL have this line at the top. So why does the -Qnew flag still exist in the cctbx.python script? It is entirely unnecessary and requires any and all dependencies to be programmed to a behavior that is not standard for the language. James p.s. It is not reasonable to ask users who install your packages to patch the cctbx.python script or to patch dependencies that don’t conform to this non-standard behavior.

6 13

building cctbx with external boost
by Marcin Wojdyr 19 Dec '13

19 Dec '13

Hi I vaguely remember a discussion some time ago about building cctbx with stand-alone boost libraries. Do you know what boost versions are compatible? Is using the --scan-boost option enough? I see that Gentoo has extra patch to unbundle boost from cctbx: http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sci-libs/cctbx/file… Is it necessary? I'm asking because at this moment we have in CCP4 two versions of boost, one in cctbx and the other for building DSSP (it uses different subset of boost libs than cctbx). It would make sense to use a single version for both, if possible. Cheers Marcin

2 3

Re: [cctbxbb] cctbxbb Digest, Vol 59, Issue 1
by Zhenwei Luo 08 Dec '13

08 Dec '13

Hi Nat, Thanks for your reply! I have tried the first method you mentioned--adding an index value to the return method. It did work if I only needed a single array for future calculation. But I found it's not very convenient for my program. It's convenient and more efficient if other classes in it could process all the one-dimensional arrays at the same time. Otherwise, I needed to write more control code in python, and the c++ part could only deal with a single array now. Therefore, I hoped that I could pass such a multidimensional array in my program. Sincerely, Zhenwei On Sun, Dec 8, 2013 at 2:00 PM, <cctbxbb-request(a)phenix-online.org> wrote: > Send cctbxbb mailing list submissions to > cctbxbb(a)phenix-online.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://phenix-online.org/mailman/listinfo/cctbxbb > or, via email, send a message with subject or body 'help' to > cctbxbb-request(a)phenix-online.org > > You can reach the person managing the list at > cctbxbb-owner(a)phenix-online.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cctbxbb digest..." > > > Today's Topics: > > 1. Passing array between python and C++ (Zhenwei Luo) > 2. Re: Passing array between python and C++ (Nathaniel Echols) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 7 Dec 2013 22:25:08 -0600 > From: Zhenwei Luo <tluozhenwei(a)gmail.com> > To: cctbxbb(a)phenix-online.org > Subject: [cctbxbb] Passing array between python and C++ > Message-ID: <90518A19-AC1B-493B-8B27-79D31EDDF055(a)gmail.com> > Content-Type: text/plain; charset=us-ascii > > Hello Everyone, > > I am writing a refinement program using the cctbx. I have > encountered with a problem when I wanted to pass an array between c++ and > python. The type of array I used in my c++ source file is > af::shared<af::shared<vec3> >, I would like to pass it into my python > script, thus I could do some other operations to this array. > I wrote a method in my class to return this array. > > af::shared<af::shared<vec3<double> > return_modes( ) { return modes; } > > I wrote another method in the *_ext file to wrap this method to make it > callable from python. > > .def("return_modes", &nm_init::return_modes) > > But when I use it in python, > eigenvector = nm_init_manager.return_modes() > > It gave me an error: > > TypeError: No to_python (by-value) converter found for c++ type: > scitbx::af::shared<scitbx::af::shared<scitbx::vec3<double> > > > > What I want to realize is just creating an two dimensional array using > vec3 as its element. I can also using the select method in this array to > select some elements from its column, just like in the > af::shared<vec3<double> > array. Namely, the column of this array should > have some same properties like af::shared<vec3<double> >. Can anyone help > me figure out what I should do? > > Thank you very much! Any help will be greatly appreciated. > > Sincerely, > Zhenwei Luo > Rice University > > > ------------------------------ > > Message: 2 > Date: Sat, 7 Dec 2013 20:45:10 -0800 > From: Nathaniel Echols <nechols(a)lbl.gov> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] Passing array between python and C++ > Message-ID: > <CALeAa1MOL1wbW5Vb=M= > jXsnGTaCBQNJjcOzDwMhM27x5PP+XNA(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On Sat, Dec 7, 2013 at 8:25 PM, Zhenwei Luo <tluozhenwei(a)gmail.com> wrote: > > > I am writing a refinement program using the cctbx. I have > > encountered with a problem when I wanted to pass an array between c++ and > > python. The type of array I used in my c++ source file is > > af::shared<af::shared<vec3> >, I would like to pass it into my python > > script, thus I could do some other operations to this array. > > I wrote a method in my class to return this array. > > > > af::shared<af::shared<vec3<double> > return_modes( ) { return modes; } > > > > I wrote another method in the *_ext file to wrap this method to make it > > callable from python. > > > > .def("return_modes", &nm_init::return_modes) > > > > But when I use it in python, > > eigenvector = nm_init_manager.return_modes() > > It gave me an error: > > TypeError: No to_python (by-value) converter found for c++ type: > > scitbx::af::shared<scitbx::af::shared<scitbx::vec3<double> > > > > > > What that error means is that boost.python doesn't know how to present this > new type to Python. For the data structures that we pass back and forth, > we have to explicitly provide wrappers for them - including new > instantiations of scitbx::array_family templates. > > What I want to realize is just creating an two dimensional array using vec3 > > as its element. I can also using the select method in this array to > select > > some elements from its column, just like in the af::shared<vec3<double> > > > array. Namely, the column of this array should have some same properties > > like af::shared<vec3<double> >. Can anyone help me figure out what I > should > > do? > > > > In general the scitbx::array_family templates aren't really intended to > every be multidimensional; for inherently 2D and 3D data types like images > and maps, we use flex.grid to provide a multi-dimensional accessor on top > of the 1D array. The grid may not be the most appropriate in your case but > if I'm making the correct assumptions about the data you're trying to pass > back and forth, I think it might be easiest to simply define a custom class > that contains a std::vector of af::shared<vec3> arrays, and add an index > parameter to the return_modes method, which would be modified to return > only a single array of modes at a time. You can also do things like > wrapping a method as __getitem__ with boost.python, which would mimic a 2D > array. (I can probably dig up/cook up some code examples tomorrow if this > is unclear, since it was certainly not intuitive to me when I started > working with CCTBX and it would be nice to have documentation for the > future.) > > -Nat >

1 0

Passing array between python and C++
by Zhenwei Luo 07 Dec '13

07 Dec '13

Hello Everyone, I am writing a refinement program using the cctbx. I have encountered with a problem when I wanted to pass an array between c++ and python. The type of array I used in my c++ source file is af::shared<af::shared<vec3> >, I would like to pass it into my python script, thus I could do some other operations to this array. I wrote a method in my class to return this array. af::shared<af::shared<vec3<double> > return_modes( ) { return modes; } I wrote another method in the *_ext file to wrap this method to make it callable from python. .def("return_modes", &nm_init::return_modes) But when I use it in python, eigenvector = nm_init_manager.return_modes() It gave me an error: TypeError: No to_python (by-value) converter found for c++ type: scitbx::af::shared<scitbx::af::shared<scitbx::vec3<double> > > What I want to realize is just creating an two dimensional array using vec3 as its element. I can also using the select method in this array to select some elements from its column, just like in the af::shared<vec3<double> > array. Namely, the column of this array should have some same properties like af::shared<vec3<double> >. Can anyone help me figure out what I should do? Thank you very much! Any help will be greatly appreciated. Sincerely, Zhenwei Luo Rice University

2 1

libtbx.run_tests_parallel
by Graeme.Winter＠diamond.ac.uk 14 Nov '13

14 Nov '13

Folks, Would anyone mind if I added an option to libtbx.run_tests_parallel to automatically make a temporary directory, chdir to there and run? It would mean I could write a little update / build / run tests script more easily. auto_run_dir=true springs to mind as a Phil option. Any thoughts? Thanks & best wishes, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

iotbx newbie question
by wtempel 07 Nov '13

07 Nov '13

Hello all, please consider the scenario of a manuscript that, among other things, describes 20 crystal structures. Suppose the journal editor requires a Table 1. I now looking for a tool to prepare such table with high accuracy and little effort. My PDB deposition workflow includes the preparation of mmcif coordinates with pdb_extract. The mmcif file is complete in the sense that after upload to RCSB-ADIT, virtually no additional information is required for deposition. My intuition was to combine CSV and IOTBX python modules and produce the table via input of a list of mmcif files. The current phenix table_1 tool provides some, but not all of that functionality, and I thought once familiar with the python modules, I would quickly write the script. After confirming that the example at http://cctbx.sourceforge.net/iotbx_cif/ works with my cctbx installation, I attempted to transform that code to read in PDB mmcif coordinates: <code> test_structure = iotbx.cif.reader( file_path="/tmp/1NXC.cif").build_crystal_structures()["1NXC"] </code> which throws an exception: <output> Traceback (most recent call last): File "1NXC.py", line 3, in <module> test_structure = iotbx.cif.reader(file_path="/tmp/1NXC.cif").build_crystal_structures()["1NXC"] KeyError: '1NXC' </output> How do I need to write the code so that it returns a python object with access to all the cif items? Thank you, Wolfram Tempel

2 1

compiler warning in dxtbx
by Oleg Dolomanov 04 Nov '13

04 Nov '13

Hi guys, there was big change in dxtbx/model/panel.h on the 30th of October (r18499) and apparently the != operator became recursive by calling itself rather than the == operator (line 378) - whoever is responsible - please check/fix, Cheers, Oleg.

3 8

Request for a fix in CC1/2 calculation in phenix.merging_statistics
by Keitaro Yamashita 28 Oct '13

28 Oct '13

Dear CCTBX developers, I found a big difference in CC1/2 value (say, 77.3 vs 68.3) between phenix.merging_statistics and XSCALE when more than one *anomalous* dataset was merged in XSCALE. phenix.merging_statistics seems to assume that equivalent (in case of anomalous, here I do not consider Friedel pairs as equivalent) reflections are sorted (equivalent reflections are not separeted) in the input file. Usually, this assumption should be correct. However, if multiple anomalous datasets were merged in XSCALE, this assumption is not correct. Because Friedel pairs are not necessarily separated. This is a real example (in P43212). These lines are extracted from .HKL file by XSCALE: > 1 -41 1 1.749E+04 1.657E+05 2052.0 2467.7 78.8 -113.23 1 > 41 -1 -1 4.569E+03 6.014E+03 454.1 1112.8 132.2 116.90 1 > 41 1 -1 8.721E+03 2.637E+03 473.0 1064.4 130.4 116.11 2 > 41 1 1 7.634E+03 2.417E+03 498.2 1010.8 135.3 115.65 2 > -1 -41 -1 4.847E+03 1.997E+03 1892.9 2543.0 85.9 -122.78 2 The last number indicates the dataset the reflection belongs to. Here, (1 -41 1), (41 -1 -1), (41, 1, 1), and (-1, -41, -1) are equivalent reflections, but (41, 1, -1) in the middle is not when anomalous data! Therefore, in phenix.merging_statistics, subsets calculation for (41,1,1) is performed twice (for first two and last two). When I changed the cctbx code to sort reflections after mapping to a.s.u., CC1/2 values by phenix.merging_statistics became closer to XSCALE. In merging_stats.__init__() in iotbx/merging_statistics.py, I inserted a line > array = array.sort("packed_indices") after > array = array.customized_copy(anomalous_flag=anomalous).map_to_asu() but I think there would be more correct way to fix this unexpected behavior. Best regards, Keitaro

2 3

correlation function for cctbx.miller.array
by Gabor Bunkoczi 24 Oct '13

24 Oct '13

Hi, I need to calculate the correlation function between two maps, both of them are stored as miller.array objects. There is a map_correlation method, which does something similar to what I want, but this calculates the total correlation coefficient, while what I need is another miller.array with amplitudes = f1*f2 and phases = p2-p1 (where f1, f2 are amplitudes and p1, p2 are phases of map1 and map2, respectively), i.e. not just the sum of these. Is this available somewhere? There is the additional complication that the hkl-list for map1 and map2 may not be fully identical, but the intersect of the two maps could be taken. Thanks, Gabor

2 5

Scipy / cctbx core dump
by Graeme.Winter＠diamond.ac.uk 15 Oct '13

15 Oct '13

Hi Folks, Just trying to use something here which depends on scipy but it seems scipy is not playing nice with flex: from scitbx.array_family import flex import scipy.linalg A tiny and inoffensive script however... cctbx.python boom.py show_stack(1): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/scipy/linalg/blas.py(113) <module> show_stack(2): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/scipy/linalg/misc.py(5) <module> show_stack(3): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/scipy/linalg/__init__.py(147) <module> show_stack(4): /scratch/gw56/2013/OCT/15/multi-lattice/boom/boom.py(2) <module> libc backtrace (2 frames, most recent call last): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/numpy/core/multiarray.so(PyArray_API+0) [0x7fb2064c9940] /lib64/libc.so.6() [0x37b3232920] Segmentation fault (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. I did a fresh nightly build install just to be sure :o( if I export those naughty env vars I get a core dump: cctbx.python boom.py Segmentation fault (core dumped) RHEL6.4 / scipy 0.12.0 and 0.12.1 any ideas? Thanks, Graeme Dr. Graeme Winter Senior Software Scientist Diamond Light Source +44 1235 778091 (work) +44 7786 662784 (work mobile) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Model based outlier calculation in mmtbx.scaling.outlier_rejection
by Keitaro Yamashita 08 Sep '13

08 Sep '13

Dear cctbx developers, I am interested in the implementation of model-based reflection outlier rejection. As I read the code mmtbx/scaling/outlier_rejection.py (lines 244-351), I noticed that maybe there was a discrepancy between what log_message explained and the actual code. The log_message in the code says: > Outliers are rejected on the basis of the assumption that a scaled > log likelihood differnce 2(log[P(Fobs)]-log[P(Fmode)])/Q\" is distributed > according to a Chi-square distribution (Q\" is equal to the second > derivative of the log likelihood function of the mode of the > distribution). > The outlier threshold of the p-value relates to the p-value of the > extreme value distribution of the chi-square distribution. while actual p_value is calculated for each hkl as p_value = 1 - erf(sqrt(LLG))**N, where LLG = log p(F=Fbar | Fmodel) - log p(F=Fobs | Fmodel), and N is the number of reflections. Here, Fbar is F which gives the maximum value of p(F | Fmodel). At least, Q (the second derivative of p(F=Fbar | Fmodel)) is not used in the actual calculation. Could someone please explain the meaning of the actual calculation? Why taking square-root and raising erf() result to the power of N? Thank you very much, Keitaro

4 4