February 2018 - cctbxbb - phenix-online.org

Revisiting cctbx/iotbx/dxtbx dependencies
by Aaron Brewster 20 Feb '18

20 Feb '18

Hey folks, especially Rob and Luc. So after a quick change to iotbx/libtbx_config, it turns out there's a way to get a super minimum version of cctbx, without configuring dxtbx, and it takes advantage of existing mechanisms: - python bootstrap.py --builder=cctbx hot update - python modules/cctbx_project/libtbx/auto_build/install_base_packages.py openssl python python_compatibility certifi numpy cython png libsvm pytest pythonextra jinja2 junitxml biopython docutils sphinx pyopengl misc freetype matplotlib pillow gettext glib expat fontconfig render pixman tiff cairo gtk fonts wxpython - mkdir build; cd build - ../base/bin/python ../modules/cctbx_project/libtbx/configure.py cctbx iotbx - make Now, some details as to why this works. First, the cctbx builder in boostrap.py doesn't download cbflib so cbflib is never configured. Next, the base step of bootstrap.py is simply a call to install_base_packages.py with a list of required packages. You can just run it standalone and get the packages you need. Here, I ran "python bootstrap.py --builder=cctbx base", killed it after I saw the list of packages it was going to install, then ran install_base_packages manually after removing hdf5 and lz4 (which depends on hdf5) from the list. One could imagine removing other packages depending on your use case. With that, I could run configure.py directly. I can't use cctbx by itself as the cctbx/maptbx/SConscript file has a dependency on iotbx, but configuring cctbx and iotbx together seems to work. The key trick was to make iotbx only depend on dxtbx if dxtbx is explicitly configured. I used a keyword in libtbx_configure added by Luc a bit ago: optional_modules_only_if_explicit_request. I committed that change just now. Outstanding questions: 1. Is there interest in making this a builder? It would be straightforward to add a builder named cctbx_minimal which would just do the above steps 2. Is there further need for making cbflib and hdf5 optional in dxtbx? I purport there is not as the above procedure seems to satisfy the use case, but I can be persuaded otherwise. Thanks, -Aaron

4 5

Re: [cctbxbb] [git/cctbx] master: use larger tolerance (0b81ca799)
by Graeme.Winter＠diamond.ac.uk 12 Feb '18

12 Feb '18

Pavel, This commit did not fix test on OS X High Sierra libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py" [FAIL] 1.4s Time: 1.39 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 128, in <module> exercise(f = f, gx = gx, gy = gy, gz = gz, function_type=function_type) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 99, in exercise assert approx_equal(it, 0.0, 1.e-4) AssertionError cheers Graeme On 11 Feb 2018, at 00:14, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> wrote: Repository : ssh://g18-sc-serv-03.diamond.ac.uk/cctbx On branch : master ________________________________ commit 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b Author: Pavel <pafonine(a)lbl.gov<mailto:[email protected]>> Date: Sat Feb 10 16:14:51 2018 -0800 use larger tolerance ________________________________ 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b cctbx/maptbx/tst_interpolation.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cctbx/maptbx/tst_interpolation.py b/cctbx/maptbx/tst_interpolation.py index 918ba68fd..90dec8a46 100644 --- a/cctbx/maptbx/tst_interpolation.py +++ b/cctbx/maptbx/tst_interpolation.py @@ -96,15 +96,15 @@ def exercise(f,gx,gy,gz,function_type, eps = 1.e-6): if(function_type=="L"): cntr+=1 for it in [tl,gxl,gyl,gzl, tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="Q"): cntr+=1 for it in [tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="C"): cntr+=1 for it in [tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(abs(tl)>eps and abs(tq)>eps): assert tl >=tq if(function_type=="F"): cntr+=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Re: [cctbxbb] [Dials-support] indexing success rates
by Aaron Brewster 05 Feb '18

05 Feb '18

Right, typically if I don't know the cell, I'll index a bunch of images with no target, then use cluster.unit_cell on the results to get a consensus cell, then re-index with that cell. For dials.stills_process, the default of fft1d works pretty well generally, but to get the best results, indexing.method_list=fft1d,fft3d might be better. There, if the first method fails the second is tried. real_space_grid_search can also be used. Getting a specific image in a composite file (such as hdf5) isn't a use case I've hit often, so I don't have an easy solution there. It's possible an image range parameter to dials.import would be useful? -Aaron On Sat, Feb 3, 2018 at 1:02 AM, Graeme Winter <graeme.winter(a)gmail.com> wrote: > Hi Aaron > > (and DIALS) > > For stills / single images - should dials.index switch by magic to default > fft1d? > > Of is that the responsibility of e.g. xia2 / cctbx.xfel to worry about? > > It has to be said, more often than not, particularly for screening, we do > not know cell / symmetry a priori (at diamond anyways) > > A related question > > If I have a data block / strong pickle from 1000 stills, how can I index > e.g. image 672? Do I have to split this into 1000 separate things? > > @JH - thanks for this - I think some rigorous testing will be useful here. > > Cheers Graeme > > On 3 Feb 2018, at 08:37, Aaron Brewster <asbrewster(a)lbl.gov> wrote: > > Hi James, thanks. I looked at random image 15 and after trying a bunch of > stuff it indexes fine if I use indexing.method=fft3d. The default for > stills processing is fft1d, so I find this super interesting. > > For what it's worth, I pretty much never process stills without a target > cell. I generated ~30 random images using your script and indexed them, > then I got the unit cells with "dials.show *refined*.json | grep cell". > They were all about the same. So I re-indexed random image 15 using fft1d > but I also specified known_symmetry.cell=50,60,70,90,90,90 and it indexed > fine. > > So, either indexing.method=fft3d or known_symmetry.cell=50,60,70,90,90,90 > is sufficient to index random image 15. Certainly for experimental data I > would recommend using known_symmetry as the results are always better. > However, I'd like to look deeper at fft1d vs fft3d at some point. > > Cc'ing dials-support. > > Thanks! > -Aaron > > > > On Fri, Feb 2, 2018 at 1:35 PM, James Holton <jmholton(a)lbl.gov> wrote: > >> While testing the new simTBX diffraction image simulator Aaron and I >> found that some rather beautiful-looking images still won't index. >> Subsequently, I followed up with mosflm, which does succeed in these cases, >> even without prior knowledge of the cell or space group. >> >> I have now run this 1000 times, varying only the crystal orientation, and >> found 171 cases of dials.stills_process failing to index the simulated >> image whereas mosflm had no trouble, and 5 cases where mosflm failed and >> dials.stills_process succeeded. There were no overlapping cases where both >> programs failed. >> >> I have tarballed up the relevant files here: >> http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz >> >> The tarball contains a "runme.com" shell script for reproducing these >> results, and also the tst_nanoBragg_forindex.py jiffy for making a test >> image given a provided random-number seed. For example: >> >> libtbx.python ./tst_nanoBragg_forindex.py random 15 >> >> will create a file called noiseimage_001.cbf that dials.stills_process >> cannot index (at least in my hands). For those of you who don't have >> mosflm installed, the tarball incluses a copy, and the "autoindex.com" >> script should run on any linux system. >> >> Any ideas as to why there is this discrepancy? Feels like an opportunity >> for a non-trivial improvement. >> >> Something to ponder over the weekend, I suppose. Note that I recently >> checked in a few bug fixes to simTBX, but this analysis is unaffected by >> them, so you shouldn't have to update your build to reproduce this. >> >> Cheers, >> >> -James Holton >> MAD Scientist >> >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot______ > _________________________________________ > DIALS-support mailing list > DIALS-support(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dials-support > > >

2 1

indexing success rates
by James Holton 03 Feb '18

03 Feb '18

While testing the new simTBX diffraction image simulator Aaron and I found that some rather beautiful-looking images still won't index. Subsequently, I followed up with mosflm, which does succeed in these cases, even without prior knowledge of the cell or space group. I have now run this 1000 times, varying only the crystal orientation, and found 171 cases of dials.stills_process failing to index the simulated image whereas mosflm had no trouble, and 5 cases where mosflm failed and dials.stills_process succeeded. There were no overlapping cases where both programs failed. I have tarballed up the relevant files here: http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz The tarball contains a "runme.com" shell script for reproducing these results, and also the tst_nanoBragg_forindex.py jiffy for making a test image given a provided random-number seed. For example: libtbx.python ./tst_nanoBragg_forindex.py random 15 will create a file called noiseimage_001.cbf that dials.stills_process cannot index (at least in my hands). For those of you who don't have mosflm installed, the tarball incluses a copy, and the "autoindex.com" script should run on any linux system. Any ideas as to why there is this discrepancy? Feels like an opportunity for a non-trivial improvement. Something to ponder over the weekend, I suppose. Note that I recently checked in a few bug fixes to simTBX, but this analysis is unaffected by them, so you shouldn't have to update your build to reproduce this. Cheers, -James Holton MAD Scientist

2 1

Fwd: [git/cctbx] master: Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified ...
by Graeme.Winter＠diamond.ac.uk 02 Feb '18

02 Feb '18

Tried sending before got rejected ... Begin forwarded message: From: Graeme Winter <graeme.winter(a)gmail.com<mailto:[email protected]>> Date: 1 February 2018 at 18:51:58 GMT To: cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]> Cc: CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]> Subject: Re: [git/cctbx] master: Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified for rstbx build (29fc06f71) Just after this commit all our builds died at diamond Is build bot happy? Curious if local issue Cheers Graeme On 1 Feb 2018, at 18:07, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> wrote: Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx On branch : master ________________________________ commit 29fc06f714e1059cb9f4eaac35cc2024178237dc Author: Robert Oeffner <rdo20(a)cam.ac.uk<mailto:[email protected]>> Date: Thu Feb 1 18:07:49 2018 +0000 Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified for rstbx build ________________________________ 29fc06f714e1059cb9f4eaac35cc2024178237dc boost_adaptbx/SConscript | 13 +++++++------ libtbx/SConscript | 4 +++- rstbx/SConscript | 2 +- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/boost_adaptbx/SConscript b/boost_adaptbx/SConscript index 8696c1510..b8b58ce2d 100644 --- a/boost_adaptbx/SConscript +++ b/boost_adaptbx/SConscript @@ -32,8 +32,9 @@ print("BOOST_VERSION: %i" %env_etc.boost_version) # any issue. if (libtbx.env.build_options.enable_boost_threads and env_etc.boost_thread_support): - boost_thread_uses_winthreads = (sys.platform == 'win32' and - not env_etc.compiler.endswith('mingw')) + boost_thread_uses_winthreads = (sys.platform == 'win32') + # MinGW compiler 5.3.0 appears to compile boost_threads + # with the same condition as win32. # This environment is to be used latter to build Boost libraries # So we need a clean one without extra library linking. env_boost_shlibs_base = env_base.Copy() @@ -264,7 +265,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". libs.append('rt') if boost_thread_uses_winthreads: libs.append('boost_chrono') - env_boost_threads.SharedLibrary( + env_boost_threads.StaticLibrary( target='#lib/boost_thread', source=boost_thread_src, LIBS=libs) @@ -280,7 +281,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". # For Windows (c.f. point 2 above) env_boost_system.Append( CPPDEFINES={'BOOST_SYSTEM_DYN_LINK':1}) - env_boost_system.SharedLibrary( + env_boost_system.StaticLibrary( target='#lib/boost_system', source=boost_system_src) @@ -294,7 +295,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". os.path.join(env_etc.boost_dist, 'libs', 'chrono', 'src', '*.cpp'))] # For Windows (c.f. point 2 above) env_boost_chrono.Append(CPPDEFINES={'BOOST_CHRONO_DYN_LINK':1}) - env_boost_chrono.SharedLibrary( + env_boost_chrono.StaticLibrary( target='#lib/boost_chrono', source=boost_chrono_src, LIBS=['boost_system']) @@ -308,7 +309,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". env_etc.include_registry.append( env=env_boost_threads_test, paths=[env_etc.boost_adaptbx_include]) - env_boost_threads_test.SharedLibrary( + env_boost_threads_test.StaticLibrary( target="#lib/boost_adaptbx_boost_thread_test_ext", source="tests/boost_thread_test.cpp") diff --git a/libtbx/SConscript b/libtbx/SConscript index b4fcc5df9..b956956f2 100644 --- a/libtbx/SConscript +++ b/libtbx/SConscript @@ -364,7 +364,9 @@ For newer versions, please adjust this SConscript. "/D_SECURE_SCL=0", "/D_CRT_SECURE_NO_DEPRECATE", "/DNOMINMAX", - #"/DWIN32", + "/DWIN32", + "/DDLL_EXPORTS", + "/D_WIN32_WINNT=_WIN32_WINNT_WIN7", "/bigobj", # for heavy use of template libraries "/wd4996", # warning C4996: ... was declared deprecated "/wd4068", # warning C4068: unknown pragma diff --git a/rstbx/SConscript b/rstbx/SConscript index 422a9864c..76ba4d1c5 100644 --- a/rstbx/SConscript +++ b/rstbx/SConscript @@ -40,7 +40,7 @@ if (not env_etc.no_boost_python): Import("env_boost_python_ext") env_rstbx_boost_python_ext = env_boost_python_ext.Clone() env_rstbx_boost_python_ext.Prepend( - LIBS=["rstbx", "cctbx", "scitbx_boost_python"]) + LIBS=["rstbx", "ann", "cctbx", "scitbx_boost_python"]) env_rstbx_boost_python_ext.SharedLibrary( target="#lib/rstbx_ext", source="ext.cpp") env_rstbx_boost_python_ext.SharedLibrary( -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2