- cctbxbb - phenix-online.org

Simple C++11 async example
by Tristan Croll 02 Oct '20

02 Oct '20

Hi all, For anyone who's interested: following on from my quick demo of the use of std::async to simplify threading of long-running tasks called from Python, I've put some example code at https://github.com/tristanic/async-hello-world/tree/master/src (implemented as a ChimeraX plugin for demo purposes, but the C++ code and bindings are entirely generic). To re-iterate what this is about, and why I think it's important: in brief, for most real-world problems Python-level threading is not particularly useful since (except in specific cases) the Global Interpreter Lock limits the overall performance to no better than - and often much worse than - single-threaded code. While multiprocessing can be an option for some tasks, it's (a) slow to get started, (b) memory-hungry due to the need for each process to have a complete copy of all the data, (c) rather difficult to organise with respect to intercommunication, and (d) not very cross-platform friendly (I'm looking at you, Windows!). The point of this example is to show that with C++11 organising the threading at the C++ level can often end up being both the simplest and fastest approach - it's only a dozen or so extra lines of fairly boilerplate code required to allow the Python programmer to choose between running the exact same function either in the main thread or asynchronously in a new one. Why I think it's important: I've said in the past, and continue to believe, that we should be aiming for (interactive) model building, validation and refinement to merge together into a single integrated process - as much as anything, ISOLDE is meant to be the proof of principle that this is a possibility. But for that to work, high-performance and cross-platform parallelism is a real necessity. The approach described here is how I'm currently getting ChimeraX to maintain 30+ fps graphics performance while simultaneously running interactive molecular dynamics and structure factor calculations. I suspect there are many functions in CCTBX that could potentially benefit from this approach even when not considering GUI applications. Best regards, Tristan

5 6

boost python conversions help needed
by Winter, Graeme (DLSLtd,RAL,LSCI) 30 Sep '20

30 Sep '20

Hi All, In the branch https://github.com/cctbx/cctbx_project/tree/fixed-width-int-types I am working to add a full range of fixed-width integer types to allow explicit reading of binary data without conversion from images (amongst other things) - mostly I’ve got something _working_ but tests fail because stdout / err contains Standard error: /Users/graeme/svn/cctbx/conda_base/python.app/Contents/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: to-Python converter for scitbx::af::shared<unsigned int> already registered; second conversion method ignored. has anyone got any hints where these may be registered? The details of the low level flex implementation seems to be minimally documented, and the error has no useful stack trace, so I am at a bit of a loss as to where the other unsigned int conversion is registered. Some pointers from someone who has delved into these in the past would be welcome - I have spent a couple days trying to find what I am doing wrong and have so far failed - hopefully someone on the list knows their way around in here Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 4

Documentation on caching mechanisms implemented in structure factor calculation
by Jan M. Simons 30 Sep '20

30 Sep '20

Hello fellow researchers and scientists, It's been a while since I actively worked with cctbx, but now I wonder if there is some in depth documentation on how the structure factor calculation in the cctbx xray module is done. I'm especially interested in caching strategies employed to speed up the calculation during refinement runs (especially in the case of small molecules using direct summation). So is there a kind of caching that keeps (partial) structure factor contributions of each individual scatterer cached so that only the partials of shifted scatterers have to be recalculated? If so is there a python interface for accessing those? Also is there an option to simulate/map the f_calc amplitudes onto a 1dimensional 2theta or d* axis to compare those to powder diffraction patterns? Thanks in advance for giving me some guidance on this (especially if there have been some recent additions to these modules). Cheers Jan

3 2

Warnings (by the hundredweight)
by Winter, Graeme (DLSLtd,RAL,LSCI) 24 Sep '20

24 Sep '20

Hi Folks Spent this afternoon looking at https://github.com/cctbx/cctbx_project/issues/532 which involves touching C++ header code deep in scitbx - which is fine - but every recompile generates hundreds of warnings from all over the place so your actual compiler errors are completely obscured Massive amounts of deprecation errors everywhere e.g. In file included from /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/asymmetric_map.cpp:5: In file included from /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/asymmetric_map.h:18: In file included from /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/sgtbx/direct_space_asu/proto/asymmetric_unit.h:6: In file included from /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/sgtbx/direct_space_asu/proto/direct_space_asu.h:9: /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/sgtbx/direct_space_asu/proto/facet_collection.h:72:34: warning: 'pointer' is deprecated [-Wdeprecated-declarations] faces_ = facet_collection::pointer(a.faces_->new_copy()); In file included from /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/sgtbx/direct_space_asu/proto/asymmetric_unit.h:6: In file included from /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/sgtbx/direct_space_asu/proto/direct_space_asu.h:9: /Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/sgtbx/direct_space_asu/proto/facet_collection.h:78:33: warning: 'pointer' is deprecated [-Wdeprecated-declarations] faces(const facet_collection::pointer &a) : faces_(a->new_copy()) {} ^ /Users/graeme/svn/cctbx/conda_base/bin/../include/c++/v1/memory:2130:28: note: 'auto_ptr<cctbx::sgtbx::asu::facet_collection>' has been explicitly marked deprecated here class _LIBCPP_TEMPLATE_VIS _LIBCPP_DEPRECATED_IN_CXX11 auto_ptr ^ /Users/graeme/svn/cctbx/conda_base/bin/../include/c++/v1/__config:972:39: note: expanded from macro '_LIBCPP_DEPRECATED_IN_CXX11' # define _LIBCPP_DEPRECATED_IN_CXX11 _LIBCPP_DEPRECATED ^ /Users/graeme/svn/cctbx/conda_base/bin/../include/c++/v1/__config:961:48: note: expanded from macro '_LIBCPP_DEPRECATED' # define _LIBCPP_DEPRECATED __attribute__ ((deprecated)) is there any appetite on the list to reduce the number of compiler warnings to make development easier? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Re: [cctbxbb] [EXTERNAL] cctbxbb Digest, Vol 138, Issue 2
by Mooers, Blaine H.M. (HSC) 23 Sep '20

23 Sep '20

Hi Billy, I installed cctbx in a new conda env called cctbx37: conda create --name cctbx37 python=3.7 conda activate cctbx37 conda install -c conda-forge cctbx jupyterlab I cannot find libtbx.run_tests_module. I can only find libtbx.run_tests_parallel. Best regards, Blaine Blaine Mooers, Ph.D. Associate Professor Department of Biochemistry and Molecular Biology College of Medicine University of Oklahoma Health Sciences Center S.L. Young Biomedical Research Center (BRC) Rm. 466 975 NE 10th Street, BRC 466 Oklahoma City, OK 73104-5419 ________________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of cctbxbb-request(a)phenix-online.org [cctbxbb-request(a)phenix-online.org] Sent: Wednesday, September 23, 2020 2:00 PM To: cctbxbb(a)phenix-online.org Subject: [EXTERNAL] cctbxbb Digest, Vol 138, Issue 2 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. cctbx now available on conda-forge (Billy Poon) ---------------------------------------------------------------------- Message: 1 Date: Wed, 23 Sep 2020 11:22:03 -0700 From: Billy Poon <BKPoon(a)lbl.gov> To: cctbx mailing list <cctbxbb(a)phenix-online.org>, PHENIX user mailing list <phenixbb(a)phenix-online.org> Subject: [cctbxbb] cctbx now available on conda-forge Message-ID: <CAHmYUcZwVDKZ6v1iuA-nP8AJn1cfp3sNWtiqeo=fcWPq9VbpqQ(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi everyone, The conda-forge package for cctbx is now available. There are 2 packages, "cctbx-base" includes the base code and "cctbx" which includes "cctbx-base" and adds "wxPython", "PySide2", and "ipython". The commands for installing cctbx into your conda environment are conda install -c conda-forge cctbx or conda install -c conda-forge cctbx-base If you want to use Jupyter Lab, you can now easily do this with conda install -c conda-forge cctbx-base jupyterlab jupyter lab The existing documentation will be updated to reflect the easier installation procedure with conda. After installing the conda package, you can also run some of the tests (in an empty directory) with libtbx.run_tests_module module=libtbx module=cctbx nproc=<nproc> I do know of a few issues that I want to fix in upcoming releases, but please submit issues to https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx… with steps to reproduce the error. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…

2 1

cctbx now available on conda-forge
by Billy Poon 23 Sep '20

23 Sep '20

Hi everyone, The conda-forge package for cctbx is now available. There are 2 packages, "cctbx-base" includes the base code and "cctbx" which includes "cctbx-base" and adds "wxPython", "PySide2", and "ipython". The commands for installing cctbx into your conda environment are conda install -c conda-forge cctbx or conda install -c conda-forge cctbx-base If you want to use Jupyter Lab, you can now easily do this with conda install -c conda-forge cctbx-base jupyterlab jupyter lab The existing documentation will be updated to reflect the easier installation procedure with conda. After installing the conda package, you can also run some of the tests (in an empty directory) with libtbx.run_tests_module module=libtbx module=cctbx nproc=<nproc> I do know of a few issues that I want to fix in upcoming releases, but please submit issues to https://github.com/cctbx/cctbx_project with steps to reproduce the error. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

E value calculation
by Academic Research 03 Sep '20

03 Sep '20

Hello Everyone, I am trying to find how to perform the E value calculation using CCTBX. I am unable to find the method in the documentation. Can someone assist me, please? Regards, AC Research

3 2

Changes to boost_adaptbx and
by Billy Poon 19 Aug '20

19 Aug '20

Hi everyone, Two pull requests were merged today that make some important changes. 1) The boost imports were moved to boost_adaptbx.boost. This avoids claiming the main "boost" namespace. A common piece of code that will need to be changed is import boost.python ext = boost.python.import_ext("your_boost_python_ext") That can be changed to import boost_adaptbx.boost.python as bp ext = bp.import_ext("your_boost_python_ext") Otherwise, a FutureWarning will be raised. More information can be found at https://github.com/cctbx/cctbx_project/issues/458 https://github.com/cctbx/cctbx_project/pull/470 https://github.com/cctbx/cctbx_project/pull/516 2) More dependency repositories were moved from cci.lbl.gov to GitHub. If you use bootstrap.py to build cctbx, you will want to delete annlib, ccp4io_adaptbx, clipper, eigen, gui_resources, tntbx. Using a newly updated bootstrap.py, the copies from GitHub will be used instead. More information can be found at https://github.com/cctbx/cctbx_project/pull/408 Let us know if you have any questions. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

unmerged data as mtz object
by Robert Oeffner 04 Aug '20

04 Aug '20

Hi, Does anybody know how to convert a miller array of unmerged data to an mtz object? For merged data I do something like this: mtzobj = millarr.as_mtz_dataset(column_root_label="I") But if millarr contains multiple observations with the same hkl index. Then the above code fails with the exception cctbx Error: Duplicate entries in miller_indices array. Any suggestions? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -- This email has been checked for viruses by AVG. https://www.avg.com

4 5

flex.histogram giving ArgumentError when arguments correct
by Winter, Graeme (DLSLtd,RAL,LSCI) 04 Aug '20

04 Aug '20

H = flex.histogram(coords, data_min=0.0, data_max=dim, n_slots=dim) Gives Boost.Python.ArgumentError: Please report this error to dials-support(a)lists.sourceforge.net: Python argument types in histogram.__init__(histogram) did not match C++ signature: __init__(_object*, double, double, double, scitbx::af::shared<long>, unsigned long) __init__(_object*, scitbx::histogram<double, long> other, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> data, double relative_tolerance=0.0001) __init__(_object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> data, double data_min, double data_max, unsigned long n_slots=1000, double relative_tolerance=0.0001) __init__(_object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> data, unsigned long n_slots=1000) Despite coords being <scitbx_array_family_flex_ext.double object at 0x7fa8d671b5b0> with shape (31294944,) and dim being 18093576 (i.e. integer) and h = flex.histogram(a, data_min=0, data_max=100, n_slots=100) working fine in another context i.e. the calling convention is correct This has so far cost a few hours, so what gives? How can this be holding it wrong? C++ code suggests that the bin types are long so this should be fine Any hints? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 5