- cctbxbb - phenix-online.org

Left to dry by iotbx_pdb_ext.input.construct_hierarchy()
by James Stroud 25 Mar '13

25 Mar '13

Hello all, This seems like it should qualify as a bug. If I have an instance of iotbx_pdb_ext.input and I create a hierarchy with construct_hierarchy(), then I must retain the reference to that hierarchy or apparently I can never get it back: >>> hierarchy = my_pdb.construct_hierarchy() >>> def some_local_scope_function(a_pdb): local_hierarchy = a_pdb.construct_hierarchy() >>> some_local_scope_function(my_pdb) […] RuntimeError: atom has another parent atom_group already. Wouldn't better behavior just return the previously constructed hierarchy given that multiple hierarchies can't be made? If each atom apparently remembers to which atom_group it belongs (and presumably on up), then shouldn't it make sense that an iotbx_pdb_ext.input be able to remember it's own hierarchy and produce it upon demand? Looking at a iotbx_pdb_hierarchy_ext.root, I don't find a reference to the iotbx_pdb_ext.input from which it was created. So, although passing around a hierarchy might be the obvious work-around, the end-user still needs to do some accounting if said user wants to track which roots belong to which inputs, and vice versa. James

2 1

Build Notes for cctbx on OS X 10.7.5
by James Stroud 22 Mar '13

22 Mar '13

Hello All, I spent some time to build cctbx on an OS X 10.7 box for a non-system interpreter. I ran into some kinks along the way, so I thought I'd share my notes in case anyone wants to do the same. These notes are in reStructuredText, so it was no extra work to turn them into the attached HTML file, which might make reading a little easier. James ==================================== Build Notes for cctbx on OS X 10.7.5 ==================================== These notes describe a successful build of cctbx against a specific python interpreter, namely `Enthought python`_ 7.3-2 for x86_64. Although working, versions of cctbx are available for both the system python or with a dedicated python, these canned solutions are not optimal. First, it's not advisable to modify system utilities. Although cctbx seems well-behaved in this regard, other packages may not be so well behaved. Second, the canned approach requires building many of the numerous packages that come with bundeld python distributions like the Enthought distribution (some such packages exceed even cctbx in their difficulty to build). Third, if every package required its own python, every script would need to fire-up dozens of separate interpreters just to do its job, resulting in significant computational overhead with consummately slower execution times. .. _`Enthought python`: http://www.enthought.com/products/epd.php 1. The default c++ compiler for OS X 10.7 is actually clang (``/usr/bin/c++``). Using clang exposes invalid template code. The first of many errors is:: In file included from /opt/cctbx/cctbx_sources/mmtbx/bulk_solvent/bulk_solvent_ext.cpp:7: /opt/cctbx/cctbx_sources/mmtbx/bulk_solvent/bulk_solvent.h:496:21: error: call to function 'r_factor' that is neither visible in the template definition nor found by argument-dependent lookup r.push_back(r_factor(f_obs, f_model.const_ref())); These types of errors are documented at the page http://clang.llvm.org/compatibility.html#dep_lookup .. note:: The installation directory is ``/opt/cctbx``. 2. I tried to use the HPC compilers at http://hpc.sourceforge.net/, but they do not recognize the ``-no-cpp-precomp`` flag. The web page https://svn.boost.org/trac/boost/ticket/6697 suggests the problem is with boost. However, according to the comments therein, this should have been fixed in boost 1.50. I am building against boost 1.52. Perplexed about the source of this compiler flag, I tracked it down to the file ``cctbx_sources/libtbx/SConscript``, after spending much of a day learning how scons goes about its business. There are two possible fixes: - Comment the following line in ``cctbx_sources/libtbx/SConscript``:: # env_etc.ccflags_base.extend(["-no-cpp-precomp"]) - Link ``/usr/bin/g++`` to ``c++`` in a way that it will be found first in the path. For example:: % ln -s /usr/bin/g++ ${HOME}/bin/c++ 3. I first fixed the ``-no-cpp-precomp`` error by the former, commenting the offending line in ``cctbx_sources/libtbx/SConscript`` but this failed with the error:: /opt/cctbx/cctbx_sources/scitbx/array_family/memory.h:35:47: error: operator '&&' has no right operand #elif defined(_XOPEN_SOURCE) && _XOPEN_SOURCE >= 600 && defined(_POSIX_ADVISORY_INFO) ^ The page http://forums.fedoraforum.org/archive/index.php/t-52929.html suggests that the naked ``_XOPEN_SOURCE`` wrecks the conditional when ``_XOPEN_SOURCE`` is not defined. One possible way to overcome this problem is to change the offending line into a nested if/then, hoping that the comparison never gets evaluated under circumstances when ``_XOPEN_SOURCE`` is not defined:: #elif defined(_XOPEN_SOURCE) #if _XOPEN_SOURCE >= 600 && defined(_POSIX_ADVISORY_INFO) #define SCITBX_AF_HAS_ALIGNED_MALLOC 1 #define SCITBX_AF_USE_POSIX_FOR_ALIGNED_MALLOC 1 #endif #endif However, this was unnecessary because the second fix to the ``-no-cpp-precomp`` error (linking ``/usr/bin/g++`` to ``c++``) worked and avoided changing the preprocessor source. 4. Left to its own devices, scons could not find my ``fftw3.h``, which I have installed (via fink) to ``/sw/include/fftw3.h``. To fix this problem, I inserted the second of the following two lines into the file ``cctbx_sources/libtbx/SConscript``:: env_etc.ccflags_base.extend(["-no-cpp-precomp"]) env_etc.ccflags_base.extend(["-I/sw/include"]) Although this allowed scons to find the header, it could not find the actual library, requiring the insertion of the second of the following lines into the file ``cctbx_sources/libtbx/SConscript``:: env_etc.shlinkflags_bpl = [ "-L/sw/lib", "-w", # suppress "source/lib does not exist" warning "-bundle", "-undefined", "dynamic_lookup"]

4 5

structure_factors() "d_max" equivalent
by James Stroud 21 Mar '13

21 Mar '13

Hello All, I'm new to cctbx and trying to get the most rudimentary understanding. I would like to select all of the reflections within a particular resolution shell. This line from the tutorial: f_calc = quartz_structure.structure_factors(d_min=2).f_calc_array() gets all structure factors through the highest resolution of d_min=2 Å (excluding 0 0 0). My question: is there an equivalent to a d_max argument for this function (i.e. a low resolution limit)? James

4 3

cctbx and symmetry: source code
by prasun kumar 21 Mar '13

21 Mar '13

Dear all I am developing a software for which I needed to use the unit cell generation module of the cctbx. The software will be working for small molecule as well as macromolecule. It will be a great help, if some body can suggest me the part of the program to be used as I am getting confused while selecting the .cpp and .h files. Thanx alot in advance Prasun Kumar PRASUN (ASHOKA) Desire + stability = Resolution Resolution + Hard work = Success

4 10

changes to SourceForge project
by Nathaniel Echols 19 Mar '13

19 Mar '13

As promised, the CCTBX SourceForge site has been migrated to their new backend. I am still not sure exactly what this means, but there are two practical changes: 1) The SVN URL has changed - it used to be http://cctbx.svn.sourceforge.net/svnroot/cctbx. Now we have several options. svn checkout --username=natechols svn+ssh://[email protected]/p/cctbx/code/trunk cctbx_project svn checkout --username=natechols https://svn.code.sf.net/p/cctbx/code/trunk cctbx_project svn checkout svn://svn.code.sf.net/p/cctbx/code/trunk cctbx_project The last one is read-only; for the others, replace "natechols" with your SourceForge ID. The old repository is now read-only, so you will need to do a completely new checkout before you can commit any changes you make. Hopefully this will cause minimal disruption; I will change the nightly builds ASAP. 2) As requested, we now have a wiki (completely empty right now): https://sourceforge.net/p/cctbx/wiki/Home/ Feel free to start populating this - if you would like to contribute and don't have access, sign up for a SourceForge account and email me your user ID. -Nat

2 1

Best way to compare two structures using cctbx?
by Jan Marten Simons 12 Mar '13

12 Mar '13

Hi, I think I've seen some code in cctbx which tries to put the similarity of two crystal structures into a number. Am I right on this? I want to have something similar: I'm interested in comparing the direct space atomic distances of two structures sharing the same unit cell parameters and space group and scatterers but different atomic coordinates for those. Now I'd like to have a mapping where I could compare the bond lengths between all scatterers which should eliminate any movements which might also be due to a shift in the origin of a P 1 cell. Also I'd like to have a value for mean, minimal and maximal displacements. The purpose of this is to check how good a structure approximation matches a reference structure. How would you suggest to take on this task? Or is there even some code related to this? Thanks in advance, Jan

5 8

upcoming SourceForge changes
by Nathaniel Echols 27 Feb '13

27 Feb '13

FYI for anyone using CCTBX SVN checkouts: on or about March 9th or 10th, I plan to upgrade our SourceForge project to their new platform, which they claim to want everyone to do in the first quarter of 2013. (I had hoped to do this a month ago but it would have created problems for other projects.) This should give us access to new features including a project wiki, requested by several of you. I am hoping any disruption to SVN access will be minimal and/or quickly resolved (the timing coincides with our Phenix developer workshop so those of us who are attending will be able to fix problems interactively). I'll send out another email after the switch summarizing any changes. -Nat

1 0

pdb shake - dihedral angles
by Dmytro Guzenko 20 Feb '13

20 Feb '13

Hi everyone, I wonder what would it take to write an equivalent to pdb_shake script from iotbx/examples with randomization of the backbone dihedral angles phi and psi, instead of all coordinates perturbation? I want to be able to randomize models in a more "realistic" way. Are there data structures/procedures that can be useful for this in cctbx, or should I extract the coordinates and take it from there? Thanks! Kind regards, Dmytro.

3 2

Re: [cctbxbb] unexpected behaviour of unit_cell.fractionalize
by Jan Marten Simons 14 Feb '13

14 Feb '13

Am Mittwoch 13 Februar 2013 17:01:37 you wrote: > Hi Jan, > > can this method work for you: sites_mod_positive ? I don't remember what > exactly it does but I think it either what you need or at least very close. Hi Pavel, hm, it goes in the right direction, but is not exactly what I'm looking for, as it is bound to scatterers and not to sites in general. And adding a scatterer for each site I want to convert to the structure first, then converting, and than extracting the converted sites from the scatterers afterwards seems quite inefficient to me. Also I think it would be clearer to add the functionality for this to the "fractionalize" function as people trying to convert from direct space coordinates to fractional coordinates will most likely find this functionality first. Cheers, Jan > On 2/13/13 7:17 AM, Jan Marten Simons wrote: > > Hi, > > > > When I try to convert a list of direct space coordinates into fractional > > coordinates like this: > > > > from cctbx import xray > > from cctbx import crystal > > from cctbx.array_family import flex > > > > structure = xray.structure( > > > > special_position_settings=crystal.special_position_settings( > > > > crystal_symmetry=crystal.symmetry( > > > > unit_cell=(3.14098, 4.08327, 5.33966, 83, 109, 129), > > space_group_symbol="P1"))) > > > > sites = flex.vec3_double( ((0.0, 0.1, 0.2), (1.3, 3.0, 2.3), (5.3, 0.1, > > 7.3)) ) sites_frac = structure.unit_cell().fractionalize(sites) > > print(list(sites_frac)) > > > > Now this returns a list of coordinates where x,y,z may be outside ot the > > interval [0,1[. ... quite unexpected. > > > > It would be very nice to have at least an option like "restrict_to_asu" > > in unit_cell.fractionalize which would enforce (x,y,z) to be inside the > > asymmetric unit and/or "restrict_to_cell" which would enforce (x,y,z) to > > be inside the unit cell. > > > > Could any of you more into the inner workings of the cctbx add this > > functionality, please? > > > > Thanks in advance, > > > > Jan

3 4

unexpected behaviour of unit_cell.fractionalize
by Jan Marten Simons 13 Feb '13

13 Feb '13

Hi, When I try to convert a list of direct space coordinates into fractional coordinates like this: from cctbx import xray from cctbx import crystal from cctbx.array_family import flex structure = xray.structure( special_position_settings=crystal.special_position_settings( crystal_symmetry=crystal.symmetry( unit_cell=(3.14098, 4.08327, 5.33966, 83, 109, 129), space_group_symbol="P1"))) sites = flex.vec3_double( ((0.0, 0.1, 0.2), (1.3, 3.0, 2.3), (5.3, 0.1, 7.3)) ) sites_frac = structure.unit_cell().fractionalize(sites) print(list(sites_frac)) Now this returns a list of coordinates where x,y,z may be outside ot the interval [0,1[. ... quite unexpected. It would be very nice to have at least an option like "restrict_to_asu" in unit_cell.fractionalize which would enforce (x,y,z) to be inside the asymmetric unit and/or "restrict_to_cell" which would enforce (x,y,z) to be inside the unit cell. Could any of you more into the inner workings of the cctbx add this functionality, please? Thanks in advance, Jan

2 1

Strange behaviour of resolution filter
by Jan Marten Simons 08 Feb '13

08 Feb '13

Hi, I discovered a rather strange behaviour of the miller.index_generator. Sample code showing the weirdness: -----8<--------------------------- from __future__ import division from cctbx import xray from cctbx import crystal from cctbx.array_family import flex structure = xray.structure( special_position_settings=crystal.special_position_settings( crystal_symmetry=crystal.symmetry( unit_cell=(3.14098, 4.08327, 5.33966, 83, 109, 129), space_group_symbol="P 1")), scatterers=flex.xray_scatterer([ xray.scatterer( label="Si", site=(0.,0.,0.), u=0.0)])) fc1 = structure.structure_factors(d_min=1.0).f_calc().sort() d_max, d_min = fc1.resolution_range() fc2 = structure.structure_factors(d_min=d_min).f_calc().sort() print(fc1.size(), fc2.size()) #fc1.show_array() fc1.show_comprehensive_summary() print("\nd_min: {}\n".format(d_min)) fc2.show_comprehensive_summary() #fc2.show_array() -----8<--------------------------- fc1 has 105 indices, while fc2 only has 104. Also fc2 has a higher d_min than fc1 although I'd expect both to have the same. To me it seems like the index with d==d_min is not included and only indices with d>d_min are calculated. I wonder if this behaviour is the desired one or rather a bug, as I was quite surprised when I discovered this. With regards, Dipl. Phys. Jan M. Simons Institute of Crystallography RWTH Aachen University

2 1

Using cctbx::uctbx::unit_cell in c++ code / boosted to Python
by Graeme.Winter＠diamond.ac.uk 25 Jan '13

25 Jan '13

Hi Folks, Been working on some c++ coding using boost etc (which I have to say is great) however I found one bit of weirdness. If I want to generate a cctbx::uctbx::unit_cell in c++ code then access this from Python I get Traceback (most recent call last): File "tst_x2tbx.py", line 67, in <module> tst_x2tbx(sys.argv[1]) File "tst_x2tbx.py", line 6, in tst_x2tbx import x2tbx File "/Users/graeme/svn/cctbx/sources/x2tbx/__init__.py", line 7, in <module> ext = boost.python.import_ext("x2tbx_ext", optional = False) File "/Users/graeme/svn/cctbx/sources/cctbx_project/boost_adaptbx/boost/python.py", line 36, in import_ext + [" "+p for p in sys.path])) ImportError: __import__("x2tbx_ext"): dlopen(/Users/graeme/svn/cctbx/prebuild/lib/x2tbx_ext.so, 2): Symbol not found: cctbx::uctbx::unit_cell::unit_cell(scitbx::af::tiny<double, 6ul> const&) Referenced from: /Users/graeme/svn/cctbx/prebuild/lib/x2tbx_ext.so Expected in: flat namespace in /Users/graeme/svn/cctbx/prebuild/lib/x2tbx_ext.so sys.path: …. unless I have import cctbx.uctbx # explicit import at the top of the Python code - have I done something wrong with the SConscript? I found something similar with import scitbx.array_family.flex # explicit import which libtbx.clean_clutter complains about as I don't reference it (hence the additional comment) Equally I could just leave the additional clutter there but it feels like I should not need it... Thanks, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 8

Sphinx documentation
by Graeme.Winter＠diamond.ac.uk 21 Jan '13

21 Jan '13

Hi Folks, I installed sphinx and tried building the documentation but I got an error: File ./xfel/xfel.cxi.gfx.rst already exists, skipping. File ./xfel/xfel.cxi.merging.rst already exists, skipping. File ./xfel/xfel.cxi.rst already exists, skipping. File ./xfel/xfel.metrology.rst already exists, skipping. File ./xfel/xfel.rst already exists, skipping. File ./xfel/modules.rst already exists, skipping. sphinx-build -b coverage -d _build/doctrees -D latex_paper_size=a4 . _build/coverage Making output directory... Running Sphinx v1.2pre loading pickled environment... not yet created building [coverage]: coverage overview updating environment: 206 added, 0 changed, 0 removed reading sources... [ 0%] cctbx/cctbx Exception occurred: File "/Users/graeme/svn/cctbx/phenix/phenix-dev-1241/build/mac-intel-osx-x86_64/base/Python.framework/Versions/Current/lib/python2.7/site-packages/Sphinx-1.2predev_20130121-py2.7.egg/sphinx/ext/autodoc.py", line 595, in filter_members membername in self.options.special_members: TypeError: argument of type 'object' is not iterable The full traceback has been saved in /var/folders/j1/cgsqy8fs1lxdyjggcsb3twsw0000gn/T/sphinx-err-jjM5C8.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-users/>, or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks! make: *** [coverage] Error 1 cp: ./_build/coverage/python.txt: No such file or directory cp: ./_build/coverage/c.txt: No such file or directory sphinx-build -b html -d _build/doctrees -D latex_paper_size=a4 . _build/html Making output directory... Running Sphinx v1.2pre loading pickled environment... not yet created building [html]: targets for 206 source files that are out of date updating environment: 206 added, 0 changed, 0 removed reading sources... [ 0%] cctbx/cctbx Exception occurred: File "/Users/graeme/svn/cctbx/phenix/phenix-dev-1241/build/mac-intel-osx-x86_64/base/Python.framework/Versions/Current/lib/python2.7/site-packages/Sphinx-1.2predev_20130121-py2.7.egg/sphinx/ext/autodoc.py", line 595, in filter_members membername in self.options.special_members: TypeError: argument of type 'object' is not iterable The full traceback has been saved in /var/folders/j1/cgsqy8fs1lxdyjggcsb3twsw0000gn/T/sphinx-err-lONSky.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-users/>, or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks! make: *** [html] Error 1 Did I do something wrong? I just grabbed sphinx from the internet. Should I have got it from someplace else? OS X Lion / svn cctbx. Thanks, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Phil (dumb) question
by Graeme.Winter＠diamond.ac.uk 14 Jan '13

14 Jan '13

Hi Folks, I have a toy example where I should be overriding Phil defaults: phil_defaults = ''' resolutionizer { rmerge = 0.0 .type = float completeness = 0.0 .type = float isigma = 1.0 .type = float misigma = 2.0 .type = float bins = 100 .type = int } ''' more = ''' resolutionizer { isigma = 2.0 misigma = 4.0 } ''' from libtbx.phil import parse working_phil = parse(phil_defaults) working_phil.fetch(parse(more)) working_phil.show() but it seems I am not: Graemes-MacBook-Pro:scale graeme$ cctbx.python toy.py resolutionizer { rmerge = 0.0 completeness = 0.0 isigma = 1.0 misigma = 2.0 bins = 100 } Am I doing something dumb? I hope so as I really cannot find my error. Thanks, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Simple cctbx / boost c++ extension example
by Graeme.Winter＠diamond.ac.uk 11 Jan '13

11 Jan '13

Hi Folks, Just say I wanted to write a new from the ground up module for cctbx which wanted to include some C++ code, and a SConscript etc. so I can run libtbx.configure toytbx (say) And then find from toytbx import cplusplus_functions would work.... Is there an example I could follow? (I have not looked through the newsletters, I suppose I should have... but I could not seen anything obvious in the documents) Thanks, Graeme Dr. Graeme Winter Senior Software Scientist Diamond Light Source +44 1235 778091 (work) +44 7786 662784 (work mobile) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 6

Computing E^4 from an mtz file
by Graeme.Winter＠diamond.ac.uk 14 Dec '12

14 Dec '12

Hi Folks, Trying to write some code to return the average E^4 across (some) bins for an mtz file, I got to def main(): import sys from iotbx import reflection_file_reader for argv in sys.argv[1:]: reflection_file = reflection_file_reader.any_reflection_file( file_name = argv) if reflection_file.file_type() is None: continue miller_arrays = reflection_file.as_miller_arrays() for ma in miller_arrays: E4(ma) def E4(ma): f = ma.as_intensity_array() f = f.array(data=f.data()/f.epsilons().data().as_double()) f.set_observation_type_xray_intensity() f.setup_binner(auto_binning = True) sm = f.second_moment(use_binning = True) sm.show() if __name__ == '__main__': main() but I have no idea how to actually extract the information I want from sm as this only has show() which gives Graemes-MacBook-Pro:PlayPen graeme$ cctbx.python E4.py /Users/graeme/projects/sci-397/DEFAULT/scale/AUTOMATIC_DEFAULT_scaled.mtz unused: - 20.8668 [ 0/7 ] bin 1: 20.8668 - 5.8873 [221/235] 2.3863 bin 2: 5.8873 - 4.6904 [216/221] 1.9823 bin 3: 4.6904 - 4.1026 [221/222] 1.9925 <snip> bin 47: 1.6553 - 1.6435 [153/209] 2.2051 bin 48: 1.6435 - 1.6320 [143/198] 2.2021 bin 49: 1.6320 - 1.6208 [141/204] 2.7085 unused: 1.6208 - [ 0/0 ] unused: - 20.8668 [ 0/9 ] bin 1: 20.8668 - 7.1654 [201/225] 2.2405 bin 2: 7.1654 - 5.7261 [202/213] 1.9109 bin 3: 5.7261 - 5.0137 [216/223] 2.0162 bin 4: 5.0137 - 4.5605 [207/212] 2.0043 <snip> bin 87: 1.6456 - 1.6393 [126/192] 2.5521 bin 88: 1.6393 - 1.6330 [122/210] 2.0195 bin 89: 1.6330 - 1.6269 [131/224] 2.6251 bin 90: 1.6269 - 1.6208 [120/212] 3.0912 unused: 1.6208 - [ 0/0 ] Really I would like the value programatically or at least a way of getting back an array of the binned values so I can average them myself (which is obviously trivial) Any suggestions? Thanks, Graeme ps - most of this code is ripped from iotbx.reflection_statistics :o) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 4

Writing a pdb file from an xray_structure object
by Claudia Millán Nebot 12 Dec '12

12 Dec '12

Hi everyone, i have been using the xray_structure_simple() method to get an xray.structure object from a pdb object and use it to perform some calculations on it, one of them being expand it to P1. After, i tried to write a new pdb out of this xray_structure in P1. For that purpose, i used the following method: cctbx.xray.structure.as_pdb_file(self, remark=None, remarks=[], fractional_coordinates=False, resname=None, connect=None) but using it both with the original xray structure and with the one on P1 I get very bad lookings pdbs, with incorrect formating, like that for the case of the one in P1: ATOM 1 PDB= PDB 1 12.330 -6.948 -40.820 1.00 19.26 N ATOM 2 PDB= PDB 2 49.988 55.370 3.948 1.00 19.26 N ATOM 3 PDB= PDB 3 36.092 30.710 93.483 1.00 19.26 N ATOM 4 PDB= PDB 4 55.370 49.988 175.123 1.00 19.26 N ATOM 5 PDB= PDB 5 30.710 36.092 85.587 1.00 19.26 N so i guessed that something must be wrong with the residues.... and went to check the code. That as_pdb_file method calls to iotbx.pdb.xray_structure.as_pdb_file(), and on that code i realised that as i was calling it whithout any resname, it was retrieving it from some information on the scatterers that could be wrong. Putting the value of resname to ALA, as a trial, produced almost the same output except for the name: ATOM 1 PDB= ALA 1 12.330 -6.948 -40.820 1.00 19.26 N ATOM 2 PDB= ALA 2 49.988 55.370 3.948 1.00 19.26 N ATOM 3 PDB= ALA 3 36.092 30.710 93.483 1.00 19.26 N ATOM 4 PDB= ALA 4 55.370 49.988 175.123 1.00 19.26 N ATOM 5 PDB= ALA 5 30.710 36.092 85.587 1.00 19.26 N Up to this point, i don't get what is wrong. It may be that i'm using a too simple method and without the correct parameters (xray_structure_simple()), as i guess part of the error is that the assignation of the types of atoms and residues is going crazy. There are other ways of performing this step from a pdb to an xray_structure and viceversa? Thanks in advance! Best regards, Claudia Millán (cmncri(a)ibmb.csic.es) Institut de Biologia Molecular de Barcelona (IBMB-CSIC) Barcelona, Spain

2 2

Documentation .... again
by Graeme.Winter＠diamond.ac.uk 12 Dec '12

12 Dec '12

Hi Folks, Been thinking some more about the old chestnut of documentation. The lack thereof is a well understood issue, and to be honest one which is not going to change any time soon as no one has the time to maintain a manual - this is fine and well understood. As a kind of "patch" for this I was wondering if it would be possible to have a forum where someone could ask for e.g. documentation for a currently undocumented function, which could then be added to the subversion in the code itself. For example: cctbx_project/cctbx/xray/structure.py ... def structure_factors(self, anomalous_flag=None, d_min=None, algorithm=None, cos_sin_table=False, quality_factor=None, u_base=None, b_base=None, wing_cutoff=None): what is cos_sin_table for? with things like this there is the attitude that it is safe to ignore, however if we're writing code to guide people's experiments it would be good to be sure of this. Some simple documentation explaining this would be very useful. I'm definitely not suggesting (i) that anyone writes a manual or (ii) that all undocumented functions are suddenly documented, only that there is a mechanism set up that someone could ask about this and that someone who knows the answer could commit the changes to cctbx and then reply with the diffs - perhaps the way that stackoverflow works, or an email, or something really lightweight. Even this mailing list could be used I guess? I would be happy to help with this, though the above example is not one where I know the answer! Thoughts? Best wishes, Graeme Dr. Graeme Winter Senior Software Scientist Diamond Light Source +44 1235 778091 (work) +44 7786 662784 (work mobile) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

8 24

Structure factor calculation weirdness
by Graeme.Winter＠diamond.ac.uk 10 Dec '12

10 Dec '12

Hi Folks, Some confusion here at diamond - it seems that the amplitude of F(hkl) depends on dmin ---------------- from cctbx import xray from cctbx import crystal from cctbx.array_family import flex # 14keV wavelength = 12.3984 / 14.0 spgroup = 'Fd-3m' latt = (5.4309, 5.4309, 5.4309, 90, 90, 90) atoms = [('Si', 0.125, 0.125, 0.125)] allscatterers=[] for atom in atoms: allscatterers.append(xray.scatterer( label = atom[0], site = (atom[1], atom[2], atom[3]), u = 0)) structure = xray.structure( special_position_settings = crystal.special_position_settings( crystal_symmetry = crystal.symmetry( unit_cell = latt, space_group_symbol = spgroup)), scatterers = flex.xray_scatterer(allscatterers)).expand_to_p1() for dmin in 5.0, 2.5, 1.7: print dmin f_calculations = structure.structure_factors( d_min = dmin).f_calc().amplitudes().show_array() --------------- Chopped out only 001, 010, 100 from this... 5.0 (0, 0, 1) 0.290277589693 (0, 1, 0) 0.290277589693 (1, 0, 0) 0.290277589693 2.5 (0, 0, 1) 0.0 (0, 1, 0) 0.0 (1, 0, 0) 0.0 <snip> 1.7 (0, 0, 1) 0.278378587828 (0, 1, 0) 0.278378587828 (1, 0, 0) 0.278378587828 <snip> Have we misunderstood something? Thanks, Graeme Dr. Graeme Winter Senior Software Scientist Diamond Light Source +44 1235 778091 (work) +44 7786 662784 (work mobile) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 9

UB matrix in cbf files
by Carlo.Vecchini＠diamond.ac.uk 05 Dec '12

05 Dec '12

Hello, I would like to add the orientation matrix information in the cbf file. Following the description in sections 3.2.2.2.3 of IUCr (G), I added the following lines in the cbf header: _diffrn_orient_matrix_type ub_matrix_test _diffrn_orient_matrix_UB_11 0.083424 _diffrn_orient_matrix_UB_12 -0.076007 _diffrn_orient_matrix_UB_13 -0.078787 _diffrn_orient_matrix_UB_21 -0.103374 _diffrn_orient_matrix_UB_22 -0.075939 _diffrn_orient_matrix_UB_23 -0.01146 _diffrn_orient_matrix_UB_31 -0.034434 _diffrn_orient_matrix_UB_32 0.043832 _diffrn_orient_matrix_UB_33 -0.156476 When I try to read back the values with: get_orientation_matrix() I get the following error: UBm = cbf_handle.get_orientation_matrix() File "~/CCTBX/cctbx_plus_build/lib/pycbf.py", line 3915, in get_orientation_matrix return _pycbf.cbf_handle_struct_get_orientation_matrix(self) Exception: CBFlib Error(s): CBF_NOTFOUND If I use set_orientation_matrix() to write the cbf file, I get a similar error. Does anyone have any suggestion on how to solve this? Thanks in advance, Carlo Vecchini -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

CBF compositing
by Frank Murphy 03 Dec '12

03 Dec '12

Hello, Does anyone have a CCTBX-based method for compositing multiple CBFs to a single image a la merge2cbf? Thanks in advance, Frank Murphy

2 3

OS X Lion + CCTBX + scipy
by Graeme.Winter＠diamond.ac.uk 01 Dec '12

01 Dec '12

Hi Folks, Has anyone got scipy to build against the cctbx python binary distribution on OS X Lion? I am finding that it keeps on compiling some bits of the library for PPC architecture which does not work... Thanks, Graeme Dr. Graeme Winter Senior Software Scientist Diamond Light Source +44 1235 778091 (work) +44 7786 662784 (work mobile) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

SVN broken?
by Jan Marten Simons 24 Oct '12

24 Oct '12

Hi, while trying to update to the latest svn revision or when trying a fresh checkout I encounter this error: svn co https://cctbx.svn.sourceforge.net/svnroot/cctbx/trunk cctbx_project svn: Can't find a temporary directory: Internal error According to a quick web search the problem is most likely on the svn server (either no disk space or corrupted .svn dir). Hopefully either sourceforge or any of you can fix this. Cheers, Jan

2 2

Re: [cctbxbb] Question about sigma(I) calculation in merge_equivalents
by Kay Diederichs 21 Oct '12

21 Oct '12

Dear Keitaro, Phil, Luc, after looking into the question of how to merge equivalent reflections (in the context of XDS data processing), I am now convinced that cctbx should _not_ use max(internal sigma, external sigma) by default. Why? - it is not obvious to the "end user" of the data that come out of cctbx that this formula was applied (I am not aware that the above formula is clearly documented) - the data processing programs that I know of do not use this formula, but rather use the "external sigma" . I don't imply that the "external sigma" is better, but it violates the principle of least surprise to deviate from a proven procedure. - if the observations (that are merged in cctbx) come from one of these data processing programs then there is a chance that the error model was already adjusted such that the reduced chi**2 is near 1 (at least this is the case for XDS and SCALA). If the sigmas of the merged data then are adjusted upwards again (by the formula above), I have a strong feeling that this leads to an inconsistency. - using max() seems like an ad-hoc way and it lacks a clear rationale. I would like to see examples of comparison of downstream crystallographic calculations (most importantly, experimental phasing) using different ways of calculating the sigma of the merged data. Until this proves the superiority of the above formula, I believe it should be an option, not a default. thanks, Kay > Keitaro Yamashita yamashita at castor.sci.hokudai.ac.jp > Thu Sep 6 09:20:34 PDT 2012 > > Previous message: [cctbxbb] Question about sigma(I) calculation in > merge_equivalents > Next message: [cctbxbb] Question about sigma(I) calculation in > merge_equivalents > Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] > > Dear Luc, > > Thank you for your explanation! I understand better than before. > > Usually, sigmas given by data processing programs are already > corrected based on their error model. > Sigmas are adjusted to match the actual scatter. > I think using internal variance is re-correction of sigmas. > Is it valid way? If internal variance is bigger, it suggests error > model is not perfect? > > I calculated external/internal variances using lysozyme (standard > sample in protein crystallography) data. > Intensities and sigmas are determined by XDS. > > I attached two plots, where "Imean", "wsigma", "sigma" are averaged > intensity, internal sigma, external sigma, respectively. > One plot is histogram of wsigma/sigma by multiplicity. For lower > multiplicity, we can see extreme discrepancies. (Note that each > vertical axis is not on the same scale.) > The other plot is wsigma/sigma vs intensity. Extreme discrepancies can > be seen in lower intensities. > I hope it could be interesting for you. > > > > Crystals use only the external variance by default, the reasoning > being that the internal variance being based on sample statistics is > almost always too unreliable because groups of equivalent reflections > are too small. > > Then, I think it would be nice if we can choose the way in cctbx. > I mean, option to choose "use bigger one" or "always use > external/internal variance" would be nice to have. > > I am looking forward to your comment. > > Best regards, > Keitaro > > 2012/9/6 Luc Bourhis <luc_j_bourhis at mac.com>: > > Dear Keitaro, > > > >> But it is still unclear to me why it takes the greatest of the > >> "internal" variance and "external" variance. > >> Is it based on some tests using real data? or is it theoretically > >> superior to using always external variance? > > > > Those are good questions and to be honest I do not know for sure the > answer to them. As it seems common in applied statistics, the treatment > starts with by-the-book methods relying on a well defined theory but at > the end there is always a completely heuristic twist. Particularly true > in crystallography I would argue. But let me try to give some rationales. > > > > It seems to me that the internal and external variance should not > differ too much. Let's consider the two ways this may not be true. > > > > 1. The quoted intensities of a group of equivalent reflections have a > small spread, leading to a small internal variance, but the quoted > sigma's are comparatively big, resulting in an external variance > significantly bigger than the internal one. This is a possible event but > an unlikely one: the statistical intuition in that case is to say that > the small internal variance is a fluke and to use the external one instead. > > > > 2. An external variance significantly smaller than the internal one, > should ring an alarm bell. Indeed a small external variance means that > the small quoted sigma's strongly suggests the intensities cannot spread > too much from their assumed common true value whereas the comparatively > bigger internal variance blatantly contradicts that. Thus either the > intensities or the sigma's have not been correctly determined. > Crystallographers seem to err on the side of trusting data here, i.e. to > disregard the sigma's, and therefore to choose the internal variance. > > > >> I would like to know how this method affects further > crystallographic process. > > > > I am afraid I do not have experience with your domain, protein > crystallography. I know that the small molecule program Crystals use > only the external variance by default, the reasoning being that the > internal variance being based on sample statistics is almost always too > unreliable because groups of equivalent reflections are too small. Since > Crystals is as well accepted as ShelXL to produce publishable > structures, it answer your question in at least in one corner of > crystallography, unfortunately not yours. > > > > I think it could be a simple and interesting exercise to take a > representative protein dataset of yours, then to print the redundancy, > the internal, and the external variance. Actually I would be surprised > if such a study has not already been done and published. Perhaps some of > the gurus on this forum can shed more lights onto that subject. > > > > Best wishes, > > > > Luc -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: Kay.Diederichs(a)uni-konstanz.de Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz This e-mail is digitally signed. If your e-mail client does not have the necessary capabilities, just ignore the attached signature "smime.p7s".

2 1

FYI: The Quaternion Characteristic Polynomial method
by Jeffrey Van Voorst 19 Oct '12

19 Oct '12

Greetings, If you already know about the quaternion characteristic polynomial method for a fast closed-form solution of the least squares superposition problem, this note is mostly noise. While in the process of updating links to my project, I noticed the note at the bottom of the cctbx sourceforge page (http://cctbx.sourceforge.net/). I like the historical nature presented, and I used the historical method to numerically solve quartic polynomials in a previous project at Michigan State University that ranks relatively large datasets of protein small-molecule binding sites (with respect to a given query site). The superposition method I used has a closed form and is due to BKP Horn. Later on (after writing my dissertation), I found that using the historical method to numerically solve quartic polynomials is not numerically stable or robust. Fortunately, there is an easy solution, and if you are interested in a fast closed-form solution for the least squares superposition problem you would do well (in my opinion) to visit http://theobald.brandeis.edu/qcp/. Best regards, Jeff Van Voorst

1 0