September 2012 - cctbxbb - phenix-online.org

Re: [cctbxbb] Question about sigma(I) calculation in merge_equivalents
by Kay Diederichs 21 Oct '12

21 Oct '12

Dear Keitaro, Phil, Luc, after looking into the question of how to merge equivalent reflections (in the context of XDS data processing), I am now convinced that cctbx should _not_ use max(internal sigma, external sigma) by default. Why? - it is not obvious to the "end user" of the data that come out of cctbx that this formula was applied (I am not aware that the above formula is clearly documented) - the data processing programs that I know of do not use this formula, but rather use the "external sigma" . I don't imply that the "external sigma" is better, but it violates the principle of least surprise to deviate from a proven procedure. - if the observations (that are merged in cctbx) come from one of these data processing programs then there is a chance that the error model was already adjusted such that the reduced chi**2 is near 1 (at least this is the case for XDS and SCALA). If the sigmas of the merged data then are adjusted upwards again (by the formula above), I have a strong feeling that this leads to an inconsistency. - using max() seems like an ad-hoc way and it lacks a clear rationale. I would like to see examples of comparison of downstream crystallographic calculations (most importantly, experimental phasing) using different ways of calculating the sigma of the merged data. Until this proves the superiority of the above formula, I believe it should be an option, not a default. thanks, Kay > Keitaro Yamashita yamashita at castor.sci.hokudai.ac.jp > Thu Sep 6 09:20:34 PDT 2012 > > Previous message: [cctbxbb] Question about sigma(I) calculation in > merge_equivalents > Next message: [cctbxbb] Question about sigma(I) calculation in > merge_equivalents > Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] > > Dear Luc, > > Thank you for your explanation! I understand better than before. > > Usually, sigmas given by data processing programs are already > corrected based on their error model. > Sigmas are adjusted to match the actual scatter. > I think using internal variance is re-correction of sigmas. > Is it valid way? If internal variance is bigger, it suggests error > model is not perfect? > > I calculated external/internal variances using lysozyme (standard > sample in protein crystallography) data. > Intensities and sigmas are determined by XDS. > > I attached two plots, where "Imean", "wsigma", "sigma" are averaged > intensity, internal sigma, external sigma, respectively. > One plot is histogram of wsigma/sigma by multiplicity. For lower > multiplicity, we can see extreme discrepancies. (Note that each > vertical axis is not on the same scale.) > The other plot is wsigma/sigma vs intensity. Extreme discrepancies can > be seen in lower intensities. > I hope it could be interesting for you. > > > > Crystals use only the external variance by default, the reasoning > being that the internal variance being based on sample statistics is > almost always too unreliable because groups of equivalent reflections > are too small. > > Then, I think it would be nice if we can choose the way in cctbx. > I mean, option to choose "use bigger one" or "always use > external/internal variance" would be nice to have. > > I am looking forward to your comment. > > Best regards, > Keitaro > > 2012/9/6 Luc Bourhis <luc_j_bourhis at mac.com>: > > Dear Keitaro, > > > >> But it is still unclear to me why it takes the greatest of the > >> "internal" variance and "external" variance. > >> Is it based on some tests using real data? or is it theoretically > >> superior to using always external variance? > > > > Those are good questions and to be honest I do not know for sure the > answer to them. As it seems common in applied statistics, the treatment > starts with by-the-book methods relying on a well defined theory but at > the end there is always a completely heuristic twist. Particularly true > in crystallography I would argue. But let me try to give some rationales. > > > > It seems to me that the internal and external variance should not > differ too much. Let's consider the two ways this may not be true. > > > > 1. The quoted intensities of a group of equivalent reflections have a > small spread, leading to a small internal variance, but the quoted > sigma's are comparatively big, resulting in an external variance > significantly bigger than the internal one. This is a possible event but > an unlikely one: the statistical intuition in that case is to say that > the small internal variance is a fluke and to use the external one instead. > > > > 2. An external variance significantly smaller than the internal one, > should ring an alarm bell. Indeed a small external variance means that > the small quoted sigma's strongly suggests the intensities cannot spread > too much from their assumed common true value whereas the comparatively > bigger internal variance blatantly contradicts that. Thus either the > intensities or the sigma's have not been correctly determined. > Crystallographers seem to err on the side of trusting data here, i.e. to > disregard the sigma's, and therefore to choose the internal variance. > > > >> I would like to know how this method affects further > crystallographic process. > > > > I am afraid I do not have experience with your domain, protein > crystallography. I know that the small molecule program Crystals use > only the external variance by default, the reasoning being that the > internal variance being based on sample statistics is almost always too > unreliable because groups of equivalent reflections are too small. Since > Crystals is as well accepted as ShelXL to produce publishable > structures, it answer your question in at least in one corner of > crystallography, unfortunately not yours. > > > > I think it could be a simple and interesting exercise to take a > representative protein dataset of yours, then to print the redundancy, > the internal, and the external variance. Actually I would be surprised > if such a study has not already been done and published. Perhaps some of > the gurus on this forum can shed more lights onto that subject. > > > > Best wishes, > > > > Luc -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: Kay.Diederichs(a)uni-konstanz.de Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz This e-mail is digitally signed. If your e-mail client does not have the necessary capabilities, just ignore the attached signature "smime.p7s".

2 1

Build failing
by Graeme.Winter＠diamond.ac.uk 01 Oct '12

01 Oct '12

Hi Folks, Following an svn update and several attempts to build, I am still getting this compile error /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jy.hpp:346:11: error: 'bessel_y_small_z_series' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive] In file included from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jn.hpp:17:0, from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/bessel.hpp:18, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/bessel.h:9, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/boost_python/math_ext.cpp:3: /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jy_series.hpp:139:10: note: 'template<class T, class Policy> T boost::math::detail::bessel_y_small_z_series(T, T, T*, const Policy&)' declared here, later in the translation unit In file included from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/bessel.hpp:17:0, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/bessel.h:9, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/boost_python/math_ext.cpp:3: /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jy.hpp:356:11: error: 'bessel_j_small_z_series' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive] In file included from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jn.hpp:17:0, from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/bessel.hpp:18, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/bessel.h:9, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/boost_python/math_ext.cpp:3: /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jy_series.hpp:47:10: note: 'template<class T, class Policy> T boost::math::detail::bessel_j_small_z_series(T, T, const Policy&)' declared here, later in the translation unit In file included from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/bessel.hpp:17:0, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/bessel.h:9, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/boost_python/math_ext.cpp:3: /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jy.hpp:363:11: error: 'bessel_yn_small_z' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive] In file included from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jn.hpp:17:0, from /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/bessel.hpp:18, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/bessel.h:9, from /Users/graeme/svn/cctbx/sources/cctbx_project/scitbx/math/boost_python/math_ext.cpp:3: /Users/graeme/svn/cctbx/sources/boost/boost/math/special_functions/detail/bessel_jy_series.hpp:213:3: note: 'template<class T, class Policy> T boost::math::detail::bessel_yn_small_z(int, T, T*, const Policy&)' declared here, later in the translation unit scons: *** [scitbx/math/boost_python/math_ext.o] Error 1 scons: building terminated because of errors. usr+sys time: 26.64 seconds, ticks: 51849408, micro-seconds/tick: 0.514 wall clock time: 8 minutes 5.55 seconds (485.55 seconds total) make: *** [default] Error 2 OS X Lion machine Graemes-MacBook-Pro:build graeme$ uname -a Darwin Graemes-MacBook-Pro.local 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 Graemes-MacBook-Pro:build graeme$ gcc --version gcc (GCC) 4.7.0 20120314 (prerelease) Copyright (C) 2012 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Have I missed something? Nightly builds seem to be OK. Commands ../../cctbx_plus/base/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python ../sources/cctbx_project/libtbx/configure.py --compiler=gcc labelit cctbx bpcx_regression rstbx . setpaths.sh make Thanks, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 8

dropping Python 2.4 support?
by Nathaniel Echols 27 Sep '12

27 Sep '12

Hi all-- Will it ruin anyone's work if we make Python 2.5 the minimum platform for using CCTBX? 2.4 is now almost eight years old and the lack of certain newer features (decorators, 'with') is starting to hold us back. thanks, Nat

2 2

Enabling OpenMP for phaser on OS X
by William G. Scott 17 Sep '12

17 Sep '12

Hi folks: I found if I created a symbolic link from a g++4.7 compiler to /usr/bin/g++, I could "trick" phaser into compiling with OpenMP enabled. This is kind of an ugly hack. How do I force it to use /sw/bin/g++-4 ? Is there some place in the source code for cctbx (or phaser) that I can make such a change? Thanks. Bil William G. Scott Professor Department of Chemistry and Biochemistry and The Center for the Molecular Biology of RNA 228 Sinsheimer Laboratories University of California at Santa Cruz Santa Cruz, California 95064 USA

2 4

Quick start info?
by esserlo＠helix.nih.gov 11 Sep '12

11 Sep '12

Hi all, this is a question about how to get started quickly. I have looked at the sourceforge site for cctbx and read diverse documents but so far I have not found documentation that would help me get started fast. I have a crystallographic problem which I can probably program from scratch but then again why bother if there is a crystallographic tool box around. But well despite all the power in it getting it to work seems a bit daunting. Are there really good examples? I did get the "sand box quartz model" script to work but this is still far off of what I want to do. Thanks for all advice and sorry if I missed the 'obvious' one. Lothar

4 4

Question about sigma(I) calculation in merge_equivalents
by Keitaro Yamashita 07 Sep '12

07 Sep '12

Dear CCTBX developers, I have a question about the method in CCTBX to calculate sigma(averaged I) when merging. In cctbx/miller/merge_equivalents.h (line 594 in phenix-dev-1148), sigma(averaged I)^2 will be calculated by max(mv.gsl_stats_wvariance()/values.size(), 1/mv.sum_weights()). It takes maximum value of the two options. I know the latter equation - this is just sigma(averaged I)^2 = 1/sum(w), where w = 1/sigma(I)^2. And I believe this equation is widely used in other crystallographic programs e.g. XDS. The former is, according to the comment in the code, the emulation of gsl_stats_wvariance, and we can find the information from gsl website: http://www.gnu.org/software/gsl/manual/html_node/Weighted-Samples.html I don't know how this "wvariance" is derived. Is it a better estimate of sigma(averaged I)? Why does it take the maximum from the two options? I would like to know the theory of this sigma calculation in CCTBX. Best regards, Keitaro

3 5

Debian Package
by Radostan Riedel 05 Sep '12

05 Sep '12

Dear CCTBX Maintainers, Maybe some of you remember me last year asking on this mailing list[1]. I was asked by the Debian Science Team[2] to contribute my former work to get cctbx finally packaged in Debian. The last month I was working with Baptiste Carvello and Picca Frédéric-Emmanuel to get this done. Our packaging efforts + patches can be found in our git[3]. We organize our packaging efforts in a wiki[4]. But we don't want to overlook you and give some of our work back into the main project. We have a few patches to give back upstream that would really help us and hopefully help to increase the cctbx user community. To explain a little bit what we've done. First of all we tried respectfully not to change too much about your build system. We integrated a few options in libtbx/env_config which shouldn't break anything. The primary changes are: * Option to check and use system libraries (in case any library is already installed no need to compile the bundled shlib) * Install targets for headers and shlibs and binaries: calling "scons install" * Shlib versioning: This is achieved by utilizing libtool (the GNU standard) * --install-prefix, --install-destdir and --libdir option * support for LD_FLAGS in --use_environment_flags option * Fallback to avoid using "LIBTBX_BUILD" env variable after install * Full integration of your scons build system in distutils: setup.py and sconsutils.py. You can just call 'python setup.py install' * Adaption of your test system to be called by distutils * Autogenerate pkg-config files for shlibs What we now need to know is if our changes are appreciated or if we need to spend more work on this. Keep in mind that our work is still work in progress! One big concern is always the ABI/API stability. But if we can count on the shlib versioning as mentioned here[5] it would help us and other developers a lot. Libtool should be available on most systems, works with common compilers and it provides the quasi standard for versioning shlibs. We have other issues too but I guess I won't overkill this mail to much for now. I hope we can discuss and work on this together. Please tell us your honest opinion and critics. kind regards Radostan Riedel P.S: This work is intended to work on other distributions too. We also hope that Fedora, openSUSE, Gentoo etc. can benefit from it too. [1] http://phenix-online.org/pipermail/cctbxbb/2011-June/000192.html [2] http://wiki.debian.org/DebianScience [3] http://anonscm.debian.org/gitweb/?p=debian-science/packages/cctbx.git;a=sum… [4] http://wiki.debian.org/DebianScience/cctbx [5] http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.…

12 84