- cctbxbb - phenix-online.org

Re: [cctbxbb] Question about sigma(I) calculation in merge_equivalents
by Kay Diederichs 22 Oct '12

22 Oct '12

Dear Keitaro, Phil, Luc, after looking into the question of how to merge equivalent reflections (in the context of XDS data processing), I am now convinced that cctbx should _not_ use max(internal sigma, external sigma) by default. Why? - it is not obvious to the "end user" of the data that come out of cctbx that this formula was applied (I am not aware that the above formula is clearly documented) - the data processing programs that I know of do not use this formula, but rather use the "external sigma" . I don't imply that the "external sigma" is better, but it violates the principle of least surprise to deviate from a proven procedure. - if the observations (that are merged in cctbx) come from one of these data processing programs then there is a chance that the error model was already adjusted such that the reduced chi**2 is near 1 (at least this is the case for XDS and SCALA). If the sigmas of the merged data then are adjusted upwards again (by the formula above), I have a strong feeling that this leads to an inconsistency. - using max() seems like an ad-hoc way and it lacks a clear rationale. I would like to see examples of comparison of downstream crystallographic calculations (most importantly, experimental phasing) using different ways of calculating the sigma of the merged data. Until this proves the superiority of the above formula, I believe it should be an option, not a default. thanks, Kay > Keitaro Yamashita yamashita at castor.sci.hokudai.ac.jp > Thu Sep 6 09:20:34 PDT 2012 > > Previous message: [cctbxbb] Question about sigma(I) calculation in > merge_equivalents > Next message: [cctbxbb] Question about sigma(I) calculation in > merge_equivalents > Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] > > Dear Luc, > > Thank you for your explanation! I understand better than before. > > Usually, sigmas given by data processing programs are already > corrected based on their error model. > Sigmas are adjusted to match the actual scatter. > I think using internal variance is re-correction of sigmas. > Is it valid way? If internal variance is bigger, it suggests error > model is not perfect? > > I calculated external/internal variances using lysozyme (standard > sample in protein crystallography) data. > Intensities and sigmas are determined by XDS. > > I attached two plots, where "Imean", "wsigma", "sigma" are averaged > intensity, internal sigma, external sigma, respectively. > One plot is histogram of wsigma/sigma by multiplicity. For lower > multiplicity, we can see extreme discrepancies. (Note that each > vertical axis is not on the same scale.) > The other plot is wsigma/sigma vs intensity. Extreme discrepancies can > be seen in lower intensities. > I hope it could be interesting for you. > > > > Crystals use only the external variance by default, the reasoning > being that the internal variance being based on sample statistics is > almost always too unreliable because groups of equivalent reflections > are too small. > > Then, I think it would be nice if we can choose the way in cctbx. > I mean, option to choose "use bigger one" or "always use > external/internal variance" would be nice to have. > > I am looking forward to your comment. > > Best regards, > Keitaro > > 2012/9/6 Luc Bourhis <luc_j_bourhis at mac.com>: > > Dear Keitaro, > > > >> But it is still unclear to me why it takes the greatest of the > >> "internal" variance and "external" variance. > >> Is it based on some tests using real data? or is it theoretically > >> superior to using always external variance? > > > > Those are good questions and to be honest I do not know for sure the > answer to them. As it seems common in applied statistics, the treatment > starts with by-the-book methods relying on a well defined theory but at > the end there is always a completely heuristic twist. Particularly true > in crystallography I would argue. But let me try to give some rationales. > > > > It seems to me that the internal and external variance should not > differ too much. Let's consider the two ways this may not be true. > > > > 1. The quoted intensities of a group of equivalent reflections have a > small spread, leading to a small internal variance, but the quoted > sigma's are comparatively big, resulting in an external variance > significantly bigger than the internal one. This is a possible event but > an unlikely one: the statistical intuition in that case is to say that > the small internal variance is a fluke and to use the external one instead. > > > > 2. An external variance significantly smaller than the internal one, > should ring an alarm bell. Indeed a small external variance means that > the small quoted sigma's strongly suggests the intensities cannot spread > too much from their assumed common true value whereas the comparatively > bigger internal variance blatantly contradicts that. Thus either the > intensities or the sigma's have not been correctly determined. > Crystallographers seem to err on the side of trusting data here, i.e. to > disregard the sigma's, and therefore to choose the internal variance. > > > >> I would like to know how this method affects further > crystallographic process. > > > > I am afraid I do not have experience with your domain, protein > crystallography. I know that the small molecule program Crystals use > only the external variance by default, the reasoning being that the > internal variance being based on sample statistics is almost always too > unreliable because groups of equivalent reflections are too small. Since > Crystals is as well accepted as ShelXL to produce publishable > structures, it answer your question in at least in one corner of > crystallography, unfortunately not yours. > > > > I think it could be a simple and interesting exercise to take a > representative protein dataset of yours, then to print the redundancy, > the internal, and the external variance. Actually I would be surprised > if such a study has not already been done and published. Perhaps some of > the gurus on this forum can shed more lights onto that subject. > > > > Best wishes, > > > > Luc -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: Kay.Diederichs(a)uni-konstanz.de Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz This e-mail is digitally signed. If your e-mail client does not have the necessary capabilities, just ignore the attached signature "smime.p7s".

2 1

FYI: The Quaternion Characteristic Polynomial method
by Jeffrey Van Voorst 20 Oct '12

20 Oct '12

Greetings, If you already know about the quaternion characteristic polynomial method for a fast closed-form solution of the least squares superposition problem, this note is mostly noise. While in the process of updating links to my project, I noticed the note at the bottom of the cctbx sourceforge page (http://cctbx.sourceforge.net/). I like the historical nature presented, and I used the historical method to numerically solve quartic polynomials in a previous project at Michigan State University that ranks relatively large datasets of protein small-molecule binding sites (with respect to a given query site). The superposition method I used has a closed form and is due to BKP Horn. Later on (after writing my dissertation), I found that using the historical method to numerically solve quartic polynomials is not numerically stable or robust. Fortunately, there is an easy solution, and if you are interested in a fast closed-form solution for the least squares superposition problem you would do well (in my opinion) to visit http://theobald.brandeis.edu/qcp/. Best regards, Jeff Van Voorst

1 0

--enable-boost-threads flag
by Jeffrey Van Voorst 19 Oct '12

19 Oct '12

Greetings, If someone knows the answer off the top of their head, I would be interested in the answer. Otherwise, feel free to ignore this. Does the --enable-boost-threads flag for cctbx_sources/libtbx/configure.py do anything drastic besides include -pthread in the compiling and linking steps? I am trying to understand why I would get a free() error (for the shared_ptr to root_data) under the following situation: *) the iotbx_pdb shared library is configured without the --enable-boost-threads flag (on a linux box) *) sometimes reproducible from a test C++ program that is linked using -pthread even in the case when the program is as simple as loading a pdb file and creating the hierarchy. *) gcc 4.4.5 --Jeff Van Voorst

4 14

Re: [cctbxbb] Patterson map data structure
by Ten Eyck, Lynn 17 Oct '12

17 Oct '12

Any Fourier map without an F000 term will analytically have a mean value of 0. Lynn Connected by DROID on Verizon Wireless -----Original message----- From: Nathaniel Echols <nechols(a)lbl.gov> To: cctbx mailing list <cctbxbb(a)phenix-online.org> Sent: Tue, Oct 16, 2012 17:22:17 GMT+00:00 Subject: Re: [cctbxbb] Patterson map data structure On Tue, Oct 16, 2012 at 6:47 AM, Dmytro Guzenko <Dmytro.Guzenko(a)pharm.kuleuven.be> wrote: > rmap = pmap.real_map_unpadded() > n_real = pmap.n_real() > unit_cell = pmap.unit_cell() > grid2cart = maptbx.grid2cart(n_real,unit_cell.orthogonalization_matrix()) > > n_points_total = n_real[0]*n_real[1]*n_real[2] > cart_coord = flex.double(flex.grid(3,n_points_total)) > p_weight = flex.double(n_points_total) > > for plain_ind,grid_point in enumerate(flex.nested_loop(n_real)): > cart_coord[0,plain_ind],cart_coord[1,plain_ind],cart_coord[2,plain_ind] = grid2cart(grid_point) > p_weight[plain_ind] = rmap[grid_point] There are better arrays for storing the coordinates, e.g.: cart_coord = flex.vec3_double(flex.grid(n_real)) p_weight = flex.double(rmap.size()) for plain_ind,grid_point in enumerate(flex.nested_loop(n_real)): cart_coord[grid_point] = grid2cart(grid_point) p_weight[plain_ind] = rmap[grid_point] > By the way, I noticed that many weights are negative, is this possible in a patterson map or am I doing something wrong? This is normal - I believe it's a Fourier series termination error, but I still don't fully understand the reason. An Fobs map will have the same problem; the mean will always be approximately zero. Technically, if the map were perfect with no missing data, it would be uniformly positive, but that's never actually the case. There are ways to "fix" the map so it's all-positive, such as maximum entropy cleanup, but in practice I think we just ignore the negative values (at least for Fobs/2mFo-DFc maps - not sure how Patterson maps are usually handled). -Nat _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb

1 0

Patterson map data structure
by Dmytro Guzenko 17 Oct '12

17 Oct '12

Hello all, I am playing with the Patterson function as calculated by cctbx and I am confused as to how I can extract and manipulate the data. More specifically, miller.patterson_map(...).real_map() returns a one-dimensional array of values. How do I get the corresponding 3D coordinates? I suspect it has to do with the gridding and some kind of fixed order enumeration, but I haven't found an example (the peak_search().sites() seems to be relevant, but I would like the whole thing). On a related note - is there a way to debug the python code of cctbx and step into the c++ code? Debugging is my favorite way to figure things out, but with cctbx most interesting things are hidden. I am using Eclipse + PyDev now. Thank you! Kind regards, Dmytro.

3 4

iotbx.xds
by Graeme.Winter＠diamond.ac.uk 16 Oct '12

16 Oct '12

Hi Folks, Would anyone have any objection if I added some code to iotbx.xds to read things like experimental geometry etc in a straightforward way from the various files that the program produces? I see something to read reflection files in there already. Thanks, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 4

no more Python 2.4
by Nathaniel Echols 16 Oct '12

16 Oct '12

FYI for those of you who didn't get my original email: the minimum Python version required for CCTBX is now 2.5, so for anyone contributing to CCTBX, you can now use more advanced syntax such as "with" and "try/except/else/finally" (and decorators, which were already supported in 2.4, but we haven't used them so far). Full list of changes in 2.5 here: http://docs.python.org/whatsnew/2.5.html -Nat

1 0

Python documentation
by David Waterman 11 Oct '12

11 Oct '12

Hi, I've got a few questions about the automatically generated Python documentation. 1. Is this done by doxygen like the C++ documentation? 2. What are the rules that determine whether text in source files is included as documentation or not? Clearly docstrings are, but also some comments do (only those at module scope?) 3. Similarly, what determines which packages are documented or not? For example, from http://cctbx.sourceforge.net/current/python/rstbx.html, cftbx is documented but bpcx is not. Is there at some point a manual decision about what should be documented? Cheers -- David

4 7

Going from a pdb file to an x_ray_structure object
by Claudia Millán Nebot 10 Oct '12

10 Oct '12

Hi everyone, i'm trying to get an xray.structure object from a pdb file, as i'm interested on using both this and a Miller array coming from a reflection file to calculate correlation coefficients between Fobs and Fcalc. For this purpose, i was searching for something that will do the same as the iotbx.pdb.xray_structure.as_pdb_file but the other way around. Is there such a tool? I've look for a while but as i'm new to all that maybe i missed something. Thanks in advance and my excuses if is a too evident question, Claudia Millán (cmncri(a)ibmb.csic.es) Institut de Biologia Molecular de Barcelona (IBMB-CSIC) Barcelona, Spain

3 8

Re: [cctbxbb] RE : cctbx debian package: new commit
by Radostan Riedel 04 Oct '12

04 Oct '12

On Wed, 03. Oct 11:11, Baptiste Carvello wrote: > 2) there are a few C++ applications which depend on cctbx, the one I > know is FOX ( http://vincefn.net/Fox/ , objcryst-fox in Debian). For > those use cases, should we still build a libcctbx-dev package with the > headers and static libs (for those we don't need to care about ABI > stability)? I think this is a very important point here. Let's assume I want to extent the cctbx by adding a module, for let's say absorbtion correction and I want to implement a method called "multiscan" and I want to make a fast python boost extension to integrate with my python code. I need some symbols from any of the shared libs in cctbx. How could this be done in Debian or in any other distribution? About the check for version: In my patch I did this with SCons[1]. What this does is to compile and link a test program and check for success. I saw in your code that you are doing similar things and we could write some simple c++ program that for example includes boost/version and check some "BOOST_VERSION"? This way it checks globally for the availability of a shared library. [1] http://www.scons.org/doc/2.0.1/HTML/scons-user/x4146.html

2 1