[phenixbb] XDS files in Phenix

[Kay Diederichs]

Am 10.06.2011 18:27, schrieb Ralf Grosse-Kunstleve:
> Hi Kay,
>
> Thanks for the explanations!
>
>     a) the main program ("xds", or "xds_par") writes only one type of
>     file, called XDS_ASCII.HKL. This has _unmerged_ observations, and
>     this is therefore currently _unsuitable_ for reading by Phenix
>     programs (since the Phenix routine expects merged, unique
>     reflections). If this is nevertheless tried, I see the danger that
>     Phenix might "eat" all observations of a unique reflections except
>     one, possibly without telling about this problem (? - I didn't
>     try!!) - this is definitely not what you want!
>
>
> We have a simple way of merging symmetry-equivalent observations, which
> is used (for example) when we read unmerged scalepack files.

this sounds useful - where can I see this? If this could be enabled then 
there is no reason not to read files of type a) or b) .

>
>     b) the program "xscale" can write an output file with unmerged
>     observations. This is the default, and corresponds to MERGE=FALSE in
>     XSCALE.INP . As for a), this is also not what you want!
>
>
> I've never been sure about the tradeoff between convenience and
> correctness. We could read this file type and merge the observations
> ourselves, which is convenient, but is the result as good as letting XDS
> (or scalepack or scala) do the merging?

I can only speak for XDS - and that internally does just the obviously 
correct thing:
a) reject observations with negative sigma
b) use the standard formulas - see 
http://en.wikipedia.org/wiki/Weighted_mean#Dealing_with_variance

I'd be surprised if SCALEPACK and SCALA use different formulas.

>
>     c) the program "xscale" can write an output file with merged
>     observations. This is not the default, but can be obtained by using
>     MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix
>     programs.
>
>     Thus only possibility c) will give you a "native XDS" file that is
>     ready to be read by Phenix programs.
>
>
> A reader for this file type has been part of Phenix for many years:
>
> http://cci.lbl.gov/cctbx_sources/iotbx/xds/read_ascii.py
>
> Does this still look right?

yes - except it should reject observations with negative sigma (maybe 
I'm overlooking it) but these probably don't occur in MERGE=TRUE files.

and yes - it seems to check that "MERGE=TRUE" so I guess it will 
complain in case a) and b) .

>
>     In addition to this possibility, one can use of course use "xdsconv"
>     to go from XDS_ASCII.HKL (or any file written by "xscale") to a
>     (non-native XDS) filetype that Phenix programs can read. The most
>     general should be to write a MTZ file, but CNS/X-PLOR type files can
>     also be produced.
>
>
> Great to know that you can export to MTZ files; I wasn't aware of this

in fact xdsconv writes the necessary files, and prints a few lines of 
output to the screen that just have to be cut-and-pasted to get an MTZ 
file. Of course this could also easily be scripted.

> before. This looks like the best option in most cases, since the
> symmetry information is preserved in MTZ files (but not CNS files).
>
> Ralf
>

thanks,

Kay