Dear Ralf, dear all, Recently there was some discussion about reading of XDS output files here on PhenixBB, and I'd like to try and shed some light on this. First, I'm glad to hear this is possible in principle! However, the caveat is, that three types of "native" output files are written by programs from the "XDS program package": a) the main program ("xds", or "xds_par") writes only one type of file, called XDS_ASCII.HKL. This has _unmerged_ observations, and this is therefore currently _unsuitable_ for reading by Phenix programs (since the Phenix routine expects merged, unique reflections). If this is nevertheless tried, I see the danger that Phenix might "eat" all observations of a unique reflections except one, possibly without telling about this problem (? - I didn't try!!) - this is definitely not what you want! b) the program "xscale" can write an output file with unmerged observations. This is the default, and corresponds to MERGE=FALSE in XSCALE.INP . As for a), this is also not what you want! c) the program "xscale" can write an output file with merged observations. This is not the default, but can be obtained by using MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix programs. Thus only possibility c) will give you a "native XDS" file that is ready to be read by Phenix programs. In addition to this possibility, one can use of course use "xdsconv" to go from XDS_ASCII.HKL (or any file written by "xscale") to a (non-native XDS) filetype that Phenix programs can read. The most general should be to write a MTZ file, but CNS/X-PLOR type files can also be produced. If someone would implement merging of observations in the Phenix routines (it should be straightforward: negative sigmas indicate that observations should not be used; otherwise the standard formulas could be used; symmetry-related reflections of course have to be handled correctly but they are already sorted by unique reflection), then possibilities a) and b) would also work. But until this is implemented, c) should be used. Hope this helps, Kay P.S. The filetypes and everything else around XDS is accurately documented at http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/ -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: [email protected] Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz This e-mail is digitally signed. If your e-mail client does not have the necessary capabilities, just ignore the attached signature "smime.p7s".
Kay, We have been quite happy using XDS processed data (including twinned, SAD, and MAD data) with Phenix via CCP4. We just process a data set through CORRECT, then put the XDS_ASCII.HKL file through Pointless (often to reindex different runs or crystals to the same reference data set). We scale the resulting MTZ file with Scala and the MTZ file with the merged/scaled data is used as input to Phenix. It is quick (~10-15 min depending) and works fine. Other then needing 3 different program suites, I don't see anything wrong with this route for routine work. Cheers, Michael **************************************************************** R. Michael Garavito, Ph.D. Professor of Biochemistry & Molecular Biology 513 Biochemistry Bldg. Michigan State University East Lansing, MI 48824-1319 Office: (517) 355-9724 Lab: (517) 353-9125 FAX: (517) 353-9334 Email: [email protected] **************************************************************** On Jun 10, 2011, at 11:08 AM, Kay Diederichs wrote:
Dear Ralf, dear all,
Recently there was some discussion about reading of XDS output files here on PhenixBB, and I'd like to try and shed some light on this.
First, I'm glad to hear this is possible in principle! However, the caveat is, that three types of "native" output files are written by programs from the "XDS program package":
a) the main program ("xds", or "xds_par") writes only one type of file, called XDS_ASCII.HKL. This has _unmerged_ observations, and this is therefore currently _unsuitable_ for reading by Phenix programs (since the Phenix routine expects merged, unique reflections). If this is nevertheless tried, I see the danger that Phenix might "eat" all observations of a unique reflections except one, possibly without telling about this problem (? - I didn't try!!) - this is definitely not what you want! b) the program "xscale" can write an output file with unmerged observations. This is the default, and corresponds to MERGE=FALSE in XSCALE.INP . As for a), this is also not what you want! c) the program "xscale" can write an output file with merged observations. This is not the default, but can be obtained by using MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix programs.
Thus only possibility c) will give you a "native XDS" file that is ready to be read by Phenix programs.
In addition to this possibility, one can use of course use "xdsconv" to go from XDS_ASCII.HKL (or any file written by "xscale") to a (non- native XDS) filetype that Phenix programs can read. The most general should be to write a MTZ file, but CNS/X-PLOR type files can also be produced.
If someone would implement merging of observations in the Phenix routines (it should be straightforward: negative sigmas indicate that observations should not be used; otherwise the standard formulas could be used; symmetry-related reflections of course have to be handled correctly but they are already sorted by unique reflection), then possibilities a) and b) would also work. But until this is implemented, c) should be used.
Hope this helps,
Kay
P.S. The filetypes and everything else around XDS is accurately documented at http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/ -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: [email protected] Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz
This e-mail is digitally signed. If your e-mail client does not have the necessary capabilities, just ignore the attached signature "smime.p7s".
_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
Michael, Well, the little problem arises if one wish to refine a structure with the EXPERIMENTAL data such as intensities. There are dialects ( I presume) between various packages converting intensities to structure factors. In addition, if the experimental data (such as merged or unmerged SCALEPACK output) is feed into PHENIX it can be used directly for refinement or for the structure determination used on anomalous data or else. Tom will correct me if I am wrong, but there is a local scaling hiding somewhere inside Solve-Resolve which may produce significantly better scaled data. I do not see why the same can not be done with XDS data. Dr Felix Frolow Professor of Structural Biology and Biotechnology Department of Molecular Microbiology and Biotechnology Tel Aviv University 69978, Israel Acta Crystallographica F, co-editor e-mail: [email protected] Tel: ++972-3640-8723 Fax: ++972-3640-9407 Cellular: 0547 459 608 On Jun 10, 2011, at 18:24 , R.M. Garavito wrote:
Kay,
We have been quite happy using XDS processed data (including twinned, SAD, and MAD data) with Phenix via CCP4. We just process a data set through CORRECT, then put the XDS_ASCII.HKL file through Pointless (often to reindex different runs or crystals to the same reference data set). We scale the resulting MTZ file with Scala and the MTZ file with the merged/scaled data is used as input to Phenix. It is quick (~10-15 min depending) and works fine. Other then needing 3 different program suites, I don't see anything wrong with this route for routine work.
Cheers,
Michael
**************************************************************** R. Michael Garavito, Ph.D. Professor of Biochemistry & Molecular Biology 513 Biochemistry Bldg. Michigan State University East Lansing, MI 48824-1319 Office: (517) 355-9724 Lab: (517) 353-9125 FAX: (517) 353-9334 Email: [email protected] ****************************************************************
On Jun 10, 2011, at 11:08 AM, Kay Diederichs wrote:
Dear Ralf, dear all,
Recently there was some discussion about reading of XDS output files here on PhenixBB, and I'd like to try and shed some light on this.
First, I'm glad to hear this is possible in principle! However, the caveat is, that three types of "native" output files are written by programs from the "XDS program package":
a) the main program ("xds", or "xds_par") writes only one type of file, called XDS_ASCII.HKL. This has _unmerged_ observations, and this is therefore currently _unsuitable_ for reading by Phenix programs (since the Phenix routine expects merged, unique reflections). If this is nevertheless tried, I see the danger that Phenix might "eat" all observations of a unique reflections except one, possibly without telling about this problem (? - I didn't try!!) - this is definitely not what you want! b) the program "xscale" can write an output file with unmerged observations. This is the default, and corresponds to MERGE=FALSE in XSCALE.INP . As for a), this is also not what you want! c) the program "xscale" can write an output file with merged observations. This is not the default, but can be obtained by using MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix programs.
Thus only possibility c) will give you a "native XDS" file that is ready to be read by Phenix programs.
In addition to this possibility, one can use of course use "xdsconv" to go from XDS_ASCII.HKL (or any file written by "xscale") to a (non-native XDS) filetype that Phenix programs can read. The most general should be to write a MTZ file, but CNS/X-PLOR type files can also be produced.
If someone would implement merging of observations in the Phenix routines (it should be straightforward: negative sigmas indicate that observations should not be used; otherwise the standard formulas could be used; symmetry-related reflections of course have to be handled correctly but they are already sorted by unique reflection), then possibilities a) and b) would also work. But until this is implemented, c) should be used.
Hope this helps,
Kay
P.S. The filetypes and everything else around XDS is accurately documented at http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/ -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: [email protected] Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz
This e-mail is digitally signed. If your e-mail client does not have the necessary capabilities, just ignore the attached signature "smime.p7s".
_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
Hi Felix,
In addition, if the experimental data (such as merged or unmerged SCALEPACK output) is feed into PHENIX it can be used directly for refinement or for the structure determination used on anomalous data or else. Tom will correct me if I am wrong, but there is a local scaling hiding somewhere inside Solve-Resolve which may produce significantly better scaled data.
Solve includes local scaling. Integrating solve algorithms is technically a little involved (it used to be a Fortran program and was converted only recently; but we still have some ways to go modularizing the algorithms it contains).
I do not see why the same can not be done with XDS data.
Yes... sigh... if only we didn't have the other about 10k similarly important things waiting for us. Ralf
Hi Kay, Thanks for the explanations! a) the main program ("xds", or "xds_par") writes only one type of file,
called XDS_ASCII.HKL. This has _unmerged_ observations, and this is therefore currently _unsuitable_ for reading by Phenix programs (since the Phenix routine expects merged, unique reflections). If this is nevertheless tried, I see the danger that Phenix might "eat" all observations of a unique reflections except one, possibly without telling about this problem (? - I didn't try!!) - this is definitely not what you want!
We have a simple way of merging symmetry-equivalent observations, which is used (for example) when we read unmerged scalepack files.
b) the program "xscale" can write an output file with unmerged observations. This is the default, and corresponds to MERGE=FALSE in XSCALE.INP . As for a), this is also not what you want!
I've never been sure about the tradeoff between convenience and correctness. We could read this file type and merge the observations ourselves, which is convenient, but is the result as good as letting XDS (or scalepack or scala) do the merging?
c) the program "xscale" can write an output file with merged observations. This is not the default, but can be obtained by using MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix programs.
Thus only possibility c) will give you a "native XDS" file that is ready to be read by Phenix programs.
A reader for this file type has been part of Phenix for many years: http://cci.lbl.gov/cctbx_sources/iotbx/xds/read_ascii.py Does this still look right?
In addition to this possibility, one can use of course use "xdsconv" to go from XDS_ASCII.HKL (or any file written by "xscale") to a (non-native XDS) filetype that Phenix programs can read. The most general should be to write a MTZ file, but CNS/X-PLOR type files can also be produced.
Great to know that you can export to MTZ files; I wasn't aware of this before. This looks like the best option in most cases, since the symmetry information is preserved in MTZ files (but not CNS files). Ralf
Am 10.06.2011 18:27, schrieb Ralf Grosse-Kunstleve:
Hi Kay,
Thanks for the explanations!
a) the main program ("xds", or "xds_par") writes only one type of file, called XDS_ASCII.HKL. This has _unmerged_ observations, and this is therefore currently _unsuitable_ for reading by Phenix programs (since the Phenix routine expects merged, unique reflections). If this is nevertheless tried, I see the danger that Phenix might "eat" all observations of a unique reflections except one, possibly without telling about this problem (? - I didn't try!!) - this is definitely not what you want!
We have a simple way of merging symmetry-equivalent observations, which is used (for example) when we read unmerged scalepack files.
this sounds useful - where can I see this? If this could be enabled then there is no reason not to read files of type a) or b) .
b) the program "xscale" can write an output file with unmerged observations. This is the default, and corresponds to MERGE=FALSE in XSCALE.INP . As for a), this is also not what you want!
I've never been sure about the tradeoff between convenience and correctness. We could read this file type and merge the observations ourselves, which is convenient, but is the result as good as letting XDS (or scalepack or scala) do the merging?
I can only speak for XDS - and that internally does just the obviously correct thing: a) reject observations with negative sigma b) use the standard formulas - see http://en.wikipedia.org/wiki/Weighted_mean#Dealing_with_variance I'd be surprised if SCALEPACK and SCALA use different formulas.
c) the program "xscale" can write an output file with merged observations. This is not the default, but can be obtained by using MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix programs.
Thus only possibility c) will give you a "native XDS" file that is ready to be read by Phenix programs.
A reader for this file type has been part of Phenix for many years:
http://cci.lbl.gov/cctbx_sources/iotbx/xds/read_ascii.py
Does this still look right?
yes - except it should reject observations with negative sigma (maybe I'm overlooking it) but these probably don't occur in MERGE=TRUE files. and yes - it seems to check that "MERGE=TRUE" so I guess it will complain in case a) and b) .
In addition to this possibility, one can use of course use "xdsconv" to go from XDS_ASCII.HKL (or any file written by "xscale") to a (non-native XDS) filetype that Phenix programs can read. The most general should be to write a MTZ file, but CNS/X-PLOR type files can also be produced.
Great to know that you can export to MTZ files; I wasn't aware of this
in fact xdsconv writes the necessary files, and prints a few lines of output to the screen that just have to be cut-and-pasted to get an MTZ file. Of course this could also easily be scripted.
before. This looks like the best option in most cases, since the symmetry information is preserved in MTZ files (but not CNS files).
Ralf
thanks, Kay
Hi Ralf and Kay, One small additional note: autosol can read unmerged MTZ or .sca files (without using the reflection_reader that merges these reflections) and takes advantage of the extra information that they offer for local scaling. This could be extended to XDS or other files as well. All the best, Tom T
Am 10.06.2011 18:27, schrieb Ralf Grosse-Kunstleve:
Hi Kay,
Thanks for the explanations!
a) the main program ("xds", or "xds_par") writes only one type of file, called XDS_ASCII.HKL. This has _unmerged_ observations, and this is therefore currently _unsuitable_ for reading by Phenix programs (since the Phenix routine expects merged, unique reflections). If this is nevertheless tried, I see the danger that Phenix might "eat" all observations of a unique reflections except one, possibly without telling about this problem (? - I didn't try!!) - this is definitely not what you want!
We have a simple way of merging symmetry-equivalent observations, which is used (for example) when we read unmerged scalepack files.
this sounds useful - where can I see this? If this could be enabled then there is no reason not to read files of type a) or b) .
b) the program "xscale" can write an output file with unmerged observations. This is the default, and corresponds to MERGE=FALSE in XSCALE.INP . As for a), this is also not what you want!
I've never been sure about the tradeoff between convenience and correctness. We could read this file type and merge the observations ourselves, which is convenient, but is the result as good as letting XDS (or scalepack or scala) do the merging?
I can only speak for XDS - and that internally does just the obviously correct thing: a) reject observations with negative sigma b) use the standard formulas - see http://en.wikipedia.org/wiki/Weighted_mean#Dealing_with_variance
I'd be surprised if SCALEPACK and SCALA use different formulas.
c) the program "xscale" can write an output file with merged observations. This is not the default, but can be obtained by using MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix programs.
Thus only possibility c) will give you a "native XDS" file that is ready to be read by Phenix programs.
A reader for this file type has been part of Phenix for many years:
http://cci.lbl.gov/cctbx_sources/iotbx/xds/read_ascii.py
Does this still look right?
yes - except it should reject observations with negative sigma (maybe I'm overlooking it) but these probably don't occur in MERGE=TRUE files.
and yes - it seems to check that "MERGE=TRUE" so I guess it will complain in case a) and b) .
In addition to this possibility, one can use of course use "xdsconv" to go from XDS_ASCII.HKL (or any file written by "xscale") to a (non-native XDS) filetype that Phenix programs can read. The most general should be to write a MTZ file, but CNS/X-PLOR type files can also be produced.
Great to know that you can export to MTZ files; I wasn't aware of this
in fact xdsconv writes the necessary files, and prints a few lines of output to the screen that just have to be cut-and-pasted to get an MTZ file. Of course this could also easily be scripted.
before. This looks like the best option in most cases, since the symmetry information is preserved in MTZ files (but not CNS files).
Ralf
thanks,
Kay _______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
Hi, I believe the unmerged data is very important for autosol. I had one case that phenix.autosol gave me a very good map using unmerged data from XSCALE (converted to CNS format by XDSCONV) while no meaningful results using merged data from XSCALE (converted to mtz format by XDSCONV). So it would save me the hassle of format converting if phenix.autosol is able to read unmerged XDS file directly. -- Jianghai On Jun 10, 2011, at 6:48 PM, Thomas C. Terwilliger wrote:
Hi Ralf and Kay,
One small additional note: autosol can read unmerged MTZ or .sca files (without using the reflection_reader that merges these reflections) and takes advantage of the extra information that they offer for local scaling. This could be extended to XDS or other files as well.
All the best, Tom T
Am 10.06.2011 18:27, schrieb Ralf Grosse-Kunstleve:
Hi Kay,
Thanks for the explanations!
a) the main program ("xds", or "xds_par") writes only one type of file, called XDS_ASCII.HKL. This has _unmerged_ observations, and this is therefore currently _unsuitable_ for reading by Phenix programs (since the Phenix routine expects merged, unique reflections). If this is nevertheless tried, I see the danger that Phenix might "eat" all observations of a unique reflections except one, possibly without telling about this problem (? - I didn't try!!) - this is definitely not what you want!
We have a simple way of merging symmetry-equivalent observations, which is used (for example) when we read unmerged scalepack files.
this sounds useful - where can I see this? If this could be enabled then there is no reason not to read files of type a) or b) .
b) the program "xscale" can write an output file with unmerged observations. This is the default, and corresponds to MERGE=FALSE in XSCALE.INP . As for a), this is also not what you want!
I've never been sure about the tradeoff between convenience and correctness. We could read this file type and merge the observations ourselves, which is convenient, but is the result as good as letting XDS (or scalepack or scala) do the merging?
I can only speak for XDS - and that internally does just the obviously correct thing: a) reject observations with negative sigma b) use the standard formulas - see http://en.wikipedia.org/wiki/Weighted_mean#Dealing_with_variance
I'd be surprised if SCALEPACK and SCALA use different formulas.
c) the program "xscale" can write an output file with merged observations. This is not the default, but can be obtained by using MERGE=TRUE in XSCALE.INP . This is what you want to read with Phenix programs.
Thus only possibility c) will give you a "native XDS" file that is ready to be read by Phenix programs.
A reader for this file type has been part of Phenix for many years:
http://cci.lbl.gov/cctbx_sources/iotbx/xds/read_ascii.py
Does this still look right?
yes - except it should reject observations with negative sigma (maybe I'm overlooking it) but these probably don't occur in MERGE=TRUE files.
and yes - it seems to check that "MERGE=TRUE" so I guess it will complain in case a) and b) .
In addition to this possibility, one can use of course use "xdsconv" to go from XDS_ASCII.HKL (or any file written by "xscale") to a (non-native XDS) filetype that Phenix programs can read. The most general should be to write a MTZ file, but CNS/X-PLOR type files can also be produced.
Great to know that you can export to MTZ files; I wasn't aware of this
in fact xdsconv writes the necessary files, and prints a few lines of output to the screen that just have to be cut-and-pasted to get an MTZ file. Of course this could also easily be scripted.
before. This looks like the best option in most cases, since the symmetry information is preserved in MTZ files (but not CNS files).
Ralf
thanks,
Kay _______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
participants (7)
-
Felix Frolow -
Jianghai Zhu -
Kay Diederichs -
Kay Diederichs
-
R.M. Garavito -
Ralf Grosse-Kunstleve -
Thomas C. Terwilliger