Hi Joe,<br><br>There was a very fruitful discussion about R-free and its size in ccp4bb and this was summerized (including views from Read and from Cowtan) in <a href="http://strucbio.biologie.uni-konstanz.de/ccp4wiki/index.php/Test_set">http://strucbio.biologie.uni-konstanz.de/ccp4wiki/index.php/Test_set</a><br>
<br>HTH,<br><br>Peter.<br><br><a href="http://strucbio.biologie.uni-konstanz.de/ccp4wiki/index.php/Test_set">http://strucbio.biologie.uni-konstanz.de/ccp4wiki/index.php/Test_set</a><br><br><div class="gmail_quote">On Thu, Oct 1, 2009 at 6:05 PM, Joe Krahn <span dir="ltr"><<a href="mailto:krahn@niehs.nih.gov">krahn@niehs.nih.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I propose that the issue of using test reflections be addressed by more<br>
than just speculation.<br>
<br>
Create a 10% test set for refinement, but divide that into two 5%<br>
groups. One is used as a "pure" test set, and the other subgroup is<br>
allowed to be used in non-refinement tasks. During the course of an<br>
entire structure determination, one can monitor whether the unbiased<br>
test set differs from the biased test set.<br>
<br>
My concern is that if the data is weak enough that including test<br>
reflections is required to interpret a part of the map, then the data is<br>
probably too weak to distinguish a correct model from a model-bias.<br>
<br>
For density modification, it may be possible to converge on a good map<br>
if both missing and test reflections use "Fcalc" fill-in values from the<br>
previous density-modified transformation. Maybe the people who could not<br>
get good results without the test reflections used a DM method that<br>
reset missing values to zero on every cycle.<br>
<div class="im"><br>
Pavel Afonine wrote:<br>
> Hi Joe,<br>
><br>
>> Normally, 5% for R-free is sufficient.<br>
><br>
> did anyone studied this and came to this conclusion (publication?)? I'm<br>
> not aware.<br>
</div>Is there a paper showing that 10% is required or sufficient? Maybe that<br>
needs to be studied as well.<br>
...<br>
<div class="im">>> I am sure that many people will use it when<br>
>> they find that real-space fitting and refinement tools lower R-free,<br>
>> unaware that they are cheating.<br>
>><br>
><br>
> Sure. This is why free-R flags are not used in maps calculation for<br>
> real-space refinement (my previous email).<br>
</div>Yes, but people can still use maps from PHENIX with external real-space<br>
refinement.<br>
<div class="im">><br>
>> With 10% test reflection, I suspect that difference maps used to find<br>
>> waters can easily find a few noise peaks with significant R-free<br>
>> contributions.<br>
><br>
> - I'm not aware of any systematic study on this matter, although I can<br>
> believe it in theory;<br>
> - phenix.refine uses very sophisticated filtering tools;<br>
> - I guess at some point I will switch to using Average Kick Maps for<br>
> water picking. This will remove the noise peaks, and so eliminate the<br>
> problem (I need to test this all, though).<br>
</div>I started wondering about test reflections im maps because I was<br>
removing some bad "waters" and found a bigger increase in Rfree than in<br>
Rwork. That made me wonder if they had a significant component of<br>
test-set density. But, I also have not checked this in detail.<br>
<div><div></div><div class="h5"><br>
Joe Krahn<br>
_______________________________________________<br>
phenixbb mailing list<br>
<a href="mailto:phenixbb@phenix-online.org">phenixbb@phenix-online.org</a><br>
<a href="http://www.phenix-online.org/mailman/listinfo/phenixbb" target="_blank">http://www.phenix-online.org/mailman/listinfo/phenixbb</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Peter<br>