<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<p>I can definitely see the advantages of a second pass with a
consensus cell. There will always be weak, split or contaminated
images that could benefit from the prior information. But in this
case I cranked up the spot intensity to be just shy of overloads
and we have beautiful, clearly-single-crystal patterns. If you
can't index these then something is wrong. I don't like comparing
software packages like this because it is hard not to sound
insulting to at least one group of authors, but I hope we can all
accept that if we have a clear and high-quality image and both
mosflm and XDS can get the correct cell from it, then there is no
reason why DIALS shouldn't. <br>
</p>
<p>I have now prepared a 2-image example based on seed 56. In this
case rotating the crystal by 0.01 degrees makes the difference
between dials.stills_process going from confidently arriving at a
very wrong cell in fft1d,fft3d,real_space_grid_search mode to
instead arriving at the correct one. In both orientations mosflm
and xds arrive at the correct cell. Files here:</p>
<p><a class="moz-txt-link-freetext" href="http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020418.tgz">http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020418.tgz</a><br>
</p>
<p>I believe the synthetic images do help nail down where the
problem might be. It can't be finding too few spots, because the
spots are clear. It can't be splitting, ice, zingers or other
outlier spots, because I didn't put those in. It can't be the
beam center because its in the center of the image and the spot
separation is nice and wide. It has to be something in the
indexing or cell reduction algorithms. <br>
</p>
<p>Comparing the logs dials_000.log vs dials_001.log, the problem
seems to arrive as early as the "Candidate basis vectors". A 4th
~8 A basis vector is found and seems to win out over the others
for reasons that aren't really obvious to me. Where is the code
for this step?<br>
</p>
<p>-James</p>
<br>
<div class="moz-cite-prefix">On 2/3/2018 1:11 AM, Aaron Brewster
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAAsS-D-xVazx-T1S408f9Up_gYOLpDOu3z3FLhOjNFR=M4fxjQ@mail.gmail.com">
<div dir="ltr">Right, typically if I don't know the cell, I'll
index a bunch of images with no target, then use
cluster.unit_cell on the results to get a consensus cell, then
re-index with that cell.
<div><br>
</div>
<div>For dials.stills_process, the default of fft1d works pretty
well generally, but to get the best results,
indexing.method_list=fft1d,fft3d might be better. There, if
the first method fails the second is tried.
real_space_grid_search can also be used.</div>
<div><br>
</div>
<div>Getting a specific image in a composite file (such as hdf5)
isn't a use case I've hit often, so I don't have an easy
solution there. It's possible an image range parameter to
dials.import would be useful?</div>
<div><br>
</div>
<div>-Aaron</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Feb 3, 2018 at 1:02 AM, Graeme
Winter <span dir="ltr"><<a
href="mailto:graeme.winter@gmail.com" target="_blank"
moz-do-not-send="true">graeme.winter@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
style="word-wrap:break-word;line-break:after-white-space">Hi
Aaron
<div><br>
</div>
<div>(and DIALS)</div>
<div><br>
</div>
<div>For stills / single images - should dials.index
switch by magic to default fft1d? </div>
<div><br>
</div>
<div>Of is that the responsibility of e.g. xia2 /
cctbx.xfel to worry about?</div>
<div><br>
</div>
<div>It has to be said, more often than not, particularly
for screening, we do not know cell / symmetry a priori
(at diamond anyways) </div>
<div><br>
</div>
<div>A related question</div>
<div><br>
</div>
<div>If I have a data block / strong pickle from 1000
stills, how can I index e.g. image 672? Do I have to
split this into 1000 separate things?</div>
<div><br>
</div>
<div>@JH - thanks for this - I think some rigorous testing
will be useful here. </div>
<div><br>
</div>
<div>Cheers Graeme<br>
<div><br>
<blockquote type="cite">
<div>
<div class="h5">
<div>On 3 Feb 2018, at 08:37, Aaron Brewster
<<a href="mailto:asbrewster@lbl.gov"
target="_blank" moz-do-not-send="true">asbrewster@lbl.gov</a>>
wrote:</div>
<br
class="m_2250465256792134629Apple-interchange-newline">
</div>
</div>
<div>
<div>
<div class="h5">
<div dir="ltr">Hi James, thanks. I looked at
random image 15 and after trying a bunch of
stuff it indexes fine if I use
indexing.method=fft3d. The default for
stills processing is fft1d, so I find this
super interesting.
<div><br>
</div>
<div>For what it's worth, I pretty much
never process stills without a target
cell. I generated ~30 random images using
your script and indexed them, then I got
the unit cells with "dials.show
*refined*.json | grep cell". They were
all about the same. So I re-indexed
random image 15 using fft1d but I also
specified known_symmetry.cell=50,60,70,9<wbr>0,90,90
and it indexed fine.</div>
<div><br>
</div>
<div>So, either <span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">indexing.method=fft3d
or <span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">known_symmetry.cell=50,60,7<wbr>0,90,90,90
is sufficient to index random image
15. Certainly for experimental data I
would recommend using known_symmetry
as the results are always better.
However, I'd like to look deeper at
fft1d vs fft3d at some point.</span></span></div>
<div><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></span></div>
<div><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Cc'ing
dials-support.</span></span></div>
<div><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></span></div>
<div>Thanks!</div>
<div>-Aaron</div>
<div><br>
</div>
<div><br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Feb 2,
2018 at 1:35 PM, James Holton <span
dir="ltr"><<a
href="mailto:jmholton@lbl.gov"
target="_blank"
moz-do-not-send="true">jmholton@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>While testing the
new simTBX
diffraction image
simulator Aaron and
I found that some
rather
beautiful-looking
images still won't
index.
Subsequently, I
followed up with
mosflm, which does
succeed in these
cases, even without
prior knowledge of
the cell or space
group.<br>
<br>
</div>
I have now run this
1000 times, varying
only the crystal
orientation, and found
171 cases of
dials.stills_process
failing to index the
simulated image
whereas mosflm had no
trouble, and 5 cases
where mosflm failed
and
dials.stills_process
succeeded. There were
no overlapping cases
where both programs
failed. <br>
<br>
</div>
I have tarballed up the
relevant files here:<br>
<a
href="http://bl831.als.lbl.gov/%7Ejamesh/bugreports/dials_index_020218.tgz"
target="_blank"
moz-do-not-send="true">http://bl831.als.lbl.gov/~jame<wbr>sh/bugreports/dials_index_0202<wbr>18.tgz</a><br>
<br>
</div>
The tarball contains a "<a
href="http://runme.com/"
target="_blank"
moz-do-not-send="true">runme.com</a>"
shell script for
reproducing these results,
and also the
tst_nanoBragg_forindex.py
jiffy for making a test
image given a provided
random-number seed. For
example:<br>
<br>
</div>
libtbx.python
./tst_nanoBragg_forindex.py
random 15<br>
<br>
</div>
will create a file called
noiseimage_001.cbf that
dials.stills_process cannot
index (at least in my hands).
For those of you who don't
have mosflm installed, the
tarball incluses a copy, and
the "<a
href="http://autoindex.com/"
target="_blank"
moz-do-not-send="true">autoindex.com</a>"
script should run on any linux
system.<br>
<br>
</div>
Any ideas as to why there is
this discrepancy? Feels like an
opportunity for a non-trivial
improvement.<br>
<br>
</div>
Something to ponder over the
weekend, I suppose. Note that I
recently checked in a few bug
fixes to simTBX, but this analysis
is unaffected by them, so you
shouldn't have to update your
build to reproduce this.<br>
<br>
</div>
Cheers,<br>
<div><br>
-James Holton<br>
</div>
<div>MAD Scientist<br>
<br>
</div>
<div><br>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
cctbxbb mailing list<br>
<a
href="mailto:cctbxbb@phenix-online.org"
target="_blank"
moz-do-not-send="true">cctbxbb@phenix-online.org</a><br>
<a
href="http://phenix-online.org/mailman/listinfo/cctbxbb"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://phenix-online.org/mailm<wbr>an/listinfo/cctbxbb</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
------------------------------<wbr>------------------------------<wbr>------------------<br>
Check out the vibrant tech community on one of the
world's most<br>
engaging tech sites, <a
href="http://Slashdot.org" target="_blank"
moz-do-not-send="true">Slashdot.org</a>! <a
href="http://sdm.link/slashdot_______________________________________________"
target="_blank" moz-do-not-send="true">http://sdm.link/slashdot______<wbr>______________________________<wbr>___________</a><br>
DIALS-support mailing list<br>
<a
href="mailto:DIALS-support@lists.sourceforge.net"
target="_blank" moz-do-not-send="true">DIALS-support@lists.<wbr>sourceforge.net</a><br>
<a
href="https://lists.sourceforge.net/lists/listinfo/dials-support"
target="_blank" moz-do-not-send="true">https://lists.sourceforge.net/<wbr>lists/listinfo/dials-support</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
cctbxbb mailing list
<a class="moz-txt-link-abbreviated" href="mailto:cctbxbb@phenix-online.org">cctbxbb@phenix-online.org</a>
<a class="moz-txt-link-freetext" href="http://phenix-online.org/mailman/listinfo/cctbxbb">http://phenix-online.org/mailman/listinfo/cctbxbb</a>
</pre>
</blockquote>
<br>
</body>
</html>