I can definitely see the advantages of a second pass with a consensus cell.  There will always be weak, split or contaminated images that could benefit from the prior information.  But in this case I cranked up the spot intensity to be just shy of overloads and we have beautiful, clearly-single-crystal patterns.  If you can't index these then something is wrong.  I don't like comparing software packages like this because it is hard not to sound insulting to at least one group of authors, but I hope we can all accept that if we have a clear and high-quality image and both mosflm and XDS can get the correct cell from it, then there is no reason why DIALS shouldn't. 

I have now prepared a 2-image example based on seed 56.  In this case rotating the crystal by 0.01 degrees makes the difference between dials.stills_process going from confidently arriving at a very wrong cell in fft1d,fft3d,real_space_grid_search mode to instead arriving at the correct one. In both orientations mosflm and xds arrive at the correct cell.  Files here:

http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020418.tgz

I believe the synthetic images do help nail down where the problem might be. It can't be finding too few spots, because the spots are clear.  It can't be splitting, ice, zingers or other outlier spots, because I didn't put those in.  It can't be the beam center because its in the center of the image and the spot separation is nice and wide. It has to be something in the indexing or cell reduction algorithms. 

Comparing the logs dials_000.log vs dials_001.log, the problem seems to arrive as early as the "Candidate basis vectors".  A 4th ~8 A basis vector is found and seems to win out over the others for reasons that aren't really obvious to me.  Where is the code for this step?

-James

Right, typically if I don't know the cell, I'll index a bunch of images with no target, then use cluster.unit_cell on the results to get a consensus cell, then re-index with that cell.

For dials.stills_process, the default of fft1d works pretty well generally, but to get the best results, indexing.method_list=fft1d,fft3d might be better.  There, if the first method fails the second is tried.  real_space_grid_search can also be used.

Getting a specific image in a composite file (such as hdf5) isn't a use case I've hit often, so I don't have an easy solution there.  It's possible an image range parameter to dials.import would be useful?

-Aaron

On Sat, Feb 3, 2018 at 1:02 AM, Graeme Winter <[email protected]> wrote:

Hi Aaron

(and DIALS)

For stills / single images - should dials.index switch by magic to default fft1d? 

Of is that the responsibility of e.g. xia2 / cctbx.xfel to worry about?

It has to be said, more often than not, particularly for screening, we do not know cell / symmetry a priori (at diamond anyways) 

A related question

If I have a data block / strong pickle from 1000 stills, how can I index e.g. image 672? Do I have to split this into 1000 separate things?

@JH - thanks for this - I think some rigorous testing will be useful here. 

Cheers Graeme

On 3 Feb 2018, at 08:37, Aaron Brewster <[email protected]> wrote:

Hi James, thanks.  I looked at random image 15 and after trying a bunch of stuff it indexes fine if I use indexing.method=fft3d.  The default for stills processing is fft1d, so I find this super interesting.

For what it's worth, I pretty much never process stills without a target cell.  I generated ~30 random images using your script and indexed them, then I got the unit cells with "dials.show *refined*.json | grep cell".  They were all about the same.  So I re-indexed random image 15 using fft1d but I also specified known_symmetry.cell=50,60,70,90,90,90 and it indexed fine.

So, either indexing.method=fft3d or known_symmetry.cell=50,60,70,90,90,90 is sufficient to index random image 15.  Certainly for experimental data I would recommend using known_symmetry as the results are always better.  However, I'd like to look deeper at fft1d vs fft3d at some point.

Cc'ing dials-support.

Thanks!

-Aaron

On Fri, Feb 2, 2018 at 1:35 PM, James Holton <[email protected]> wrote:

While testing the new simTBX diffraction image simulator Aaron and I found that some rather beautiful-looking images still won't index.  Subsequently, I followed up with mosflm, which does succeed in these cases, even without prior knowledge of the cell or space group.

I have now run this 1000 times, varying only the crystal orientation, and found 171 cases of dials.stills_process failing to index the simulated image whereas mosflm had no trouble, and 5 cases where mosflm failed and dials.stills_process succeeded.  There were no overlapping cases where both programs failed.

I have tarballed up the relevant files here:
http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz

The tarball contains a "runme.com" shell script for reproducing these results, and also the tst_nanoBragg_forindex.py jiffy for making a test image given a provided random-number seed.  For example:

libtbx.python ./tst_nanoBragg_forindex.py random 15

will create a file called noiseimage_001.cbf that dials.stills_process cannot index (at least in my hands).  For those of you who don't have mosflm installed, the tarball incluses a copy, and the "autoindex.com" script should run on any linux system.

Any ideas as to why there is this discrepancy?  Feels like an opportunity for a non-trivial improvement.

Something to ponder over the weekend, I suppose.  Note that I recently checked in a few bug fixes to simTBX, but this analysis is unaffected by them, so you shouldn't have to update your build to reproduce this.

Cheers,

-James Holton

MAD Scientist

_______________________________________________
cctbxbb mailing list
[email protected]
http://phenix-online.org/mailman/listinfo/cctbxbb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
DIALS-support mailing list
DIALS-support@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dials-support

_______________________________________________
cctbxbb mailing list
[email protected]
http://phenix-online.org/mailman/listinfo/cctbxbb