I’m curious about prediction in idexamajig and outlier rejection in partialator. I’m working on a dataset from CXIDB (CXIDB - Coherent X-ray Imaging Data Bank) to benchmark my merging code.
Here are the indexamajig params from the stream: CrystFEL stream format 2.3 Generated by CrystFEL 0.6.1 /home/kyotoyaku/yamashita/crystfel_builds/15ll0907_0.6.1+6aeb5402/build/bin/indexamajig -i /home/kyotoyaku/lfs01/2015-apr/se_pg250_forpaper/4_nomask/297132-0/test_11//../files.lst -g each.geom --fix-bandwidth=5.2e-3 --fix-divergence=4.0e-4 --indexing=dirax --peaks=zaef --threshold=500 --min-gradient=200000 --min-snr=1 --int-radius=4,6,8 -o out.stream -p /home/kyotoyaku/lfs01/2015-apr/se_pg250_forpaper/4_nomask/297132-0/test_11//../../merge_1_median.cell -j 12 --integration=rings-nocen-sat
It seems like this config overpredicts reflections. If I bin by the angular separation between the spot and the Ewald sphere, i find there are large number of somewhat bright reflections far off the diffracting condition.
Is this typical of indexamajig predictions or a product of specific way it was run? I can see an argument for overpredicting at this stage. It’s easy to discard reflections later.
Why do the reflections tail up as they get further away from the ewald sphere? Are these predictions bumping into other reflections?
Does partialator have any safeguards to reject reflections like these that are overly bright and way out in the tails?
Is any of this even relevant anymore? This is quite an old version of CrystFEL.
For historical context, CrystFEL 0.6.1 is from 2015, pre-dating even peakFinder8, and was the first version to do refinement of the indexing solution at all. You can therefore count on the indexing solutions being a bit wonky, which leads to over-prediction as it tries to match the observed spots using a slightly wrong orientation. Automatic profile radius determination was also a new and experimental feature at the time, having been added in version 0.6.0. There are non-negligible bandwidth and divergence values in the parameters (0.5% and 0.4 mrad), which would also increase the amount of over-prediction. On top of that, the detector geometry was probably not so good at the time. I think the amount of over-prediction would go down a lot if you were to re-process with recent CrystFEL and after running the geometry file through align_detector.
There’s no per-reflection rejection like you describe, because it always seemed to me that there’s no way to tell, based just on the intensities, between the following two cases:
A strong reflection that was hit for real once, amongst many weak measurements because of over-prediction.
A weak reflection that was correctly measured many times, together with a small number of outlier high measurements from mis-indexing/bad pixels/etc.
The way we answer this question, of course, is by getting really good at correctly predicting reflection locations. I’d be happy to discover that we can now trust the prediction enough that Case 1 is insignificant, and so assume Case 2 and eliminate outliers. But I haven’t checked myself, and I’m quite certain it wasn’t the case in 2015. The “blue sausage problem” [1] is still limiting prediction.
As for why the intensity goes up away from the Bragg condition, I’m not sure. You’ve calculated the offset in degrees, though. Does it go away if you calculate it as reciprocal space distance instead?
Thanks for the historical context! This is valuable info. I don’t have a plot handy right now, but I believe the trend disappears if you calculate the Ewald offset in inverse Angstroms.
