Spotfinding DIALS `.expt` Files with Many Stills

Software DIALS
1

I have a dataset with 5332 still images. I can run dials.import on them and get 5332 stills, but am routinely running into Out-Of-Memory errors even with 120 GB of memory during spotfinding. Are there any tips for dealing with this?

For the time being I am using dials.split_experiments and dials.combine_experiments to process 100 images at a time, but it would be useful for myself and users if I could keep this all in one file without needing to write Bash logic to split and combine files during processing.

2

Are you running dials.stills_process? If not, try it! If so, try setting composite_mode=False. This will generate way more files but won’t keep data in memory. Of course, you can also use mpi with multiple nodes, that will distribute the load and alleviate memory requirements. But that option isn’t always available.