DIALS Stills Processing Notes

Beamtime Processing ESRF 2025
1

refine.phil


refinement {
  parameterisation {
    auto_reduction {
      min_nref_per_parameter = 3
      action = fail fix *remove
    }
    beam {
      fix = *all in_spindle_plane out_spindle_plane wavelength
    }
    detector {
      fix_list = Group1Tau1
      hierarchy_level = 1
    }
    crystal.fix = all
  }
  refinery {
    engine = SimpleLBFGS LBFGScurvs GaussNewton LevMar *SparseLevMar
  }
  reflections {
    outlier {
      algorithm = null auto mcd tukey *sauter_poon
      separate_experiments = False
      separate_panels = True
    }
  }
}
2

Full DIALS Workflow using S3DF

Start session with S3DF OnDemand
Open terminal

Activate DIALS and set the split detector format


source /sdf/group/lcls/ds/tools/cctbx/setup.sh

export SPLIT_ESRF_JUNGFRAU=1

Can check that the h5s were properly reformatted with

dxtbx.print_header /path/to/hdf5_master

Do initial indexing - put in unit cell and space group parameters in process.phil

cd processing_directory
dials.stills_process /sdf/data/lcls/ds/prj/prjesrf0125/hdf5/Hsp31/0ms/run_05_ssx_injector_collection/aggregated/master_0ms-Hsp31_dense.h5 process.phil mp.nproc=120

Do metrology refinement - if you have many hits in one run, you must combine the frames in chunks

mkdir metrology
cd metrology

dials.combine_experiments ../idx-0???_indexed.refl ../idx-0???_refined.expt reference_from_experiment.detector=0 reference_from_experiment.beam=0 

dials.refine combined.expt combined.refl refine.phil

OR if you have too many frames

dials.combine_experiments ../idx-0???_indexed.refl ../idx-0???_refined.expt reference_from_experiment.detector=0 reference_from_experiment.beam=0 max_batch_size=2000

dials.refine combined_004.{expt,refl} refine.phil output.experiments=refined_004.expt output.reflections=refined_004.refl

cctbx.xfel.recompute_mosaicity refined_019* output.experiments=refinedmos_019.expt output.reflections=refinedmos_019.refl

mpirun n -1 cctbx.xfel.mpi_integrate input.path=refinedmos_0* integrate.phil input.experiments_suffix=.expt input.reflections_suffix=.refl integration.integrator=stills

Check how much the detector model changed

dxtbx.plot_detector_models combined.expt refined.expt

cctbx.xfel.detector_residuals refined.* hierarchy=1

Check unit cell

dials.unit_cell_histogram combined.expt

Re-process with updated unit cell and detector model

mkdir reprocess
cd reprocess 
vi process.phil (update detector geometry and unit cell)
dials.stills_process /sdf/data/lcls/ds/prj/prjesrf0125/hdf5/Hsp31_foil/filtered/run_01_ssx_foil_collection/aggregated/master_filtered-Hsp31_foil_dense.h5 process.phil mp.nproc=120

Repeat metrology refinement and re-processing until the detector model converges

cd refine

dials.combine_experiments ../idx-0???_indexed.refl ../idx-0???_refined.expt reference_from_experiment.detector=0 reference_from_experiment.beam=0

dials.refine combined.expt combined.refl refine.phil

dxtbx.plot_detector_models combined.expt refined.expt

dials.unit_cell_histogram combined.expt

cctbx.xfel.detector_residuals refined.* hierarchy=1

Then, process one last time but add multi-lattice

dials.stills_process /sdf/data/lcls/ds/prj/prjesrf0125/hdf5/Hsp31_foil/unfiltered/run_01_ssx_foil_collection/aggregated/master_unfiltered_01-Hsp31_foil_dense.h5 process.phil mp.nproc=120 multiple_lattice_search.max_lattices=2

Repeat this process for every run individually, then scale and merge

mpirun -n 120 --oversubscribe cctbx.xfel.merge merge.phil