Recover Intensity without Background Subtraction

Software DIALS
1

I have a reflection table from stills_process (source: CXIDB ID 81) with the following columns. Can I recover an estimate of the intensity prior to background subtraction? Should it be a simple arithmetic transform involving intensity.sum.value, background.sum.value, and maybe some of the num_pixels entries? Maybe @phyy-nx or @rahewitt would know?

[nav] In [11]: list(refl.keys()) 
Out[11]:                         
['background.dispersion',        
 'background.mean',              
 'background.mse',               
 'background.sum.value',         
 'background.sum.variance',      
 'bbox',                         
 'd',                            
 'delpsical.rad',                
 'entering',                     
 'flags',                        
 'id',                           
 'intensity.sum.value',          
 'intensity.sum.variance',       
 'miller_index',                 
 'num_pixels.background',        
 'num_pixels.background_used',   
 'num_pixels.foreground',        
 'num_pixels.valid',             
 'panel',                        
 'partial_id',                   
 'partiality',                   
 's1',                           
 'xyzcal.mm',                    
 'xyzcal.px',                    
 'xyzobs.mm.value',              
 'xyzobs.mm.variance',           
 'xyzobs.px.value',              
 'xyzobs.px.variance']
2

I think if you want true raw counts, the best thing to do would be to store shoeboxes and then use something like the following:

totals = []

for sb in reflections["shoebox"]:
    data = sb.data 
    
    # Sum all pixels
    total = data.sum()
    
    totals.append(total)

If you want details of what is in the foreground and background, use the MaskCode. You may also need to do this if you have masked bad pixels in your shoeboxes to exclude them. My understanding is that the intensity.sum.value can be changed by corrections that may/may not be applied in dials.stills_process (@phyy-nx can chime in here), and that background.sum.value can reflect the background model which may not be strictly accurate to summing with intensity.sum.value to the raw photon counts.

3

For more context, the num_pixels.* columns are just a raw count of how many pixels follow certain MaskCode values. If you just need an intensity estimate and not the raw photon counts, then intensity.sum.value + background.sum.value should be sufficient, but note that the worse the signal is relative to background, the more error-prone this estimate will be due to background modelling inaccuracies. For strong reflections I wouldn’t worry too much about it but for weak reflections I would be concerned.

4

dials.stills_process (or d.sp if you are feeling spicy) uses the linear2d model for background, which fits a plane and then subtracts the plane from every pixel prior to summation of the foreground pixels. I don’t think you can unwind that without the shoebox, as @rahewitt says.

Are these the indexed.refl files? Those should have shoeboxes. If they are integrated.refl files, those have had the shoeboxes removed on purpose because they take up a lot of file space. If you can reintegrate, you can specify integration.debug.output=True and integration.debug.separate_files=False to retain the shoeboxes.

5

Thanks both! This answers my question. I will try re-integrating some data and retaining shoeboxes.