Author(s)

• scale_and_merge: Tom Terwilliger

Purpose

scale_and_merge is a tool for scaling unmerged anomalous data or multiple data files and creating a scaled dataset and two scaled half-datasets. This tool normally is used in combination with anomalous_signal to create a scaled dataset and analyze anomalous signal in a SAD dataset. It can also be used to scale any other group of datasets.

Usage

How scale_and_merge works:

• You supply scale_and_merge with a directory containing datafiles or the name of a single unmerged datafile.
• You can optionally also specify a pair of labels that identifies datasets that are to be kept together. For example if you collected your data as pairs of datafiles with inverse beam geometry, you might have called the members of a pair data_1_0_w1.HKL and data_1_0_w2.HKL, related by w1 and w2.
• scale_and_merge will first check the cell dimensions of all the datasets. Normally it will choose the largest set of similar crystals (you can have it keep all datasets with only_similar_datasets=False). It will also check the anisotropy in all the datafiles and calculate the average anisotropy (to be applied by default to all datafiles before scaling).
• scale_and_merge will then scale all the data together. It will split your datafiles into smaller files if your datafiles contain duplicate measurements of the same indices. The intensities in each datafile will be adjusted for anisotropy to match the average anisotropy of all the datafiles. In this way all the datafiles are matched but the overall character of the data is not changed.
• scale_and_merge will then scale each individual file with local scaling, producing files in the temporary directory with names ending in _scale.mtz. and then merge them all together.
• Merging of the individual datasets is done twice, and then optionally two additional times to optimize anomalous differences.
• In the first merging the individual datasets are simply averaged with weights based on the sigmas for each reflection. Then the individual datasets are compared with the overall mean dataset and dataset variances (estimates of systematic differences between datasets and the mean) are calculated based on the rms differences from the mean and the individual sigmas. In essence the dataset variance plus the individual variances are then to be used as estimates of the total variance for each reflection. The second merging uses these total variances in weighting rather than the original sigmas. The correlations of each individual dataset with the merged dataset are calculated.
• If anomalous differences are optimized (default with optimize_anomalous=True), merging is carried out another time in order to optimize the weighting of anomalous differences in the merging step. For each unique reflection in the asymmetric unit of each dataset, the mean I+ and mean I- are used to calculate anomalous differences. The anomalous differences from each individual dataset are then compared with the anomalous differences from the merged dataset to estimate individual dataset anomalous difference variances. Then the anomalous differences from each individual dataset are averaged, with weights based on the original sigmas and the dataset variances. These merged anomalous differences are then used to replace the anomalous differences in the merged dataset above (for example, a reflection in the merged dataset above that has I+,sigI+,I-,sigI- would get new values of I+ and I- that have a difference equal to the appropriate merged anomalous difference, but the same mean as before.) The correlation of the anomalous differences in the original merged dataset and after optimization is printed (this should be high, for example 0.80).
• The original datasets are then split into two parts for creation of two half-datasets. These half-datasets are useful for estimating the quality of the data and are used in anomalous_signal for this purpose.
• The splitting into half-datasets is done in one of three ways, and the method is chosen based on the number of anomalous differences available for comparison using each method. With each method the data in each half-dataset are scaled just as the entire dataset was scaled. The preferred method is to split by files. Half of the datafiles are used to create each half-dataset. The next preferred method is to split with the first half of each dataset in one half-dataset and the second half in the other. The third preferred method is splitting alternate reflections with each unique index (after mapping to the asymmetric unit) into the two half-datasets. The last method is to randomly assign reflections to the two half-datasets. The reason for this hierarchical approach is that reflections measured close in time, within the same dataset are better matched than those measured further in time or within different datasets. These approaches for splitting the datafiles attempt to pair anomalous differences measured close in time and in the same dataset.
• After half-datasets are obtained, the correlation of anomalous differences between half-datasets is estimated.

Output from scale_and_merge

scale_and_merge provides a summary half-dataset correlations in your dataset. Here is an example with very weak anomalous signal (but present):

Scale and merge...analysis of multi-dataset SAD data
input\_files {
  data = "dd"
  data\_labels = None
  paired\_group\_ids = None
}
output\_files {
  output\_file = "scaled\_data.mtz"
  output\_half\_dataset\_a = "half\_dataset\_a.mtz"
  output\_half\_dataset\_b = "half\_dataset\_b.mtz"
  output\_file\_format = *mtz sca
}
crystal\_info {
  resolution = None
  low\_resolution = None
  space\_group = None
  unit\_cell = None
}
data\_selection {
  minimum\_datafile\_fraction = 0.3
  require\_fpfm = True
  only\_similar\_datasets = True
  relative\_length\_tolerance = None
  absolute\_angle\_tolerance = None
  choose\_optimal\_datasets = False
  sort\_datasets\_by\_anomalous\_cc = True
}
scaling {
  make\_anisotropy\_uniform = True
  overallscale = False
  skip\_scaling = False
  lowest\_resolution\_range = 6
}
merging {
  optimize\_anomalous = True
  use\_best\_group\_as\_target = False
  rescale\_sigmas = False
}
half\_dataset\_cc {
  get\_half\_dataset\_cc = True
  half\_dataset\_cc\_by\_files = True
  split\_as\_first\_second\_half = True
  split\_alternately = True
  split\_randomly = True
}
directories {
  temp\_dir = None
  output\_dir = ""
  gui\_output\_dir = None
}
control {
  verbose = False
  random\_seed = 714215
  comparison\_file = None
  clean\_up = True
}
Comparing crystal symmetries and noting anisotropy in data
Relative length tolerance:  0.010  Angle tolerance: 1.000 degrees
Taking initial symmetry from /Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_205\_w2\_2.9\_90.HKL
with 3 similar files


            File             SG         A       B       C      Alpha   Beta   Gamma

Symmetry group 1
WNV\_NS1\_202\_w1\_2.9\_90.HKL    P 3 2 1  166.997 166.997  94.156  90.000  90.000 120.000
WNV\_NS1\_202\_w2\_2.9\_90.HKL    P 3 2 1  166.704 166.704  94.113  90.000  90.000 120.000
WNV\_NS1\_205\_w1\_2.9\_90.HKL    P 3 2 1  167.578 167.578  93.934  90.000  90.000 120.000
WNV\_NS1\_205\_w2\_2.9\_90.HKL    P 3 2 1  167.639 167.639  93.859  90.000  90.000 120.000

Scaling 4 datasets

Scaling data files with local scaling

Files to scale:
/Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_202\_w1\_2.9\_90.HKL
/Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_202\_w2\_2.9\_90.HKL
/Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_205\_w1\_2.9\_90.HKL
/Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_205\_w2\_2.9\_90.HKL

Selecting just reflections that have both F+ and F- in the same
dataset or are centric

Splitting datafiles into sub-files with one copy of each unique hkl

              File                    Refl       (rejected)   B1      B2     B3        B-avg

TEMP0/WNV\_NS1\_202\_w1\_2.9\_90\_1.sca:   138180     (    6378)    63.9    63.9   55.00      60.9
TEMP0/WNV\_NS1\_202\_w1\_2.9\_90\_2.sca:      229     (       0)    63.9    63.9   55.00      60.9
TEMP0/WNV\_NS1\_202\_w2\_2.9\_90\_1.sca:   136254     (    6096)    59.4    59.4   49.78      56.2
TEMP0/WNV\_NS1\_202\_w2\_2.9\_90\_2.sca:      206     (       0)    59.4    59.4   49.78      56.2
TEMP0/WNV\_NS1\_205\_w1\_2.9\_90\_1.sca:   148714     (    4464)    73.7    73.7   63.92      70.4
TEMP0/WNV\_NS1\_205\_w1\_2.9\_90\_2.sca:      239     (       0)    73.7    73.7   63.92      70.4
TEMP0/WNV\_NS1\_205\_w2\_2.9\_90\_1.sca:   147256     (    4560)    70.3    70.3   60.50      67.0
TEMP0/WNV\_NS1\_205\_w2\_2.9\_90\_2.sca:      241     (       0)    70.3    70.3   60.50      67.0

Notes:
Splitting /Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_202\_w1\_2.9\_90.HKL into 2 files:
Splitting /Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_202\_w2\_2.9\_90.HKL into 2 files:
Splitting /Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_205\_w1\_2.9\_90.HKL into 2 files:
Splitting /Users/terwill/unix/misc/scale\_and\_merge/dd/WNV\_NS1\_205\_w2\_2.9\_90.HKL into 2 files:
Keeping split datafiles with at least 44614 reflections

High-resolution limit:    2.89

Scaling data in batches from individual data files

List of scaled data files:
TEMP0/WNV\_NS1\_205\_w1\_2.9\_90\_1\_scale.mtz (30844 refl)
TEMP0/WNV\_NS1\_205\_w2\_2.9\_90\_1\_scale.mtz (30828 refl)
TEMP0/WNV\_NS1\_202\_w1\_2.9\_90\_1\_scale.mtz (29641 refl)
TEMP0/WNV\_NS1\_202\_w2\_2.9\_90\_1\_scale.mtz (29527 refl)


Scaling and merging data files with overall scale factor

Files to scale:
TEMP0/WNV\_NS1\_205\_w1\_2.9\_90\_1\_scale.mtz
TEMP0/WNV\_NS1\_205\_w2\_2.9\_90\_1\_scale.mtz
TEMP0/WNV\_NS1\_202\_w1\_2.9\_90\_1\_scale.mtz
TEMP0/WNV\_NS1\_202\_w2\_2.9\_90\_1\_scale.mtz

 ****** Putting all data on common scale ******

Standard dataset: 1  I/sigma:    8.30  Nrefl: 59133


Total of 4 datasets to be used

Scale factors for data groups:
 ID    Scale   file\_name
  1  1.000  TEMP0/WNV\_NS1\_205\_w1\_2.9\_90\_1\_scale.mtz
  2  0.683  TEMP0/WNV\_NS1\_205\_w2\_2.9\_90\_1\_scale.mtz
  3  1.198  TEMP0/WNV\_NS1\_202\_w1\_2.9\_90\_1\_scale.mtz
  4  1.366  TEMP0/WNV\_NS1\_202\_w2\_2.9\_90\_1\_scale.mtz

Mean I of scaled unmerged datasets:
1:   9940.78 (N=59133)
2:   9926.98 (N=59085)
3:  10203.75 (N=56663)
4:  10305.51 (N=56463)


Getting overall merged dataset using original sigmas


Datasets to be merged in estimation of variances:  1 2 3 4
Datasets to be merged in final merging step:  1 2 3 4

Getting scale factors, dataset variances and scaled intensities

Merged mean I:  8966.65 (N=64581)


Dataset variances
(RMS difference from target dataset after accounting for sigmas)
                          Dataset
   Resolution        1        2        3        4
 48.40 -  4.94   2923.90  2353.89  6175.92  8461.67
  4.94 -  3.92   1348.27  1407.39  1191.09  1267.29
  3.92 -  3.43      0.00     0.00     0.00     0.00
  3.43 -  3.11      0.00     0.00     0.00     0.00
  3.11 -  2.89      0.00     0.00     0.00     0.00

  ALL            2515.66  1935.84  3450.29  5499.59

Getting merged dataset including dataset variances

Dataset correlations with merged dataset
                          Dataset
   Resolution        1        2        3        4
 48.40 -  4.94      1.00     1.00     1.00     0.99
  4.94 -  3.92      1.00     1.00     1.00     0.99
  3.92 -  3.43      0.99     0.99     0.98     0.97
  3.43 -  3.11      0.93     0.92     0.87     0.85
  3.11 -  2.89      0.84     0.94     0.69     0.55

  ALL               1.00     1.00     1.00     0.99

New merged mean I:  9035.38 (N=64581) and I/sigma:   10.89

NOTE: I/sigma cannot be directly compared to original due to including
estimates of dataset variances and changes in number of reflections.


Optimizing anomalous differences

Merged mean anomalous difference:    -3.48 (N=31009)


Dataset variances (anomalous differences)
(RMS difference from target dataset after accounting for sigmas)
                          Dataset
   Resolution        1        2        3        4
 47.38 -  4.94      0.00  1652.13   993.28     0.00
  4.94 -  3.92    732.44     0.00   503.86     0.00
  3.92 -  3.43    690.71     0.00     0.00     0.00
  3.43 -  3.11    272.68     0.00     0.00     0.00
  3.11 -  2.89      0.00     0.00     0.00     0.00

  ALL             468.90     0.00   435.50     0.00

Getting merged dataset including dataset variances

Dataset correlations with merged dataset (anomalous differences)
                          Dataset
   Resolution        1        2        3        4
 47.38 -  4.94      0.77     0.79     0.69     0.72
  4.94 -  3.92      0.65     0.73     0.64     0.67
  3.92 -  3.43      0.70     0.70     0.63     0.66
  3.43 -  3.11      0.72     0.71     0.70     0.66
  3.11 -  2.89      0.73     0.81     0.71     0.78

  ALL               0.73     0.75     0.67     0.70
Anom correlation on I of std merged and anom scaled:     0.83 (N=31009)
Merged scaled data optimized for anomalous differences: scaled\_data.mtz


==============================================================================

Splitting data into groups for half-dataset CC

Keeping datafiles intact within each half-dataset

Half-dataset groups:

Group A
 TEMP0/WNV\_NS1\_205\_w1\_2.9\_90\_1\_scale.mtz
TEMP0/WNV\_NS1\_202\_w1\_2.9\_90\_1\_scale.mtz

Group B
 TEMP0/WNV\n_NS1\_205\_w2\_2.9\_90\_1\_scale.mtz
TEMP0/WNV\_NS1\_202\_w2\_2.9\_90\_1\_scale.mtz

---- Calculating anomalous CC between half-datasets A,B----

Half-dataset A: half\_dataset\_a.mtz
Half-dataset B: half\_dataset\_b.mtz
Unique reflections:
half\_dataset\_a.mtz: 30242
half\_dataset\_b.mtz: 30222
Reflections in common:    29455
Overall resolution:    2.90 A  Nrefl: 29455
Overall anomalous correlation:   -0.018

Anomalous correlation with varying high-resolution
limits

  d\_min     ---- CC ------       ----  N -----
           Shell   Cumulative   Shell  Cumulative
   6.00     0.080    0.080      3392      3392
   5.50    -0.060    0.043      1052      4444
   5.00    -0.115   -0.006      1519      5963
   4.50    -0.040   -0.018      2256      8219
   4.00    -0.065   -0.036      3540     11759
   3.50    -0.016   -0.026      5862     17621
   3.00    -0.009   -0.018      9997     27618
   2.90    -0.022   -0.018      1837     29455

Possible Problems

Literature

List of all available keywords

• input_files
  • data = None Anomalous data (I+ and I- or F+ and F-). Any standard format is fine. You can have any number of input files. If you have multiple files, specify the directory that contains them and nothing else.
  • data_labels = None Optional label specifying which columns of anomalous data to use. Not necessary if your input files have only one set of anomalous data. If you have multiple input files they need to have the same labels.
  • assume_shelx_observation_type_is = *amplitudes intensities For shelx files you need to specify if they are amplitudes or intensities
  • paired_group_ids = None Optional pair of identifiers that specify datasets that should be grouped together. The paired group ids must be the only characters that differ between datasets that should be grouped together, and the characters must uniquely define what datasets to group. For example paired_group_ids='w1 w2' would be the pair for the two datasets se_w1.sca and se_w2.sca. In this example you could also use paired_group_ids='w1.sca w2.sca' and that would have the same effect. Use this option when some of your datafiles are related to others by inverse-beam data collection for example. This matching of datasets can be very helpful.
  • check_file_type = False You can request that the data type of input files be checked if a data directory is supplied. Default is to read everything in the data directory.
  • reference_data = None Reference data for use in scaling. All datasets will be scaled to this reference dataset. Useful if you have very accurate native data.
  • reference_data_labels = None Optional label specifying which columns of reference data to use.
• output_files
  • output_file = scaled_data.mtz Output data file name for merged scaled data
  • output_half_dataset_a = half_dataset_a.mtz Output data file name for half dataset A. Only written if get_half_datset_cc=True. This is the scaled merged data from half of the input data.
  • output_half_dataset_b = half_dataset_b.mtz Output data file name for half dataset B. Only written if get_half_datset_cc=True. This is the scaled merged data from the other half of the input data.
  • output_file_format = *mtz sca Output format for scaled data. Choices are MTZ and unmerged scalepack formats. Note: unmerged scalepack format not normally used because indices may be changed from original values and that data from within one input file may be merged, so that this unmerged data is partially merged and not the same as the original data.
• crystal_info
  • resolution = None high-resolution limit
  • low_resolution = None Low-resolution limit
  • space_group = None Space group (normally read from the data file)
  • unit_cell = None Unit Cell (normally read from the data file)
• data_selection
  • anomalous = True Consider data as anomalous (otherwise average Bijvoet mates).
  • minimum_datafile_fraction = 0.3 You can reject any datasets (after splitting) that have less than this fraction of the maximum number of reflections found in any datafile
  • require_fpfm = True If set, then when files are split into unique sets of indices only centric reflections and reflections with both F+ and F- present are written out. Note that in most space groups there may be more than one F+ or F- symmetry mate present. If any one is present for F+ and any one for F- the reflection is written out.
  • only_similar_datasets = True Group data by lattice similarity and take biggest group only. If space_group and unit_cell are set, then take those only that match this space_group and unit_cell. Use relative_length_tolerance and absolute_angle_tolerance to decide if two datasets are similar.
  • relative_length_tolerance = None Tolerance for considering two cells similar (fractional) You can allow anything with only_similar_datasets=False
  • resolution_length_tolerance = None Tolerance for considering two cells similar (fraction of resolution. For example normalized_length_tolerance=0.25 means the unit cells must match within 1/4 of the resolution. You can allow anything with only_similar_datasets=False
  • absolute_length_tolerance = None
  • absolute_angle_tolerance = None Tolerance for considering two cell angles similar (degrees) You can allow anything with only_similar_datasets=False
  • choose_optimal_datasets = False After initial merging, sort datasets by correlation to merged dataset, remove datasets until overall I/sigI is maximized, including dataset systematic errors in I/sigI.
  • sort_datasets_by_anomalous_cc = True For choosing optimal datasets, sort the datsets by anomalous CC. Alternative is sorting by non-anomalous CC.
• scaling
  • make_anisotropy_uniform = True Calculate anisotropy for all datasets and adjust all of them to the average anisotropy. This is designed to improve scaling without changing the overall character of the data (i.e., not sharpening the data or removing anisotropy).
  • overallscale = False If set, only use overall scale and B (no local scaling)
  • skip_scaling = False You can skip the scaling step and just get half-dataset CC for a pair of existing files that you have put in a directory by themselves.
  • lowest_resolution_range = 6. Lowest value of the high-resolution cutoff for CC as a function of high-resolution cutoff with constant low-resolution cutoff
  • min_common_refl = 200 Minimum number of reflections in common to scale datasets
  • sort_on_batch_number = True You can sort XDS and MTZ reflections by frame number. Otherwise scale and merge assumes reflections are collected in the order in the data file.
  • scale_reference_data = True Scale reference data to maximum of 99999 if writing intermediate scalepack data files
• merging
  • include_dataset_variance = True Scale datasets together including inter-dataset variance in in weighting. This takes into account the systematic differences among datasets and allows automatic down-weighting of datasets that are different than all the others.
  • optimize_anomalous = True Scale datasets together including inter-dataset variance in anomalous differences in weighting. Involves separation of mean intensity of each reflection from anomalous differences and recombination after scaling to recreate I+ and I-. Alternative is to include inter-dataset variance in intensities.
  • use_best_group_as_target = False Estimate inter-dataset variance by using the best dataset (highest I/sigma) as target for all datasets. This amounts to using all the other datasets to optimally estimate the data for this one crystal.
  • rescale_sigmas = False Rescale sigmas assuming that at high resolution, there are no anomalous differences. This can sometimes be useful if the sigmas are all too high or low and the data are collected to sufficiently high resolution that there really are essentially no anomalous differences (only noise) at high resolution.
• half_dataset_cc
  • get_half_dataset_cc = True Split datasets and get half-dataset CC if data are available
  • half_dataset_cc_by_files = True Split datasets into groups based on keeping all data in a file together (only applies if multiple data files are supplied.)
  • half_dataset_cc_match_inverses = False Split datasets into groups keeping (hkl, -h-k-l) together. Applies if only one file or half_dataset_cc_by_files=False. Does not apply if less than half of the reflections have (-h-k-l) pairs and other methods below have more matches.
  • split_as_first_second_half = True split datasets into groups based on using the first or second half of each dataset in each half-dataset. Applies if half_dataset_cc_by_files=False or only one dataset is supplied and half_dataset_cc_match_inverses=False. Also applies if half_dataset_cc_by_files and half_dataset_cc_match_inverses have failed to yield sufficient anomalous differences.
  • split_alternately = True split datasets into groups based on sending observations of equivalent reflections alternately to the first or second half-dataset. Applies if split_as_first_second_half=False and half_dataset_cc_by_files=False and only one dataset is supplied. Also applies if other methods have failed to yield sufficient anomalous differences
  • split_randomly = True split datasets into groups based on sending observations of equivalent reflections randomly to the first or second half-dataset. Applies if split_as_first_second_half=False and half_dataset_cc_by_files=False and split_alternately is False and only one dataset is supplied. Also applies if other methods have failed to yield sufficient anomalous differences
• directories
  • temp_dir = None Optional temporary work directory. If specified, it will be used for temporary files. If clean_up=True and it did not exist before, it will be removed at the end of the run. If it did exist, it will be left.
  • output_dir = "" Output directory where files are to be written
  • gui_output_dir = None GUI use only - does not apply to command line version
• control
  • job_title = None Job title in PHENIX GUI, not used on command line
  • verbose = False Verbose output
  • random_seed = 714215 random seed
  • comparison_file = None Data file with comparison anomalous differences
  • multiprocessing = *multiprocessing sge lsf pbs condor pbspro slurm Choices are multiprocessing (single machine) or queuing systems
  • queue_run_command = None run command for queue jobs. For example qsub.
  • nproc = 1 Number of processors to use in scaling
  • solve_size = None You can set the size of Solve. Normally done automatically.
  • clean_up = True Clean up and remove temp directory after finishing