3
GUIDELINES FOR REPEATABILITY ESTIMATION
1. No collaborators should be removed, except for known cause. Such causes may include:
1) does not meet inclusion criteria for protocol, if protocols used are known; or 2)
provable contamination. Statistical identification of outliers or influential data is not
grounds for removal, only for investigation.
2. Replication may range from 2 to 4 replicates per collaborator. Repeatability should be
estimated in the usual way as the pooled standard deviation of the combined set of data.
3. Alternatively, replication may exceed 4 for some collaborators, but each estimate of
repeatability standard deviation should be assigned the minimum degrees of freedom
across all collaborators, and this number should be used in pooling and reporting.
4. The final number of degrees of freedom assigned to the pooled estimate must be 8 or
more.
5. There must be at least 3 collaborators with replication.
6. Robust statistics associated with repeatability may be estimated and reported (such as
interquartile range), but not estimates which attempt to convert such statistics to standard
deviations by, e. g., constant factors under a normality assumption. Reporting such robust
estimates for designed collaborative studies should be encouraged so that comparative
results may be accumulated over time.
7. Boxplots and half-normal plots are encouraged.
GUIDELINES FOR REPRODUCIBILITY ESTIMATION
1. No collaborators should be removed, except for known cause. Such causes may include:
1) does not meet inclusion criteria for protocol, if protocols used are known; or 2)
provable contamination. Statistical identification of outliers or influential data is not
grounds for removal, only for investigation.
2. The number of collaborators providing included data must be 8 or more.
3. If replication is present for most or all collaborators, repeatability, among-collaborator
variability and reproducibility should be estimated as standard deviations estimated in the
usual way from 1-way analysis of variance (additive model). No more than 4 replicates
should be used for any collaborator.
4. If replication is not present, reproducibility only may be estimated (as the standard
deviation of collaborator results).
5. Robust statistics associated with reproducibility may be estimated and reported (such as
interquartile range), but not estimates which attempt to convert such statistics to standard
deviations by, e. g., constant factors under a normality assumption. Reporting such robust
estimates for designed collaborative studies should be encouraged so that comparative
results may be accumulated over time.
6. Boxplots and half-normal plots are encouraged.
Recommended to OMB by Committee on Statistics: 07-17-2013
Reviewed and approved by OMB: 07-18-2013
9
by Committee on S atistics: 07-17-2 13
40
