Microbiology-PTM-OMA_Modules_1-5
Module M1: PTM and OMA Inclusivity and Exclusivity Studies
PTM and OMA SLV Study Designs
Method Type Study Design
Statistics
Acceptance Criteria Expert Judgement
50 target strains (100 if Salmonella spp) 100x LOD; all claimed enrichments
None ‐ specifically report negative inclusivity and positive exclusivity strains None ‐ specifically report negative inclusivity and positive exclusivity strains
Qualitative
30 non‐target species stationary growth in non‐selective media Quantitative 50 target strains (100 if Salmonella spp) 100x LOD in non‐selective media
Expert Judgement
30 non‐target species stationary growth in non‐selective media Identification* 25‐200 target strains depending on target None ‐ specifically report 100 non‐target strains *Preference is to follow ISO 16140‐6 for number and types of strains, dependent on the analyte claim. negative inclusivity and positive exclusivity strains
Expert Judgement
OMA Collaborative Study Design
Method Type
# Valid Data Sets
Study Design
Statistics
ID Method
≥10
≥12 Organisms All claimed agars (Include non‐specific agar if not claimed)
# Correct Identifications Specify strains with incorrect identifications
1
Panel Requirements The types of strains required depends on the analyte claim. Example 1: Salmonella spp. claim for screening method = 100 serovars for inclusivity; 30 species for exclusivity
Inclusivity: ≥2 serovars S. bongori
≥3 serovars S. enterica subsp arizonae ≥3 serovars S. enterica subsp diarizonae ≥3 serovars S. enterica subsp houtenae
≥3 serovars S. enterica subsp indica ≥3 serovars S. enterica subsp salamae Remainder is 1 strain per serovar of S. enterica subsp enterica
Exclusivity: 1 strain per species non‐ Salmonella
Slide 4 1
Could a slide or language be added for new or new to AOAC analytes or those that are not as common as Salmonella and E. coli O157? Zerlinde Balverde, 1/25/2019
Panel Requirements Example 2: E. coli O157:H7 screening method = 50 inclusivity strains; 30 exclusivity strains
Inclusivity: 50 strains of E. coli O157:H7 including at least 10 strains of E. coli O157:NM (if claimed)
Exclusivity:Other serotypes of pathogenic E. coli (regulated and non‐regulated) Non‐pathogenic E. coli (1 strain) Other Escherichia species (1 strain per species)
Other coliforms (1 strain per species) Non‐coliforms (1 strain per species)
Panel Requirements Example 3: Legionella sp. screening method = 25‐30 inclusivity strains; 30 exclusivity strains
Inclusivity: 25‐30 strains of claimed Legionella sp. including at least 10 strains of L. pneumophila
Exclusivity:Non‐claimed Legionella sp. (1 strain per species)
Waterborne Gram‐negative strains (1 strain per species)
Panel Requirements Example 4: Vibrio species screening method = 50 inclusivity strains; 30 exclusivity strains
Inclusivity: 50 strains of Vibrio species including at least 10 strains of V. cholera, V. parahaemolyticus, V. vulnificus (if claimed)
Exclusivity: Vibrio species except cholera, parahaemolyticus and vulnificus Enterobacteriaceae (1 strain per species) Non‐EB (1 strain per species)
Panel Requirements Example 5: Hepatitis A or Norovirus screening method = To be determined with the Volunteer Expert
Inclusivity: TBD # of strains of claimed serotypes (ex. GI, GII, HAV)
Exclusivity:Non‐target viral strains (one strain per species; minimum 10‐15) Bacterial strains (one strain per species; minimum 5‐10) Parasites (one strain per species)
Panel Requirements • Panel requirements for various analyte claims are currently being codified and will be posted to the RI website after review and approval by the Microbiology Volunteer Experts. • Preference is for ~50% food isolates • Report strain number, source, and origin for each organism. Where appropriate, also report presence/absence of virulence genes.
Testing Requirements • For qualitative methods:
• Grow inclusivity organisms according to all candidate method enrichment protocols. If the only difference is time of enrichment, follow the shortest enrichment time. Dilute, if necessary, to 100x LOD. • Grow exclusivity organisms under appropriate conditions (broth(s)/temperature) to encourage growth. Test at highest growth level achieved. • For quantitative methods: • Grow all organisms under appropriate conditions (broth(s)/temperature) to encourage growth. • Dilute inclusivity organisms to 100x LOD. • Test exclusivity organisms at the highest growth level achieved. • For ID Methods: • Streak inclusivity and exclusivity organisms to all claimed agars and/or dilute in all claimed buffers. • Also include one non‐selective agar, if not already included in claim.
Testing and Reporting Requirements • For all methods: • Randomize and blind code all inclusivity and exclusivity cell suspensions. • Test 1 replicate of each blinded sample. • Report number of inclusivity organisms correctly detected or identified and specify those that were not detected or were misidentified. • Report the number of exclusivity organisms correctly not detected or not identified and specify those that were incorrectly detected or identified. • Exclusivity organisms incorrectly detected can be retested under method‐specific enrichment conditions (applies to qualitative methods).
Testing and Reporting Requirements • Testing by the reference method is not required, but can be included if desired. • Confirmation testing is not required, but the identification of each strain should be documented in the lab. • Strain purity should be verified before use. • Testing may be carried out by the method developer or independent lab.
Module M2: PTM and OMA Matrix Studies
Overview • The Matrix Study determines the method performance in the claimed matrices. • Matrix Studies for microbiology are typically a method comparison between the candidate method and an appropriate reference method for each matrix. • In rare cases, a reference method may not exist. • Method comparisons can be paired studies or unpaired studies. • Paired: Same enrichment or sample preparation procedure used for both methods. Both methods analyze the same test portion. • Unpaired: Different enrichment or sample preparation procedures used for the candidate and reference methods. The two methods analyze distinct test portions.
Terminology
Qualitative Food Matrix Studies
Quantitative Food Matrix Studies
PTM and OMA SLV Matrix Study Designs
Method Type Study Design
Statistics
Acceptance Criteria
Qualitative If artificially contaminated: 5 Replicates High (all positive)
Probability of Detection (POD)
95% CI on dPOD must include 0
20 Replicates Low (fractional positives*) 5 Replicates Noninoculated If naturally contaminated: 2 Lots 20 Replicates/lot One lot must have fractional positive results
Quantitative 5 Replicates High
Log(10) transform
5 Replicates Medium 5 Replicates Low [5 Replicates Noninoculated]
Difference of Means (DOM) with 95% CI
[In development]
Expert Judgement
S r
*Fractional positives are defined as POD = 0.25‐0.75
OMA Collaborative (MLV) Study Designs
Method Type # Valid Data Sets Study Design per Collaborator
Statistics
Qualitative
≥10
1 Matrix 12 Replicates High 12 Replicates Low (fractional pos.*) 12 Replicates Noninoculated
LPOD, dLPOD + CI dLPOD CI must include “0” S r with 95% CI S R with 95% CI S L FP/FN rates in text of report
Quantitative ≥8
1 Matrix 2 Replicates High 2 Replicates Medium 2 Replicates Low [2 Replicates Noninoculated]
Log(10) transform DOM + 95% CI S r S R
*Fractional positives defined as LPOD = 0.25‐0.75
Matrix Study Designs – Recent Trends
• Candidate methods must always be confirmed • 1 colony from 1 agar plate per test portion is acceptable to follow through to identification. • All instrument platforms must be tested in the matrix study. • Composite test portions (e.g., 375 g) for the candidate method are compared to 25 g test portions for the reference method (if applicable). • Temperature claims • Enrichments ‐ if both incubation temperatures have a ± 1°C range and incubation is conducted at 41.5°C (EU), then it is acceptable to claim for 42°C in the US • Selective agars – incubation at 36 ± 2°C is acceptable if the US method is 35 ± 1°C and the EU method is 37 ± 1°C. • ISPAM approved equivalence of 35°C and 37°C for Salmonella primary enrichments.
Matrix Study Designs – Recent Trends
• FDA and USDA will accept studies done according to ISO 16140‐2 if BAM (FDA) and MLG (USDA) reference methods are used. • Validation of Environmental Surfaces • Comparison to FDA BAM method • For single surface claims, test with and without background organism present. • 12” x 12” surface claims – comparison is to 4” x 4” for the reference method with the same number of organisms applied, but in 9‐fold larger volume for 12” x 12” (analogous to validating a composite test portion). • New proposal under consideration to add sanitizer to the environmental surface evaluation scheme.
Matrix Study Designs – Recent Trends
• Collaborative Studies • Up to 3 collaborators per site using the same equipment if work is performed independently • Independent test portions for each collaborator. • Independent analyses performed by each collaborator. • Single matrix must be the most challenging matrix/enrichment combination from the SLV or PTM study. • 8 replicates per collaborator is acceptable for qualitative studies harmonized with ISO 16140‐2, but 12 replicates is preferred.
Choosing Matrices
Choose matrices: • Likely to contain the target analyte
• Associated with outbreaks • Where analyte can survive
• pH, water activity, spices, etc. • Based on target market or customer for candidate method
Competitive Flora
• At least one food item and one surface must be tested with competing flora (natural or artificial) • Naturally present in raw foods • Artificially added for environmental surfaces • Simulates real‐world conditions • If artificially contaminating, competing organism should be at least ten times more abundant than target • Competing strain should be appropriate to the target
Food Preparation • Purchase sufficient amount of single lot for candidate method analyses, reference method analyses, MPN determination (if applicable) and aerobic plate count. • Screen food item for natural contamination • Recommend using both the candidate and reference methods. • Naturally contaminated matrices are preferred • Naturally contaminated food may be temperature abused to increase level of contamination or diluted with uncontaminated matrix to decrease level of contamination. • If naturally contaminated, test two lots of matrix, 20 replicates each. • One lot must meet fractional positive requirement • No 0 CFU level required. • If natural contamination not found, artificially inoculate each matrix with a different target organism (strain, serovar, species).
Artificial Contamination of Foods • Use food‐borne isolates if possible, preferably from a similar food type • Choose inoculation species/serovars based on prevalence in the matrix or outbreak data • Choose a different inoculation species/serovar for each claimed matrix • For single target assays, inoculate matrix with a single target strain • For multiplex assays, inoculate at least one matrix with all target strains • Strain source • Internationally recognized source (e.g., ATCC) • Laboratory isolate (must have documentation verifying identity of strain) • Bulk inoculation required whenever possible • Exceptions for “unit” matrices such as chicken carcasses and whole cantaloupe
Artificial Contamination of Foods
• Chop, cut, melt, temper, etc.
• Maintain the integrity of the matrix • Use the same lot of matrix for inoculated materials and noninoculated material • Do not dilute matrix with inoculum • For qualitative methods, inoculate: • 1 material at a low level to achieve fractional positives • 1 material at a high level to aim for all positives (2‐5x inoculum of low level) • For quantitative methods, inoculate:
• 1 material at a low level within the method range • 1 material at a medium level within the method range • 1 material at a high level within the method range
Preparation of Test Materials
Preparation of Inoculum • Raw and cold‐processed foods should be inoculated with unstressed organisms • Heat‐processed foods should be inoculated with heat‐stressed organisms • Heat culture at 50˚C for ~10 min • The heat stress must achieve 50 – 80% injury of the inoculum • The degree of injury is calculated as follows: 1 – ( n select / n nonselect ) x 100 n select = mean number of colonies on selective agar n nonselect = mean number of colonies on nonselective agar
Inoculation of Liquid Matrices
• Add diluted liquid culture to large quantity and mix • Use fresh culture for unprocessed liquids • Use heat‐stressed culture for heat‐processed liquids • Stabilize 48‐72 h at 2‐8°C • For quantitative Listeria store for 24 h at 2‐8°C
Inoculation of Moist Solid Matrices • Use fresh culture for raw foods
• Use heat‐stressed culture for heat‐processed foods • Add diluted culture • Drop‐wise • Spray • Incorporate by kneading, mixing, etc. • Stabilize 48‐72 h at 2‐8°C • For quantitative Listeria store for 24 h at 2‐8°C
Inoculation of Dried Foods
• Add lyophilized culture to small amount of matrix and mix • roll, shake, etc. • Add to bulk matrix and mix again • Stabilize 2 weeks at RT • Unless known, recommend aging study of organism in matrix
Inoculation of Frozen Foods • Use fresh culture for raw foods
• Use heat‐stressed culture for heat‐processed foods • Thaw food product • Add diluted culture • Incorporate by stirring • Refreeze product • Stabilize for 2 weeks at ‐20°C
Inoculation of Chocolate
• Understand how to temper chocolate ( http://www.instructables.com/id/How‐to‐ Temper‐Chocolate/?ALLSTEPS ) • When melted, add a small volume of liquid culture or add dried culture to avoid ‘seizing’ the chocolate and mix well
• Allow to reharden (temper) • Stabilize for 2 weeks at RT
Inoculation in Special Cases
Inoculate whole cantaloupe on the outer rind
Inoculate chicken carcasses in the cavity
Inoculation of Surfaces
• Dilute inoculum with stabilizer to high and low levels • 10% NFDM solution, hot dog juice, etc.
• Low level inoculum must yield fractional positive results for one of the methods • High level inoculum, 2‐5X concentration of low level, aims for all positives • Noninoculated level receives stabilizer only. • Perform and report plate count of inoculum • Inoculate surface areas in a random blinded manner with a volume of the appropriate inoculum that allows even distribution across the surface without excessive pooling of liquid • 2.25 mL for 12” x 12” area
• 250 µL for 4” x 4” area • 100 µL for 1” x 1” area
• Dry overnight (≥16 h) at room temperature – surfaces must be visibly dry • Recovery can vary depending on strain/surface combination and laboratory conditions (temperature and humidity) – preliminary range finding is critical
Swiping of Surfaces
• Swipe 12” x 12” and 4” x 4” surface areas using sponges • Swipe 1” x 1” areas using swabs • Premoisten sponge or swab with neutralizing buffer as specified in each method • Swipe entire surface area by swabbing/sponging back and forth 10 times horizontally and 10 times vertically • Hold sponges in sterile bags or swabs in sterile tubes with neutralizing buffer for 2 h at room temperature before initiating enrichment
Confirmations • Confirmations must follow the Reference Method procedures • Biochemical gallery methods with Official Method status may be used in place of traditional biochemical methods. • Other identification methods with Official Method status may be used for identification of isolated colonies. • Alternative confirmation procedures can be validated • Confirmations must include isolation of suspect colonies • Regulatory agencies require a cultural isolate for defensible proof
Performing the Study – Test Portions
• From each material, remove the replicate test portions needed for the Candidate and Reference Methods • Artificially or naturally contaminated matrix • Paired or unpaired analyses • Qualitative or Quantitative method comparison • Randomize and blind code the test portions for each method • From Materials 2 and 3, remove replicate test portions needed for Most Probable Number (MPN) determination – Qualitative methods only • Not required for noninoculated material • Need at least 5 replicates at 3 levels for MPN • Reference method only • Can use the reference method replicates from the matrix study as one level of the MPN
MPN Replicates – Qualitative methods
• For MPN determination of Material 2 (low level), remove: • 5 replicate large MPN test portions 2‐5x larger than the reference method test portions and • 5 replicate small MPN test portions 2‐5x smaller than the reference method test portions • For MPN determination of Material 3 (high level), remove: • 5 replicate medium MPN test portions 2‐5x smaller than the reference method test portions • 5 replicate small MPN test portions 2‐5x smaller than the medium MPN test portions • Analyze by the reference method, maintaining normal test portion to enrichment media ratio. • MPNs not applicable to environmental surfaces, chicken carcasses, whole cantaloupe, or other “unitized” matrixes.
MPN Replicates ‐ Example
Large Test Portions 5 x 50 g 5 x 25 g*
Medium Test Portions
Small Test Portions
Reference Method Enrichment
Matrix
Material
2 (Low level) 3 (High level)
20 x 25 g*
5 x 10 g 5 x 5 g
Raw ground chicken
1:9 BPW 35°C, 20‐24 h
MLG 4.10
5 x 10 g
*Test portions from matrix study analyzed by the reference method.
Replicates for Composite Test Portions
• Compositing is the combining of individual test portions for the purpose of performing a single enrichment for cost savings when the probability of any individual test portion being positive is very low • For example, laboratories may composite 15 individual 25‐g test portions into a 375 g composite test portion • The composite enrichment must be positive if even one of the individual 25‐g test portions is positive • Same needle now in a bigger haystack – must now detect one target cell in 375 g • Candidate method composite test portions are compared to reference method individual (25 g) test portions
Replicates for Composite Test Portions
• Inoculate materials at high and low levels • Low level must achieve fractional positive results for 25‐g test portions • For the reference method, analyze 25‐g test portions of all materials • For the candidate method, mimic composite by combining one 25‐g test portion from a test material with 300 g noninoculated matrix (for a 325 g composite) or 350 g noninoculated matrix (for a 375‐g composite)
Composite Test Portion Example
Performing the Study – Qualitative Methods
• Begin all enrichments on the same day • Blind coded candidate method test portions or environmental swabs/sponges • Blind coded reference method test portions or environmental swabs/sponges • MPN test portions (if applicable) • Follow each method as written • Use most current published version of the reference method • Confirm every enrichment (candidate, reference, and MPN enrichments) according to the reference method procedures • Confirm candidate method enrichments according to alternate candidate method confirmation procedure, if applicable
Analyzing Data – Qualitative Methods
• Use Probability of Detection (POD) statistics • POD is the proportion of positive analytical outcomes for a qualitative method for a given matrix at a given bacterial level or concentration • POD = x/N • Unblind the data • Analyze each level of each matrix separately • Analyze data separately if there are multiple time points, multiple instruments, manual and automated methods, etc. • Calculate POD and 95% confidence interval (CI) for each method • POD CP for the candidate method presumptive results • POD CC for the candidate method confirmation results • POD C for the candidate method final results (accounting for presumptive and confirmatory results) • POD R for the reference method results
Analyzing Data – Qualitative Method SLV • Method comparison achieved by estimating bias of candidate method • Bias estimated as dPOD, the difference between two POD values • dPOD CP = POD CP – POD CC = bias between presumptive and confirmatory results of the candidate method • dPOD C = POD C – POD R = bias between candidate and reference method results • Calculate 95% confidence intervals on dPOD values • Calculation depends on whether comparison is paired or unpaired • If the confidence interval on dPOD does not contain zero, then the two POD values are statistically different (bias is significant).
Data Table – Qualitative Method SLV
Table 1. Comparison of presumptive and confirmation candidate method results
Presumptive
Confirmation
MPN b / test portion N a
Matrix (APC i )
Strain
Instrument
dPOD CP f
95% CI g
POD CP d (95% CI g ) 0.00 (0.00, 0.43) 0.00 (0.00, 0.43) 0.80 (0.58, 0.92) 0.80 (0.58, 0.92) 1.00 (0.57, 1.00) 1.00 (0.57, 1.00)
POD CC e (95% CI g ) 0.00 (0.00, 0.43) 0.00 (0.00, 0.43) 0.80 (0.58, 0.92) 0.80 (0.58, 0.92) 1.00 (0.57, 1.00) 1.00 (0.57, 1.00)
X c
X c
A
0
0
0.00 (‐0.47, 0.47)
N/A
5
B
0
0
0.00 (‐0.47, 0.47)
Smoked Salmon (7.0 x 10 5 CFU/g)
A
16
16
0.00 (‐0.13, 0.13)
L. monocytogenes ATCC 15313
0.91 (0.64, 1.76)
20
B
16
16
0.00 (‐0.13, 0.13)
A
5
5
0.00 (‐0.47, 0.47)
7.34 (3.65, 16.76)
5
B 0.00 (‐0.47, 0.47) a N = Number of test potions. b MPN = Most Probable Number with 95% confidence interval. c x = Number of positive test portions. d POD CP = probability of detection for the candidate presumptive results. e POD CC = probability of detection for the candidate confirmed results. f dPOD CP = POD CP minus POD CC . g 95% Confidence Interval. i APC = Aerobic Plate Count. 5 5
Note : Comparison of presumptive and confirmation results is always a paired comparison.
Data Table – Qualitative Method SLV
Table 2. Comparison of candidate and reference method results
Confirmed Candidate
Reference
MPN b / test portion N a
Matrix
Strain
Instrument
dPOD C f
95% CI g
POD C d (95% CI g ) 0.00 (0.00, 0.43) 0.00 (0.00, 0.43) 0.80 (0.58, 0.92) 0.80 (0.58, 0.92) 1.00 (0.57, 1.00) 1.00 (0.57, 1.00)
POD R e (95% CI g ) 0.00 (0.00, 0.43) 0.00 (0.00, 0.43) 0.70 (0.48, 0.85) 0.70 (0.48, 0.85) 1.00 (0.57, 1.00) 1.00 (0.57, 1.00)
X c
X c
A
0
0
0.00 (‐0.43, 0.43)
N/A
5
B
0
0
0.00 (‐0.43, 0.43)
Smoked Salmon (7.0 x 10 5 CFU/g)
A
16
14
0.10 (‐0.17, 0.35)
L. monocytogenes ATCC 15313
0.91 (0.64, 1.76)
20
B
16
14
0.10 (‐0.17, 0.35)
A
5
5
0.00 (‐0.43, 0.43)
7.34 (3.65, 16.76)
5
B 0.00 (‐0.43, 0.43) a N = Number of test potions. b MPN = Most Probable Number with 95% confidence interval. c x = Number of positive test portions. d POD C = Probability of Detection for the confirmed candidate results . e POD R = probability of detection for the reference method results . f dPOD C = POD C minus POD R . g 95% Confidence Interval. h N/A not applicable. 5 5
Analyzing Data – Qualitative Method MLV
• First, analyze data from each laboratory and each material individually for POD CP , POD CC , POD C , and POD R . • Calculate dPOD CP , and dPOD C with appropriate confidence intervals (paired or unpaired) for each laboratory and each material. • Look for any statistically significant differences • Combine data across laboratories and calculate LPOD CP , LPOD CC , LPOD C , and LPOD R with 95% confidence intervals. • Determine error estimates s r , s L , and s R and report with 95% confidence intervals. • Determine the p ‐value of the T test based on the X 2 distribution and compare to 0.10 to determine whether the interlaboratory effect, s L , is statistically significant. This is the homogeneity test of laboratory POD values. • Finally, determine dLPOD CP and dLOPD C with appropriate confidence intervals (paired or unpaired) for each material. • In the text of the report, include the false positive and false negative rates for the comparison of candidate method presumptive and confirmation results across all laboratories.
Data Table – Qualitative Method MLV
Table 1. Individual collaborator results for detection of Salmonella in milk chocolate
Candidate confirmed (CC)
Candidate result (C)
Reference result (R)
dPOD C (95% CI) Unpaired
MPN/Test portion (95% CI)
dPOD CP (95% CI) Paired
Candidate presumptive (CP)
Lab.
N x
POD CP
N x
POD CC
N x
POD C
N x
POD R
1 2 3
12 12 12
2 5 4
0.17 12 0.42 12 0.33 12
2 6 3
0.17 12 0.50 12 0.25 12
2 5 3
0.17 12 0.42 12 0.25 12
4 2 4
0.33
0.00 (‐0.21, 0.21) ‐0.17 (‐0.47, 0.18)
0.17 ‐0.08 (‐0.34, 0.18) 0.25 (‐0.11, 0.54)
0.33
0.08 (‐0.18, 0.34) ‐0.08 (‐0.40, 0.26)
4
12
3
0.25 12
4
0.33 12
3
0.25 12
7
0.58 ‐0.08 (‐0.34, 0.18) ‐0.33 (‐0.61, 0.05)
5
12
3
0.25 12
3
0.25 12
3
0.25 12
5
0.42
0.00 (‐0.21, 0.21) ‐0.17 (‐0.48, 0.19)
6
12
1
0.08 12
1
0.08 12
1
0.08 12
3
0.25
0.00 (‐0.21, 0.21) ‐0.17 (‐0.46, 0.15)
0.46 (0.35, 0.59)
7
12
3
0.25 12
4
0.33 12
3
0.25 12
6
0.50 ‐0.08 (‐0.34, 0.18) ‐0.25 (‐0.54, 0.12)
8
12
3
0.25 12
3
0.25 12
3
0.25 12
6
0.50
0.00 (‐0.21, 0.21) ‐0.25 (‐0.54, 0.12)
9
12
5
0.42 12
4
0.33 12
4
0.33 12
5
0.42
0.08 (‐0.18, 0.34) ‐0.08 (‐0.41, 0.27)
NA NA
NA NA NA
NA NA NA NA NA NA
NA
NA
NA
10 a
11 12 12 12 13 12 14 12
5 5 6 7
0.42 12 0.42 12 0.50 12 0.58 12
5 5 6 7
0.42 12 0.42 12 0.50 12 0.58 12
5 5 6 7
0.42 12 0.42 12 0.50 12 0.58 12
6 4 2 3
0.50 0.33 0.17 0.25
0.00 (‐0.21, 0.21) ‐0.08 (‐0.42, 0.28) 0.00 (‐0.21, 0.21) 0.08 (‐0.27, 0.41) 0.00 (‐0.21, 0.21) 0.33 (‐0.04, 0.61) 0.00 (‐0.21, 0.21) 0.33 (‐0.05, 0.61)
a Laboratory did not complete testing
Data Table – Qualitative Method MLV
Table 2. Collaborative study data summary and statistical analyses for detection of Salmonella in milk chocolate
Parameter
Material 1
Material 2
Material 3
MPN/Test Portion
0.46 (0.35, 0.59)
3.81 (3.07, 5.49)
Candidate Presumptive (CP) x/N
2/156
52/156
152/156
0.01 (0.00, 0.05) 0.11 (0.10, 0.15) 0.00 (0.00, 0.04) 0.11 (0.10, 0.13)
0.33 (0.26, 0.41) 0.47 (0.42, 0.52) 0.03 (0.00, 0.19) 0.47 (0.43, 0.52)
0.97 (0.94, 0.99) 0.16 (0.14, 0.18) 0.03 (0.00, 0.07) 0.16 (0.14, 0.18)
LPOD CP
s r s L s R
p
0.5167
0.3875
0.1955
Candidate Confirmed (CC) x/N
0/156
53/156
152/156
0.00 (0.00, 0.02) 0.00 (0.00, 0.15) 0.00 (0.00, 0.15) 0.00 (0.00, 0.21)
0.34 (0.26, 0.42) 0.47 (0.42, 0.52) 0.04 (0.00, 0.19) 0.48 (0.43, 0.52)
0.97 (0.94, 0.99) 0.16 (0.14, 0.18) 0.03 (0.00, 0.07) 0.16 (0.14, 0.18)
LPOD CC
s r s L s R
p
1.0000
0.3709
0.1955
Candidate Result (C) x/N
0/156
50/156
152/156
0.00 (0.00, 0.02) 0.00 (0.00, 0.15) 0.00 (0.00, 0.15) 0.00 (0.00, 0.21)
0.32 (0.24, 0.40) 0.47 (0.42, 0.52) 0.04 (0.00, 0.19) 0.47 (0.42, 0.52)
0.97 (0.94, 0.99) 0.16 (0.14, 0.18) 0.03 (0.00, 0.07) 0.16 (0.14, 0.18)
LPOD CC
s r s L s R
p
1.0000
0.3769
0.1955
Reference Result (R) x/N
0/156
57/156
153/156
0.00 (0.00, 0.02) 0.00 (0.00, 0.15) 0.00 (0.00, 0.15) 0.00 (0.00, 0.21) 0.00 (‐0.02, 0.02) 0.01 (‐0.01, 0.05) 1.0000
0.37 (0.29, 0.44) 0.48 (0.43, 0.52) 0.00 (0.00, 0.18) 0.48 (0.44, 0.52) ‐0.05 (‐0.15, 0.06) ‐0.01 (‐0.12, 0.10) 0.5145
0.98 (0.95, 0.99) 0.13 (0.12, 0.15) 0.03 (0.00, 0.07) 0.14 (0.12, 0.16) ‐0.01 (‐0.05, 0.03) 0.00 (‐0.04, 0.04) 0.0877
LPOD R
s r s L s R
p
dLPOD CP dLPOD C
Performing the Study – Quantitative Methods
• Begin all analyses on the same day • Blind coded candidate method test portions • Blind coded reference method test portions • Paired or unpaired study • Follow each method as written • Use most current published version of the reference method • Use all claimed dilution buffers • Confirm every blind coded test portion according to the reference method procedures, if applicable • Confirm blind coded candidate method test portions according to alternate candidate method confirmation procedure, if applicable
Analyzing Data – Quantitative Methods
• Unblind the data • Analyze each level of each matrix separately • Perform a logarithmic transformation on the reported CFU/g or CFU/mL: Log 10 [CFU/g + (0.1)f] Where f is the reported CFU/unit corresponding to the smallest reportable result and unit is the reported unit of measure (e.g., g, mL, in 2 , etc.). • Perform outlier tests (Cochran and Grubbs) to identify significantly outlying data points. Data points may only be removed, however, for justifiable cause.
Analyzing Data – Quantitative Methods • Plot the candidate method result (x‐axis) vs. the reference method result (y‐axis) for each matrix. Calculate the slope and square of the linear correlation coefficient (r 2 ) for each plot. • Calculate repeatability as the standard deviation of replicates at each concentration of each matrix for each method. • Calculate the mean difference between the candidate and reference method transformed results with 95% confidence interval for each concentration of each matrix.
Data Analysis – Quantitative Methods
Chicken breast, E. coli
0 1 2 3 4 5 6 7 8
y = 1.0073x + 0.2026 R² = 0.995
Candidate Method, log CFU/g
0
1
2
3
4
5
6
7
ISO 16649‐2, log CFU/g
Figure 1. Plot of candidate method versus reference method for enumeration of E. coli in raw chicken breast
Data Table – Quantitative Method SLV
Table 1. Matrix study results for enumeration of E. coli
Candidate Method
Reference Method
DOM (Δlog CFU/g)
Matrix
Level
N
95% CI
Mean (log CFU/g)
Mean (log CFU/g)
s r
s r
Low
5
2.236
0.125
2.261
0.084
‐0.025
(‐0.268, 0.217)
Raw ground pork
Med
5
4.526
0.094
4.239
0.041
0.287
(0.154, 0.420)
High
5
6.465
0.056
6.210
0.109
0.256
(0.102, 0.409)
Low
5
2.388
0.079
2.198
0.067
0.190
(0.097, 0.284)
Raw chicken breast
Med
5
4.541
0.079
4.263
0.044
0.278
(0.161, 0.394)
High
5
6.432
0.078
6.200
0.181
0.232
(0.007, 0.456)
Low
5
2.322
0.151
2.245
0.146
0.077
(‐0.122, 0.276)
Raw chicken breast a
Med
5
4.371
0.042
4.211
0.066
0.160
(0.069, 0.251)
High
5
6.356
0.048
6.139
0.118
0.217
(0.092, 0.341)
Note : Acceptance criteria for DOM (difference of means) and 95% CI on DOM are currently under consideration. a Matrix tested by the Independent Laboratory
Data Table – Quantitative Method MLV
Table 1. Log transformed counts for Enterobacteriaceae in powdered infant formula with probiotics
Material 1
Material 2
Material 3
Material 4
Reference Log 10 CFU/g A B
Candidate Log 10 CFU/g
Reference Log 10 CFU/g
Candidate Log 10 CFU/g
Reference Log 10 CFU/g
Candidate Log 10 CFU/g
Reference Log 10 CFU/g
Candidate Log 10 CFU/g
Collaborator A B 1 0.000 0.000 0.000 0.000 1.041 1.041 1.176 1.013 2.179 2.281 1.908 2.179 3.117 3.233 3.149 3.233 2 0.000 0.000 0.000 0.000 0.000 0.000 0.519 0.519 3.042 2.807 2.852 2.479 3.634 3.533 3.324 3.149 3 0.000 0.000 0.000 0.000 1.041 1.041 1.176 0.954 2.281 2.569 2.507 2.533 3.417 3.507 3.004 3.083 4 0.000 0.000 0.000 0.000 1.041 1.322 1.013 1.013 2.614 2.045 2.682 2.004 3.507 3.654 3.464 3.603 5 0.000 0.000 0.000 0.000 1.041 1.322 1.398 1.013 2.558 2.179 2.382 2.207 3.558 3.464 3.464 3.464 6 0.000 0.000 0.000 0.000 1.041 0.000 0.954 0.519 2.382 2.281 2.324 2.149 3.324 3.569 3.281 3.624 7 0.000 0.000 0.000 0.000 1.041 1.322 1.643 1.643 2.117 2.258 2.004 2.045 3.149 3.117 2.996 2.991 8 0.000 0.000 0.000 0.000 1.041 1.041 0.519 0.681 2.449 2.464 2.479 2.382 3.258 3.179 3.045 3.083 9 0.000 0.000 0.000 0.000 1.041 1.041 1.398 1.398 2.887 2.493 2.364 2.004 3.281 3.581 3.083 3.207 10 0.000 0.000 0.000 0.000 0.000 1.041 0.519 1.009 2.52 2.644 2.814 2.045 3.233 3.433 3.045 3.083 11 0.000 0.000 0.000 0.000 1.041 1.041 1.398 1.643 2.581 2.344 2.364 2.004 3.493 3.303 3.149 3.117 A a B A B A B A B A B A B
a A and B indicate duplicate test portions.
Data Table – Quantitative Method MLV
Table 2. Interlaboratory study result summary and statistics
Candidate Method
Reference Method
Mean Log 10 CFU/g
Mean
Matrix
Material
DOM b
95% CI c
N a
s R
N
s r
s R
s r
Log 10
CFU/g
1
11 0.000 0.00 0.00 11 0.000 0.00 0.00
0.00
(0.00, 0.00)
Powdered Infant Formula with Probiotics 24 h Inc. Powdered Infant Formula with Probiotics 48 h Inc.
2
11 0.890 0.33 0.45 11 1.051 0.18 0.39
‐0.16
(‐0.31, ‐0.01)
3
11 2.453 0.20 0.26 11 2.305 0.27 0.27
0.15
(0.05, 0.25)
4
11 3.388 0.12 0.18 11 3.211 0.10 0.20
0.18
(0.11, 0.25)
1
11 0.000 0.00 0.00 11 0.000 0.00 0.00
0.00
(0.00, 0.00)
2
11 0.903 0.34 0.45 11 1.051 0.18 0.39
‐0.15
(‐0.31, 0.1)
3
11 2.467 0.25 0.25 11 2.305 0.27 0.27
0.16
(0.06, 0.26)
4
11 3.395 0.11 0.17 11 3.211 0.10 0.20
0.18
(0.12, 0.25)
a N = Number of collaborators that reported complete results b DOM = Difference of means c CI = Confidence interval
Module M3: Robustness Study
Overview • This study evaluates the ability of the method to remain unaffected by small variations in method parameters that might be expected to occur when the method is performed by an end user. • The method developer, in conjunction with the AOAC Project Manager, is expected to make a good faith effort to choose parameters that can be influenced by the end user and are most likely to affect the analytical performance and determine the range of variation that can occur without adversely affecting the analytical results • Parameters are varied and deviations tested in a factorial study and compared to the baseline results of the test method. • 3 parameters, whenever possible, are chosen to be tested. • Results are compared to the baseline test results. • May utilize pure culture or may require inoculating a matrix depending on parameters chosen.
Examples of Robustness Parameters • REAGENTS • SAMPLE
• temperature of reagent at time of analysis • concentrations of reagents at specified limits • source of reagents • age of reagents • quantity of reagents • mixing times • reaction times • temperature of analysis • flow rates • variance in measuring apparatus
• temperature of sample at time of analysis • accuracy of obtaining sample quantity • age of sample prior to analysis • extraction time • Test portion enrichment time or temperature
Examples Study: Parameters to be Varied
Robustness Test Parameter
Baseline Value (Package Insert Instructions)
Low Value
High Value
Enrichment time
14 h
18 h
22 h
Lysis time
10 min
15 min
20 min
Time between start of PCR and hybridization
1 h
1.5 h
2 h
Examples Study: Factorial Design
Treatment Combination
Time Between Start of PCR and Hybridization
Enrichment Time
Lysis Time
1 2 3 4 5 6 7 8
14 h 14 h 14 h 14 h 22 h 22 h 22 h 22 h
10 min 10 min 20 min 20 min 10 min 10 min 20 min 20 min
1 h
2h
1 h 2 h 1 h 2 h 1 h 2 h
9
18 h
15 min
1.5 h
Robustness Study Designs
Method Type
Study Design
Statistics
Qualitative
Artificially contaminated: 10 Replicates Low (fractional positives a ) 10 Replicates Noninoculated or Inoculated with Background Organism
Probability of Detection (POD)
Quantitative
5 Replicates High 5 Replicates Low 5 Replicates Noninoculated
Log(10) transform
a Fractional positives are defined as POD = 0.25‐0.75 b dPOD values are calculated using nominal value as the baseline
Study • Inoculate matrices in the same manner as the matrix study. • Analyze each treatment combination/matrix separately compared to the baseline test method results. • No confirmation of test portions is required
Conclusions • Method developers should be encouraged that statistically significant findings in this experiment are not indicative of a faulty method, and the discovery of significance is not a roadblock to successful method validation. • Success of this experiment is not conditional on a conclusion of “no significant differences were observed.” • Any findings at this stage will be used to modify method kit instructions, emphasize areas of caution, or even possibly to widen specific parameter options.
Module M4: Product Consistency and Stability
Product Consistency (Lot‐to‐Lot Variability)
• Study to confirm that the manufacturing of the test kit and its components is consistent among lots • Testing is performed on 3 unique lots • Each lot must be a uniquely manufactured lot or consist of uniquely prepared reagents, supplies, and kit components • Can be combined with stability testing in a single study; if tested independent of stability testing, age of test kit is not a variable in this study, only variation among lots
Product Stability
• Study to support the shelf life statement and confirm that there is no observable change in performance of the method over the shelf life under normal storage conditions. Three possible study designs are: 1. Testing is performed on lots representing the full span of the shelf life of the kits (newly manufactured, middle of term, near expiration date). • This design allows the stability study to be performed concurrently with the product consistency study when unique lots are tested 2. Testing is performed on 1 lot by testing at least 5 time points over the shelf life of the kit 3. Preliminary stability testing is conducted using accelerated studies • Provides only a rough estimate of shelf‐life • Real time data supporting the entire shelf life of the kit under normal storage conditions must be submitted as soon as available
Product Consistency & Stability: Combined Real‐ Time Study Design for Quantitative Methods
• Three lots of test kits
• One lot near the expiration date • One lot near the middle of the expiration period • One lot recently manufactured. • Conducted using pure culture, matrix, DNA, etc. • Fifteen (15) total replicates evaluated for each lot. • 5 replicates of the target analyte at a high level • 5 replicates of the target analyte at a low level • 5 replicates of a non‐target analyte at a high level • Test in a randomized blind coded fashion. • Decode results and analyze for effects on bias and repeatability. • Data demonstrating no statistical difference in detection between lots and no significant time slope are required.
Quantitative 3 lots of kits • 5 replicates of target analyte at low level • 5 replicates of target analyte at high level • 5 replicates of blanks Bias & repeatability
Product Consistency & Stability: Combined Real‐Time Study Design for Qualitative Methods • Three lots of test kits • One lot near the expiration date • One lot near the middle of the expiration period • One lot recently manufactured. • Conducted using pure culture, inoculated matrix, DNA, etc. • Twenty (20) total replicates evaluated for each lot.
Qualitative 3 lots of kits • 10 replicates of target material • 10 replicates of non‐ target material at high level POD + CI
• 10 replicates of the target analyte at a fractional level (2‐8) • 10 replicates of the non‐target analyte at an undiluted level • Test in a randomized blind coded fashion. • Decode, calculate POD values and confidence intervals and analyze data for variable detection between lots/time points.
Product Consistency & Stability: Combined Real‐ Time Study Design for Identification Methods
• Three lots of test kits
• One lot near the expiration date • One lot near the middle of the expiration period • One lot recently manufactured. • Conducted using pure culture, matrix, DNA, etc. • Fifteen (15) total replicates evaluated for each lot. • 10 of the target analyte • 5 of the non‐target analyte • Test in a randomized blind coded fashion. • Decode results and determine number of correct identification
ID Method 3 lots of kits
• 10 target strains • 5 non‐target strains # correct results
Independent Product Consistency Study
• For test methods that have multiple components with different lot identification numbers • Lots of the individual components will be interchanged prior to analysis • Testing will be conducted in same manner as combined consistency/stability study (20 replicates for qualitative, etc.)
Kit Component 1 Lot 1 Lot 2 Lot 3 Lot 1 Lot 2 Lot 3 Lot 1 Lot 2 Lot 3
Kit Component 2 Lot 1 Lot 1 Lot 1 Lot 2 Lot 2 Lot 2 Lot 3 Lot 3 Lot 3
Kit Component 3 Lot 3 Lot 2 Lot 1 Lot 1 Lot 2 Lot 3 Lot 2 Lot 1 Lot 3
Product Consistency & Stability: Independent Stability Study
Qualitative
Quantitative
ID Method
Real time or accelerated
Real time or accelerated
Real time or accelerated
1 lot of kits
1 lot of kits
1 lot of kits
• 10 replicates of target material • 10 replicates of non‐target material at high level
• 5 replicates of target material at high level • 5 replicates of target material at low level • 5 replicates of non‐target material at high level 5 time points over the shelf life of the test kit
• 10 target strains • 5 non‐target strains
5 time points over the shelf life of the test kit
5 time points over the shelf life of the test kit
POD + CI
Bias & repeatability
# correct results
Accelerated Study Design
Accelerated Stability Study
Claimed Shelf Life
Required Storage Temperature
Storage Temperature
Component
Shelf Life
Time Points
Test Solution 1
25°C
9 mos. (270 d)
55°C
0, 2, 5, 8, 11, 14 d
Test Solution 2
‐20°C
9 mos. (270 d)
25°C
0, 1, 2, 4, 6, 8 d
Test Solution 3
‐20°C
9 mos. (270 d)
25°C
0, 1, 2, 4, 6, 8 d
• Data generated is based on the Arrhenius model • Assuming Ea = 20 kcal ex. 1 year at 5°C ≈ 32 days at 25°C, and 1 year at 25°C ≈ 45 days at 45°C • Test kit is stored at an elevated temperature to age the product more quickly • Number of replicates/time points is consistent with standard stability study
Parting Thoughts
• Product consistency and stability testing can be conducted independently or combined • Stored enrichments or diluents from the matrix studies may be used for this study • Work with your AOAC Technical Consultant to ensure that the testing performed meets the study design requirements for an AOAC validation
Module M5: Harmonization Studies
Harmonization Studies: Overview
• What is a harmonization study? • A single validation that allows for certification from AOAC and an ISO 16140 approval organization (AFNOR, MicroVal, NordVal). • The study is designed to meet all the requirements need for certification including • Data requirements • Statistical analysis • Submission guidelines • The goal is to achieve optimal efficiency and avoid duplication of efforts in order to meet regulatory and product safety testing requirements. • Reference standards • Testing requirements
Harmonization Studies: Guidelines & Standards
• Appendix J: AOAC INTERNATIONAL Methods Committee Guidelines for Validation of Microbiological Methods for Food and Environmental Surfaces (2012)
• Performance Tested Methods SM (PTM) • Official Methods of Analysis SM (OMA) http://www.eoma.aoac.org/app_j.pdf
• AOAC Research Institute uses guidelines and references developed by AOAC INTERNATIONAL and AOAC volunteer subject matter experts for its testing protocols and data evaluation
Harmonization Studies: Guidelines & Standards
• ISO 16140‐2 (2016): Microbiology of the food chain — Method validation — Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method https://www.iso.org/standard/54870.html • ISO/DIS 16140‐6 (2018): Microbiology of the food chain — Method validation — Part 6: Protocol for the validation of alternative (proprietary) methods for Microbial Confirmation and Typing Procedures https://www.iso.org/standard/66327.html • The ISO 16140 series (1‐6), developed by ISO TC34/SC9/WG3 forms the basis for the European certification of alternative methods, thus fulfilling European legislation.
Harmonization Studies: Overview
• Combines aspects of AOAC PTM and AOAC OMA programs that are also required for ISO 16140 • Inclusivity & Exclusivity • Matrix studies (referred to RLOD studies in ISO) • Interlaboratory study • Matrix studies, inclusivity & exclusivity, and ILS are conducted by the Expert Lab, no internal data is allowed (16140 requirement) • Since all matrix study work is done by the Expert Lab, an additional independent lab study is not required for PTM • Additional PTM studies (lot‐to‐lot, stability, robustness, instrument variation) are performed by the method developer • Data not submitted as part of ISO 16140 certification • Stability, lot‐to‐lot covered under manufacturing standard
Harmonization Studies: Expert Laboratory Requirements
• AOAC Approved Lab • MicroVal Approved Lab (if performing AOAC/MicroVal) • AFNOR Approved Lab (if performing AOAC/AFNOR) • ISO 17025 accreditation (required by AFNOR & MicroVal) • Scope of ISO 17025 Accreditation must contain reference standards performed in validation
Harmonization Studies: Application Process
AOAC • Application is submitted for consulting services • Technical consultant is assigned and will work with you to develop the AOAC protocol • Protocol can be PTM, Harmonized PTM/OMA, or OMA (direct pathway) • AOAC TC will work with Expert Lab to ensure matrix selection and inclusivity/exclusivity lists meet AOAC and ISO Technical Committee requirement • AOAC TC will submit Protocol for review by Volunteer Reviewer and members of the Expert Review Panel
• After feedback is received, the AOAC TC will update protocol • Upon approval, the testing protocol is provided to the Expert Laboratory for performing the analysis • To reduce the chance of additional work it is highly recommended to have the Expert Laboratory and AOAC TC complete the study protocols before any testing is initiated
Made with FlippingBook - Online catalogs