Chemistry-PTM-OMA_Modules_1-4

Module C1: Selectivity and  Calibration Experiments

Calibration Experiment (Quantitative) • The calibration of the candidate method relates the response of an  instrument to the concentration of an analyte. The relationship between  concentration and response can be linear or non‐linear (curvilinear). • Determine the dose response curve (instrument response versus  concentration) using at least six concentrations (including a blank) over the  concentration range of interest. Test at least four replicates at each  concentration. • Fit calibration curve with an appropriate regression model.  Plot residuals  vs. concentration and examine for non‐random patterns.  Residuals should  be at <15% for each concentration but can be at 20% for the lowest  nonzero concentration.  If a non‐random pattern is observed, use a  different calibration model.

Calibration Examples

RU

Biotin Response Curve

Claimed Method Range 30‐150

1430

180

y = 1.0082x ‐ 2.8026 R² = 0.9979

160

1230

140

120

1030

100

830

80

Response

60

630

Method Result (ppm)

40

430

20

0

230

0

20

40

60

80

100

120

140

160

0 10 20 30 40 50 60 70 80 90 100 110

Sodium Sulfite Concentration (ppm)

ng/ml

Concentration

Calibration Residuals

Claimed Method Range 30‐150 ppm Residual Plot

2

1.5

1

0.5

0

Residual

0

20

40

60

80

100

120

140

160

‐0.5

‐1

‐1.5

‐2

Sodium Sulfite Concentration (ppm)

Residuals are all <2%.

Selectivity (Quantitative and Qualitative)

Selectivity is a study designed to demonstrate: • The method’s ability to detect or determine the full range of target  analytes • The method’s ability to detect or determine the target analyte(s)  without interference from matrix or components with similar  properties • The method’s ability to not detect or determine molecules similar  in chemical structure or activity to the analyte that are likely to be  present in matrices of interest.

Selectivity (Quantitative and Qualitative)

• Develop a list of potential interferents.  • Test potential interferents in the presence and absence of the full  range of analytes.   • Report any observed positive or negative interference. • Positive interference is the enhancement of the analytical response in the  presence or absence of the analyte. • Negative interference is the reduction of the analytical response in the  presence of the analyte.

Selectivity Example

Histamine Concentration, ppm

Compound, 1000 ppm

Raw  Sardine

Raw  Mackerel

Raw  Anchovy

Cooked  Tuna

Canned Tuna in  Water

Canned Tuna  in Oil

Canned Mackerel  in Tomato

Canned Pickled  Sardine

Canned Salted  Anchovy

Raw Tuna

Water

3 0 0 0 0 1 0 1 1 1 1

4 9 1 6 1 9 0

5 0 0 0 0 0 0

3 4 0 8 4 2 0

0 0 0 0 0 0 1 7 6 0 6

2 0 2 2 4 3 2 4 6 0 8

5 0 0 0 3 5 0 1 1 1 3

7 4 0 4 5 5 0

1 0 1 0 1 0 0

19 11 10 16 13 19 15

3‐Methylhistamine

Tyramine

L‐Phenylalanine L‐Histidine L‐Tyrosine Tryptamine L‐Tryptophan Cadaverine

0

ND

ND

ND

ND

ND

ND

ND 11 17

ND

ND 18 10 13 19 33 48 63 64 65 63 62 65 ND

ND a

7

0

5

3

Putrescine Anserine Carnosine Agmatine

26

15

26

17

4 4

0 1

3 2

2 5

0 2

250

42 46 60 56 57 46 58 50

49 50 49 47 47 46 47 45

42 45 56 52 53 48 52 37

42 48 53 55 54 52 56 48

39 50 51 56 51 55 58 49

50 50 53 48 48 54 59 36

24 45 45 51 54 52 48 40

190

Water

51 50 47 40 49 56 36

50 53 52 54 53 56 51

3‐Methylhistamine

Tyramine

L‐Phenylalanine L‐Histidine L‐Tyrosine Tryptamine L‐Tryptophan Cadaverine

50

ND 49 46 50 53

ND 47 80 49 52

ND 49 54 48 46

ND 55 80 49 52

ND 54 54 52 55

ND 54 55 51 58

ND 50 51 50 57

ND 59 67 48 54

ND 56 66 54 54

65 69 65 65 71

Putrescine Anserine Carnosine Agmatine

>200

>200

145

>200

>200

>200

>200

>200

>200

a ND = Not determined due to insolubility

Module C2: PTM and OMA  Matrix Studies

Overview

• The Matrix Study determines the method performance in the claimed  matrices. • Matrix Studies can include the use of reference materials, reference  standards and/or reference methods.  • Reference material (RM) – Material, sufficiently homogeneous and stable  with respect to one or more specified properties, which has been  established to be fit for its intended use in a measurement process (NIST). • Certified Reference Material (CRM) – RM characterized by a metrologically  valid procedure for one or more specified properties, accompanied by a  certificate that provide the value of the specified property, its associated  uncertainty, and a statement of metrological traceability.

Overview • Standard Reference Material (SRM) – A CRM issued by NIST that also  meets additional NIST‐specific certification criteria and is issued with  a certificate of analysis that reports the results of its characterizations  and provides information regarding the appropriate use(s) of the  material. Uses include method development, calibration, or ensuring  the integrity of measurement quality assurance programs. • Reference Standard – A traceable standard preparation of analyte  with documented identity and purity. • Reference Method – A pre‐existing official regulatory or other  recognized analytical method against which the candidate method  will be compared.

Overview

• RMs, CRMs, and SRMs that are relevant matrixes can be used as test  materials in a validation study. • Reference standards can be used to prepare solutions to fortify  matrixes to specified analyte concentrations for use as test materials  in a validation study. • Reference methods applicable to the relevant matrixes can be used in  a method comparison validation study. • In rare cases, such as new or complex analytes, reference methods,  RMs, CRMs, SRMs, and reference standards may not be available. In  these cases, alternate strategies should be explored.

Sources of Reference Materials and  Reference Standards • National Institute of Standards and Technology (NIST) https://www.nist.gov/srm • Environmental Protection Agency (EPA)  https://cfpub.epa.gov/si/si_public_record_Report.cfm?Lab=ORD&dirEntryId=38019 • Canadian National Research Council (NRC) https://nrc.canada.ca/en/certifications‐ evaluations‐standards/certified‐reference‐materials/certified‐reference‐materials • US Pharmacopeia (USP) http://www.usp.org/reference‐standards/reference‐standards‐ catalog • Institute of Reference Materials and Measurements (IRMM)  https://ec.europa.eu/jrc/en/science‐update/welcome‐irmm‐reference‐materials‐ catalogue • Chemical Suppliers

Sources of Reference Methods

• AOAC Official Methods of Analysis (OMA) http://www.eoma.aoac.org/subscriber/main.asp • FDA Laboratory Methods https://www.fda.gov/Food/FoodScienceResearch/LaboratoryMethods/default.htm • USDA Food Safety Inspection Service (FSIS) Chemistry Laboratory Guidebook (CLG)  https://www.fsis.usda.gov/wps/portal/fsis/topics/science/laboratories‐and‐procedures/guidebooks‐and‐ methods/chemistry‐laboratory‐guidebook/chemistry‐laboratory‐guidebook • USDA Federal Grain Inspection Service (FGIS) https://www.ams.usda.gov/services/fgis/standardization/reference‐methods • CEN/ISO Methods https://www.iso.org/store.html • Health Canada Health Products and Food Branch (HPFB) Compendium of Methods https://www.canada.ca/en/health‐ canada/services/food‐nutrition/research‐programs‐analytical‐methods/analytical‐methods/compendium‐methods.html • American Public Health Association (APHA) Standard Methods for the Examination of Dairy Products  https://ajph.aphapublications.org/doi/book/10.2105/9780875530024 • APHA Standard Methods for the Examination of Water and Wastewater https://www.standardmethods.org • US Pharmacopeia (USP)/National Formulary https://www.uspnf.com

Terminology

PTM and OMA SLV Matrix Study Designs

Method Type Study Design

Statistics

Acceptance Criteria Determined by  SMEs; suggested  recovery and  precision criteria in  OMA Appendix K 95% CI on dPOD must  include 0 if method  comparison If no reference  method, determine  lowest concentration  that yields POD = 1.0.

Recovery, bias, LOD,  LOQ, s r Regression analysis  per matrix

Quantitative All claimed matrixes in MD lab; 1/5 matrixes  in IL; co‐fortification if multianalyte method 4‐6 concentrations (incl 0) 5 replicates per concentration Qualitative All claimed matrixes in MD lab; 1/5 matrixes  in IL; co‐fortification if multianalyte method ≥6 concentrations (incl 0) – 2 concs yielding  fractional positive results* 20‐30 replicates per concentration *Fractional positives are defined as POD = 0.25‐0.75

Intermediate  reproducibility Probability of  Detection (POD)

OMA Collaborative (MLV) Study Designs

Method Type # Valid Data Sets Study Design per Collaborator

Statistics

Recovery, bias, LOD, LOQ, s r , s R

Quantitative ≥8

≥5 Materials 2 Replicates per material

Qualitative

≥10

1 Matrix 12 Replicates High 12 Replicates Low (fractional pos.*) 12 Replicates Non‐spiked

LPOD s r

with 95% CI with 95% CI

s R

s L If method comparison:  dLPOD + CI (CI must include “0”)

*Fractional positives defined as LPOD = 0.25‐0.75

Choosing Matrices

Choose matrices: • Regulatory requirements • Likely to contain the target analyte • Recent public interest (e.g., outbreaks, recalls, known health risks) • Associated with trade disputes • Marketing considerations

Preparation of Test Materials

• Obtain sufficient amount of single lot of matrix for candidate method analyses and  reference method analyses (if applicable) • Screen food item for endogenous analyte • Recommend using both the candidate and reference methods • Naturally contaminated or incurred matrices are preferred • If analyte not found, or the contamination level is too low, then fortify the matrix  with the target analyte. • Incorporate into matrix using an appropriate technique with appropriate holding conditions • Details are analyte/matrix dependent and will be specified in protocols • Analyte concentrations should cover the intended range of the candidate method

Contamination of Surfaces

• Contaminate surface areas in a random blinded manner with a  volume of the appropriate material that allows even  distribution across the surface without excessive pooling of  liquid • 2.25 mL for 12” x 12” area • 250 µL for 4” x 4” area • 100 µL for 1” x 1” area • Dry overnight (≥16 h) at room temperature – surfaces must be  visibly dry • Swipe surface area with sponge or swab according to the  method.

Performing the Study – Test Portions

• From each material, select the replicate test portions needed for the  Candidate and Reference Methods • Randomize and blind code the test portions for each method • Analyze test portions for each method • In some cases, intermediate reproducibility may be desired • Second analyst, different day, separate equipment

Analyzing Data – Quantitative Methods

• Unblind the data • Plot candidate method results vs. known concentration (or reference  method results) • Perform regression analysis • Perform outlier tests (Cochran and Grubbs) to identify significantly  outlying data points. Data points should rarely be removed, and only for  justifiable cause.

Analyzing Data – Quantitative Method SLV

Calculate for each concentration of each matrix: • Mean – the average result from replicate test portions • Repeatability (s r • Relative Standard Deviation of Repeatability (RSD r ) ‐ s r

) ‐ the standard deviation from replicate test portions

divided by the 

mean candidate method result  • Bias ‐ the difference between the mean candidate method result and  the true value (known fortification level or reference method result) • Recovery – the ratio of the mean candidate method result and the true  value (known fortification level or reference method result)

Limit of Detection (LOD) and  Limit of Quantitation (LOQ) Limit of Detection (LOD)

• The measured quantity value, obtained by a given measurement  procedure, for which the probability of falsely claiming the absence of  a component in a material is β, given a probability α of falsely  claiming its presence • IUPAC recommends default values for αand β equal to 0.05 (or 95%  confidence) Limit of Quantitation (LOQ), also called Limit of Determination • The lowest amount of analyte in a laboratory sample which can be  quantitatively determined with a defined confidence (predetermined  goals for bias and imprecision are met)

Limit of Detection (LOD) and  Limit of Quantitation (LOQ) Estimate LOD and LOQ for each matrix one of several ways: 1. Plot s r vs. mean result from matrix study and perform regression a. Estimate LOD as                , where       is the mean value of blank replicates,      is the  intercept, and      is the slope b. Estimate LOQ as LOD x 3. 2. Analyze 10 blank test portions and 10 test portions containing a low    0 3.3 1 1.65 b X s m   b s m 0 X

concentration of analyte (near the expected LOQ) a. Estimate LOD as the mean blank result + 3 SD low conc b. Estimate LOQ as the mean blank result + 10 SD low conc 3. Analyze 10 blank test portions a. Estimate LOD as the mean blank result + 3 SD b. Estimate LOQ as the mean blank result + 10 SD

Limit of Detection (LOD) and  Limit of Quantitation (LOQ) Next, validate the estimated LOQ (LOQ est ): • Fortify matrix at the LOQ est concentration  • Analyze 10 replicates • Demonstrate RSD r <20%

Data Table – Quantitative Method SLV Candidate Method Results for Histamine in Frozen Tuna Compared to a Known Spike Value

Test Portion

Spike  ppm

Result ppm

Spike  ppm

Result ppm

Spike  ppm

Result ppm

Spike  ppm

Result ppm

Spike  ppm

Result ppm

Spike ppm

Result ppm

1 2 3 4 5 6 7 8 9

0 0 0 0 0 0 0 0 0 0

0.0

2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5

2.1 2.3 2.1 2.4 2.1 2.3 2.2 2.3 2.1 2.3

5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0

5.4 5.1 4.8 5.3 5.2 4.9 4.9 5.6 5.0 5.2

10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0

10.5 10.2 11.0 11.5 10.1 12.1 10.8 10.5 11.1 10.8 0.69 9.8

20.0 20.0 20.0 20.0 20.0 20.0 20.0 20.0 20.0 20.0

24.4 19.7 24.0 22.4 20.2 21.0 22.9 23.6 21.4 23.0 22.3 1.62

50.0 50.0 50.0 50.0 50.0 50.0 50.0 50.0 50.0 50.0

49.5 57.3 51.2 54.9 58.9 59.4 53.6 55.0 56.3 51.7 54.8 3.31

‐0.1 0.0 0.3 0.1 0.2 0.3 0.2 0.3 ‐0.1

10

Mean 0.12

Mean

2.22 0.11

Mean

5.14 0.25

Mean

Mean

Mean

s r

0.16

s r

s r

s r

s r

s r

RSD r 6.04% Recovery 88.8% Recovery 102.8% Recovery 108% Recovery 111.3% Recovery 109.6% Bias ‐0.28 Bias 0.14 Bias 0.8 Bias 2.3 Bias 4.8 4.95% RSD r 4.86% RSD r 6.39% RSD r 7.26% RSD r

Data Graph – Quantitative Method SLV

Candidate Method Results for Histamine in Frozen Tuna Compared  to a Known Spike Value

70

60

50

40

30

y = 1.1016x ‐ 0.1851 R² = 0.9939

20

10

0

Candidate Method Result, ppm

0

10

20

30

40

50

60

-10

Histamine Concentration, ppm 

Data Graph – LOD and LOQ est

Determination of LOD and LOQ est

4

y = 0.0604x + 0.0646 R² = 0.99

3

2

1

Standard Deviation

0

0

10

20

30

40

50

60

Mean Concentration, ppm

Calculation of LOD and LOQ est From the graph of s r vs. mean cand :

y = 0.0604x + 0.0646

  0.12 3.3 0.0646 1 1.65(0.0604)  

  s m

3.3 1 1.65  

X

LOD =                    =                              = 0.37 ppm 0 b

LOQ est

= 3 x LOD = 3 x 0.37 = 1.1 ppm

Analyzing Data – Quantitative Method MLV

Calculate for each material: • Grand Mean – the mean result across all laboratories • Repeatability (s r portions among laboratories • Relative Standard Deviation of Repeatability (RSD r ) ‐ s r

) ‐ the mean standard deviation from duplicate test 

divided by the 

mean candidate method result  • Reproducibility (s R

) ‐ the standard deviation across all laboratories

• Relative Standard Deviation of Reproducibility (RSD R ) – s R the mean candidate method result

divided by 

Data Table – Quantitative Method MLV Raw Data

From Feng et al. (2018)  J AOAC Int . 101, 1566‐1577

Data Table – Quantitative Method MLV  Statistical Summary

From Feng et al. (2018)  J AOAC Int . 101, 1566‐1577

Analyzing Data – Qualitative Methods

• Use Probability of Detection (POD) statistics • POD is the proportion of positive analytical outcomes for a qualitative method  for a given matrix at a given bacterial level or concentration • POD = x/N • Unblind the data • Analyze each level of each matrix separately • Calculate POD and 95% confidence interval (CI) for each method • POD C for the candidate method results • POD R for the reference method results

Analyzing Data – Qualitative Method SLV

• Method comparison achieved by estimating bias of candidate method • Bias estimated as dPOD, the difference between two POD values • dPOD C = POD C – POD R = bias between candidate and reference method results • Calculate 95% confidence intervals on dPOD values • If the confidence interval on dPOD does not contain zero, then the two POD values  are statistically different (bias is significant). • If no reference method, determine lowest level that achieves POD = 1.0. • For antibiotics in milk (FDA CVM program), use probit analysis to determine  concentration that yields 90% POD with 95% confidence.

Data Table – Qualitative Method SLV Matrix Pork (wt %) N a x b POD c 95% CI d Beef 0.00 % 20 0 0.00 0.00, 0.16 0.10 % 20 1 0.05 0.01, 0.24 0.50 % 20 19 0.95 0.76, 0.99 0.99 % 20 20 1.00 0.84, 1.00 2.44 % 20 20 1.00 0.84, 1.00 4.76 % 20 20 1.00 0.84, 1.00 Beef e 0.00 % 20 0 0.00 0.00, 0.16 0.10 % 20 0 0.00 0.00, 0.16 0.50 % 20 12 0.60 0.33, 0.78 1.00 % 20 20 1.00 0.84, 1.00 2.50 % 20 20 1.00 0.84, 1.00 5.00 % 20 20 1.00 0.84, 1.00 Bovine Liver 0.00 % 20 0 0.00 0.00, 0.16 0.10 % 20 0 0.00 0.00, 0.16 0.50 % 20 1 0.01 0.01, 0.24 1.00 % 20 19 0.95 0.76, 0.99 2.49 % 20 20 1.00 0.84, 1.00 4.99 % 20 20 1.00 0.84, 1.00

a N = Number of test portions b x = Number of positive test portions c POD = x/N d 95 % CI = 95 % confidence interval e Data from independent laboratory

POD Graphs

Data Table – Qualitative Method SLV Antibiotics in Milk Drug Concentration (µg/kg) N a x b POD c Conc. at 90 % POD with  95 % Conf. (µg/kg) d Amoxicillin 0 60 0 0.00 5.8 2 30 2 0.067 4 30 15 0.50 6 30 30 1.00 7 30 30 1.00 7.5 30 30 1.00 8 30 30 1.00 Penicillin G 0 60 0 0.00 2.9 1 30 1 0.033 2 30 11 0.40 2.5 30 24 0.80 2.75 30 27 0.90 3 30 29 0.967 4 30 30 1.00 5 30 30 1.00

a N = Number of test portions b x = Number of positive test portions c POD = x/N d By probit analysis

POD Graphs

Detection of Amoxicillin in Raw Milk

Detection of Penicillin G in Raw Milk

1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

POD

POD

0.20

0.20

0.00

0.00

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

Amoxicillin Concentration, µg/kg

Penicillin G Concentration, µg/kg

Analyzing Data – Qualitative Method MLV

• First, analyze data from each laboratory and each material individually • Determine POD values and 95 % Confidence Intervals • If method comparison, calculate dPOD C statistically significant differences • Combine data across laboratories and calculate LPOD with 95% confidence  interval for each material • If method comparison, calculate dLPOD C with 95 % confidence intervals and look for any  statistically significant differences • Determine error estimates s r , s L , and s R and report with 95% confidence  intervals. • Determine the p ‐value of the t‐test based on the X 2 distribution and compare  to 0.10 to determine whether the interlaboratory effect, s L , is statistically  significant. This is the homogeneity test of laboratory POD values. with 95 % confidence intervals and look for any 

Data Table – Qualitative Method MLV Raw Data

From Lacorn et al. (2016) J AOAC Int., 99, 730‐737

Data Table – Qualitative Method MLV  Statistical Summary

From Lacorn et al. (2016) J AOAC Int., 99, 730‐737

POD Graphs – Qualitative Method MLV

From Lacorn et al. (2016) J AOAC Int., 99, 730‐737

Module C3: Robustness Study

Overview • This study evaluates the ability of the method to remain unaffected  by small variations in method parameters that might be expected to  occur when the method is performed by an end user. • The method developer, in conjunction with the AOAC Project  Manager, is expected to make a good faith effort to choose  parameters that can be influenced by the end user and are most likely  to affect the analytical performance and determine the range of  variation that can occur without adversely affecting the analytical  results

Overview Parameters are varied and deviations tested by an appropriate  experimental design: • Plackett‐Burman design (e.g., 7 parameters, 8 treatments) • Full factorial design (k parameters, 2 k   treatments, e.g., 3 parameters, 8  treatments) • Fractional factorial design of choice, 2 k‐1 or  2 k‐2  (k=4‐6 parameters)

Factorial Design

• Full factorial design • For selected parameters, vary conditions above and below nominal test  conditions • Test every possible combination of all variables • Requires 2 n  de terminations, where n is the number of parameters varied • Fractional factorial design • For selected parameters, vary conditions above and below nominal test  conditions • Test a subset of a full factorial design depending upon the fractional design  (e.g.,  ½,  ¼) • Study materials prepared in the same manner as the matrix study materials

Plackett‐Burman Design

• W.J. and Steiner, E.H. (1975), Statistical Manual of the Association of  Official Analytical Chemists, p 50‐55. • Wernimont, Grant T. (1985), Use or Statistics to Develop and Evaluate  Analytical Methods, p 78‐82. • For each parameter, vary conditions above and below nominal test  conditions • For example, test 8 combinations of 7 parameters and analyze data by  parameter  • Study materials prepared in the same manner as the matrix study  materials

Examples of Robustness Parameters Parameters that should be varied will be method dependent.    Examples of parameters that might be varied are as follows:

• temperature at any temperature  critical step  (e.g., storage, extraction,  incubation, mixing) • times for any time critical step (e.g.,  storage, extraction, incubation,  mixing) • order of reagent addition

• test portion mass/volume • laboratory sample holding time  • test portion/solvent ratio • reagent volume(s) • reagent concentration

Plackett‐Burnam Design  (8 combinations, 7 variables)

Determination #

Parameter

1 A B C D

2 A B

3 A b C d E

4 A b

5 a B C d e F g

6 a B

7 a b C D

8 a b

A or a B or b C or c D or d E or e F or f G of g

c

c

c

c

D

d e F

d E

D

E F

e

e

E F g

f

f

f

f

G

g

g

G

G

G

Observed Result

s

t

u

v

w

x

y

z

Fractional Factorial Design  (4 variables, 2 4‐1  factorial design – 1/2  fraction resulting in 8 determination)

Determination #

Parameter

1 A B C D

2 a b C D

3 a B

4 A b

5 a B C d

6 A b C d

7 A B

8 a b

A or a B or b C or c D or d

c

c

c

c

D

D

d

d

Observed Result

s

t

u

v

w

x

y

z

Fractional Factorial Design  (6 variables, 2 4‐2  factorial design ‐ 1/4  fraction resulting in 16 determination) Determination # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 A a A a A a A a A a A a A a A a B B b b B B b b B B b b B B b b Parameter

A or a B or b C or c D or d E or e F or f

C C C C c c c

c

C C C C c c c c

D D D D D D D D d d d d d d d d

E e e E e E E e E

e e E e E E e

F F f

f

f

f

F F

f

f

F F F F f

f

Observed Result

k l

m n o p q r

s

t

u v w x y z

p

Method Parameter Robustness Examples The greater the reduction in combinations from a full factorial, the wider the variation in method parameters  is needed to examine the effect of the parameter variation.

Robustness Test  Parameter

Baseline Value  (Package Insert  Instructions)

Low Value

High Value

Full factorial

4.5 g 4.0 g

5.5 g 5.0 g

Test portion mass

5 g ± 0.5g

Plackett‐Burnam

Full factorial

110 min

130 min

Extraction time

120 min ± 10 min

Plackett‐Burnam

100 min

140 min

Full factorial

10 min

20 min

Time between  extraction and  instrument reading

15 min ± 5 min

Plackett‐Burnam

7 min

25 min

Robustness

Quantitative Methods • Use 1 test portion per treatment  combination • Analyze according to candidate  method with appropriate parameters  altered • Analyze data by appropriate data  analysis: ANOVA, multi‐factor  regression, or generalized linear  model software 

Qualitative Methods • Use 1 test material at analyte concentration that yields fractional  positives (2‐4 positives/5 replicates) • Test 5 replicates per treatment  combination • Use generalized linear model for data  analysis

Conclusions • Method developers should be encouraged that statistically significant  findings in this experiment are not indicative of a faulty method, and  the discovery of significance is not a roadblock to successful method  validation.  • Success of this experiment is not conditional on a conclusion of “no  significant differences were observed.” • Any findings at this stage may be used to modify method parameters,  emphasize areas of caution, or ease or tighten specific parameter  ranges.

Module C4: Product Consistency  and Stability

Product Consistency (Lot‐to‐Lot Variability)

• Study to confirm that the manufacturing of the test  kit and its components is consistent among lots • Testing is performed on 3 unique lots • Each lot must be a uniquely manufactured lot  or consist of uniquely prepared reagents,  supplies, and kit components • Can be combined with stability testing in a  single study; if tested independent of stability  testing, age of test kit is not a variable in this  study, only variation among lots

Product Stability

• Study to support the shelf life statement and confirm that there is no  observable change in performance of the method over the shelf life  under normal storage conditions. Three possible study designs are: 1. Testing is performed on lots representing the full span of the  shelf life of the kits (newly manufactured, middle of term, near  expiration date).   • This design allows the stability study to be performed concurrently  with the product consistency study when unique lots are tested 2. Testing is performed on 1 lot by testing at least 5 time points  over the shelf life of the kit 3. Preliminary stability testing is conducted using accelerated  studies  • Provides only a rough estimate of shelf‐life • Real time data supporting the entire shelf life of the kit under normal  storage conditions must be submitted as soon as available

Product Consistency & Stability: Combined Real‐ Time Study Design for Quantitative Methods

• Three (3) lots of test kits 

• One lot near the expiration date • One lot near the middle of the expiration period • One lot recently manufactured.  • Conducted using a single material (e.g., matrix, calibration  standards, reference standards, reference material)  • Five (5) test portions evaluated for each lot.   • 5 replicates of the target analyte at a high level • 5 replicates of the target analyte at a low level • 5 replicates of blanks • Test in a randomized blind coded fashion. • Decode results and analyze for effects on bias and repeatability. • Data demonstrating no statistical difference in detection  between lots and no significant time slope are required.

Quantitative 3 lots of kits • 5 replicates of target  analyte at low level • 5 replicates of target  analyte at high level • 5 replicates of blanks Bias & repeatability

Product Consistency & Stability: Combined Real‐ Time Study Design for Qualitative Methods

• Three lots of test kits 

• One lot near the expiration date • One lot near the middle of the expiration period • One lot recently manufactured.  • Conducted using a single material (e.g., matrix, reference  standards, reference material, calibration standards)  • Twenty (20) total replicates evaluated for each lot.   • 10 replicates of the target analyte at a fractional level (2‐8) • 10 replicates of blanks • Test in a randomized blind coded fashion. • Decode, calculate POD values and confidence intervals and analyze  data for variable detection between lots/time points.

Qualitative 3 lots of kits • 10 replicates of target material at low level • 10 replicates of blanks

POD + CI

Independent Product Consistency Study

• For test methods that have multiple components with different lot identification numbers • Lots of the individual components will be interchanged prior to analysis • Testing will be conducted in same manner as combined consistency/stability study

Kit Component 1 Lot 1 Lot 2 Lot 3 Lot 1 Lot 2 Lot 3 Lot 1 Lot 2 Lot 3

Kit Component 2 Lot 1 Lot 1 Lot 1 Lot 2 Lot 2 Lot 2 Lot 3 Lot 3 Lot 3

Kit Component 3 Lot 3 Lot 2 Lot 1 Lot 1 Lot 2 Lot 3 Lot 2 Lot 1 Lot 3

Independent Stability Study

Quantitative

Qualitative

Real time or accelerated

Real time or accelerated

1 lot of kits

1 lot of kits

• 5 replicates of target material at  low level • 5 replicates of target material at  high level • 5 replicates of blanks 5 time points over the shelf life of  the test kit

• 10 replicates of target material at  low level • 10 replicates of blanks

5 time points over the shelf life of  the test kit

Bias & repeatability

POD + CI

NT3

Accelerated Stability Study Design

Accelerated Stability Study

Claimed Shelf Life

Required Storage  Temperature

Storage  Temperature

Component

Shelf Life

Time Points

Test Solution 1

25°C

9 mos. (270 d)

55°C

0, 2, 5, 8, 11, 14 d

Test Solution 2

‐20°C

9 mos. (270 d)

25°C

0, 1, 2, 4, 6, 8 d

Test Solution 3

‐20°C

9 mos. (270 d)

25°C

0, 1, 2, 4, 6, 8 d

• Data generated is based on the Arrhenius model  • Assuming Ea = 20 kcal ex.  1 year at 5°C ≈ 32 days at 25°C, and 1 year at 25°C ≈ 45 days at 45°C • Test kit is stored at an elevated temperature to age the product more quickly • Number of replicates/time points is consistent with standard stability study

Slide 8 NT3

Does this even apply to chemistry? Maybe it is only micro. I had never heard of it before. Nancy Thiex, 5/21/2019

Parting Thoughts

• Product consistency and stability testing can be conducted independently or  combined • Fortified materials, extracts or diluents from the matrix studies may be used for this  study • Work with your AOAC Technical Consultant to ensure that the testing performed  meets the study design requirements for an AOAC validation