Attribute agreement analysis

Method

• Purpose: an attribute agreement analysis checks a visual or go/no-go inspection, in which appraisers rate samples with categories (Pass / Fail, G / NG, defect types or grades 1–5) instead of measuring them. It shows whether each appraiser repeats his or her own ratings, whether the appraisers agree with each other and, when the correct rating of each sample is known, whether they agree with it. The layout and statistics follow Minitab's Attribute Agreement Analysis (Stat > Quality Tools).
• Data: long format, one rating per row, with the columns Sample, Appraiser and Response, and optionally Standard (the known correct rating of the sample; one value per sample, the same in each of its rows) and Trial. Every column is read as text, so “1” and “Pass” are both category labels; capital letters count (“pass” ≠ “Pass”, a note points it out). Every appraiser must rate every sample the same number of times; missing and repeated combinations are listed. Without a Trial column the ratings of an appraiser and sample are taken as trials 1, 2, … in the order of the rows.
• Categories: the distinct responses and standards. Their order (for the tables and the two-category disagreement columns) is numeric when every label is a number, otherwise the order of first appearance; it can be typed, labels separated by “;”. Kappa does not depend on the order.
• Assessment agreement: # Inspected = samples, # Matched = samples where, within appraisers, the appraiser gave the same rating in every trial (needs 2 or more trials); each appraiser vs standard, every trial equals the standard; between appraisers, every rating of every appraiser in every trial is the same; all appraisers vs standard, every one of them equals the standard. Percent = 100 · matched / inspected with an exact Clopper-Pearson interval: lower = Beta(α/2; x, n − x + 1), upper = Beta(1 − α/2; x + 1, n − x), 0 for x = 0 and 100% for x = n (confidence level 95% by default, as Minitab).
• Assessment disagreement (each appraiser vs standard): with two categories, “# A / B” counts the samples with standard B that the appraiser rated A in every trial, as a percent of the samples whose standard is B; “# Mixed” counts the samples whose trials differ, as a percent of all samples. With more categories: “# Other than standard” (the same rating in every trial, but not the standard) and “# Mixed”. Matched + disagreements = samples.
• Fleiss' kappa (Fleiss 1971) for n samples each rated m times, x_ij = ratings of sample i in category j, p_j = Σ x_ij / (n m), q_j = 1 − p_j: κ_j = 1 − Σ_i x_ij(m − x_ij) / (n m (m − 1) p_j q_j) per category, and overall κ = Σ p_j q_j κ_j / Σ p_j q_j. Standard errors under H0: κ = 0 (Fleiss, Nee & Landis 1979): SE(κ_j) = √(2 / (n m (m − 1))) and SE(κ) = √2 / (Σ p_jq_j · √(n m (m − 1))) · √((Σ p_jq_j)² − Σ p_jq_j(q_j − p_j)). Z = κ / SE and P(vs > 0) = P(Z > z), one-sided, as Minitab reports it. A category that occurs in none or in all of the ratings compared has no kappa (shown as —).
• Which ratings are compared: within an appraiser, the m = trials ratings of each sample; between appraisers, all m = appraisers × trials ratings of each sample. Each appraiser vs standard: the Fleiss kappa of each trial with the standard (m = 2: the rating and the standard) is computed for every trial and the kappas are averaged, with SE = √(Σ SE_t²) / T over the T trials (the trials taken as independent); z and p follow from these. All appraisers vs standard: the same mean over every appraiser and trial. This averaging is this tool's documented convention. Minitab's own description of these two cases could not be checked here, and the match with Minitab's output is not verified.
• Cohen's kappa is added where exactly two ratings of each sample are compared: within an appraiser with exactly 2 trials (trial 1 vs trial 2), each appraiser vs standard with 1 trial, and between appraisers when there are exactly 2 appraisers with 1 trial each. κ = (p_o − p_e) / (1 − p_e) from the two-way table of the two raters, p_e = Σ r_j c_j; per category from the table collapsed to “j / any other”. SE under H0 (Fleiss, Cohen & Everitt 1969): Var = (p_e + p_e² − Σ r_jc_j(r_j + c_j)) / (N (1 − p_e)²); z and the one-sided p as for Fleiss' kappa.
• Reading kappa: 1 is perfect agreement, 0 is the agreement expected by chance, below 0 is worse than chance. Guideline of Fleiss (1981): above 0.75 excellent, 0.40 to 0.75 fair to good, below 0.40 poor; the AIAG MSA manual quotes the same bands. Both sources are cited from memory and were not checked against the books here. These are rules of thumb, not requirements: use the limits of your customer or procedure. A kappa whose p-value is 0.05 or more is not shown to be above chance and is pointed out.
• Not included: Kendall's coefficient of concordance and correlation for ordinal ratings (Minitab offers them for ordered categories), the AIAG effectiveness, miss rate and false alarm rate, and weighted kappa.
• Charts: Minitab's “Assessment Agreement” graph: the percent matched of each appraiser (dot) with its confidence interval (line), within appraisers and appraiser vs standard side by side.
• The ratings are text, so the results use a decimal point unless “Decimal” is set to comma by hand (the confidence level field accepts either sign). The results CSV lists every table at full precision; a decimal comma gives “;” between columns. The templates (CSV and XLSX) hold the example in the long format with all five columns.
• Files: CSV, TSV and TXT are read as if pasted (UTF-8, UTF-16, otherwise Windows-1257). XLSX: choose the sheet and the columns or a cell range; formula cells use the value Excel last saved, dates are kept as text. Files up to 20 MB are read in this browser only and never uploaded.
• XLSX files are read with SheetJS Community Edition 0.20.3, Apache License 2.0 (license), loaded only when an Excel file is opened or created.