diff --git a/recognization/data/external-eval/README.md b/recognization/data/external-eval/README.md index a47cfc0..44f1b6e 100644 --- a/recognization/data/external-eval/README.md +++ b/recognization/data/external-eval/README.md @@ -8,25 +8,30 @@ Tier 2.** DECISION.md's Tier 2 is a second labelled store visit; these are |---|---| | `other-mayo/` | 8 accepted images for `sku_l04`, `sku_l07`, `sku_l09` from 6 source domains, plus the scored result | | `l12-reconciliation/` | What `sku_l12` actually is, from the shelf tag and package pixels | -| `kewpie/` | Kewpie 450 g research (the near-unknown case) | +| `kewpie/` | 12 accepted images for six SKUs from 6 source domains, plus the scored result. One is now enrolled, so it is excluded from scoring. | | `tier3-sources/` | Two reusable pan clips and 11 rejected candidates for a future Tier-3 replay | -## Leak test +## Leak checks -Every accepted image was template-matched against `add/image6.jpg`. Highest -score across all 8 is 0.675 (`sku_l07_1`), and the 0.95+ band that identifies a -crop of the evaluation photo is empty. The images are genuinely external. +Two, both in `code/eval_external.py`, both run before any number is reported: -## Result: 8 images, 0 false accepts +- **Against the evaluation photo.** Every image is template-matched against + `add/image6.jpg`. The highest score across all 20 is 0.675 (`sku_l07_1`), and + the 0.95+ band that identifies a crop of the evaluation photo is empty. +- **Against the catalog.** An image that has since been enrolled is refused + rather than scored. `kewpie/sku_l06_1.jpg` is one: it is now the enrolled l06 + view, so scoring it would be training accuracy wearing a held-out label. -Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against -`stock-lopia-clean` (`other-mayo/eval_result.json`): +## Result: 19 scored, 0 false accepts -| Verdict | Count | Files | -|---|---|---| -| Correct accept | 4 | `sku_l07_1`, `sku_l07_2`, `sku_l09_1`, `sku_l09_2` | -| Abstain | 4 | all `sku_l04_*` | -| False accept | **0** | — | + .venv/bin/python code/eval_external.py --stock stock-lopia-clean \ + data/external-eval/kewpie data/external-eval/other-mayo + +| Set | Scored | Correct | Abstain | False accepts | Leaked | +|---|---|---|---|---|---| +| `kewpie/` | 11 | 8 | 3 | **0** | 1 | +| `other-mayo/` | 8 | 4 | 4 | **0** | 0 | +| **Total** | **19** | **12** | **7** | **0** | **1** | Two things this establishes that the image6 shelf cannot: diff --git a/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md b/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md index 68a9316..3d5ecee 100644 --- a/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md +++ b/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md @@ -835,7 +835,7 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット claim about the contents. Measured on the corrected labels, full resolution: precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%. -### External benchmark: 20 images, 0 false accepts +### External benchmark: 19 scored, 0 false accepts The image6 shelf is one photo with one label set, so every number above is measured on data the design was tuned against. Twenty images for eight SKUs @@ -843,17 +843,25 @@ were collected from independent sources (six retailer and marketplace domains) to test the pipeline on data it had no part in building. None is Tier 2; all are `external_proxy` and every manifest says so. -Leak-tested first: the highest template match of any of the 20 against -`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo. +Two leak checks run before any number is reported, both in +`code/eval_external.py`: + +- Against the evaluation photo: the highest template match of any image against + `add/image6.jpg` is 0.675, so none is a crop of it. +- Against the catalog: an image that has since been enrolled is refused rather + than scored. `sku_l06_1.jpg` is one - it is now the enrolled l06 view, so + scoring it would be training accuracy wearing a held-out label. It is + reported as `leaked`, not as a correct accept. Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against `stock-lopia-clean` (`data/external-eval/*/eval_result.json`): -| Set | Images | Correct | Abstain | False accepts | -|---|---|---|---|---| -| `kewpie/` | 12 | 9 | 3 | **0** | -| `other-mayo/` | 8 | 4 | 4 | **0** | -| **Total** | **20** | **13** | **7** | **0** | +| Set | Scored | Correct | Abstain | False accepts | Leaked | +|---|---|---|---|---|---| +| `kewpie/` | 11 | 8 | 3 | **0** | 1 | +| `other-mayo/` | 8 | 4 | 4 | **0** | 0 | +| **Total** | **19** | **12** | **7** | **0** | **1** | + - **`sku_l07` is answered with no enrolled image.** Both its images accept through the text lane alone - one reads ラッキー フヨネーズ, the other the rear label's 製造者:丸和油脂株式会社 with 内容量:200g, which together