docs(decision): report the external benchmark on the corrected basis

19 scored rather than 20: sku_l06_1.jpg is now the enrolled l06 view, so it is
reported as leaked instead of as a held-out correct accept. 12 correct,
7 abstain, 0 false accepts.

Both leak checks are named in the record and in the external README, since the
enrollment that closed F-12 is exactly the kind of change that silently
converts a held-out row into training data.
This commit is contained in:
kiyo9w 2026-09-29 03:30:16 +09:00
parent e25fcda67d
commit 56cff09292
2 changed files with 34 additions and 21 deletions

View file

@ -8,25 +8,30 @@ Tier 2.** DECISION.md's Tier 2 is a second labelled store visit; these are
|---|---| |---|---|
| `other-mayo/` | 8 accepted images for `sku_l04`, `sku_l07`, `sku_l09` from 6 source domains, plus the scored result | | `other-mayo/` | 8 accepted images for `sku_l04`, `sku_l07`, `sku_l09` from 6 source domains, plus the scored result |
| `l12-reconciliation/` | What `sku_l12` actually is, from the shelf tag and package pixels | | `l12-reconciliation/` | What `sku_l12` actually is, from the shelf tag and package pixels |
| `kewpie/` | Kewpie 450 g research (the near-unknown case) | | `kewpie/` | 12 accepted images for six SKUs from 6 source domains, plus the scored result. One is now enrolled, so it is excluded from scoring. |
| `tier3-sources/` | Two reusable pan clips and 11 rejected candidates for a future Tier-3 replay | | `tier3-sources/` | Two reusable pan clips and 11 rejected candidates for a future Tier-3 replay |
## Leak test ## Leak checks
Every accepted image was template-matched against `add/image6.jpg`. Highest Two, both in `code/eval_external.py`, both run before any number is reported:
score across all 8 is 0.675 (`sku_l07_1`), and the 0.95+ band that identifies a
crop of the evaluation photo is empty. The images are genuinely external.
## Result: 8 images, 0 false accepts - **Against the evaluation photo.** Every image is template-matched against
`add/image6.jpg`. The highest score across all 20 is 0.675 (`sku_l07_1`), and
the 0.95+ band that identifies a crop of the evaluation photo is empty.
- **Against the catalog.** An image that has since been enrolled is refused
rather than scored. `kewpie/sku_l06_1.jpg` is one: it is now the enrolled l06
view, so scoring it would be training accuracy wearing a held-out label.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against ## Result: 19 scored, 0 false accepts
`stock-lopia-clean` (`other-mayo/eval_result.json`):
| Verdict | Count | Files | .venv/bin/python code/eval_external.py --stock stock-lopia-clean \
|---|---|---| data/external-eval/kewpie data/external-eval/other-mayo
| Correct accept | 4 | `sku_l07_1`, `sku_l07_2`, `sku_l09_1`, `sku_l09_2` |
| Abstain | 4 | all `sku_l04_*` | | Set | Scored | Correct | Abstain | False accepts | Leaked |
| False accept | **0** | — | |---|---|---|---|---|---|
| `kewpie/` | 11 | 8 | 3 | **0** | 1 |
| `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
| **Total** | **19** | **12** | **7** | **0** | **1** |
Two things this establishes that the image6 shelf cannot: Two things this establishes that the image6 shelf cannot:

View file

@ -835,7 +835,7 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット
claim about the contents. Measured on the corrected labels, full resolution: claim about the contents. Measured on the corrected labels, full resolution:
precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%. precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
### External benchmark: 20 images, 0 false accepts ### External benchmark: 19 scored, 0 false accepts
The image6 shelf is one photo with one label set, so every number above is The image6 shelf is one photo with one label set, so every number above is
measured on data the design was tuned against. Twenty images for eight SKUs measured on data the design was tuned against. Twenty images for eight SKUs
@ -843,17 +843,25 @@ were collected from independent sources (six retailer and marketplace domains)
to test the pipeline on data it had no part in building. None is Tier 2; all to test the pipeline on data it had no part in building. None is Tier 2; all
are `external_proxy` and every manifest says so. are `external_proxy` and every manifest says so.
Leak-tested first: the highest template match of any of the 20 against Two leak checks run before any number is reported, both in
`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo. `code/eval_external.py`:
- Against the evaluation photo: the highest template match of any image against
`add/image6.jpg` is 0.675, so none is a crop of it.
- Against the catalog: an image that has since been enrolled is refused rather
than scored. `sku_l06_1.jpg` is one - it is now the enrolled l06 view, so
scoring it would be training accuracy wearing a held-out label. It is
reported as `leaked`, not as a correct accept.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
`stock-lopia-clean` (`data/external-eval/*/eval_result.json`): `stock-lopia-clean` (`data/external-eval/*/eval_result.json`):
| Set | Images | Correct | Abstain | False accepts | | Set | Scored | Correct | Abstain | False accepts | Leaked |
|---|---|---|---|---| |---|---|---|---|---|---|
| `kewpie/` | 12 | 9 | 3 | **0** | | `kewpie/` | 11 | 8 | 3 | **0** | 1 |
| `other-mayo/` | 8 | 4 | 4 | **0** | | `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
| **Total** | **20** | **13** | **7** | **0** | | **Total** | **19** | **12** | **7** | **0** | **1** |
- **`sku_l07` is answered with no enrolled image.** Both its images accept - **`sku_l07` is answered with no enrolled image.** Both its images accept
through the text lane alone - one reads ラッキー フヨネーズ, the other the through the text lane alone - one reads ラッキー フヨネーズ, the other the
rear label's 製造者:丸和油脂株式会社 with 内容量:200g, which together rear label's 製造者:丸和油脂株式会社 with 内容量:200g, which together