docs(decision): report the external benchmark on the corrected basis

19 scored rather than 20: sku_l06_1.jpg is now the enrolled l06 view, so it is
reported as leaked instead of as a held-out correct accept. 12 correct,
7 abstain, 0 false accepts.

Both leak checks are named in the record and in the external README, since the
enrollment that closed F-12 is exactly the kind of change that silently
converts a held-out row into training data.
This commit is contained in:
kiyo9w 2026-09-29 03:30:16 +09:00
parent e25fcda67d
commit 56cff09292
2 changed files with 34 additions and 21 deletions

View file

@ -8,25 +8,30 @@ Tier 2.** DECISION.md's Tier 2 is a second labelled store visit; these are
|---|---|
| `other-mayo/` | 8 accepted images for `sku_l04`, `sku_l07`, `sku_l09` from 6 source domains, plus the scored result |
| `l12-reconciliation/` | What `sku_l12` actually is, from the shelf tag and package pixels |
| `kewpie/` | Kewpie 450 g research (the near-unknown case) |
| `kewpie/` | 12 accepted images for six SKUs from 6 source domains, plus the scored result. One is now enrolled, so it is excluded from scoring. |
| `tier3-sources/` | Two reusable pan clips and 11 rejected candidates for a future Tier-3 replay |
## Leak test
## Leak checks
Every accepted image was template-matched against `add/image6.jpg`. Highest
score across all 8 is 0.675 (`sku_l07_1`), and the 0.95+ band that identifies a
crop of the evaluation photo is empty. The images are genuinely external.
Two, both in `code/eval_external.py`, both run before any number is reported:
## Result: 8 images, 0 false accepts
- **Against the evaluation photo.** Every image is template-matched against
`add/image6.jpg`. The highest score across all 20 is 0.675 (`sku_l07_1`), and
the 0.95+ band that identifies a crop of the evaluation photo is empty.
- **Against the catalog.** An image that has since been enrolled is refused
rather than scored. `kewpie/sku_l06_1.jpg` is one: it is now the enrolled l06
view, so scoring it would be training accuracy wearing a held-out label.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
`stock-lopia-clean` (`other-mayo/eval_result.json`):
## Result: 19 scored, 0 false accepts
| Verdict | Count | Files |
|---|---|---|
| Correct accept | 4 | `sku_l07_1`, `sku_l07_2`, `sku_l09_1`, `sku_l09_2` |
| Abstain | 4 | all `sku_l04_*` |
| False accept | **0** | — |
.venv/bin/python code/eval_external.py --stock stock-lopia-clean \
data/external-eval/kewpie data/external-eval/other-mayo
| Set | Scored | Correct | Abstain | False accepts | Leaked |
|---|---|---|---|---|---|
| `kewpie/` | 11 | 8 | 3 | **0** | 1 |
| `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
| **Total** | **19** | **12** | **7** | **0** | **1** |
Two things this establishes that the image6 shelf cannot:

View file

@ -835,7 +835,7 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット
claim about the contents. Measured on the corrected labels, full resolution:
precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
### External benchmark: 20 images, 0 false accepts
### External benchmark: 19 scored, 0 false accepts
The image6 shelf is one photo with one label set, so every number above is
measured on data the design was tuned against. Twenty images for eight SKUs
@ -843,17 +843,25 @@ were collected from independent sources (six retailer and marketplace domains)
to test the pipeline on data it had no part in building. None is Tier 2; all
are `external_proxy` and every manifest says so.
Leak-tested first: the highest template match of any of the 20 against
`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo.
Two leak checks run before any number is reported, both in
`code/eval_external.py`:
- Against the evaluation photo: the highest template match of any image against
`add/image6.jpg` is 0.675, so none is a crop of it.
- Against the catalog: an image that has since been enrolled is refused rather
than scored. `sku_l06_1.jpg` is one - it is now the enrolled l06 view, so
scoring it would be training accuracy wearing a held-out label. It is
reported as `leaked`, not as a correct accept.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
`stock-lopia-clean` (`data/external-eval/*/eval_result.json`):
| Set | Images | Correct | Abstain | False accepts |
|---|---|---|---|---|
| `kewpie/` | 12 | 9 | 3 | **0** |
| `other-mayo/` | 8 | 4 | 4 | **0** |
| **Total** | **20** | **13** | **7** | **0** |
| Set | Scored | Correct | Abstain | False accepts | Leaked |
|---|---|---|---|---|---|
| `kewpie/` | 11 | 8 | 3 | **0** | 1 |
| `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
| **Total** | **19** | **12** | **7** | **0** | **1** |
- **`sku_l07` is answered with no enrolled image.** Both its images accept
through the text lane alone - one reads ラッキー フヨネーズ, the other the
rear label's 製造者:丸和油脂株式会社 with 内容量:200g, which together