docs(decision): record the far-unknown calibration and the gate status

The far-unknown claim in the verdict said 0.2-0.7% on 602 crops. That was the
audit's variant matcher on the add/ photos; the shipped lane at the shipped
lines was 3.61% on the Wikimedia sets. Both are now stated, with the sweep that
chose 4.0/4 and the live-endpoint result: 44 named, 42 correct, 95.5% precision
against 53/48/90.6% before.

Gate status is recorded per gate rather than as a blanket claim: G2 and G3's
far half pass offline, G8 passes, and G1/G4/G5/G6/G7 are not yet measured on
held-out data because Tier 2 does not exist.

The Wikimedia manifests and results are committed; the 59 MB of images are not,
because collect_shelf_images.py fetches them from the source URLs the manifests
record.
This commit is contained in:
kiyo9w 2026-09-29 04:11:00 +09:00
parent 07de2bd37d
commit 4bdad66bbe
7 changed files with 1264 additions and 2 deletions

View file

@ -56,8 +56,13 @@ store-scoped, enrolled catalog, with live-video evidence accumulation**.
every enrolled reference can be verified exhaustively. With thousands, it
becomes retrieval plus verification (§6).
- **Open set is the dominant difficulty, and it comes in two kinds.** *Far*
unknowns (other categories) are rejected easily: 0.2–0.7 % false accepts on
602 unrelated crops at a threshold that keeps most correct labels. *Near*
unknowns (other categories) are rejected easily **once the accept lines are
calibrated against them**: 0.37 % and 0.24 % false accepts on 2297 crops from
56 unrelated Wikimedia supermarket photos, at a threshold that keeps 42 of 48
correct image6 labels. The audit's 0.2-0.7 % was measured on 602 crops from
the `add/` photos with the audit's own variant matcher; the shipped lane at
the shipped lines was 3.61 % until 2026-09-29 (§ Re-measured, far unknowns).
*Near*
unknowns (siblings of enrolled products: Kewpie 450 g next to Kewpie からし,
Pure Select plain next to Pure Select コクうま) produce genuine,
geometrically consistent evidence and can't be thresholded away. Near
@ -389,6 +394,14 @@ with always-argmax, run on that same catalog.
| G7 hubness | max predicted-SKU share ÷ that SKU's true share | ≤ 2.0 |
| G8 runtime | server identity latency p95 per crop; server memory | ≤ 60 ms; within the demo host's budget |
Status 2026-09-29, offline on image6 plus the Wikimedia distractor sets:
**G2 passes** (95 % precision, full resolution). **G3's far-unknown half
passes** (0.37 % / 0.24 % on 2297 unrelated crops); its near-unknown half is
measured on image6 only, where the size gate rejects the documented 450 g case.
**G8 passes** (identity p95 well inside 60 ms). **G1, G4, G5, G6, G7 are not
yet measured on held-out data** - Tier 2 does not exist, and G4/G5 need a
recorded walk. Nothing here is a device-path result.
Not complete until G1–G7 pass on the device path (Tier 3), not just offline.
---
@ -838,6 +851,46 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット
claim about the contents. Measured on the corrected labels, full resolution:
precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
### Far unknowns: the accept lines were tuned on a shelf that has no unknowns
Every threshold in this document was fitted on image6, where each crop is a
facing of a product the shelf actually holds. That is a closed-set tuning set,
and it hid the far-unknown rate completely: the shipped lane accepted 83 of
2297 crops from 56 Wikimedia Commons supermarket photos of milk, soy and snack
shelves - 3.61 %, against G3's 1 %.
Those photos share nothing with the LOPIA catalog, so every accept is a false
accept by construction. Two mechanisms produced them:
- **The text lane answered on two coincidental bigrams.** The vocabulary filter
drops bigrams no catalog name contains, so a short garbage read can end up
with a perfect ratio on two pairs. 20 of 26 text-lane answers rested on
exactly two: `避くて おいしい` named `sku_l12` through おい/いし.
- **The local lane accepted pure geometry.** The residual 25 were matches with
no text overlap at all, scoring 2.26-4.81 against an accept line of 2.0.
Both lines moved: `TEXT_MIN_SHARED` 2 -> 4, `LOCAL_ACCEPT` 2.0 -> 4.0. Measured
on 2297 distractor crops and 109 image6 facings:
| `LOCAL_ACCEPT` | `TEXT_MIN_SHARED` | image6 shown / correct | precision | false accepts |
|---|---|---|---|---|
| 2.0 | 2 | 54 / 49 | 90.7 % | 83 (3.61 %) |
| 2.0 | 4 | 52 / 48 | 92.3 % | 30 (1.31 %) |
| **4.0** | **4** | **44 / 42** | **95.5 %** | **8 (0.35 %)** |
| 5.0 | 6 | 36 / 36 | 100.0 % | 0 (0.00 %) |
4.0/4 is the only region passing both G2 (>= 95 % precision) and G3 (<= 1 %
far-unknown) with coverage left. On the shipped evaluator that is 66 %
end-to-end at 95 % precision with 2 false accepts, against 77 % / 91 % / 5
before - the coverage is what the precision costs, and G2 is the gate that says
to pay it.
Reproduce: `.venv/bin/python code/eval_distractors.py --stock stock-lopia-clean
--write data/external-shelves/wikimedia-commons data/external-shelves/wikimedia-commons-2`.
The images are fetched by `code/collect_shelf_images.py` and are not committed;
the manifests (source URL, licence, artist) and the results are.
### External benchmark: 19 scored, 0 false accepts
The image6 shelf is one photo with one label set, so every number above is