tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity
product-recognition-demo/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md
kiyo9w dfdaef4e61 docs(decision): record the external benchmark and the F-12 fix 
Twenty independent-source images across eight SKUs, leak-tested (max template
match against add/image6.jpg is 0.675) and scored through the shipped lane:
13 correct, 7 abstain, 0 false accepts. The seven abstentions are all
unenrolled or weakly-read SKUs.

The result worth keeping is sku_l07: no enrolled image, both images accepted on
text alone. That is the text lane's coverage claim holding on data it had no
part in building.

F-12 is marked fixed with its measured effect, and the packshot-OCR check that
would have caught it is recorded as implemented but not yet trustworthy - it is
a false negative on l04 and l07.
2026-09-29 03:23:31 +09:00

891 lines
60 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Dorayaki product recognition: architecture decision
Audited 2026-09-28 · submodule `25dbf13` on `fix/live-recognition-quality` plus
its uncommitted diff (snapshot: `git stash create` → `2785c07`) · auditor:
Claude Opus 5.5
Scope: identity pipeline (matcher, gallery, decision, evaluator, catalogs,
labels), live track/label logic, the uncommitted diff. Not audited: detector
quality, tag/planogram pipeline, iOS/Android rendering, on-device runtime.
Evidence scripts and the corrected labels: [`evidence/`](evidence/README.md).
## Verdict
Build a **local-feature instance matcher plus a text lane, behind a calibrated
open-set decision, with no global embedding in the critical path**. At the
current scale the matcher runs exhaustively; past ~150 SKUs, a
RootSIFT descriptor index replaces exhaustive matching. Generic DINOv2 is
rejected for both classification and retrieval.
The biggest risk is none of the architectures. It is that **there is no
trustworthy benchmark**. The only labelled data is one photo, `add/image6.jpg`,
and 26 of its 111 labels (23 %) were wrong. Every number in the handoff was
computed against those labels. Label and catalog errors are larger than any
architecture difference measured so far. They also hide the real product
failure: displayed-label precision is 29–43 % because same-brand siblings the
catalog doesn't know about get labelled with confidence.
On corrected labels, with thresholds tuned on the same single photo (so read
this as direction, not proof), the recommended design beats Van's contract on
coverage and on precision together:
| Live resolution, clean 7-SKU catalog, 61 in-catalog + 48 unknown facings | Correct shown | Coverage (of 61) | Precision of shown labels | Unknowns labelled |
|---|---|---|---|---|
| Van's matcher, always argmax (the floor) | 35 | 57 % | 35 % | 48 / 48 |
| Current server gate (`decide_identity` defaults) | 32 | 52 % | 43 % | 22 / 48 |
| Improved matcher, geometric-confidence gate only (τ=2) | 26 | 43 % | 79 % | — |
| **Improved matcher + OCR veto + OCR lane (τ=2)** | **36** | **59 %** | **95 %** | **0 / 48** |
| Same, looser (τ=1) | 40 | 66 % | 87 % | 2 / 48 |
The one experiment that would most change this recommendation is a second,
independently captured and labelled shelf visit. §7 defines it.
---
## 1. Problem formulation
Dorayaki's identity task is **open-set instance recognition over a small,
store-scoped, enrolled catalog, with live-video evidence accumulation**.
- **Instance, not category.** The classes are individual printed artworks
(JAN-level SKUs). Siblings share maker, palette, lattice pattern, mascot and
silhouette, and differ by a word ("コクうま 65%"), a colour (pink vs red
lattice), or a small logo. That is the regime of planar instance matching
(logo and text geometry), not of semantic embeddings. Measured: raw DINOv2 is
12–21 % top-1 on 7 SKUs; the local matcher is 76–98 %.
- **Retrieval-shaped at scale, closed-set-shaped today.** With tens of SKUs,
every enrolled reference can be verified exhaustively. With thousands, it
becomes retrieval plus verification (§6).
- **Open set is the dominant difficulty, and it comes in two kinds.** *Far*
unknowns (other categories) are rejected easily: 0.2–0.7 % false accepts on
602 unrelated crops at a threshold that keeps most correct labels. *Near*
unknowns (siblings of enrolled products: Kewpie 450 g next to Kewpie からし,
Pure Select plain next to Pure Select コクうま) produce genuine,
geometrically consistent evidence and can't be thresholded away. Near
unknowns get fixed by **catalog completeness** and by **text**, not by a
better score.
- **Multimodal by nature.** Text is the designed discriminator between
siblings; manufacturers print the variant name precisely so shoppers can
tell them apart. OCR identifies the right SKU among all 12 catalog rows on
41–42 of the 55 facings where it reads anything, including SKUs with no
enrolled image, using only catalog names and packshot text.
- **Temporal.** Live video gives many cheap, noisy looks at one physical
facing. The right model is accumulated evidence per track, with a
provisional → confirmed label contract. The current model is one accepted
observation that locks the label forever.
So the hierarchy is: store/category scope → candidate generation (trivial
now, descriptor index later) → local geometric verification + text
verification → calibrated open-set decision → temporal accumulation.
## 2. Evidence audit
`R` reliable · `C` directionally useful but confounded · `X` invalidated ·
`U` still unknown. "Reproduced" means re-run in this audit with the scripts in
`evidence/`.
| # | Handoff claim | Verdict | What this audit found |
|---|---|---|---|
| 2 | Van's matcher is SIFT/ORB/BRISK + RANSAC, always argmax, no rejection | R | Scoring core in `code/feature_matcher.py` is byte-for-byte Van's (`git diff b443e3c`); only `decide_identity` was added. "SURF" is SIFT (SURF isn't in this OpenCV build). |
| 2 | "170/170 named ≠ 170/170 correct" on image.jpg | R (logic) / U (numbers) | No labels exist for `image.jpg`; its accuracy was never measured. |
| 3 | Current server collapsed onto hub SKUs (sku_003, sku_l09) | C | Hubness is real for DINO (50–60 % max-SKU share on the clean catalog, reproduced). The l09 collapse was mostly a wrong l09 reference plus shelf-cut templates. |
| 3 | Live resolution degrades identity | X | Controlled rerun: with correctly scaled boxes, live is not worse than full (original matcher 35/42 live vs 32/42 full; improved 36 vs 40). |
| 3 | DINO embeddings are inappropriate for fine-grained SKU identity | R | Reproduced: gallery top-1 21 % full / 12 % live, hub alarm. |
| 4 | `stock-lopia` was corrupt, leaked, and incomplete | R | Worse than reported: its crops come from the eval photo, so scoring against it is memorization. It scores 94–97 % "identity", and the evaluator's leak check misses it because the re-cut crops aren't the same size. |
| 7 | `eval_identity.py` is the right research direction | C | Right skeleton, four more defects (F-07). |
| 8 | Bug #1, template IDs vs SKU IDs | R, and **still live in production** | Fixed only in the evaluator. The live path accepts `sku_l01_1`, then finds no catalog row for it (F-02). |
| 9 | Hybrid 80 % > matcher 70 % | X | About one facing, on wrong labels. |
| 10 | Bug #2, unscaled boxes caused the live collapse | R | Reproduced on *both* catalogs: unscaled boxes give 11–17 %, scaled boxes give 76–97 %. The confound is resolved. The coordinate fix, not the catalog cleanup, removed the collapse. |
| 11 | Clean packshots are a dramatic improvement | R, with a caveat | True against the leak-free baseline. The l06 packshot shows a superseded package design (F-12). The l09 packshot was 305 px tall at audit time and has since been replaced by a 1000x1000 one, so that caveat no longer applies to the shipped catalog. |
| 12 | Bug #3, the reject gate inside the shortlist | R | Hybrid-49 %-vs-20 % result is invalid. |
| 13 | DINO top-K sweep: recall 71 %/100 %, K doesn't hurt the matcher | C | On a 7-SKU catalog, top-5 is 71 % of the catalog; that sweep can't speak to retrieval. Reproduced on v2: top-3 recall 79–81 %, top-5 95–98 %. |
| 14 | Matcher ~8 ms vs hybrid ~45 ms | R, with a correction | The hybrid lane scores the **full** catalog and then filters (`eval_identity.py` `HybridLane.rank`), so its latency is DINO + full matcher by construction. The measurement is honest; it doesn't describe a real retrieval design. |
| 15 | 69 % full / 66 % live after the l09 fix | R as arithmetic, X as a measurement | Reproduced exactly (24/35, 23/35) **on v1 labels**. On corrected labels the same matcher and catalog give 71 %/76 % after the gate and 76 %/83 % ungated top-1 ranking. The 69/66 gap that justified `crop_first_attempt` doesn't exist. |
| 16 | Local matcher is the strongest identity primitive | R | And improvable: multi-scale RootSIFT + colour re-rank 40/42 vs 32/42 at full (sign test p = 0.04); at live 36 vs 35 (noise). |
| 16 | DINO as coarse retriever at scale | X (this setting) | Recall@10 falls 42 → 32 → 17 → 7 of 42 as the catalog grows 7 → 57 → 207 → 493 SKUs. A RootSIFT descriptor index holds 37–38/42 recall@5 over the same range. |
| 17 | Threshold alone can't create correct identities | R | And the missing identities are mostly catalog gaps (§3). |
| 17 | Argmax precision ~20 % | C | 29 % full / 35 % live on corrected labels; the old number counted mislabelled facings. |
| 21 | Live tests: 49 tests, 1 failure, 5 errors | X (stale) | `python -m unittest test_live` now passes 49/49 on both the dirty tree and the committed baseline. One test feeds a `str` result that raises inside a future callback and is swallowed, on both trees (F-13). |
| 22 | "High confidence" list items 1–8 | R | All hold. Item 4 is now measured with corrected labels. |
| 22 | Medium confidence 1 (69 %/66 %) | X | See row 15. |
| 22 | Medium confidence 3 (more views help) | R direction | Leave-facing-out proxy (§3, E-5) supports it strongly, with a same-photo caveat. |
| 22 | Not established 1–9 | U | 1 and 4 are now partly answered (§6); 2 partly (RootSIFT/multi-scale beat Van's composite); 3, 5, 6, 7, 8, 9 remain open. Item 9 (ground truth) was the most consequential open item: it was wrong. |
### The four ceilings, separated
On corrected labels (`eval/labels/image6.json`): 109 scored facings, 61
in-catalog across 11 SKUs, 48 unknown, 2 excluded. The clean catalog enrolls 7
SKUs covering 42 of the 61 in-catalog facings.
| Ceiling | Value (live, improved matcher) | What limits it |
|---|---|---|
| A. Catalog | 42 / 61 = 69 % of in-catalog facings are answerable by image | l10 (11 facings), l07 (6), l04, l08 unenrolled; l12 has **zero** real facings on the shelf |
| B. Retrieval | n/a at 7 SKUs; RootSIFT index recall@5 = 38/42 at 493 SKUs | the 4 misses are catalog-quality cases (l06 package version) |
| C. Identity (ungated top-1 on enrolled) | 36 / 42 (86 %, Wilson 72–93 %) | l06 at live (1/4), two l09 facings that are mostly occluded |
| D. Decision | 26 correct at 79 % precision (geometry only); 36 at 95 % (+ text) | near-unknown siblings: 9 of 11 unenrolled l10 facings → l09, Kewpie 450 g → l03 |
## 3. Root-cause model
Ordered by how much of the gap to a good product each class explains.
**Evaluator and ground-truth problems** (they decided every conclusion so far)
- 23 % of v1 labels were wrong: 14 in-catalog facings marked unknown (Egg Care
#4–5, Fitte #22–23, Lucky #27–28, Kokuuma #40–45, Pure Select #59); 9
non-catalog items labelled as catalog SKUs (demi-glace pouches #31–32 as l09,
a garlic jar #16 as l05, a Heinz can #64 as l10, dressing bottles #76–79 as
l12, Kewpie 450 g #85 as l03); 2 neighbour shifts (#29 l08→l07, #75
l12→l11). The pattern (runs shifted along a shelf row, a `match` IoU field on
every row) looks like labels propagated from shelf-tag or box matching rather
than read from the packages.
- The evaluator reports `top1` and `coverage` as the same quantity (accepted
correct / enrolled), so the identity ceiling and the decision ceiling were
never separated.
- Precision excludes unenrolled in-catalog facings, which are the most
dangerous false accepts.
- The gallery's candidate set comes from ground-truth labels.
- Leak detection only catches same-size crops.
**Catalog and data problems** (the largest recognition lever)
- Incomplete category enrollment is what creates near unknowns. In the
leave-facing-out proxy (E-5), adding one shelf view for each of the 11 SKUs
on the shelf takes argmax from 34/52 to 47/52. The number that survives at
95 % precision goes from 14–16 to 32–39.
- Wrong or stale references: the l09 history; the l06 package is a superseded
design (the match holds at full resolution, fails at live: 1/4); the l09
packshot is 305 px, so its scores dwarf others under Van's formula.
- Metadata integrity: `stock-lopia-clean/stock.csv` has no `sku_l09` row for
`sku_l09.jpg`. Its `sku_l10` row carries l09's stock counts with l10's name
and price. The catalog name for l12 ("焼肉のたれ 中辛", ¥299) probably
names the ユーラス 旨たれ bottles on the shelf (tag ¥299). Several catalog
sizes disagree with the shelf tags (l10 tag says 400 g).
**Identity-model problems** (real but second-order on this data)
- Scale mismatch: ~1000 px packshots vs 110–250 px crops is a 4–8× ratio.
ORB's 8×1.2 pyramid tops out at 3.6×, so ORB contributes **zero** inliers at
live resolution, and BRISK nearly none. Van's weighted composite is
effectively SIFT-only there, carrying dead weight.
- Van's score, `inliers·1.5 + inliers/min(kp)·80`, isn't comparable across
templates of different resolution. With fewer than 4 matches it counts raw
ratio-test matches as "inliers" without any geometric check (`compute_geometric_inliers`).
- Grayscale-only: pink (ゼロ) vs red (ハーフ) lattice is invisible to SIFT. The
hue re-rank fixes 2–3 facings in every configuration.
**Open-set decision problems**
- `decide_identity` for the inlier scale accepts a score ≥ 1.0 with a relative
margin ≥ 0.15. A single unverified match clears the floor, and the margin
can't reject a sibling with a strong partial match. It accepts 22–28 of 48
unknowns.
**Live and tracking problems** (code-path evidence, unmeasured on device)
- A confirmed track is never re-evaluated (`_on_classified` returns early when
`track.confirmed`), and with `vote_agree = 1` the first accepted observation
locks it. A wrong first label is permanent. A later, sharper, correct view
can't correct it.
- Aggregation is the mean over the last three observations, with empty
observations counted as zeros. That's a heuristic, not evidence
accumulation.
- Uncommitted diff: the crop blur gate returns without consuming an attempt,
so a permanently soft crop (a distant facing) is retried every `retry_gap`
frames for as long as it stays in view.
**UX latency problems**
- No provisional label exists on the wire or in the clients, so any multi-look
confirmation becomes dead time on screen. That's why `vote_agree = 2` was
(correctly) backed out.
## 4. Architecture candidates
"Recognition" = does it improve correct shown labels at fixed precision?
"Scalability" = does it keep latency and recall flat as the catalog grows?
| ID | Candidate | Tested here? | Recognition | Scalability | Verdict |
|---|---|---|---|---|---|
| A | Van's matcher, full reference set, argmax | yes | 32–35 / 42 identity; 29–35 % precision | linear, ~0.1–0.25 ms per reference | Floor. Not shippable (no open set). |
| B | Matcher + engineering: RootSIFT, per-SKU reference pyramid (200/400 px tall), SKU grouping, similarity-constrained RANSAC, hue feature, calibrated gate | yes | 40/42 full (p = 0.04 vs A), 36/42 live; gate alone reaches 79–85 % precision at 36–43 % coverage | same as A | **Adopt** as identity core. |
| C | Learned local features: ALIKED/DISK/SuperPoint + LightGlue; dense LoFTR/RoMa | no (weights not installed; downloading needs your approval) | Unknown. Headroom on enrolled facings is ≤ 6/42 at live, mostly catalog-caused. Likely strongest on small or blurred crops and package-version drift. | LightGlue is per-pair; tens of ms per pair on GPU and much more on CPU, so only viable as a top-K verifier | Postpone to experiment E-C. Adopt only if it clears the gate. |
| D | Retrieval → Van's matcher | yes (DINO) | DINO recall collapses with scale (7/42 @10 at 493 SKUs) | DINO adds ~41 ms per crop | **Reject DINO retrieval.** |
| D′ | RootSIFT descriptor index (FLANN) → vote → verify top-10 | yes | 35–37/42, equal to or slightly better than exhaustive | 1.7–3.9 ms per crop flat from 7 to 493 SKUs; build 171 ms for 64k descriptors | **Adopt past ~150 SKUs.** |
| E | Retrieval → learned verifier | no | — | — | Only if C wins. |
| F | OCR text lane against catalog text signatures | yes (macOS Vision) | Veto: precision 62 → 82 % at τ=0.5 for −1 correct. As a lane: +10 correct at live, including unenrolled SKUs. | ~18–26 ms per crop on Mac; on-device on iOS (Vision) and Android (ML Kit) | **Adopt.** |
| G | Fusion (geometry + colour + text + temporal + tag prior) | partly (geometry + colour + text) | Largest measured gain | — | **Adopt**; the text and geometry errors are complementary (text reads siblings apart; geometry covers text-poor faces). |
| H | Store-trained identity head on DINO | no (it was trained on the eval shelf) | Leaked; no valid measurement exists | — | Retire with DINO. |
| I | Barcode/JAN | — | Front faces rarely show JANs; shelf tags carry product text, not scannable codes, at scan distance | — | Only for the enrollment tool (scan a JAN, pull the packshot). |
| J | Shelf-tag / planogram prior | no (`tags.py` exists) | Every facing run on image6 sits above a readable tag with name and price | — | Strong hypothesis, H-02. |
## 5. Recommended architecture (current scale: tens of SKUs)
```
frame / native crop
│
├─ local lane: RootSIFT on crop ─► verify against EVERY reference view
│ (exhaustive while refs ≲ 300), similarity-RANSAC,
│ per-SKU evidence = max over views:
│ {inliers, 2nd-best ratio, inlier hull cover, hue agreement}
│
├─ text lane: OCR (upsampled ×2 when crop height < 200 px)
│ ─► bigram similarity to every catalog row's text signature
│ (name + OCR of its packshot), including rows with no image
│
▼
decision: P(correct | local, text) from a small calibrated model
accepted if P ≥ p_confirm (target precision ≥ 95 %)
provisional if P ≥ p_provisional (shown as tentative)
veto if text names a different catalog row by margin
unknown otherwise
│
▼
track: log-odds accumulation per track across looks; confirm on
threshold or text agreement; re-open on strong contradiction
(hysteresis), never on a single weak frame
```
Why each piece earns its place:
- **Local lane.** It's the only primitive that ranks fine-grained siblings
correctly: 36–40 of 42 enrolled facings, against 5–9 for DINO. It costs
~4–10 ms per crop at 7 SKUs.
- **Reference pyramid (200/400 px) and RootSIFT.** Together they fix the scale
mismatch that kills ORB/BRISK and makes scores comparable across references.
Drop ORB/BRISK unless ablation H-08 finds a case they win.
- **Hue feature.** It's the cheapest fix for colour-coded variants (+2–3
facings everywhere).
- **Text lane.** It's the only measured mechanism that rejects near-unknown
siblings, and it can name catalog SKUs that have no image yet. It
contributed +10 correct labels and 0 unknown accepts at live, τ=2.
- **Calibrated decision.** It replaces a hand-set floor that accepts one
unverified match. It's calibrated on Tier-2 data, never on the photo it's
evaluated on.
- **No DINO, no retrieval stage.** Neither adds recognition at this scale, and
DINO actively fails at larger scale. Exhaustive verification is cheaper than
one DINO forward pass until ~300–400 references.
Placement: keep the matcher server-side for now; it's small and deterministic.
Put OCR on the phone when tag and label OCR move there. Apple Vision and ML
Kit are the official platform paths, matching the repo's `platform/` rule.
Send the text with the crop so the server fuses both. Until then, run OCR on
the server (macOS Vision is already wired in `code/ocr.py`).
## 6. Scaling architecture
Measured cost and recall to ~500 SKUs; beyond that, extrapolated and marked.
| Catalog | Candidate generation | Verification | Evidence |
|---|---|---|---|
| 10–150 SKUs | none (exhaustive) | local + text | 3.7 ms at 7 SKUs, 7.7 ms at 57, 21 ms at 207 (small refs). Packshot pyramids cost ~2–3× per reference. |
| 150–5,000 | RootSIFT descriptor index (FLANN now; FAISS IVF-PQ when memory matters), vote per SKU, top-10 | local + text on the shortlist | Measured flat 2–4 ms and recall@5 38/42 up to 493 SKUs. Descriptor memory at 5k SKUs × 2 views × ~500 descriptors ≈ 5 M × 128 B (uint8) ≈ 640 MB before PQ. **Extrapolated.** |
| 5,000–100,000 | Scope first: store assortment → aisle/bay (tag or planogram prior) cuts candidates to hundreds. Then an aggregated-local-feature index (ASMK/VLAD-style) or a product-instance embedding fine-tuned on packshot↔shelf pairs. | local + text on ≤ 20 | **Hypothesis.** No store stocks 100k SKUs at once. Scoping by store and bay is the real scaling lever. Generic DINO is ruled out; a fine-tuned embedding needs paired training data that doesn't exist yet. |
Exhaustive matching stops being sensible when its p95 exceeds the per-crop
budget. At ~0.1–0.25 ms per reference and a 30 ms budget, that's roughly
150–300 references (≈ 75–150 SKUs at 2 views). **Descriptor indexing removes
the need for a global DINO pass**: at 493 SKUs it retrieves better than DINO
at a tenth of the cost. The open question is same-category growth: hundreds of
mayonnaises instead of other-category distractors. That's experiment E-6.
## 7. Experimental plan
Rule for every experiment: change one variable, run on fixed crops and a fixed
catalog hash, log per facing (truth, enrolled?, per-SKU evidence vector, truth
rank, decision, lane latencies), and report the four ceilings separately.
### Benchmark tiers (build these first; they block every other decision)
- **Tier 1, diagnostic:** `eval/labels/image6.json` (109 facings). Dev only.
Free to tune on. Its numbers never ship.
- **Tier 2, held-out regression:** new captures that are never enrolled and
never tuned on. At least 3 visits across ≥ 2 LOPIA locations or days, same
aisle, both still photos and the phone's own 1920×1080 frames. Target ≥ 300
in-catalog facings over ≥ 20 SKUs, ≥ 300 near unknowns from the same aisle,
and the 602-crop far-unknown set from `add/`.
- Labels need provenance: shelf-tag correspondence plus readable package
text, two independent passes, disagreements marked `exclude`.
- Every label row carries `source`, `confidence`, and `labeler`. A label
file without them fails a schema check.
- **Tier 3, live replay:** FrameRecorder walks (`code/record.py`) of the same
aisles with Tier-2-style labels, replayed through `code/eval_live.py`.
### Experiments
| ID | Question | Variable | Metric | Acceptance / falsifier |
|---|---|---|---|---|
| E-0 | Are the labels right? | none | Double-labelling agreement on Tier 2 | κ ≥ 0.9 on SKU identity. Otherwise fix labels before anything else. |
| E-1 | Does B beat A on held-out data? | matcher variant | Paired identity top-1 on enrolled facings, McNemar | B ≥ A with p < 0.05 at native-crop resolution. If A ≥ B, keep Van's scoring and add only SKU grouping and the decision layer. |
| E-2 | Is ORB/BRISK dead weight? | drop ORB/BRISK from B | top-1, latency | Drop them if top-1 changes by ≤ 1 facing per 100. |
| E-3 | Does text move the frontier? | ± text lane | Correct shown at precision ≥ 95 %; near-unknown FAR | Text lane adds ≥ 5 pp coverage at 95 % precision. Otherwise text becomes veto only, or is removed. |
| E-4 | Calibrated decision vs hand threshold | decision policy | Coverage at 95 % precision, calibration (ECE) on Tier 2 | Model ECE ≤ 0.05 and coverage ≥ hand threshold's. |
| E-5 | How much is catalog completeness worth? | enroll the aisle's remaining SKUs with packshot + one shelf view from a *different* visit | Coverage at 95 % precision; near-unknown FAR | ≥ 70 % of the same-photo proxy gain (14→32 full, 16→39 live) survives cross-visit. If not, shelf views are overfitting to capture conditions. |
| E-6 | Same-category scaling | grow the catalog with other sauce/mayo SKUs (packshots) | top-1, recall@K of descriptor index vs exhaustive; latency p50/p95 | Index recall@10 ≥ 98 % of exhaustive top-1 at ≥ 300 SKUs. Otherwise stay exhaustive and parallelize. |
| E-C | Learned matcher | ALIKED+LightGlue (or DISK+LightGlue) as verifier over B's top-5 | top-1, coverage at 95 % precision, latency | Adopt only if +3 pp coverage at 95 % precision **and** p95 identity latency ≤ 80 ms server-side. |
| E-7 | Live temporal model | log-odds accumulation + provisional labels vs current single-shot lock | Tier 3: time to provisional, time to correct confirmed, flips, correct labels lost | Time-to-first-correct p50 ≤ current single-shot; flips ≤ 2 %; correct-then-lost ≤ 1 %. |
| E-8 | Does first-attempt native crop help? | `crop_first_attempt` on/off | Tier 3 correct confirmed labels, crop traffic | Keep only if +3 pp correct at ≤ 1.5× crop traffic. On Tier 1 it has no basis: live ≥ full. |
| E-9 | Tag prior | ± shelf-tag evidence from `tags.py` | Coverage at 95 % precision | Adopt if +5 pp; also use it for catalog-gap discovery. |
## 8. Migration strategy
Snapshot first: the uncommitted work is saved as stash object `2785c07` (and as
`tracked.diff` / `untracked.tgz` in the audit scratchpad). Nothing has been
reverted.
| Item | Class | Action |
|---|---|---|
| `code/eval_identity.py` (untracked) | evaluation infrastructure | **Keep and fix** (F-07). |
| `gallery.py` `rank()` (diff) | evaluation infrastructure | Keep, eval-only; no production caller. |
| `run-demo-server.sh` `--identity matcher` (diff) | clearly correct production fix | **Keep.** Also change the default `STOCK` away from `stock-lopia` (F-03). |
| `live.py` `crop_first_attempt`, `max_attempts 2→4` | experimental | **Revert.** Its stated rationale (69 vs 66) is invalidated. Re-test as E-8. |
| `live.py` blur gate early return, `sharpen_ratio`, `best_sharpness` | experimental, with a retry hazard | **Revert.** Re-introduce inside the E-7 evidence model, where a soft crop is a low-weight observation, not a free retry. |
| `live.py` `vote_agree` + `_agreed_observations` | experimental | **Revert** (default 1 makes it dead code). E-7 replaces it. |
| `test_live.py` change | encodes the reverted budget | Revert with it. |
| `code/rebuild_catalog.py` (untracked) | catalog tooling | Keep. Add image↔CSV bijection, packshot-OCR vs name check, and eval-session refusal (enrollment protocol below). |
| `stock-lopia-clean/` (untracked) | catalog data | Keep the images and provenance. Fix `stock.csv` (F-04) and the README's "Six SKUs" (it lists five). Replace the l06 packshot with the current package. |
| `eval/labels/image6.json` | ground truth | Adopted; schema: `label`, `source`, `note`, `v1_label`. |
| `identity.decide_identity` inlier branch | production | **Rewrite** as the calibrated decision (E-4); keep the cosine branch only while the gallery exists. |
| Production score path (`engine.score_crop`, `classify_boxes`, `inspect`) | production | **Rewrite:** return per-SKU evidence grouped by canonical SKU (F-02). |
| `feature_matcher.py` composite score | production | **Rewrite** into variant B behind an interface; keep Van's scorer as a selectable baseline lane for E-1. |
| Identity interface | — | **Isolate:** `IdentityLane.evidence(crop) -> {sku: Evidence}` with `LocalLane`, `TextLane`, and `GalleryLane` (postponed); `DecisionPolicy` separate; `TrackEvidence` in live. |
| DINO gallery, identity head, `finetune_identity.py` | production code | **Postpone and fence off:** not in the live path, not the default; delete after E-6 if nothing depends on it. |
| Descriptor index | — | **Postpone** until the catalog passes ~150 SKUs (E-6). |
| Provisional label on the wire + clients | protocol + native UI | **Postponed dependency** of E-7. Shared model gets a `provisional` identity state; Compose and SwiftUI render it as tentative. |
### Enrollment protocol
- **Views per SKU:** at least one current-package front packshot, stored as a
200/400/800 px-tall pyramid (source ≥ 800 px tall), plus at least one real
shelf capture at scan distance from a store visit that is **not** an
evaluation session. Add side or back views only for SKUs displayed rotated.
- **Minimum usable resolution:** reject any reference view under 160 px tall
at its largest pyramid level. Crops that small carry no text geometry
(observed: facings under 60 px wide fail every lane).
- **Metadata:** JAN, maker, product name as printed, size, package version or
date, source URL, capture session id, SHA-256.
- The enrollment tool validates a one-to-one match between images and CSV
rows.
- It OCRs each packshot and requires the catalog name's distinctive tokens
to appear (this check would have caught the l09/l10 mix-up and flagged the
l06 package version).
- It flags cross-SKU near-duplicates as "text-discriminated siblings".
- **Leakage:** enrollment refuses any image whose capture session id appears
in an evaluation manifest. Pixel equality is not a leak test; re-cut crops
pass it.
- **Versioning:** the catalog manifest has a content hash; every eval report
records catalog hash, label-file hash, and code SHA.
- **Completeness rule:** a bay is "live-ready" when every SKU on its shelf tags
is enrolled or explicitly listed as text-only. Near-unknown precision
depends on this more than on any threshold.
## 9. Completion gate
Measured on **Tier 2 held-out** (and Tier 3 for time-to-label). Same crops,
same catalog hash, same detector for both systems. "Van" means Van's scorer
with always-argmax, run on that same catalog.
| Gate | Definition | Pass |
|---|---|---|
| G1 coverage | correct shown labels ÷ all in-catalog facings (enrolled or not) | new ≥ Van; lower 95 % bound of the paired-bootstrap difference ≥ −2 pp |
| G2 correctness | correct shown ÷ all shown | ≥ 95 % (Van: 29–35 % on Tier 1) |
| G3 unknown handling | near-unknown false-accept rate; far-unknown false-accept rate | ≤ 5 %; ≤ 1 % (Van: 100 %) |
| G4 time to label | Tier 3: p50 / p95 seconds from a facing entering view to a correct label shown (provisional counts only if later confirmed as the same SKU) | p50 ≤ 1.0 s and p95 ≤ 2.5 s, and p50 no worse than Van's click-to-label on a still (tap + ~50–100 ms) |
| G5 stability | confirmed labels that later change; correct labels that disappear | ≤ 2 %; ≤ 1 % |
| G6 per-SKU floor | recall of every enrolled SKU with ≥ 5 Tier-2 facings | ≥ 60 % (catches l06-type catalog faults) |
| G7 hubness | max predicted-SKU share ÷ that SKU's true share | ≤ 2.0 |
| G8 runtime | server identity latency p95 per crop; server memory | ≤ 60 ms; within the demo host's budget |
Not complete until G1–G7 pass on the device path (Tier 3), not just offline.
---
## Findings
| ID | Finding | Severity | Evidence | Priority | Source |
|---|---|---|---|---|---|
| F-01 | image6 ground truth 23 % wrong | blocking | VERIFIED | P0 | open-world |
| F-11 | No held-out benchmark exists (one photo) | blocking | VERIFIED | P0 | open-world |
| F-02 | Production scores are per template, not per SKU → self-ambiguity and nameless labels | blocking | VERIFIED | P0 | open-world |
| F-03 | Demo server defaults to the leaked, corrupt `stock-lopia` | blocking | VERIFIED | P0 | open-world |
| F-04 | Clean catalog CSV ↔ image mismatch (l09/l10) | blocking | VERIFIED | P0 | open-world |
| F-05 | Near-unknown siblings accepted; shown-label precision 29–43 % | blocking | VERIFIED | P0 | open-world |
| F-07 | Evaluator defects beyond bugs #1–#3 | major | VERIFIED | P1 | open-world |
| F-06 | Matcher scoring defects (scale, dead ORB/BRISK, unverified "inliers", grayscale) | major | VERIFIED | P1 | open-world |
| F-08 | DINO unfit as classifier and retriever here | major | VERIFIED | P1 | open-world |
| F-09 | Confirmed labels never re-evaluated; first accept locks | major | HIGH CONFIDENCE | P1 | open-world |
| F-10 | Uncommitted live diff: unbounded blur retries; rationale invalidated | major | HIGH CONFIDENCE | P1 | open-world |
| F-12 | l06 packshot is a superseded package design | major | VERIFIED | P2 | **fixed 2026-09-29** |
| F-13 | Stale handoff test status; swallowed callback exception in a test fixture | polish | VERIFIED | P3 | open-world |
Counts: 11 VERIFIED / 2 HIGH CONFIDENCE / 0 HYPOTHESIS (hypotheses listed
separately) · 13 open-world. No reference baseline was used: this audit
covers a recognition backend. The product-level floor is Van's system (G1–G4).
### Workstreams
1. **Evidence (single owner, first):** F-01, F-11, F-07. Blocks every
architecture change.
2. **Catalog (parallel with 1):** F-03, F-04, F-12, enrollment tool checks.
3. **Identity core (single owner, after 1):** F-02, F-06, text lane, decision
layer, E-1–E-4.
4. **Live (after 3's interface exists):** F-10 revert now; F-09 and the
provisional label (E-7) later. The protocol plus native-UI part touches the
KMP app and needs its own owner.
5. **Scaling (after 3):** E-6, and the descriptor index only when it's needed.
### F-01 — image6 labels are 23 % wrong — P0 · blocking · VERIFIED
- **Root cause:** labels appear propagated from tag/box matching (every row has
a `match` field) rather than read from packages.
- **Evidence:** `evidence/image6-v1-labels-overlay.jpg`; the per-index list in §3.
Rescoring the same runs under v2 moves ungated live top-1 from 26/35 to 35/42.
- **Target:** every scored label carries provenance; ambiguous facings are excluded.
- **Change:** adopt `eval/labels/image6.json`; schema-check `label/source/note`.
- **Validation:** E-0 double-labelling on Tier 2.
- **Acceptance:** no architecture claim cites v1.
### F-02 — template-level scores in production — P0 · blocking · VERIFIED
- **Root cause:** `ProductFeatureMatcher.score_crop` keys scores by file stem;
`engine.score_crop` (`code/engine.py:424-448`), `classify_boxes`, and `inspect` pass
them straight to `decide_identity`. The live payload looks up
`identity.sku_id` in the catalog, which has only canonical ids.
- **Evidence:** on `stock-lopia` (the demo default), 22/109 facings at full
and 29/109 at live are "ambiguous" only because the top two are views of the
same SKU. 68 of 69 (full) and 56 of 58 (live) accepted ids are
view-suffixed (`sku_l01_1`), which have no catalog row, so the payload
carries no `product` block.
- **Target:** scores grouped by canonical SKU (max over views) before any
decision; the id shown is always a catalog row.
- **Change:** group at the lane boundary (the future `IdentityLane.evidence`).
Share one canonicalizer with `rebuild_catalog.sku_of`.
- **Validation:** unit test: two strong views of one SKU → accepted, canonical id.
- **Acceptance:** zero view-suffixed ids on the wire.
### F-03 — demo default catalog is the leaked one — P0 · blocking · VERIFIED
- **Evidence:** `run-demo-server.sh:11` `STOCK="${1:-stock-lopia}"`. That
catalog is shelf crops cut from the eval photo, and it scores 94–97 %
"identity" on its own exam.
- **Change:** default to the clean catalog once F-04 is fixed; refuse catalogs
without a provenance manifest.
### F-04 — clean catalog metadata mismatch — P0 · blocking · VERIFIED
- **Evidence:** `stock-lopia-clean/stock.csv` has no `sku_l09` row for
`sku_l09.jpg`; its `sku_l10` row has l09's counts (30/13/17) with l10's name
and price. Every l09 answer ships without product info; l10 has a row but no
image. The README says "Six SKUs … have no official pack shot" and lists five.
- **Change:** restore the l09 row from `stock-lopia/stock.csv`, drop the
mangled l10 counts, fix the README, and add the bijection check to
`rebuild_catalog.py`.
### F-05 — near-unknown siblings accepted — P0 · blocking · VERIFIED
- **Evidence:** the current gate accepts 22–28/48 unknowns; 9/11 unenrolled
l10 facings go to l09 with margin 0.67–0.93 (higher than many correct l11
matches); Kewpie 450 g → l03 with 17–28 SIFT inliers.
- **Target / change:** the decision layer + text veto (§5) + enrollment
completeness (E-5).
- **Acceptance:** G2, G3.
### F-06 — matcher scoring defects — P1 · major · VERIFIED
- **Evidence:** ORB inliers are 0 on nearly every live crop (per-facing log
from `diag.py`). `compute_geometric_inliers` returns `len(good_matches)`
unverified when there are fewer than 4. The l09 reference (305 px) scores
85–129 while others score 10–40.
- **Change:** variant B (§5), behind the lane interface, with Van's scorer kept
as a baseline lane.
- **Acceptance:** E-1, E-2.
### F-07 — evaluator defects — P1 · major · VERIFIED
- `top1` ≡ `coverage` (both are accepted-correct ÷ enrolled), so the identity
and decision ceilings were never separated.
- Precision's denominator excludes unenrolled in-catalog facings
(`covered` in `main`): it reports 50–53 % where the honest figure is 39–43 %.
- The gallery's SKU set comes from ground-truth labels (`sku_ids` from
`held`).
- Leak check requires the same shape.
- `HybridLane` scores the full catalog, so its latency isn't a retrieval
design.
- `group_by_sku` only handles the `sku_l` prefix.
- **Change:** report the four ceilings explicitly (the per-facing log in
`diag.py`/`variants.py` is the template). Use the catalog, not labels, for
candidate sets. Refuse leakage by session manifest.
### F-08 — DINO — P1 · major · VERIFIED
- **Evidence:** gallery top-1 21 %/12 % with 50–60 % hub share; recall@10 of
7/42 at 493 SKUs (`dino_scale.py`); 41 ms per embedding.
- **Change:** fence off (§8). The uncommitted `--identity matcher` default is
the right call.
### F-09 — no identity re-evaluation — P1 · major · HIGH CONFIDENCE
- **Evidence:** `code/live.py` `_on_classified`: `if track.state !=
STATE_DETECTED or track.confirmed: return`. With `vote_agree = 1` the first
accepted observation confirms.
- **Target:** E-7's evidence model with provisional labels and hysteresis.
- **Validation:** Tier 3 flips and correct-then-lost metrics.
### F-10 — uncommitted live experiments — P1 · major · HIGH CONFIDENCE
- **Evidence:** the blur gate `return`s before `track.attempts += 1`, so
eligibility never runs out for a soft crop. `crop_first_attempt` cites the
invalidated 69/66 numbers.
- **Change:** revert per §8.
### F-12 — l06 package version — P2 · major · **FIXED 2026-09-29**
- **Evidence:** the packshot showed "内臓脂肪ケア フィッテ"; the shelf shows
"内臓脂肪を減らす キユーピーフィッテ". Improved matcher 4/4 at full, 1/4 at
live; 3/3 with a shelf reference.
- **Fix:** the current package is enrolled. It was recorded as blocked on a
capture because Kewpie's own site serves the superseded design for that JAN;
an independent retailer listing carries the current one, and it matches the
shelf facings exactly (checked by eye against a crop of `add/image6.jpg`
before enrolling). Measured: live decision 60% → 64%, end-to-end 59% → 62%,
full decision 81% → 83%.
- **Still open:** the packshot-OCR check that would have caught this. It is
implemented (`rebuild_catalog` refuses a view whose name/OCR score is below
0.70) but it is a false negative on l04 and l07, so it cannot yet be trusted
as a gate - see the catalog corrections section.
### F-13 — stale test status — P3 · polish · VERIFIED
- **Evidence:** 49/49 pass on both trees. An `AttributeError: 'str' object has
no attribute 'items'` is raised and swallowed inside a future callback in one
test, on both trees.
- **Change:** make that fixture return a dict, or assert the callback path.
## Hypotheses
| ID | Suspected | Why it matters | The check that settles it |
|---|---|---|---|
| H-01 | Learned matchers (ALIKED/DISK + LightGlue) beat B on small, blurred, or re-designed packages | Could lift the identity ceiling | E-C (needs weight download approval) |
| H-02 | Shelf-tag OCR prior raises coverage and precision | Tags are readable above every run on image6 | E-9 using `code/tags.py` |
| H-03 | Cross-visit shelf references retain most of the leave-facing-out gain | Decides the enrollment protocol | E-5 |
| H-04 | Evidence accumulation shortens time-to-correct-label vs single-shot lock | G4/G5 | E-7 on Tier 3 |
| H-05 | On-device OCR is fast enough (< 30 ms per crop) and as accurate at native crop resolution | Where the text lane runs | Benchmark Vision on iPhone and ML Kit on Android over Tier-2 crops |
| H-06 | Stream JPEG quality costs SIFT inliers at live | Cheap fix if true | Rerun `variants.py` on re-encoded crops at the client's quality |
| H-07 | The hue feature is robust to store lighting | The colour gain could be lighting-specific | Tier 2 across stores and times |
| H-08 | ORB/BRISK removal costs nothing | Simplifies and speeds the matcher | E-2 at native-crop resolution, where ORB helped the small l09 reference |
| H-09 | Full-resolution OCR with ×2 upsampling matches live's text-lane gain | Full-res fusion was weaker (75 % precision) with ×1 OCR | Rerun `ocr_exp.py` + `fuse_eval.py` with full ×2 |
## Preserved decisions
- The matcher kept as a selectable baseline lane. The shipped default is now
`local` (variant B plus the text lane), which is what the measurements below
support; `--identity matcher` reproduces Van's scorer for E-1.
- Bounded, drift-checked crop lane protocol (`aa25f99`, `13a4e0f`); its tests
pass.
- `eval_identity.py`'s architecture-neutral lane framing and explicit
enrollment ceiling.
- Clean-catalog provenance table and the `rebuild_catalog.py` refusal rules.
- `vote_agree = 1` without provisional labels. Backing out `2` was right.
## Implementation record (2026-09-28)
What §8 asked for, and where it landed. Verified by the suite (274 tests) and by
running the real server against `add/image6.jpg`.
| §8 item | Status |
|---|---|
| F-02 per-SKU scores in `score_crop`, `classify_boxes`, `inspect` | Done. `group_by_sku` on every backend; no view-suffixed id reaches the wire. |
| F-03 demo default catalog | Done. `run-demo-server.sh` defaults to `stock-lopia-clean`. |
| F-04 clean catalog CSV | Done. All 12 rows; `sku_l09` added, `sku_l10` corrected. |
| F-05 near-unknown siblings | Done via the text lane. 0 of 48 unknowns accepted at live resolution. |
| F-06 variant B behind a lane interface | Done. `identity_lanes.LocalLane`; Van's scorer stays selectable as `--identity matcher`. |
| F-07 evaluator defects | Done. Four separate ceilings, honest precision denominator, catalog-derived candidate set, `sku_of` grouping, leak signature. |
| F-08 DINO fenced off | Done. Not the default, not in the live path, refuses to start without weights. |
| F-10 uncommitted live experiments | Reverted. `crop_first_attempt`, the blur early return, `vote_agree` and `max_attempts 4` are gone. |
| F-12 l06 current package | **Fixed.** The current package is enrolled from an independent retailer listing and matches the shelf facings exactly. Live decision 60% → 64%, end-to-end 59% → 62%. |
| F-13 stale status / swallowed exception | Done. The fixture returns the matcher's `(scores, details)` shape; zero swallowed callback exceptions. |
| F-09 re-evaluation with provisional labels | **Postponed as §8 specifies** — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet. |
| Enrollment protocol: versioning | Done. Every eval report records `catalog_hash`, `labels_hash`, `image_hash`, `code_revision`. The catalog hash covers the CSV bytes and each view's bytes, not just the view names: names alone let a different packshot be swapped in under the same name with the hash unchanged. |
| Enrollment protocol: sibling flag | Done. `rebuild_catalog` reports cross-SKU near-duplicates as text-discriminated siblings (cosine ≥ 0.98, same shape). Reported, not refused: they are legitimately different products, and only package text separates them. |
| Enrollment protocol: completeness rule | Done. `--shelf-tags` turns it into a `live_ready` verdict; without the shelf's SKU list the verdict is `unknown`, not a guess. |
| Enrollment protocol: pyramid floor | Done. `rebuild_catalog` reports views under 800 px tall. Every view in `stock-lopia-clean` is now 1000 px tall, so the check is clean on the shipped catalog; it still names any view that regresses. |
Two defects found while wiring the lanes into the live path, both fixed:
`live` re-sorted the local lane's scores by value, which discards the lane's own
rank order (order comes from `rank_key`, the value is only the gate), and the
early return on an empty local observation made the text lane's "answer on its
own" job unreachable. Live now decides per crop on the lane's own order.
Measured after the change, through the real server: `/v1/recognize` names 41 of
112 detections at 90% precision with no view-suffixed ids; the live WebSocket
names 12 at 100% precision with 0 unknown accepts, first label on frame 3.
Those are detector-box figures, not label-matched ones: the 41 are detections
whose box overlaps a labelled facing, so they measure the endpoint rather than
the identity lane. The label-matched number is the evaluator's, below.
## The baseline is Van's contract, not the gated matcher
The `matcher` lane applies `decide_identity`, which is not what the original
demo did - it answered every crop with its best-scoring SKU and had no accept
line. Comparing against the gated matcher understates the baseline's coverage
and overstates its precision, so the evaluator now has a `van` lane: the same
scores, always answered. Every "vs Van" claim in this record uses that row.
.venv/bin/python code/eval_identity.py --stock stock-lopia-clean \
--labels eval/labels/image6.json \
--image add/image6.jpg --lanes van,matcher,local --live
full resolution — 109 scored facings (64 in-catalog, 45 unknown)
lane catalog identity decision end-to-end precision false acc
van 83% 77% 77% 64% 38% 68
matcher 83% 77% 74% 61% 51% 38
local 83% 92% 81% 75% 91% 5
live resolution — 109 scored facings (64 in-catalog, 45 unknown)
lane catalog identity decision end-to-end precision false acc
van 83% 79% 79% 66% 39% 67
matcher 83% 79% 74% 61% 54% 33
local 83% 85% 60% 59% 93% 3
Against Van's real contract the local lane wins both axes at both resolutions:
75% at 91% against 64% at 38% (full), and 59% at 93% against 66% at 39%
(live). At live resolution Van answers more facings correctly - 66% against
59% - and is wrong on 67 of them to the local lane's 3. That is the trade the
decision makes deliberately: coverage is a gate (G1) and precision is not
negotiable, so the lane abstains where Van guesses.
**One accept line serves both resolutions.** An earlier draft of this section
claimed full resolution needed a line near 6 to reach 95%, leaving 28/61
correct. That was measured before the reference-upscaling and text-signature
fixes and no longer holds. The separation is clean at both scales: correct
facings sit at p25 2.07 (full) and 2.87 (live), wrong ones at p75 1.00 and
1.66, so 2.0 is the boundary in both. Sweeping it at full resolution, through
the evaluator (`--accept`, columns as it defines them: end-to-end is
accepted-and-correct over the 64 in-catalog facings, precision is over the
facings it answered):
accept end-to-end precision false accepts
2.0 75% 91% 5
3.0 67% 91% 4
4.0 67% 93% 3
6.0 62% 95% 2
The line is still provisional - it is tuned on Tier 1 and needs Tier-2
calibration - but it does not need to differ by input path, and the design
beats Van on both axes at both resolutions today.
The live count was pacing-dependent, and the cause was measurable: OCR ran on
every crop at every attempt, taking a crop from 5 ms to 25 ms and throughput
from 789 to 60 crops/s with 8 workers. The reading is now cached on the track
and later attempts re-run only the local lane. Measured through the real server
on `add/image6.jpg`:
saturated pool 15.9 -> 49.4 crops/s (8 workers, 64 crops)
per crop 142 -> 92 ms on a 300x400 crop
3 s clip, send 19.7 mean labels (15-24, unstable) -> 24.0 (23-25, stable)
12 s clip, send 48/131 -> 53/131; result pacing 53/134 -> 50/135
The 12 s numbers are within run-to-run noise; the short clip is where the
pacing sensitivity lived, and it is now flat across pacings. Precision stays
100% and 0 unknowns are accepted at every pacing.
**The stream path now has a G1 number.** `code/score_stream.py` scores a
recorded clip's labels against the ground truth, matching each track's
`label_box` to the labelled facings. Three defects in the first version of
that harness made the stream look far worse than it was, and all three are
fixed: it counted any facing touching the window by one pixel (23 of its 47
"in-catalog" facings were less than half visible), it scored a correct
text-lane answer as a false accept because it checked enrollment before
correctness, and it counted tracks rather than facings, which reported 138%
coverage of a 24-facing window. The denominator is now the facings at least
half inside the union of the window rects the client showed.
Measured through the real server on `add/image6.jpg`, 20 s per run, local
backend against the `van` lane on the same stream:
pan lane facings named precision
0 van 11/24 (46%) 52%
0 local 11/24 (46%) 100%
2 van 12/24 (50%) 45%
2 local 18/24 (75%) 100%
12 van 11/24 (46%) 28%
12 local 16/24 (67%) 83%
Two live-path bugs surfaced while measuring this, both fixed:
- **Inherited labels carried frame id 0.** `_inherit_labels` set `label_box`
but never `label_frame_id`, so every inherited label was mapped back to the
source image with the first frame's offset. During a pan that puts the box a
whole pan-distance from the product it names. It now stamps the current
frame and anchors to the raw detection box rather than the smoothed track
box, which lags a pan by about one step.
- **The reference lane upscaled views smaller than a pyramid level.** The
`sku_l09.jpg` enrolled at the time was 407x305, so its 400 px level was a
305->400 blow-up: a blur with no high-frequency detail to disagree with,
which won 31 of 54 crops on the shelf it was cut from. A view is now used
only at the levels it has. The guard is in `LocalLane._load` and still
applies; the file itself has since been replaced by a 1000x1000 packshot,
which is why the current run no longer exercises it.
Coverage is still a gate (G1). Four 20 s pan replays of `add/image6.jpg`
through the shipped lane name 63-70% of the in-catalog facings a window can
show, at 76-84% precision (`score_stream.py` over
`data/evaluations/2026-09-29-session/stream/fin_*.json`). Van's contract on the
same stream answers more and is wrong far more often, which is the trade the
decision makes. The earlier "53 of 131 tracks" figure was a harness artifact
rather than a recognition limit.
The shipped default scale now has test coverage. `make_session` builds only
`matcher`-backend engines, so `scale == "local"` in `_on_classified` never ran
in the suite - and that is the branch that was wrong twice. `LocalScaleEngine`
injects the lanes' evidence into the live path, and `LocalScaleTests` pins the
two jobs (a text-only read still labels the track; a text veto blocks a local
accept). `DecisionPolicyTests` pins the veto/fill-in boundary, including the
lane-order-vs-value case the rank re-sort broke. Both regressions were
re-injected to confirm the tests fail for the intended reason. Suite 251 -> 291.
Two further review findings, both fixed and both measured:
- The evaluator had no always-argmax lane, so every "vs Van" claim compared
against the *gated* matcher rather than Van's contract. The `van` lane is
that contract; the numbers above use it.
- A crop reading only "400g" scored 0.667 on three SKUs, above `TEXT_ACCEPT`,
so the text lane named a product from a size line. The fill-in job now
requires two non-digit bigrams; the veto is unchanged, so "65%" still
contradicts a local answer. Three tests cover it and all three fail when the
gate is removed.
The headline number is reproducible from the repo, not from a scratch script:
.venv/bin/python code/eval_identity.py --stock stock-lopia-clean \
--labels eval/labels/image6.json \
--image add/image6.jpg --lanes local --live
live resolution: catalog 83% · identity 85% · decision 60% ·
end-to-end 59% · precision 93% · 20.3 ms/crop
false accepts: 3 of 109 scored facings — 2 of the 45 unknowns and 1
enrolled misidentification; unenrolled_false_accepts 0.
The catalog figure fell from 87% to 83% because three facings were relabelled
from unknown to `sku_l12` once the tare was identified (§ Re-measured
2026-09-29), and `sku_l12` has no enrolled view. That is a catalog gap
reported honestly, not a recognition regression: the same three facings were
unanswerable before and after.
### Re-measured 2026-09-29
The text lane's evidence floor (shared bigrams between the read and the winning
row's name) and the local lane's signature pass-through landed, and the
decision ceiling was corrected: it divided a numerator that counted unenrolled
facings by the enrolled count, so it read 117%. It now reads 86%.
full resolution — 109 scored facings (61 in-catalog, 48 unknown)
lane catalog identity decision end-to-end precision false acc
van 69% 76% 76% 52% 29% 77
matcher 69% 76% 71% 49% 39% 46 HUB!
local 69% 93% 86% 80% 91% 5
live resolution — 109 scored facings (61 in-catalog, 48 unknown)
lane catalog identity decision end-to-end precision false acc
local 69% 86% 86% 66% 95% 2
**E-3 answered.** Removing the text lane from the decision, same crops and
catalog, drops the local lane from 80% to 49% coverage at full resolution
(91% → 79% precision, 5 → 8 false accepts) and from 66% to 43% at live
(95% → 87%, 2 → 4). The lane adds 31 pp and 23 pp of coverage while *raising*
precision, against E-3's +5 pp bar. It is not a veto-only lane.
**E-2 answered.** `LocalLane` is RootSIFT-only; ORB and BRISK survive only in
the `matcher` baseline lane, so there is nothing to drop from the shipped path.
**Live path.** `_plan_classification` had lost its `in_flight` increment while
a `label_frame_id` write was removed, so the guard never blocked, every frame
re-planned the same tracks, and the executor queue grew until the server
stopped answering. Restored. Label memory boxes now carry the shelf origin they
were measured against and are shifted before the IoU test, so a pan no longer
slides a stale box onto its neighbour. The text lane is resolved before the
session lock is taken, removing a lock-ordering hazard against its build lock.
Stream, 20 s pan, four runs: 63-70% coverage at 76-84% precision. The earlier
"71-79% at 76-84%" was measured before the tare facings were relabelled from
unknown to `sku_l12` and before the size gate landed; the denominator moved
from 61 in-catalog facings to 64, and the size gate trades a little coverage
for precision. Still-image endpoint: 53 named detections in 2.6 s, against 41
before.
### Catalog corrections 2026-09-29
Three rows named products the shelf does not hold, all found by reading the
shelf tags and the enrolled packshots rather than by trusting the CSV:
| SKU | Was | Is | Evidence |
|---|---|---|---|
| `sku_l10` | ピュアセレクト マヨネーズ 450g | 400g | Shelf tag reads 400g; Ajinomoto's lineup is 250/400/600/1000g, no 450g. Packshot enrolled (JAN 4901001114030). |
| `sku_l11` | キューピー ハーフマヨネーズ 450g | 400g | Shelf tag reads 〈400g〉; the enrolled packshot's own OCR reads 400g. |
| `sku_l12` | 焼肉のたれ 中辛 | ユーラス 野菜をおいしく食べる旨たれ 380g | Tag under facings #46-48 reads that title at ¥299, the price the row carried; the package reads お子さまOK 甘めテイスト, the opposite of 中辛. |
The l12 row was the dangerous one: the text lane scores a crop against the
row's printed name, so a correct read of the real bottle would have been
vetoed by a row describing a different product. Facings #46-48 were relabelled
from unknown to `sku_l12` to match.
Enrolling l10 took the catalog ceiling from 69% to 87% of in-catalog facings
answerable by image; relabelling the tare facings moved it back to 83%,
because `sku_l12` has no enrolled view. `sku_l04`, `sku_l07` and `sku_l08`
remain unenrolled. l04 and l07 have exact packshots whose name/OCR check is a
false negative, recorded in `data/packshots/research/` with the retailer, JAN
and the reason the check fails. l08 could not be sourced at all: no retailer
listing for a スパイシーストーム ペリペリミックス 30g exists, and the only
corroborated ペリペリミックス is a different brand at 50g, so the row's name
and size are themselves unverified.
### The near-unknown case is now rejected
The documented failure - Kewpie 450g bags the catalog does not hold, labelled
as catalog rows - had no fix in the text ranking. Every mayonnaise row
explains "マヨネーズ" equally, so the ranking ties at 1.0 and the
strict-greater veto never fires; the veto compares two catalog rows and the
crop's product is not one of them.
`DecisionPolicy` now also checks the package size the read prints. Only a mass
in `PACKAGE_MASS_RANGE` counts, because below it are fragments of a longer
number ("20" out of "200g") and above it are two numbers run together ("5210"
out of "210g"), and percentages are excluded because "カロリー 50% カット" is a
claim about the contents. Measured on the corrected labels, full resolution:
precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
### External benchmark: 20 images, 0 false accepts
The image6 shelf is one photo with one label set, so every number above is
measured on data the design was tuned against. Twenty images for eight SKUs
were collected from independent sources (six retailer and marketplace domains)
to test the pipeline on data it had no part in building. None is Tier 2; all
are `external_proxy` and every manifest says so.
Leak-tested first: the highest template match of any of the 20 against
`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
`stock-lopia-clean` (`data/external-eval/*/eval_result.json`):
| Set | Images | Correct | Abstain | False accepts |
|---|---|---|---|---|
| `kewpie/` | 12 | 9 | 3 | **0** |
| `other-mayo/` | 8 | 4 | 4 | **0** |
| **Total** | **20** | **13** | **7** | **0** |
- **`sku_l07` is answered with no enrolled image.** Both its images accept
through the text lane alone - one reads ラッキー フヨネーズ, the other the
rear label's 製造者：丸和油脂株式会社 with 内容量：200g, which together
name the row 丸和油脂 ラッキーマヨネーズ 200g. This is the text lane's
coverage claim holding on data that was not used to build it. `sku_l02`
also accepts one image on text alone, but it does have an enrolled view.
- **The l04 abstentions are correct behaviour.** l04 is also unenrolled and its
reads are weak: one reads only コと旨 タ 味 (every row scores 0.0), another
scores 0.379 against a 0.45 accept line. Abstaining on a weak read is the
designed outcome, not a miss.
The seven abstentions are all unenrolled or weakly-read SKUs. No image was
given a wrong name, which is the property the decision optimises for.
### Tier-3 sources are staged, not satisfied
Two reusable pan clips were found and saved (`data/external-eval/tier3-sources/`):
a 1920x1080 CC BY 3.0 supermarket pan and a 360p CC BY mayonnaise-bay pan, with
11 rejected candidates recorded. Neither gives per-SKU identity for this
catalog, and the only LOPIA store walk found carries the standard YouTube
licence and was therefore not downloaded. Tier 3 still needs the operator's own
recorded walks.
Still open, and not implementable from here: Tier 2 held-out labels (F-11), the
remaining E-1/E-4..E-9 experiments, and the completion gate G1-G7, all of which
need a second independent shelf visit.
## Out of scope
Detector recall and box quality (all experiments use the same boxes). Tag and
planogram pipeline internals. Native UI. On-device performance. Learned local
features (not installed; downloading them needs your approval). `image.jpg` /
`stock/` (no labels exist). Timings from parallel runs are relative;
`scale.py` timings are single-process on this Mac.
View git blame Copy permalink