product-recognition-demo/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md

Dorayaki product recognition: architecture decision

Audited 2026-09-28 · submodule 25dbf13 on fix/live-recognition-quality plus its uncommitted diff (snapshot: git stash create → 2785c07) · auditor: Claude Opus 5.5 Scope: identity pipeline (matcher, gallery, decision, evaluator, catalogs, labels), live track/label logic, the uncommitted diff. Not audited: detector quality, tag/planogram pipeline, iOS/Android rendering, on-device runtime. Evidence scripts and the corrected labels: evidence/.

Verdict

Build a local-feature instance matcher plus a text lane, behind a calibrated open-set decision, with no global embedding in the critical path. At the current scale the matcher runs exhaustively; past ~150 SKUs, a RootSIFT descriptor index replaces exhaustive matching. Generic DINOv2 is rejected for both classification and retrieval.

The biggest risk is none of the architectures. It is that there is no trustworthy benchmark. The only labelled data is one photo, add/image6.jpg, and 26 of its 111 labels (23 %) were wrong. Every number in the handoff was computed against those labels. Label and catalog errors are larger than any architecture difference measured so far. They also hide the real product failure: displayed-label precision is 29–43 % because same-brand siblings the catalog doesn't know about get labelled with confidence.

On corrected labels, with thresholds tuned on the same single photo (so read this as direction, not proof), the recommended design beats Van's contract on coverage and on precision together:

Live resolution, clean 7-SKU catalog, 61 in-catalog + 48 unknown facings	Correct shown	Coverage (of 61)	Precision of shown labels	Unknowns labelled
Van's matcher, always argmax (the floor)	35	57 %	35 %	48 / 48
Current server gate (decide_identity defaults)	32	52 %	43 %	22 / 48
Improved matcher, geometric-confidence gate only (τ=2)	26	43 %	79 %	—
Improved matcher + OCR veto + OCR lane (τ=2)	36	59 %	95 %	0 / 48
Same, looser (τ=1)	40	66 %	87 %	2 / 48

The one experiment that would most change this recommendation is a second, independently captured and labelled shelf visit. §7 defines it.

---

1. Problem formulation

Dorayaki's identity task is open-set instance recognition over a small, store-scoped, enrolled catalog, with live-video evidence accumulation.

• Instance, not category. The classes are individual printed artworks (JAN-level SKUs). Siblings share maker, palette, lattice pattern, mascot and silhouette, and differ by a word ("コクうま 65%"), a colour (pink vs red lattice), or a small logo. That is the regime of planar instance matching (logo and text geometry), not of semantic embeddings. Measured: raw DINOv2 is 12–21 % top-1 on 7 SKUs; the local matcher is 76–98 %.
• Retrieval-shaped at scale, closed-set-shaped today. With tens of SKUs, every enrolled reference can be verified exhaustively. With thousands, it becomes retrieval plus verification (§6).
• Open set is the dominant difficulty, and it comes in two kinds. Far unknowns (other categories) are rejected easily: 0.2–0.7 % false accepts on 602 unrelated crops at a threshold that keeps most correct labels. Near unknowns (siblings of enrolled products: Kewpie 450 g next to Kewpie からし, Pure Select plain next to Pure Select コクうま) produce genuine, geometrically consistent evidence and can't be thresholded away. Near unknowns get fixed by catalog completeness and by text, not by a better score.
• Multimodal by nature. Text is the designed discriminator between siblings; manufacturers print the variant name precisely so shoppers can tell them apart. OCR identifies the right SKU among all 12 catalog rows on 41–42 of the 55 facings where it reads anything, including SKUs with no enrolled image, using only catalog names and packshot text.
• Temporal. Live video gives many cheap, noisy looks at one physical facing. The right model is accumulated evidence per track, with a provisional → confirmed label contract. The current model is one accepted observation that locks the label forever.

So the hierarchy is: store/category scope → candidate generation (trivial now, descriptor index later) → local geometric verification + text verification → calibrated open-set decision → temporal accumulation.

2. Evidence audit

R reliable · C directionally useful but confounded · X invalidated · U still unknown. "Reproduced" means re-run in this audit with the scripts in evidence/.

#	Handoff claim	Verdict	What this audit found
2	Van's matcher is SIFT/ORB/BRISK + RANSAC, always argmax, no rejection	R	Scoring core in code/feature_matcher.py is byte-for-byte Van's (git diff b443e3c); only decide_identity was added. "SURF" is SIFT (SURF isn't in this OpenCV build).
2	"170/170 named ≠ 170/170 correct" on image.jpg	R (logic) / U (numbers)	No labels exist for image.jpg; its accuracy was never measured.
3	Current server collapsed onto hub SKUs (sku_003, sku_l09)	C	Hubness is real for DINO (50–60 % max-SKU share on the clean catalog, reproduced). The l09 collapse was mostly a wrong l09 reference plus shelf-cut templates.
3	Live resolution degrades identity	X	Controlled rerun: with correctly scaled boxes, live is not worse than full (original matcher 35/42 live vs 32/42 full; improved 36 vs 40).
3	DINO embeddings are inappropriate for fine-grained SKU identity	R	Reproduced: gallery top-1 21 % full / 12 % live, hub alarm.
4	stock-lopia was corrupt, leaked, and incomplete	R	Worse than reported: its crops come from the eval photo, so scoring against it is memorization. It scores 94–97 % "identity", and the evaluator's leak check misses it because the re-cut crops aren't the same size.
7	eval_identity.py is the right research direction	C	Right skeleton, four more defects (F-07).
8	Bug #1, template IDs vs SKU IDs	R, and still live in production	Fixed only in the evaluator. The live path accepts sku_l01_1, then finds no catalog row for it (F-02).
9	Hybrid 80 % > matcher 70 %	X	About one facing, on wrong labels.
10	Bug #2, unscaled boxes caused the live collapse	R	Reproduced on both catalogs: unscaled boxes give 11–17 %, scaled boxes give 76–97 %. The confound is resolved. The coordinate fix, not the catalog cleanup, removed the collapse.
11	Clean packshots are a dramatic improvement	R, with a caveat	True against the leak-free baseline. The l06 packshot shows a superseded package design (F-12); the l09 packshot is only 305 px tall.
12	Bug #3, the reject gate inside the shortlist	R	Hybrid-49 %-vs-20 % result is invalid.
13	DINO top-K sweep: recall 71 %/100 %, K doesn't hurt the matcher	C	On a 7-SKU catalog, top-5 is 71 % of the catalog; that sweep can't speak to retrieval. Reproduced on v2: top-3 recall 79–81 %, top-5 95–98 %.
14	Matcher ~8 ms vs hybrid ~45 ms	R, with a correction	The hybrid lane scores the full catalog and then filters (eval_identity.py HybridLane.rank), so its latency is DINO + full matcher by construction. The measurement is honest; it doesn't describe a real retrieval design.
15	69 % full / 66 % live after the l09 fix	R as arithmetic, X as a measurement	Reproduced exactly (24/35, 23/35) on v1 labels. On corrected labels the same matcher and catalog give 71 %/76 % after the gate and 76 %/83 % ungated top-1 ranking. The 69/66 gap that justified crop_first_attempt doesn't exist.
16	Local matcher is the strongest identity primitive	R	And improvable: multi-scale RootSIFT + colour re-rank 40/42 vs 32/42 at full (sign test p = 0.04); at live 36 vs 35 (noise).
16	DINO as coarse retriever at scale	X (this setting)	Recall@10 falls 42 → 32 → 17 → 7 of 42 as the catalog grows 7 → 57 → 207 → 493 SKUs. A RootSIFT descriptor index holds 37–38/42 recall@5 over the same range.
17	Threshold alone can't create correct identities	R	And the missing identities are mostly catalog gaps (§3).
17	Argmax precision ~20 %	C	29 % full / 35 % live on corrected labels; the old number counted mislabelled facings.
21	Live tests: 49 tests, 1 failure, 5 errors	X (stale)	python -m unittest test_live now passes 49/49 on both the dirty tree and the committed baseline. One test feeds a str result that raises inside a future callback and is swallowed, on both trees (F-13).
22	"High confidence" list items 1–8	R	All hold. Item 4 is now measured with corrected labels.
22	Medium confidence 1 (69 %/66 %)	X	See row 15.
22	Medium confidence 3 (more views help)	R direction	Leave-facing-out proxy (§3, E-5) supports it strongly, with a same-photo caveat.
22	Not established 1–9	U	1 and 4 are now partly answered (§6); 2 partly (RootSIFT/multi-scale beat Van's composite); 3, 5, 6, 7, 8, 9 remain open. Item 9 (ground truth) was the most consequential open item: it was wrong.

The four ceilings, separated

On corrected labels (evidence/image6_v2.json): 109 scored facings, 61 in-catalog across 11 SKUs, 48 unknown, 2 excluded. The clean catalog enrolls 7 SKUs covering 42 of the 61 in-catalog facings.

Ceiling	Value (live, improved matcher)	What limits it
A. Catalog	42 / 61 = 69 % of in-catalog facings are answerable by image	l10 (11 facings), l07 (6), l04, l08 unenrolled; l12 has zero real facings on the shelf
B. Retrieval	n/a at 7 SKUs; RootSIFT index recall@5 = 38/42 at 493 SKUs	the 4 misses are catalog-quality cases (l06 package version)
C. Identity (ungated top-1 on enrolled)	36 / 42 (86 %, Wilson 72–93 %)	l06 at live (1/4), two l09 facings that are mostly occluded
D. Decision	26 correct at 79 % precision (geometry only); 36 at 95 % (+ text)	near-unknown siblings: 9 of 11 unenrolled l10 facings → l09, Kewpie 450 g → l03

3. Root-cause model

Ordered by how much of the gap to a good product each class explains.

Evaluator and ground-truth problems (they decided every conclusion so far)

• 23 % of v1 labels were wrong: 14 in-catalog facings marked unknown (Egg Care #4–5, Fitte #22–23, Lucky #27–28, Kokuuma #40–45, Pure Select #59); 9 non-catalog items labelled as catalog SKUs (demi-glace pouches #31–32 as l09, a garlic jar #16 as l05, a Heinz can #64 as l10, dressing bottles #76–79 as l12, Kewpie 450 g #85 as l03); 2 neighbour shifts (#29 l08→l07, #75 l12→l11). The pattern (runs shifted along a shelf row, a match IoU field on every row) looks like labels propagated from shelf-tag or box matching rather than read from the packages.
• The evaluator reports top1 and coverage as the same quantity (accepted correct / enrolled), so the identity ceiling and the decision ceiling were never separated.
• Precision excludes unenrolled in-catalog facings, which are the most dangerous false accepts.
• The gallery's candidate set comes from ground-truth labels.
• Leak detection only catches same-size crops.

Catalog and data problems (the largest recognition lever)

• Incomplete category enrollment is what creates near unknowns. In the leave-facing-out proxy (E-5), adding one shelf view for each of the 11 SKUs on the shelf takes argmax from 34/52 to 47/52. The number that survives at 95 % precision goes from 14–16 to 32–39.
• Wrong or stale references: the l09 history; the l06 package is a superseded design (the match holds at full resolution, fails at live: 1/4); the l09 packshot is 305 px, so its scores dwarf others under Van's formula.
• Metadata integrity: stock-lopia-clean/stock.csv has no sku_l09 row for sku_l09.jpg. Its sku_l10 row carries l09's stock counts with l10's name and price. The catalog name for l12 ("焼肉のたれ 中辛", ¥299) probably names the ユーラス 旨たれ bottles on the shelf (tag ¥299). Several catalog sizes disagree with the shelf tags (l10 tag says 400 g).

Identity-model problems (real but second-order on this data)

• Scale mismatch: ~1000 px packshots vs 110–250 px crops is a 4–8× ratio. ORB's 8×1.2 pyramid tops out at 3.6×, so ORB contributes zero inliers at live resolution, and BRISK nearly none. Van's weighted composite is effectively SIFT-only there, carrying dead weight.
• Van's score, inliers·1.5 + inliers/min(kp)·80, isn't comparable across templates of different resolution. With fewer than 4 matches it counts raw ratio-test matches as "inliers" without any geometric check (compute_geometric_inliers).
• Grayscale-only: pink (ゼロ) vs red (ハーフ) lattice is invisible to SIFT. The hue re-rank fixes 2–3 facings in every configuration.

Open-set decision problems

• decide_identity for the inlier scale accepts a score ≥ 1.0 with a relative margin ≥ 0.15. A single unverified match clears the floor, and the margin can't reject a sibling with a strong partial match. It accepts 22–28 of 48 unknowns.

Live and tracking problems (code-path evidence, unmeasured on device)

• A confirmed track is never re-evaluated (_on_classified returns early when track.confirmed), and with vote_agree = 1 the first accepted observation locks it. A wrong first label is permanent. A later, sharper, correct view can't correct it.
• Aggregation is the mean over the last three observations, with empty observations counted as zeros. That's a heuristic, not evidence accumulation.
• Uncommitted diff: the crop blur gate returns without consuming an attempt, so a permanently soft crop (a distant facing) is retried every retry_gap frames for as long as it stays in view.

UX latency problems

• No provisional label exists on the wire or in the clients, so any multi-look confirmation becomes dead time on screen. That's why vote_agree = 2 was (correctly) backed out.

4. Architecture candidates

"Recognition" = does it improve correct shown labels at fixed precision? "Scalability" = does it keep latency and recall flat as the catalog grows?

ID	Candidate	Tested here?	Recognition	Scalability	Verdict
A	Van's matcher, full reference set, argmax	yes	32–35 / 42 identity; 29–35 % precision	linear, ~0.1–0.25 ms per reference	Floor. Not shippable (no open set).
B	Matcher + engineering: RootSIFT, per-SKU reference pyramid (200/400 px tall), SKU grouping, similarity-constrained RANSAC, hue feature, calibrated gate	yes	40/42 full (p = 0.04 vs A), 36/42 live; gate alone reaches 79–85 % precision at 36–43 % coverage	same as A	Adopt as identity core.
C	Learned local features: ALIKED/DISK/SuperPoint + LightGlue; dense LoFTR/RoMa	no (weights not installed; downloading needs your approval)	Unknown. Headroom on enrolled facings is ≤ 6/42 at live, mostly catalog-caused. Likely strongest on small or blurred crops and package-version drift.	LightGlue is per-pair; tens of ms per pair on GPU and much more on CPU, so only viable as a top-K verifier	Postpone to experiment E-C. Adopt only if it clears the gate.
D	Retrieval → Van's matcher	yes (DINO)	DINO recall collapses with scale (7/42 @10 at 493 SKUs)	DINO adds ~41 ms per crop	Reject DINO retrieval.
D′	RootSIFT descriptor index (FLANN) → vote → verify top-10	yes	35–37/42, equal to or slightly better than exhaustive	1.7–3.9 ms per crop flat from 7 to 493 SKUs; build 171 ms for 64k descriptors	Adopt past ~150 SKUs.
E	Retrieval → learned verifier	no	—	—	Only if C wins.
F	OCR text lane against catalog text signatures	yes (macOS Vision)	Veto: precision 62 → 82 % at τ=0.5 for −1 correct. As a lane: +10 correct at live, including unenrolled SKUs.	~18–26 ms per crop on Mac; on-device on iOS (Vision) and Android (ML Kit)	Adopt.
G	Fusion (geometry + colour + text + temporal + tag prior)	partly (geometry + colour + text)	Largest measured gain	—	Adopt; the text and geometry errors are complementary (text reads siblings apart; geometry covers text-poor faces).
H	Store-trained identity head on DINO	no (it was trained on the eval shelf)	Leaked; no valid measurement exists	—	Retire with DINO.
I	Barcode/JAN	—	Front faces rarely show JANs; shelf tags carry product text, not scannable codes, at scan distance	—	Only for the enrollment tool (scan a JAN, pull the packshot).
J	Shelf-tag / planogram prior	no (tags.py exists)	Every facing run on image6 sits above a readable tag with name and price	—	Strong hypothesis, H-02.

5. Recommended architecture (current scale: tens of SKUs)

frame / native crop
  │
  ├─ local lane:  RootSIFT on crop ─► verify against EVERY reference view
  │               (exhaustive while refs ≲ 300), similarity-RANSAC,
  │               per-SKU evidence = max over views:
  │               {inliers, 2nd-best ratio, inlier hull cover, hue agreement}
  │
  ├─ text lane:   OCR (upsampled ×2 when crop height < 200 px)
  │               ─► bigram similarity to every catalog row's text signature
  │               (name + OCR of its packshot), including rows with no image
  │
  ▼
decision:  P(correct | local, text) from a small calibrated model
           accepted  if P ≥ p_confirm            (target precision ≥ 95 %)
           provisional if P ≥ p_provisional      (shown as tentative)
           veto      if text names a different catalog row by margin
           unknown   otherwise
  │
  ▼
track:     log-odds accumulation per track across looks; confirm on
           threshold or text agreement; re-open on strong contradiction
           (hysteresis), never on a single weak frame

Why each piece earns its place:

• Local lane. It's the only primitive that ranks fine-grained siblings correctly: 36–40 of 42 enrolled facings, against 5–9 for DINO. It costs ~4–10 ms per crop at 7 SKUs.
• Reference pyramid (200/400 px) and RootSIFT. Together they fix the scale mismatch that kills ORB/BRISK and makes scores comparable across references. Drop ORB/BRISK unless ablation H-08 finds a case they win.
• Hue feature. It's the cheapest fix for colour-coded variants (+2–3 facings everywhere).
• Text lane. It's the only measured mechanism that rejects near-unknown siblings, and it can name catalog SKUs that have no image yet. It contributed +10 correct labels and 0 unknown accepts at live, τ=2.
• Calibrated decision. It replaces a hand-set floor that accepts one unverified match. It's calibrated on Tier-2 data, never on the photo it's evaluated on.
• No DINO, no retrieval stage. Neither adds recognition at this scale, and DINO actively fails at larger scale. Exhaustive verification is cheaper than one DINO forward pass until ~300–400 references.

Placement: keep the matcher server-side for now; it's small and deterministic. Put OCR on the phone when tag and label OCR move there. Apple Vision and ML Kit are the official platform paths, matching the repo's platform/ rule. Send the text with the crop so the server fuses both. Until then, run OCR on the server (macOS Vision is already wired in code/ocr.py).

6. Scaling architecture

Measured cost and recall to ~500 SKUs; beyond that, extrapolated and marked.

Catalog	Candidate generation	Verification	Evidence
10–150 SKUs	none (exhaustive)	local + text	3.7 ms at 7 SKUs, 7.7 ms at 57, 21 ms at 207 (small refs). Packshot pyramids cost ~2–3× per reference.
150–5,000	RootSIFT descriptor index (FLANN now; FAISS IVF-PQ when memory matters), vote per SKU, top-10	local + text on the shortlist	Measured flat 2–4 ms and recall@5 38/42 up to 493 SKUs. Descriptor memory at 5k SKUs × 2 views × ~500 descriptors ≈ 5 M × 128 B (uint8) ≈ 640 MB before PQ. Extrapolated.
5,000–100,000	Scope first: store assortment → aisle/bay (tag or planogram prior) cuts candidates to hundreds. Then an aggregated-local-feature index (ASMK/VLAD-style) or a product-instance embedding fine-tuned on packshot↔shelf pairs.	local + text on ≤ 20	Hypothesis. No store stocks 100k SKUs at once. Scoping by store and bay is the real scaling lever. Generic DINO is ruled out; a fine-tuned embedding needs paired training data that doesn't exist yet.

Exhaustive matching stops being sensible when its p95 exceeds the per-crop budget. At ~0.1–0.25 ms per reference and a 30 ms budget, that's roughly 150–300 references (≈ 75–150 SKUs at 2 views). Descriptor indexing removes the need for a global DINO pass: at 493 SKUs it retrieves better than DINO at a tenth of the cost. The open question is same-category growth: hundreds of mayonnaises instead of other-category distractors. That's experiment E-6.

7. Experimental plan

Rule for every experiment: change one variable, run on fixed crops and a fixed catalog hash, log per facing (truth, enrolled?, per-SKU evidence vector, truth rank, decision, lane latencies), and report the four ceilings separately.

Benchmark tiers (build these first; they block every other decision)

• Tier 1, diagnostic: evidence/image6_v2.json (109 facings). Dev only. Free to tune on. Its numbers never ship.
• Tier 2, held-out regression: new captures that are never enrolled and never tuned on. At least 3 visits across ≥ 2 LOPIA locations or days, same aisle, both still photos and the phone's own 1920×1080 frames. Target ≥ 300 in-catalog facings over ≥ 20 SKUs, ≥ 300 near unknowns from the same aisle, and the 602-crop far-unknown set from add/.
  • Labels need provenance: shelf-tag correspondence plus readable package text, two independent passes, disagreements marked exclude.
  • Every label row carries source, confidence, and labeler. A label file without them fails a schema check.
• Tier 3, live replay: FrameRecorder walks (code/record.py) of the same aisles with Tier-2-style labels, replayed through code/eval_live.py.

Experiments

ID	Question	Variable	Metric	Acceptance / falsifier
E-0	Are the labels right?	none	Double-labelling agreement on Tier 2	κ ≥ 0.9 on SKU identity. Otherwise fix labels before anything else.
E-1	Does B beat A on held-out data?	matcher variant	Paired identity top-1 on enrolled facings, McNemar	B ≥ A with p < 0.05 at native-crop resolution. If A ≥ B, keep Van's scoring and add only SKU grouping and the decision layer.
E-2	Is ORB/BRISK dead weight?	drop ORB/BRISK from B	top-1, latency	Drop them if top-1 changes by ≤ 1 facing per 100.
E-3	Does text move the frontier?	± text lane	Correct shown at precision ≥ 95 %; near-unknown FAR	Text lane adds ≥ 5 pp coverage at 95 % precision. Otherwise text becomes veto only, or is removed.
E-4	Calibrated decision vs hand threshold	decision policy	Coverage at 95 % precision, calibration (ECE) on Tier 2	Model ECE ≤ 0.05 and coverage ≥ hand threshold's.
E-5	How much is catalog completeness worth?	enroll the aisle's remaining SKUs with packshot + one shelf view from a different visit	Coverage at 95 % precision; near-unknown FAR	≥ 70 % of the same-photo proxy gain (14→32 full, 16→39 live) survives cross-visit. If not, shelf views are overfitting to capture conditions.
E-6	Same-category scaling	grow the catalog with other sauce/mayo SKUs (packshots)	top-1, recall@K of descriptor index vs exhaustive; latency p50/p95	Index recall@10 ≥ 98 % of exhaustive top-1 at ≥ 300 SKUs. Otherwise stay exhaustive and parallelize.
E-C	Learned matcher	ALIKED+LightGlue (or DISK+LightGlue) as verifier over B's top-5	top-1, coverage at 95 % precision, latency	Adopt only if +3 pp coverage at 95 % precision and p95 identity latency ≤ 80 ms server-side.
E-7	Live temporal model	log-odds accumulation + provisional labels vs current single-shot lock	Tier 3: time to provisional, time to correct confirmed, flips, correct labels lost	Time-to-first-correct p50 ≤ current single-shot; flips ≤ 2 %; correct-then-lost ≤ 1 %.
E-8	Does first-attempt native crop help?	crop_first_attempt on/off	Tier 3 correct confirmed labels, crop traffic	Keep only if +3 pp correct at ≤ 1.5× crop traffic. On Tier 1 it has no basis: live ≥ full.
E-9	Tag prior	± shelf-tag evidence from tags.py	Coverage at 95 % precision	Adopt if +5 pp; also use it for catalog-gap discovery.

8. Migration strategy

Snapshot first: the uncommitted work is saved as stash object 2785c07 (and as tracked.diff / untracked.tgz in the audit scratchpad). Nothing has been reverted.

Item	Class	Action
code/eval_identity.py (untracked)	evaluation infrastructure	Keep and fix (F-07).
gallery.py rank() (diff)	evaluation infrastructure	Keep, eval-only; no production caller.
run-demo-server.sh --identity matcher (diff)	clearly correct production fix	Keep. Also change the default STOCK away from stock-lopia (F-03).
live.py crop_first_attempt, max_attempts 2→4	experimental	Revert. Its stated rationale (69 vs 66) is invalidated. Re-test as E-8.
live.py blur gate early return, sharpen_ratio, best_sharpness	experimental, with a retry hazard	Revert. Re-introduce inside the E-7 evidence model, where a soft crop is a low-weight observation, not a free retry.
live.py vote_agree + _agreed_observations	experimental	Revert (default 1 makes it dead code). E-7 replaces it.
test_live.py change	encodes the reverted budget	Revert with it.
code/rebuild_catalog.py (untracked)	catalog tooling	Keep. Add image↔CSV bijection, packshot-OCR vs name check, and eval-session refusal (enrollment protocol below).
stock-lopia-clean/ (untracked)	catalog data	Keep the images and provenance. Fix stock.csv (F-04) and the README's "Six SKUs" (it lists five). Replace the l06 packshot with the current package.
eval/labels/image6.json (untracked)	ground truth	Replace with evidence/image6_v2.json, schema: label, source, note, v1_label.
identity.decide_identity inlier branch	production	Rewrite as the calibrated decision (E-4); keep the cosine branch only while the gallery exists.
Production score path (engine.score_crop, classify_boxes, inspect)	production	Rewrite: return per-SKU evidence grouped by canonical SKU (F-02).
feature_matcher.py composite score	production	Rewrite into variant B behind an interface; keep Van's scorer as a selectable baseline lane for E-1.
Identity interface	—	Isolate: IdentityLane.evidence(crop) -> {sku: Evidence} with LocalLane, TextLane, and GalleryLane (postponed); DecisionPolicy separate; TrackEvidence in live.
DINO gallery, identity head, finetune_identity.py	production code	Postpone and fence off: not in the live path, not the default; delete after E-6 if nothing depends on it.
Descriptor index	—	Postpone until the catalog passes ~150 SKUs (E-6).
Provisional label on the wire + clients	protocol + native UI	Postponed dependency of E-7. Shared model gets a provisional identity state; Compose and SwiftUI render it as tentative.

Enrollment protocol

• Views per SKU: at least one current-package front packshot, stored as a 200/400/800 px-tall pyramid (source ≥ 800 px tall), plus at least one real shelf capture at scan distance from a store visit that is not an evaluation session. Add side or back views only for SKUs displayed rotated.
• Minimum usable resolution: reject any reference view under 160 px tall at its largest pyramid level. Crops that small carry no text geometry (observed: facings under 60 px wide fail every lane).
• Metadata: JAN, maker, product name as printed, size, package version or date, source URL, capture session id, SHA-256.
  • The enrollment tool validates a one-to-one match between images and CSV rows.
  • It OCRs each packshot and requires the catalog name's distinctive tokens to appear (this check would have caught the l09/l10 mix-up and flagged the l06 package version).
  • It flags cross-SKU near-duplicates as "text-discriminated siblings".
• Leakage: enrollment refuses any image whose capture session id appears in an evaluation manifest. Pixel equality is not a leak test; re-cut crops pass it.
• Versioning: the catalog manifest has a content hash; every eval report records catalog hash, label-file hash, and code SHA.
• Completeness rule: a bay is "live-ready" when every SKU on its shelf tags is enrolled or explicitly listed as text-only. Near-unknown precision depends on this more than on any threshold.

9. Completion gate

Measured on Tier 2 held-out (and Tier 3 for time-to-label). Same crops, same catalog hash, same detector for both systems. "Van" means Van's scorer with always-argmax, run on that same catalog.

Gate	Definition	Pass
G1 coverage	correct shown labels ÷ all in-catalog facings (enrolled or not)	new ≥ Van; lower 95 % bound of the paired-bootstrap difference ≥ −2 pp
G2 correctness	correct shown ÷ all shown	≥ 95 % (Van: 29–35 % on Tier 1)
G3 unknown handling	near-unknown false-accept rate; far-unknown false-accept rate	≤ 5 %; ≤ 1 % (Van: 100 %)
G4 time to label	Tier 3: p50 / p95 seconds from a facing entering view to a correct label shown (provisional counts only if later confirmed as the same SKU)	p50 ≤ 1.0 s and p95 ≤ 2.5 s, and p50 no worse than Van's click-to-label on a still (tap + ~50–100 ms)
G5 stability	confirmed labels that later change; correct labels that disappear	≤ 2 %; ≤ 1 %
G6 per-SKU floor	recall of every enrolled SKU with ≥ 5 Tier-2 facings	≥ 60 % (catches l06-type catalog faults)
G7 hubness	max predicted-SKU share ÷ that SKU's true share	≤ 2.0
G8 runtime	server identity latency p95 per crop; server memory	≤ 60 ms; within the demo host's budget

Not complete until G1–G7 pass on the device path (Tier 3), not just offline.

---

Findings

ID	Finding	Severity	Evidence	Priority	Source
F-01	image6 ground truth 23 % wrong	blocking	VERIFIED	P0	open-world
F-11	No held-out benchmark exists (one photo)	blocking	VERIFIED	P0	open-world
F-02	Production scores are per template, not per SKU → self-ambiguity and nameless labels	blocking	VERIFIED	P0	open-world
F-03	Demo server defaults to the leaked, corrupt stock-lopia	blocking	VERIFIED	P0	open-world
F-04	Clean catalog CSV ↔ image mismatch (l09/l10)	blocking	VERIFIED	P0	open-world
F-05	Near-unknown siblings accepted; shown-label precision 29–43 %	blocking	VERIFIED	P0	open-world
F-07	Evaluator defects beyond bugs #1–#3	major	VERIFIED	P1	open-world
F-06	Matcher scoring defects (scale, dead ORB/BRISK, unverified "inliers", grayscale)	major	VERIFIED	P1	open-world
F-08	DINO unfit as classifier and retriever here	major	VERIFIED	P1	open-world
F-09	Confirmed labels never re-evaluated; first accept locks	major	HIGH CONFIDENCE	P1	open-world
F-10	Uncommitted live diff: unbounded blur retries; rationale invalidated	major	HIGH CONFIDENCE	P1	open-world
F-12	l06 packshot is a superseded package design	major	VERIFIED	P2	open-world
F-13	Stale handoff test status; swallowed callback exception in a test fixture	polish	VERIFIED	P3	open-world

Counts: 11 VERIFIED / 2 HIGH CONFIDENCE / 0 HYPOTHESIS (hypotheses listed separately) · 13 open-world. No reference baseline was used: this audit covers a recognition backend. The product-level floor is Van's system (G1–G4).

Workstreams

1. Evidence (single owner, first): F-01, F-11, F-07. Blocks every architecture change.
2. Catalog (parallel with 1): F-03, F-04, F-12, enrollment tool checks.
3. Identity core (single owner, after 1): F-02, F-06, text lane, decision layer, E-1–E-4.
4. Live (after 3's interface exists): F-10 revert now; F-09 and the provisional label (E-7) later. The protocol plus native-UI part touches the KMP app and needs its own owner.
5. Scaling (after 3): E-6, and the descriptor index only when it's needed.

F-01 — image6 labels are 23 % wrong — P0 · blocking · VERIFIED

• Root cause: labels appear propagated from tag/box matching (every row has a match field) rather than read from packages.
• Evidence: evidence/image6-v1-labels-overlay.jpg; the per-index list in §3. Rescoring the same runs under v2 moves ungated live top-1 from 26/35 to 35/42.
• Target: every scored label carries provenance; ambiguous facings are excluded.
• Change: adopt evidence/image6_v2.json; schema-check label/source/note.
• Validation: E-0 double-labelling on Tier 2.
• Acceptance: no architecture claim cites v1.

F-02 — template-level scores in production — P0 · blocking · VERIFIED

• Root cause: ProductFeatureMatcher.score_crop keys scores by file stem; engine.score_crop (code/engine.py:424-448), classify_boxes, and inspect pass them straight to decide_identity. The live payload looks up identity.sku_id in the catalog, which has only canonical ids.
• Evidence: on stock-lopia (the demo default), 22/109 facings at full and 29/109 at live are "ambiguous" only because the top two are views of the same SKU. 68 of 69 (full) and 56 of 58 (live) accepted ids are view-suffixed (sku_l01_1), which have no catalog row, so the payload carries no product block.
• Target: scores grouped by canonical SKU (max over views) before any decision; the id shown is always a catalog row.
• Change: group at the lane boundary (the future IdentityLane.evidence). Share one canonicalizer with rebuild_catalog.sku_of.
• Validation: unit test: two strong views of one SKU → accepted, canonical id.
• Acceptance: zero view-suffixed ids on the wire.

F-03 — demo default catalog is the leaked one — P0 · blocking · VERIFIED

• Evidence: run-demo-server.sh:11 STOCK="${1:-stock-lopia}". That catalog is shelf crops cut from the eval photo, and it scores 94–97 % "identity" on its own exam.
• Change: default to the clean catalog once F-04 is fixed; refuse catalogs without a provenance manifest.

F-04 — clean catalog metadata mismatch — P0 · blocking · VERIFIED

• Evidence: stock-lopia-clean/stock.csv has no sku_l09 row for sku_l09.jpg; its sku_l10 row has l09's counts (30/13/17) with l10's name and price. Every l09 answer ships without product info; l10 has a row but no image. The README says "Six SKUs … have no official pack shot" and lists five.
• Change: restore the l09 row from stock-lopia/stock.csv, drop the mangled l10 counts, fix the README, and add the bijection check to rebuild_catalog.py.

F-05 — near-unknown siblings accepted — P0 · blocking · VERIFIED

• Evidence: the current gate accepts 22–28/48 unknowns; 9/11 unenrolled l10 facings go to l09 with margin 0.67–0.93 (higher than many correct l11 matches); Kewpie 450 g → l03 with 17–28 SIFT inliers.
• Target / change: the decision layer + text veto (§5) + enrollment completeness (E-5).
• Acceptance: G2, G3.

F-06 — matcher scoring defects — P1 · major · VERIFIED

• Evidence: ORB inliers are 0 on nearly every live crop (per-facing log from diag.py). compute_geometric_inliers returns len(good_matches) unverified when there are fewer than 4. The l09 reference (305 px) scores 85–129 while others score 10–40.
• Change: variant B (§5), behind the lane interface, with Van's scorer kept as a baseline lane.
• Acceptance: E-1, E-2.

F-07 — evaluator defects — P1 · major · VERIFIED

• top1 ≡ coverage (both are accepted-correct ÷ enrolled), so the identity and decision ceilings were never separated.
• Precision's denominator excludes unenrolled in-catalog facings (covered in main): it reports 50–53 % where the honest figure is 39–43 %.
• The gallery's SKU set comes from ground-truth labels (sku_ids from held).
• Leak check requires the same shape.
• HybridLane scores the full catalog, so its latency isn't a retrieval design.
• group_by_sku only handles the sku_l prefix.
• Change: report the four ceilings explicitly (the per-facing log in diag.py/variants.py is the template). Use the catalog, not labels, for candidate sets. Refuse leakage by session manifest.

F-08 — DINO — P1 · major · VERIFIED

• Evidence: gallery top-1 21 %/12 % with 50–60 % hub share; recall@10 of 7/42 at 493 SKUs (dino_scale.py); 41 ms per embedding.
• Change: fence off (§8). The uncommitted --identity matcher default is the right call.

F-09 — no identity re-evaluation — P1 · major · HIGH CONFIDENCE

• Evidence: code/live.py _on_classified: if track.state != STATE_DETECTED or track.confirmed: return. With vote_agree = 1 the first accepted observation confirms.
• Target: E-7's evidence model with provisional labels and hysteresis.
• Validation: Tier 3 flips and correct-then-lost metrics.

F-10 — uncommitted live experiments — P1 · major · HIGH CONFIDENCE

• Evidence: the blur gate returns before track.attempts += 1, so eligibility never runs out for a soft crop. crop_first_attempt cites the invalidated 69/66 numbers.
• Change: revert per §8.

F-12 — l06 package version — P2 · major · VERIFIED

• Evidence: the packshot shows "内臓脂肪ケア フィッテ"; the shelf shows "内臓脂肪を減らす キユーピーフィッテ". Improved matcher 4/4 at full, 1/4 at live; 3/3 with a shelf reference.
• Change: enroll the current package; adopt the packshot-OCR check.

F-13 — stale test status — P3 · polish · VERIFIED

• Evidence: 49/49 pass on both trees. An AttributeError: 'str' object has no attribute 'items' is raised and swallowed inside a future callback in one test, on both trees.
• Change: make that fixture return a dict, or assert the callback path.

Hypotheses

ID	Suspected	Why it matters	The check that settles it
H-01	Learned matchers (ALIKED/DISK + LightGlue) beat B on small, blurred, or re-designed packages	Could lift the identity ceiling	E-C (needs weight download approval)
H-02	Shelf-tag OCR prior raises coverage and precision	Tags are readable above every run on image6	E-9 using code/tags.py
H-03	Cross-visit shelf references retain most of the leave-facing-out gain	Decides the enrollment protocol	E-5
H-04	Evidence accumulation shortens time-to-correct-label vs single-shot lock	G4/G5	E-7 on Tier 3
H-05	On-device OCR is fast enough (< 30 ms per crop) and as accurate at native crop resolution	Where the text lane runs	Benchmark Vision on iPhone and ML Kit on Android over Tier-2 crops
H-06	Stream JPEG quality costs SIFT inliers at live	Cheap fix if true	Rerun variants.py on re-encoded crops at the client's quality
H-07	The hue feature is robust to store lighting	The colour gain could be lighting-specific	Tier 2 across stores and times
H-08	ORB/BRISK removal costs nothing	Simplifies and speeds the matcher	E-2 at native-crop resolution, where ORB helped the small l09 reference
H-09	Full-resolution OCR with ×2 upsampling matches live's text-lane gain	Full-res fusion was weaker (75 % precision) with ×1 OCR	Rerun ocr_exp.py + fuse_eval.py with full ×2

Preserved decisions

• The matcher kept as a selectable baseline lane. The shipped default is now local (variant B plus the text lane), which is what the measurements below support; --identity matcher reproduces Van's scorer for E-1.
• Bounded, drift-checked crop lane protocol (aa25f99, 13a4e0f); its tests pass.
• eval_identity.py's architecture-neutral lane framing and explicit enrollment ceiling.
• Clean-catalog provenance table and the rebuild_catalog.py refusal rules.
• vote_agree = 1 without provisional labels. Backing out 2 was right.

Implementation record (2026-09-28)

What §8 asked for, and where it landed. Verified by the suite (251 tests) and by running the real server against add/image6.jpg.

§8 item	Status
F-02 per-SKU scores in score_crop, classify_boxes, inspect	Done. group_by_sku on every backend; no view-suffixed id reaches the wire.
F-03 demo default catalog	Done. run-demo-server.sh defaults to stock-lopia-clean.
F-04 clean catalog CSV	Done. All 12 rows; sku_l09 added, sku_l10 corrected.
F-05 near-unknown siblings	Done via the text lane. 0 of 48 unknowns accepted at live resolution.
F-06 variant B behind a lane interface	Done. identity_lanes.LocalLane; Van's scorer stays selectable as --identity matcher.
F-07 evaluator defects	Done. Four separate ceilings, honest precision denominator, catalog-derived candidate set, sku_of grouping, leak signature.
F-08 DINO fenced off	Done. Not the default, not in the live path, refuses to start without weights.
F-10 uncommitted live experiments	Reverted. crop_first_attempt, the blur early return, vote_agree and max_attempts 4 are gone.
F-12 l06 current package	Blocked on a capture. Kewpie serves the superseded design for JAN 4901577083600, so the current package cannot be enrolled from the official URL. The text lane ranks l06 first on all 4 shelf facings but abstains below the accept line, so no wrong label is shown.
F-13 stale status / swallowed exception	Done. The fixture returns the matcher's (scores, details) shape; zero swallowed callback exceptions.
F-09 re-evaluation with provisional labels	Postponed as §8 specifies — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet.

Two defects found while wiring the lanes into the live path, both fixed: live re-sorted the local lane's scores by value, which discards the lane's own rank order (order comes from rank_key, the value is only the gate), and the early return on an empty local observation made the text lane's "answer on its own" job unreachable. Live now decides per crop on the lane's own order.

Measured after the change, through the real server: /v1/recognize names 41 of 112 detections at 90% precision with no view-suffixed ids; the live WebSocket names 12 at 100% precision with 0 unknown accepts, first label on frame 3. Offline on the corrected labels: 36/61 (59%) at 95% precision, 0/48 unknowns, 11 of those labels sourced from the text lane alone.

The headline number is reproducible from the repo, not from a scratch script:

.venv/bin/python code/eval_identity.py --stock stock-lopia-clean \
  --labels docs/research/2026-09-28-recognition-architecture/evidence/image6_v2.json \
  --image add/image6.jpg --lanes local --live

live resolution: catalog 69% · identity 86% · decision 86% ·
end-to-end 59% · precision 95% · 20.6 ms/crop

false accepts: 2 of 109 scored facings — 0 of the 48 unknowns, and 1
unenrolled in-catalog facing plus 1 enrolled misidentification. The
evaluator's own columns: unknown_false_accepts 0, unenrolled_false_accepts 1.

Still open, and not implementable from here: Tier 2 held-out labels (F-11), the E-1..E-9 experiments, and the completion gate G1-G7, all of which need a second independent shelf visit.

Out of scope

Detector recall and box quality (all experiments use the same boxes). Tag and planogram pipeline internals. Native UI. On-device performance. Learned local features (not installed; downloading them needs your approval). image.jpg / stock/ (no labels exist). Timings from parallel runs are relative; scale.py timings are single-process on this Mac.