product-recognition-demo/recognization/docs/research/2026-09-29-shelf-intelligence/DECISION.md

Dorayaki: shelf-intelligence decision

Measured 2026-09-29 · server product-recognition-demo/recognization at fbf31d7 plus the coverage-first diff described in §F · evidence in evidence/ · every number below is read from a JSON in that directory, named inline.

Scope: the live decision path (group inference, OCR scheduling, decision levels), the catalog the server matches against, and the two native overlays. Not in scope: detector quality, tag/planogram pipeline, on-device runtime.

The previous phase (../2026-09-28-recognition-architecture/DECISION.md) settled what matches a product: a local-feature instance matcher plus a text lane behind a calibrated open-set gate. This phase settles how much of a shelf to label, how fast, and how to spend compute doing it.

Verdict

Label every detected product with its best candidate immediately, mark it provisional, and refine it as stronger evidence arrives. Keep the accepted line where it is; stop treating abstention as quality. Group adjacent facings so one strong observation can answer for a run, and spend OCR on group representatives and ambiguous tracks instead of every crop.

The measured case for it, on add/image6.jpg at live resolution (ocr-ablation.json, anytime-policies.json):

Policy	Shown	Correct	Precision	Coverage of 64 in-catalog facings	Blank
strict / individual	39	38	97.4 %	59.4 %	64.2 %
coverage-first (shipped)	63	55	87.3 %	86.0 %	42.2 %

The shipped policy shows 24 more products and gets 17 more of them right, at the cost of 7 more wrong labels. That is the trade the product asked for: a shelf where 86 % of products carry a name reads as working; one where 59 % do reads as broken.

A. Where the live time goes

replay.py on the running server, image6 at 8 fps (replay-image6-8fps-*.json):

Stage	p50	p95
decode	8–12 ms	20 ms
detect	67–107 ms	131–278 ms
track	1–3 ms	6 ms
total	90–140 ms	200–300 ms

Detection dominates frame cost by an order of magnitude; tracking is free. Identity work is off the frame path (a worker pool), so it shows up as label latency, not frame latency. scheduler-benchmark.json prices it: 218 classification tasks, worker p50 147 ms with OCR on every attempt versus 19.6 ms with OCR once per track. OCR is ~87 % of per-task identity cost.

Ranked bottlenecks: (1) OCR invocations, (2) matcher invocations per facing, (3) detector. Only the first two are ours to schedule.

B. Compute and evidence schedule

scheduler-benchmark.json, same 218 tasks:

Schedule	OCR calls	Worker p50	First facing	All first facings	Wall	CPU
every attempt	218	147 ms	1177 ms	1890 ms	4.32 s	4.48 s
once per track	109	20 ms	1235 ms	2680 ms	2.77 s	3.42 s
ambiguous track only	83	16 ms	806 ms	1951 ms	2.05 s	2.97 s
group representative	49	16 ms	454 ms	1168 ms	1.30 s	2.64 s

Group representatives cut OCR calls 4.4× and wall time 3.3×, and they make the first label appear 2.6× sooner. The accuracy cost is small and measured (ocr-ablation.json, group metrics): full-resolution coverage 57.8 % → 56.0 %, precision 87.3 % → 80.3 %; at live resolution coverage rises 56.0 % → 58.7 % and precision 90.2 % → 87.5 %. Ship the group-representative schedule; the ambiguous_track schedule is the fallback when grouping is unavailable.

The shipped live path already does the cheap half of this: OCR runs once per track (live.py caches text_read on the track), and the local lane re-runs on every attempt.

C. Repeated-facing group detection

analysis.json (resolutions.full and .live) scores eleven pairwise signals on 83 adjacent pairs from image6 (72 same-SKU, 3 sibling, 8 unrelated), with leave-one-row-out thresholds:

Signal	AUC same-vs-sibling	AUC same-vs-unrelated	Precision	Recall	Boundary acc.
hue histogram	0.977	0.967	98.3 %	80.6 %	81.9 %
logo pixels	0.968	0.981	100 %	93.1 %	94.0 %
pixels	0.944	0.979	100 %	88.9 %	90.4 %
inliers	0.944	0.944	100 %	88.9 %	90.4 %
height	0.972	0.775	92.6 %	87.5 %	83.1 %
aspect	0.403	0.691	93.9 %	43.1 %	48.2 %
ocr_jaccard	0.667	0.806	97.7 %	58.3 %	62.7 %
combined	0.991	0.990	98.5 %	91.7 %	91.6 %

Use the combined signal: it is the only one that is simultaneously strong on siblings (the hard case) and on unrelated pairs. Hue alone is nearly as good and is the cheapest single signal; logo pixels is the best single signal but needs a logo detector. Aspect ratio and OCR similarity are actively misleading on siblings (AUC 0.40 and 0.67) — do not gate a group on either.

At live resolution the combined signal holds: AUC 0.991 / 0.984, precision 96.9 %, recall 87.5 %, boundary accuracy 86.7 %.

D. Group-to-individual inference

representative.json prices propagating a group identity to its members, on image6 (109 facings, 64 in-catalog):

Arm	Matcher calls	Total ms	Shown	Correct	Precision	Coverage	Rescued	Damaged
sharpest-one	47	1114	50	46	92.0 %	45.9 %	1	8
sharpest-two	70	1413	52	48	92.3 %	47.7 %	1	6
all (no grouping)	109	2024	63	55	87.3 %	57.8 %	2	0

At full resolution, recognising only the sharpest member of each group damages 8 correct labels to rescue 1. At live resolution the same arm rescues 3 and damages 8. Group propagation is therefore not a coverage win on this shelf: the members are individually recognisable, and the group's answer is only as good as its weakest member when the group is wrong.

Decision: use groups to schedule compute (C, B), not to overwrite individual labels. Propagate a group identity only to members whose own evidence is empty, and never over a member that produced a confident individual answer. The rescued_correct column is the ceiling for that rule; it is small here and should be re-measured on a shelf with more blur and occlusion.

E. Sibling boundaries

The combined signal's boundary accuracy is 91.6 % at full resolution, 86.7 % at live (analysis.json). The failure mode is a boundary missed, not a boundary invented: precision 98.5 % vs recall 91.7 %. A missed boundary merges two sibling runs into one group.

Because §D says groups do not overwrite individual labels, a missed boundary is now a scheduling error (one representative OCR'd for two SKUs), not a labelling error. That is the safe direction to fail. Two rules keep it safe:

• A member whose own individual answer disagrees with the group's is a boundary candidate, not an outlier to be smoothed away.
• A group never spans a detected price-tag change or a gap wider than the shelf's own facing pitch.

F. Provisional and confirmed transitions

Shipped in live.py _on_classified, identity.py, identity_lanes.py:

• accepted — the decision cleared its gate. state: recognized, identity_level: accepted, confirmed = True, never re-classified.
• provisional — the decision produced a candidate but did not clear the gate. state: recognized, identity_level: provisional, confirmed = False, and the track stays eligible for re-classification so a later, sharper attempt can upgrade it to accepted or correct it.
• unrecognized — no candidate at all, or the attempt budget is spent with no provisional label on screen. state: unrecognized, no chip.

A provisional label is never blanked once shown: when the budget runs out the track keeps its guess and simply stops being re-classified. An inherited provisional label stays re-classifiable; an inherited accepted label does not.

Measured on the wire (replay-image6-8fps-0.5x-0.json): 65 of 79 tracks labelled, 742 accepted + 309 provisional observations, first label at 1.27 s. identity_level is the only new wire field; an older client that ignores it renders a provisional label as a normal one, which is the intended degradation.

Both overlays render provisional distinctly: amber chip and dashed box (LiveScanOverlayPolicy.isProvisional(identity), ScanScreen.kt, LiveScanOverlay.swift).

G. Operating recommendation

far-unknown-policies.json prices the accept floor on 4580 crops from 64 unrelated Wikimedia shelf photos:

Policy	local_accept	Accepts	Rate
strict	4.0	3	0.066 %
balanced	2.0	8	0.175 %
coverage-first	0.5	83	1.81 %
near-argmax	0.0 + fallback	119	2.60 %

The shipped LOCAL_ACCEPT stays at 4.0 for the accepted level: it is the line that keeps far-unknown false accepts at 0.066 %, and the accepted level is what the product treats as settled. Coverage comes from the provisional level, which is a separate, much looser rule and is not gated by this number.

This is the key structural choice: do not lower the accept line to raise coverage. Lowering it to 0.5 would raise far-unknown false accepts 27× and would still leave the shelf's own hard cases (siblings) unresolved. Showing the best candidate as provisional gets the coverage without moving the line that protects precision.

anytime-policies.json confirms the shape: the strict policy reaches 59.4 % coverage and stays there at every time budget; coverage-first reaches 86 % and is already at 28 % of facings by 250 ms.

H. Measured improvement

Metric	Before	After
Catalog the server matches	7 SKUs / 13 views	60 SKUs / 86 views
Tracks labelled, image6 8 fps 0.5×	57 / 80	65 / 79
In-catalog facings shown	59.4 %	86.0 %
Precision of shown labels	97.4 %	87.3 %
First label	623 ms	1.27 s
OCR calls per 218 tasks	218	49 (group-representative schedule)
Wall time per 218 tasks	4.32 s	1.30 s

The catalog change is code/build_stock_all.py, which merges stock, stock-lopia, stock-mediya and stock-milk150 into stock-all (111 images, 60 SKUs; stock-demo is byte-identical to mediya+milk150 and is not a source). SKU ids are unique across the folders, so the merge is a copy plus a CSV concat.

Limitations

• Every shelf number is one photo, add/image6.jpg, and its labels are the corrected ones from the previous phase. The group signals and the representative arms are measured on that photo's own facings.
• The far-unknown rates come from a cache that records no catalog, so the catalog match is read off the worktree, not attested by the cache.
• scheduler-benchmark.json and anytime-policies.json are diagnostic counterfactuals: they price schedules against frozen evidence, not against a live deployment. Transport and grouping time are excluded.
• The native overlays were changed but not compiled: no JDK is installed on this machine, so ./gradlew cannot run. The shared signature change is mechanical and every caller was updated, but this is unverified by a compiler.
• Group propagation (§D) is deliberately not shipped as a labelling rule. Its ceiling on this shelf is 1–3 rescued labels against 8 damaged; re-measure on a shelf with real blur and occlusion before revisiting.