The enrolled sku_l06.jpg showed the superseded 内臓脂肪ケア wrapper. The shelf
and the current retail package both read 内臓脂肪を減らす キューピーフィッテ
210g, so the reference image was of a design the store no longer sells - the
catalog was pointing at the wrong package.
F-12 was recorded as blocked on a capture because Kewpie's own site serves the
old design for that JAN. An independent retailer listing carries the current
one, and it matches the shelf facings exactly (verified by eye against a crop
of add/image6.jpg before enrolling).
Measured effect on the corrected labels: live decision 60% -> 64%, end-to-end
59% -> 62%, full-resolution decision 81% -> 83%. Suite still 307.
Also lands 12 more external-proxy images (six SKUs, two source domains each)
and their scored result. Across both external sets - 20 images - the lane is
13 correct, 7 abstain, 0 false accepts.
The row named a product that is not on the shelf. The tag under facings #46-48
reads ユーラス 野菜をおいしく食べる旨タレ (380g) at ¥299 - the exact price the
l12 row already carried - and the package reads お子さまOK 甘めテイスト, the
opposite of 中辛. Category, heat level and brand were all wrong.
This one is worse than a wrong size: the text lane scores a crop against the
row's printed name, so a correct read of the real bottle would have been
vetoed by a row describing a different product.
The 2023 green-label predecessor (380ml, JAN 4571233473887, checksum verified)
is recorded in the manifest as predecessor-only evidence; no current JAN is
claimed, because the current white package's rear label has not been
photographed. One in-store rear-label photo would close that.
Both CSVs called sku_l11 キューピー ハーフマヨネーズ 450g. The shelf tag under
the l11 row in add/image6.jpg reads キユーピー ハーフマヨネーズ〈400g〉, and the
enrolled packshot's own OCR reads 400g, so the catalog shipped a size the
product does not have - the same defect class as sku_l10.
Measured on the corrected image6 labels, local lane: full 86% precision (was
84%), live 95% (was 91%), false accepts 8 and 2. 303 tests pass.
Ajinomoto ピュアセレクト マヨネーズ has no 450g size: the official lineup is
250g/400g/600g/1kg, the shelf tag in add/image6.jpg reads 400g, and DECISION.md
already recorded the mismatch. Both CSVs carried 450g, so every l10 answer
shipped a wrong size.
The packshot is the exact 400g front (JAN 4901001114030, 1000x1000) with its
source URL and the 450g rejection recorded in
data/packshots/research/sku_l10/manifest.json.
Enrolling it moves the catalog ceiling from 69% to 87% of in-catalog facings
answerable by image (l10 was 11 unenrolled facings). Measured on the corrected
image6 labels: local lane 92% identity at 84% precision full, 85% at 91% live.
Also points external-eval defaults at the repository-local data path; the
licensed Wikimedia corpus remains uncommitted pending manifest review.
The user's objection was right and it changed the design: the tag channel is
sparse in the real world (measured: 0 tags readable at ~2 m on the main shelf), so
packaging recognition has to carry identity. Measured on a 12-class, 57-facing
evaluation set read off the demo shelf, the packaging channel as shipped cannot:
views per SKU embedding top-1 shipped matcher top-1
1 57 % 33 %
2 72 % 27 %
3 75 % 20 %
4 76 % 5 %
Two things follow. First, two views per SKU is where most of the gain is - that is
the enrollment spec (a first scan already produces several facings per product).
Second, the matcher gets *worse* with more views, because its raw inlier score is
not calibrated to be aggregated and it cannot express two views of one SKU at all:
it keys templates by filename.
So identity becomes retrieval: `gallery.py` embeds every enrolled view with DINOv2
ViT-S/14, scores a SKU by its best view, and rejects when nothing is close enough
(an unknown product must come back unknown, not as the nearest stranger). Measured
with held-out views - the gallery never contains its own queries - 2 views per SKU
over 12 SKUs gives 69 % top-1 with rejection working, and the threshold moves the
rejection line.
It is affordable in the live loop only on the GPU: 159 ms per crop on CPU versus
3.6 ms on the M5's MPS, and a live frame only needs its new tracks embedded.
157 tests pass; test_gallery.py covers the retrieval contract (best view wins,
several views are one answer, strangers are rejected) with injected embeddings so
it needs no torch.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Stage 2 of the plan is "identity as retrieval", so the first thing measured was
whether retrieval can even work here. It cannot, on its own: an off-the-shelf
DINOv2 embedding does group identical products (two orange juices, two apple
juices, two almond drinks) but the fine-grained cases collapse - 32 cartons into
one cluster, and nine different bottled drinks grouped by *container shape*. That
matches the literature (RP2K, RPC all fine-tune; nobody ships off-the-shelf
features for SKU identity).
Meanwhile the shelf has been carrying the answer. Every product has a tag printed
under it, and the tag spells the product's name out in clean text, once per
product - the store's own label. This adds that as an identity channel:
- a tag with a readable name is matched against the catalog by name similarity;
- facings are assigned to the *nearest tag along their row* (the tags mark where
products sit), not by geometric grouping, which was measured to merge different
products into a 945 px "product" that then swallowed the wrong tag;
- rows are clustered first, because the same column on six shelves would
otherwise become one product.
Then the demo catalog itself had to be fixed: it listed another store's products
(senbei, Tropicana) while the shelf holds mayonnaise, so nothing could match. The
catalog for this shelf was read off the tags - 12 products with names and prices -
and the products were enrolled as packaging templates by cropping the facings the
tags had just named.
Measured through the real still path on add/image6.jpg (111 detections):
tag named the product 28 facings (all 28 verified correct by eye)
packaging named the product 74 facings
both spoke 23 facings, agreeing 16 (69 %)
facings with any identity 79
So packaging recognition makes a ~30 % error rate on the facings the tag can
check, and every disagreement carried a low packaging score. The tag therefore
wins a conflict, and the conflict is recorded (`identity_conflict`) instead of
hidden - a product whose tag says one thing and whose packaging says another is a
real finding, not noise to suppress.
144 tests pass; test_tag_identity.py adds 6.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
