product-recognition-demo/recognization/docs/EXTERNAL-EVAL.md

Evaluation on real store photos from Wikimedia Commons

Everything else in this repository is measured on two photographs. This evaluates the pipeline on 54 real store photos it has never seen, collected from Wikimedia Commons with their licences (CC0 / CC BY / CC BY-SA / public domain) recorded in a manifest.

Reproduce:

python code/collect_shelf_images.py ~/dorayaki-data/shelf-images   # needs network
python code/eval_external_images.py ~/dorayaki-data/shelf-images

1. Detection transfers, and well

Images with at least one detection	46 of 54
Total detections	2,258
Book scans, illustrations, a photo of a shop front	0 detections (correct)

The detector is the teammate's single-class YOLO trained on one shelf, and it found individual cartons, bottles and packets on dairy cases, soy-sauce shelves, spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - with tight boxes. Detection is not the weak link.

2. Identity transfers when the product is genuinely present

The original 21-SKU catalog contains 明治 おいしい牛乳. On File:2011年4月の牛乳 (5588310773).jpg - a milk case photographed in 2011, entirely unseen - the pipeline found 6 facings of 明治 おいしい牛乳 and labelled them correctly, and on File:Kawasaki-Inryo-1.jpg it found 野菜生活100 and プロビオヨーグルト R-1. So the matcher does transfer to new photographs of products it has templates for.

3. Precision is the failure, and rejection is missing entirely

The same run labelled these as 明治 おいしい牛乳, in the same photograph:

• 低脂肪殺菌牛乳 (a タカナシ carton, top left)
• 有機豆乳 (top left corner)
• 特濃4.5 (three cartons, top middle)
• 無調整豆乳 (top right)
• 北海道牛乳 (several, bottom right)

They are all white milk cartons. The matcher has no way to say "unknown", so every look-alike becomes a claim.

Measured over 24 foreign shelves whose products are not in the catalog: 451 facings accepted as recognised products, with the score distribution overlapping the real one and its extremes landing on the wrong images:

	accepted	median score	p10	p90
The shelf the catalog describes (image6)	71	18.2	5.2	372.4
Foreign shelves (24 images)	451	6.0	—	—
A bento (「198円 中華弁当」)	1	50.5
A supermarket interior	1	80.6
Deer cookies at Nara	6	33.1		189.9

No threshold on this score separates the two: it would have to sit above 5 (to catch the foreign p10) and below 6 (to keep the real median), and the worst offenders score higher than any correct match on the demo shelf.

4. What this means for the plan

It confirms, on real data, the two conclusions the literature and the review pointed at - and it makes them concrete:

1. The matcher's raw inlier score is not a confidence. It cannot be used for rejection or for ranking, and it is the reason packaging recognition cannot name products on its own.
2. Fine-grained look-alikes are the failure mode: milk cartons, soy-milk cartons, and the twenty near-identical mayonnaise packs from the earlier benchmark. The GroceryVision study of 190 open VLMs found the same shape (94.5 % Recall@5, a 17.5-point drop at Recall@1): global embeddings cluster categories well and rank visually similar SKUs badly.

So step 4 of the plan is not a refinement, it is the fix: a shortlist followed by local re-ranking (PRISM's shape: SigLIP shortlist, then LightGlue verification) and a calibrated rejection on an embedding distance where same-product and different-product are actually separable (measured earlier: above 0.6 and below 0.5 respectively, unlike this score).

Attribution

The images are from Wikimedia Commons under their individual licences (CC0, CC BY, CC BY-SA, public domain); manifest.json records the author, licence and source URL of each one. They are evaluation material and are not redistributed here.

---

Part 2: the identity stack, measured on a shelf where the product IS present

Part 1 measured shelves whose products are absent. This part uses File:2011年4月の牛乳 (5588310773).jpg - a milk case where 14 of 107 facings are 明治 おいしい牛乳, a product the original catalog holds - so precision and recall can be computed instead of just false accepts. The 107 crops were labelled by eye from a contact sheet.

The shipped matcher

	claims	correct	precision	recall
matcher	31	12	39 %	86 %

It finds almost every true facing and makes 19 false claims, on cartons of 特濃4.5, 低脂肪殺菌牛乳, 明治ラブ and 北海道牛乳. Correct where the product is, wrong on the look-alikes: the same shape as Part 1.

The embedding gallery: rejection, but not precision

threshold	false accepts on foreign shelves	milk shelf: precision	recall
0.45	29 %	28 %	79 %
0.50	18 %	29 %	64 %
0.55	11 %	35 %	50 %
0.60	5 %	45 %	36 %
0.65	2 %	0 %	0 %

The gallery's value is the controllable rejection: on shelves where nothing matches, false accepts fall from the matcher's 47.8 % to 5 % at 0.60, and the curve is smooth enough to pick an operating point. But on the shelf where the product is present its precision (28-45 %) is no better than the matcher's 39 % - it buys honesty about the unknown, not finer discrimination.

Enrolling a shelf from itself does not fix it

The flywheel's first step is to enroll a store's own shelf. Measured by taking 1-3 views per product from the labelled milk shelf and testing on the rest:

views per SKU	top-1	wrong	rejected
1	26 %	30	29
2	25 %	27	27
3	23 %	24	26

Flat at ~25 %, and far below the 72-76 % measured on the demo shelf with the same number of views. The demo shelf's facings are near-identical crops of the same photograph; real milk cartons differ by small text and a colour band. Off-the-shelf DINOv2 is not discriminative enough for fine-grained SKUs, which is what the literature says (RP2K and RPC both fine-tune; GroceryVision reports a 17.5-point drop from Recall@5 to Recall@1). Fine-tuning is not a refinement of the plan, it is a requirement - and its labels are what the tag channel and the confirmation loop produce.

The packaging's own text is a real channel, and resolution-bound

The cartons differ by the text printed on them, so OCR was tested as an identity channel - first single-scale, then with the tag reader's trick (two upscales, the red channel, pooled):

product	single scale	multi-scale + red
MEGMILK (large Latin text)	85 %	85 %
北海道牛乳 (large kanji)	65 %	65 %
明治ラブ	0 %	75 %
特濃4.5	0 %	50 %
明治 おいしい牛乳 (small vertical text)	7 %	28 %
低脂肪殺菌牛乳 / 無調整豆乳 / 酪農3.6	0 %	0 %
overall	51 %	60 %

Text works where it is large and fails where it is small - the same resolution wall as the price tags. This is a capture-spec question, not a model question.

The identity architecture this implies

channel	what it is good for	measured
Detection	finding facings on unseen shelves	46/54 images, tight boxes
Tag (corroborated)	exact identity and price, where readable	15/15 fields on the demo shelf; sparse in the field
Packaging text	brand-level discrimination	60 % recall, higher when text is large
Embedding gallery	rejection - saying "unknown"	5 % false accepts at 0.60
Fine-tuned embedding	fine-grained SKUs (milk cartons)	not built; off-the-shelf is ~25 %
Confirmation loop	everything the above cannot settle	not built

So the honest design is: reject by default, name the product from the tag when it is readable, from the packaging text when it is legible, from a fine-tuned embedding otherwise - and hand the remainder to the operator, whose answers become the next enrollment.