product-recognition-demo/recognization/docs/EXTERNAL-EVAL.md

Evaluation on real store photos from Wikimedia Commons

Everything else in this repository is measured on two photographs. This evaluates the pipeline on 54 real store photos it has never seen, collected from Wikimedia Commons with their licences (CC0 / CC BY / CC BY-SA / public domain) recorded in a manifest.

Reproduce:

python code/collect_shelf_images.py ~/dorayaki-data/shelf-images   # needs network
python code/eval_external_images.py ~/dorayaki-data/shelf-images

1. Detection transfers, and well

Images with at least one detection	46 of 54
Total detections	2,258
Book scans, illustrations, a photo of a shop front	0 detections (correct)

The detector is the teammate's single-class YOLO trained on one shelf, and it found individual cartons, bottles and packets on dairy cases, soy-sauce shelves, spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - with tight boxes. Detection is not the weak link.

2. Identity transfers when the product is genuinely present

The original 21-SKU catalog contains 明治 おいしい牛乳. On File:2011年4月の牛乳 (5588310773).jpg - a milk case photographed in 2011, entirely unseen - the pipeline found 6 facings of 明治 おいしい牛乳 and labelled them correctly, and on File:Kawasaki-Inryo-1.jpg it found 野菜生活100 and プロビオヨーグルト R-1. So the matcher does transfer to new photographs of products it has templates for.

3. Precision is the failure, and rejection is missing entirely

The same run labelled these as 明治 おいしい牛乳, in the same photograph:

• 低脂肪殺菌牛乳 (a タカナシ carton, top left)
• 有機豆乳 (top left corner)
• 特濃4.5 (three cartons, top middle)
• 無調整豆乳 (top right)
• 北海道牛乳 (several, bottom right)

They are all white milk cartons. The matcher has no way to say "unknown", so every look-alike becomes a claim.

Measured over 24 foreign shelves whose products are not in the catalog: 451 facings accepted as recognised products, with the score distribution overlapping the real one and its extremes landing on the wrong images:

	accepted	median score	p10	p90
The shelf the catalog describes (image6)	71	18.2	5.2	372.4
Foreign shelves (24 images)	451	6.0	—	—
A bento (「198円 中華弁当」)	1	50.5
A supermarket interior	1	80.6
Deer cookies at Nara	6	33.1		189.9

No threshold on this score separates the two: it would have to sit above 5 (to catch the foreign p10) and below 6 (to keep the real median), and the worst offenders score higher than any correct match on the demo shelf.

4. What this means for the plan

It confirms, on real data, the two conclusions the literature and the review pointed at - and it makes them concrete:

1. The matcher's raw inlier score is not a confidence. It cannot be used for rejection or for ranking, and it is the reason packaging recognition cannot name products on its own.
2. Fine-grained look-alikes are the failure mode: milk cartons, soy-milk cartons, and the twenty near-identical mayonnaise packs from the earlier benchmark. The GroceryVision study of 190 open VLMs found the same shape (94.5 % Recall@5, a 17.5-point drop at Recall@1): global embeddings cluster categories well and rank visually similar SKUs badly.

So step 4 of the plan is not a refinement, it is the fix: a shortlist followed by local re-ranking (PRISM's shape: SigLIP shortlist, then LightGlue verification) and a calibrated rejection on an embedding distance where same-product and different-product are actually separable (measured earlier: above 0.6 and below 0.5 respectively, unlike this score).

Attribution

The images are from Wikimedia Commons under their individual licences (CC0, CC BY, CC BY-SA, public domain); manifest.json records the author, licence and source URL of each one. They are evaluation material and are not redistributed here.

---

Part 2: the identity stack, measured on a shelf where the product IS present

Part 1 measured shelves whose products are absent. This part uses File:2011年4月の牛乳 (5588310773).jpg - a milk case where 14 of 107 facings are 明治 おいしい牛乳, a product the original catalog holds - so precision and recall can be computed instead of just false accepts. The 107 crops were labelled by eye from a contact sheet.

The shipped matcher

	claims	correct	precision	recall
matcher	31	12	39 %	86 %

It finds almost every true facing and makes 19 false claims, on cartons of 特濃4.5, 低脂肪殺菌牛乳, 明治ラブ and 北海道牛乳. Correct where the product is, wrong on the look-alikes: the same shape as Part 1.

The embedding gallery: rejection, but not precision

threshold	false accepts on foreign shelves	milk shelf: precision	recall
0.45	29 %	28 %	79 %
0.50	18 %	29 %	64 %
0.55	11 %	35 %	50 %
0.60	5 %	45 %	36 %
0.65	2 %	0 %	0 %

The gallery's value is the controllable rejection: on shelves where nothing matches, false accepts fall from the matcher's 47.8 % to 5 % at 0.60, and the curve is smooth enough to pick an operating point. But on the shelf where the product is present its precision (28-45 %) is no better than the matcher's 39 % - it buys honesty about the unknown, not finer discrimination.

Enrolling a shelf from itself does not fix it

The flywheel's first step is to enroll a store's own shelf. Measured by taking 1-3 views per product from the labelled milk shelf and testing on the rest:

views per SKU	top-1	wrong	rejected
1	26 %	30	29
2	25 %	27	27
3	23 %	24	26

Flat at ~25 %, and far below the 72-76 % measured on the demo shelf with the same number of views. The demo shelf's facings are near-identical crops of the same photograph; real milk cartons differ by small text and a colour band. Off-the-shelf DINOv2 is not discriminative enough for fine-grained SKUs, which is what the literature says (RP2K and RPC both fine-tune; GroceryVision reports a 17.5-point drop from Recall@5 to Recall@1). Fine-tuning is not a refinement of the plan, it is a requirement - and its labels are what the tag channel and the confirmation loop produce.

The packaging's own text is a real channel, and resolution-bound

The cartons differ by the text printed on them, so OCR was tested as an identity channel - first single-scale, then with the tag reader's trick (two upscales, the red channel, pooled):

product	single scale	multi-scale + red
MEGMILK (large Latin text)	85 %	85 %
北海道牛乳 (large kanji)	65 %	65 %
明治ラブ	0 %	75 %
特濃4.5	0 %	50 %
明治 おいしい牛乳 (small vertical text)	7 %	28 %
低脂肪殺菌牛乳 / 無調整豆乳 / 酪農3.6	0 %	0 %
overall	51 %	60 %

Text works where it is large and fails where it is small - the same resolution wall as the price tags. This is a capture-spec question, not a model question.

The identity architecture this implies

channel	what it is good for	measured
Detection	finding facings on unseen shelves	46/54 images, tight boxes
Tag (corroborated)	exact identity and price, where readable	15/15 fields on the demo shelf; sparse in the field
Packaging text	brand-level discrimination	60 % recall, higher when text is large
Embedding gallery	rejection - saying "unknown"	5 % false accepts at 0.60
Fine-tuned embedding	fine-grained SKUs (milk cartons)	not built; off-the-shelf is ~25 %
Confirmation loop	everything the above cannot settle	not built

So the honest design is: reject by default, name the product from the tag when it is readable, from the packaging text when it is legible, from a fine-tuned embedding otherwise - and hand the remainder to the operator, whose answers become the next enrollment.

---

Part 3: fine-tuning the embedding, and what it buys

Part 2 ended on the number that matters: enrolling a real shelf from itself leaves off-the-shelf DINOv2 at ~25 % top-1, because real milk cartons differ by small text and a colour band. The literature's answer is fine-tuning, so this measures it.

code/finetune_identity.py trains a small projection head (384 -> 128, cosine classifier, angular margin) on frozen DINOv2 features, with augmentation standing in for the second visit we do not have. Two enrolment views per product, twelve augmented copies each, 72 held-out facings, 8 products. Balanced accuracy is reported because the shelf is unbalanced (MEGMILK has 34 facings, 酪農3.6 has two - a system that always answered "MEGMILK" would look respectable on the raw rate).

	top-1	balanced	answered	of those right
raw DINOv2 cosine	2/72 (2 %)	1 %	72	3 %
raw, rejecting below 0.55	2/72 (2 %)	1 %	44	5 %
fine-tuned + margin	28/72 (38 %)	73 %	72	39 %
fine-tuned, rejecting below 0.4	17/72 (23 %)	58 %	18	94 %

Three things follow.

1. Fine-tuning is not a refinement, it is the difference between working and not. Balanced accuracy goes from 1 % to 73 % on the same held-out facings. Off-the- shelf features cannot separate these products; a head trained on sixteen crops can.
2. With a rejection threshold it becomes precise: answering only 18 of 72 facings and being right on 94 % of them is exactly the operating point a store wants - name what you are sure of, and hand the rest to the operator. The raw model cannot do this at any threshold (5 % of answered).
3. Where it fails is where the labels are coarse. Per class, with rejection: 明治ラブ 2/2, おいしい無調整豆乳 2/2, 低脂肪 1/1, 特濃4.5 1/2, 北海道牛乳 8/21, 明治 おいしい牛乳 2/12, and MEGMILK 1/32. MEGMILK is the class I labelled most coarsely - the contact sheet shows what are probably several different products under one brand - so the class is internally inconsistent and the model correctly refuses most of it. Better labels, not a better model, is what that needs.

The honest limit

Enrolment and held-out facings come from one photograph. Augmentation stands in for a second visit, which is not the same thing: a real second visit changes the light, the angle and the stock. So 73 % balanced and 94 % precision are upper bounds, and the only way to know the real number is to record a second visit (docs/DATA.md).

What is settled is the mechanism: a head trained on labels the pipeline can produce itself - tag-named facings, plus operator confirmations - moves fine-grained identity from useless to usable, and a rejection threshold turns it into something a store can act on.

---

Part 4: making it usable, and the trap that came with it

The head is now part of a store: python code/finetune_identity.py --stock stock-lopia trains on that store's enrolled views and writes <stock>/identity_head.pt, which ProductGallery loads automatically. The first attempt exposed a trap worth writing down.

A trained head destroys rejection

Measured on the same two sets (the home shelf it was enrolled from, and 20 shelves whose products are absent):

	home shelf accepted	foreign shelves accepted
raw features, threshold 0.55	78 %	5 %
trained head, threshold 0.40	58 %	44 %

A discriminative head has no "none of the above": trained with an angular margin to separate seven products, it maps everything confidently onto its seven prototypes, including cartons it has never seen. Rejection, which was the gallery's whole point, disappears.

The fix is two stages, and it needs no retraining

The two mechanisms fail in opposite directions, so each does what it is good at:

• raw features decide whether to answer (they generalise: 5 % false accepts on strangers),
• the head decides what to answer (it discriminates: 73 % balanced on fine-grained products against 1 % raw).

	home shelf accepted	foreign shelves accepted
raw gate, then head ranking	47 %	5 %

The home-shelf acceptance is lower than raw-only because the gate is strict, which is the honest trade: fewer claims, and the claims are worth more.

Still open

• A joint accuracy number on the LOPIA store needs held-out labels there; its enrolment views came from the facings the tags named, so scoring on those would be scoring on the training set.
• Training the head with out-of-distribution negatives (crops from other stores) should let it keep rejecting on its own, which would remove the need for the gate.
• Everything above is still one photograph per store. The second visit remains the measurement that settles it.