tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

feat: let the shelf's own price tag name the product

Browse source 
Stage 2 of the plan is "identity as retrieval", so the first thing measured was
whether retrieval can even work here. It cannot, on its own: an off-the-shelf
DINOv2 embedding does group identical products (two orange juices, two apple
juices, two almond drinks) but the fine-grained cases collapse - 32 cartons into
one cluster, and nine different bottled drinks grouped by *container shape*. That
matches the literature (RP2K, RPC all fine-tune; nobody ships off-the-shelf
features for SKU identity).

Meanwhile the shelf has been carrying the answer. Every product has a tag printed
under it, and the tag spells the product's name out in clean text, once per
product - the store's own label. This adds that as an identity channel:

- a tag with a readable name is matched against the catalog by name similarity;
- facings are assigned to the *nearest tag along their row* (the tags mark where
  products sit), not by geometric grouping, which was measured to merge different
  products into a 945 px "product" that then swallowed the wrong tag;
- rows are clustered first, because the same column on six shelves would
  otherwise become one product.

Then the demo catalog itself had to be fixed: it listed another store's products
(senbei, Tropicana) while the shelf holds mayonnaise, so nothing could match. The
catalog for this shelf was read off the tags - 12 products with names and prices -
and the products were enrolled as packaging templates by cropping the facings the
tags had just named.

Measured through the real still path on add/image6.jpg (111 detections):

  tag named the product          28 facings   (all 28 verified correct by eye)
  packaging named the product    74 facings
  both spoke                     23 facings, agreeing 16 (69 %)
  facings with any identity      79

So packaging recognition makes a ~30 % error rate on the facings the tag can
check, and every disagreement carried a low packaging score. The tag therefore
wins a conflict, and the conflict is recorded (`identity_conflict`) instead of
hidden - a product whose tag says one thing and whose packaging says another is a
real finding, not noise to suppress.

144 tests pass; test_tag_identity.py adds 6.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
kiyo9w 2026-09-24 23:28:36 +09:00
parent cf7e8a5a88
commit 52905e73f6
11 changed files with 507 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

173
recognization/code/reconcile.py
View file

@ -300,6 +300,32 @@ def group_facings(
return groups
def cluster_into_rows(tracks: Sequence[Dict[str, Any]]) -> List[List[Dict[str, Any]]]:
"""Split facings into shelf rows by vertical overlap.
`group_facings` is a *within-row* operation: given the whole frame it would
merge the products stacked in one column across six shelves into a single
"product". Identity assignment needs rows first.
"""
rows: List[List[Dict[str, Any]]] = []
for track in sorted(
(track for track in tracks if track.get("box") or track.get("raw_box")),
key=lambda track: (track.get("box") or track.get("raw_box"))[1],
):
box = track.get("box") or track.get("raw_box")
placed = False
for row in rows:
row_box = _group_box(row)
overlap = min(row_box[3], box[3]) - max(row_box[1], box[1])
if overlap > 0.5 * min(row_box[3] - row_box[1], box[3] - box[1]):
row.append(track)
placed = True
break
if not placed:
rows.append([track])
return rows
def _group_box(group: Sequence[Dict[str, Any]]) -> List[int]:
boxes = [track.get("box") or track.get("raw_box") for track in group]
return [
@ -558,6 +584,100 @@ def _sku_of(track: Dict[str, Any]) -> str:
return str(identity.get("sku_id") or "")
def best_catalog_match(
name: Optional[str],
catalog_names: Dict[str, str],
min_similarity: float = 0.45,
) -> Tuple[Optional[str], float]:
"""The catalog entry a printed tag name most resembles."""
if not name:
return None, 0.0
normalized = normalize_name(name)
best_sku, best_score = None, 0.0
for sku_id, catalog_name in catalog_names.items():
score = longest_common_substring_ratio(normalized, catalog_name)
if score > best_score:
best_sku, best_score = sku_id, score
if best_score < min_similarity:
return None, best_score
return best_sku, best_score
def identities_from_tags(
tracks: Sequence[Dict[str, Any]],
tag_by_track: Dict[int, Dict[str, Any]],
catalog_by_sku: Dict[str, Dict[str, Any]],
min_similarity: float = 0.40,
all_tags: Optional[Sequence[Dict[str, Any]]] = None,
) -> Dict[int, Dict[str, Any]]:
"""Identify a product from the tag printed under it.
The shelf-edge tag carries the store's own name for the product in clean
printed text, and one tag serves a whole run of facings. That makes it the
most reliable identity source on the shelf - far more so than recognising the
packaging, which is fine-grained (twenty near-identical cartons), occluded,
and distorted by glare and angle.
Measured with an off-the-shelf DINOv2 embedding: identical products do cluster
(two orange juices, two apple juices, two almond drinks) but fine-grained
cases do not - 32 cartons collapsed into one cluster and nine different
bottled drinks grouped by *container shape*.
Assignment is by *tag position*, not by geometric grouping: the tags on a rail
mark where each product sits, so a facing belongs to the nearest tag along the
row. Grouping facings by size and spacing was measured to merge different
products (a 945 px "product" spanning half a shelf), which then swallowed the
wrong tag.
"""
catalog_names = {
sku_id: normalize_name(product.get("product_name"))
for sku_id, product in catalog_by_sku.items()
if product.get("product_name")
}
if not catalog_names:
return {}
tags = [dict(tag) for tag in (all_tags or []) if tag.get("box")]
for tag in tags:
if tag.get("sku_from_name") is None:
sku_id, score = best_catalog_match(tag.get("name"), catalog_names, min_similarity)
tag["sku_from_name"] = sku_id
tag["sku_score"] = score
if not tags:
return {}
found: Dict[int, Dict[str, Any]] = {}
for row in cluster_into_rows(tracks):
row_box = _group_box(row)
row_height = max(1, row_box[3] - row_box[1])
# The rail's tags sit just under the row they describe.
row_tags = [
tag
for tag in tags
if row_box[3] - 0.5 * (tag["box"][3] - tag["box"][1])
<= tag["box"][1]
<= row_box[3] + 1.5 * (tag["box"][3] - tag["box"][1])
]
if not row_tags:
continue
for track in row:
box = track.get("box") or track.get("raw_box")
center = (box[0] + box[2]) / 2.0
nearest = min(
row_tags,
key=lambda tag: abs(center - (tag["box"][0] + tag["box"][2]) / 2.0),
)
sku_id = nearest.get("sku_from_name")
if not sku_id:
continue
found[int(track["track_id"])] = {
"sku_id": sku_id,
"score": round(float(nearest.get("sku_score") or 0.0), 3),
"source": "tag_name",
}
return found
def attach_tags_to_boxes(
tag_service: Any,
image: Any,
@ -594,11 +714,63 @@ def attach_tags_to_boxes(
"track_id": int(box["id"]),
"box": list(box["box"]),
"identity": (identities or {}).get(int(box["id"])) or {},
"tag_identity": {},
}
for box in boxes
]
tag_by_track, all_tags, counts = tag_service.attach(tracks)
# Tag-first identity, applied before reconciliation so the comparison below
# sees the identity the tag itself provides.
tag_identities = identities_from_tags(
tracks, tag_by_track, catalog_by_sku, all_tags=all_tags
)
for track in tracks:
derived = tag_identities.get(int(track["track_id"]))
if not derived:
continue
# Always keep the tag's claim, even when packaging recognition already
# named the product: two independent identity channels that can be
# compared are worth more than one that silently wins. A disagreement is
# itself a finding (a misplaced tag, or a product in the wrong place).
track["tag_identity"] = dict(derived)
current = track["identity"] or {}
if current.get("state") == "recognized":
if current.get("sku_id") != derived["sku_id"]:
# Two channels disagree. Measured on the demo shelf: where both
# named the same facing they agreed only 69% of the time, and the
# packaging side of every disagreement carried a low score - so
# the store's own label wins, and the disagreement is recorded
# rather than hidden (a misplaced tag is a real finding).
track["identity_conflict"] = {
"packaging": current.get("sku_id"),
"tag": derived["sku_id"],
"packaging_score": current.get("score"),
}
track["identity"] = {
"state": "recognized",
"sku_id": derived["sku_id"],
"identity_level": "tag_over_packaging",
"confidence": derived["score"],
"score": derived["score"],
"source": "tag_name",
"candidates": [],
"attempts": 1,
"method": "tag_name",
}
continue
track["identity"] = {
"state": "recognized",
"sku_id": derived["sku_id"],
"identity_level": "tag_derived",
"confidence": derived["score"],
"score": derived["score"],
"source": "tag_name",
"candidates": [],
"attempts": 1,
"method": "tag_name",
}
reconciliations: Dict[int, Dict[str, Any]] = {}
for track in tracks:
track_id = track["track_id"]
@ -615,6 +787,7 @@ def attach_tags_to_boxes(
"tags": all_tags,
"counts": counts,
"reconciliations": reconciliations,
"tag_identities": tag_identities,
"bands": [{"row_index": band.row_index, "box": band.box} for band in bands],
"bands_objects": list(bands),
"compliance": {