f74366d6a4
Everything else here is measured on two photographs. This collects real store shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY, CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on them. 46 of 54 images carry a shelf; the rest are book scans and a shop front. What it found, and it is not what I expected: - **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves, spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - tight boxes on images the model has never seen, and correctly nothing on the book scans. - **Identity transfers when the product is genuinely there.** On a 2011 milk-case photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and R-1 elsewhere. The matcher does reach new photographs of products it has templates for. - **Precision is the failure, and rejection is missing.** In that same photograph it labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 - they are all white milk cartons. Across 24 shelves whose products are not in the catalog it accepted 451 identities, and the score cannot be thresholded to fix it: the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and 189.9 (deer cookies). A threshold that catches the foreigners also throws away the real matches. That confirms on real data what the literature and the review predicted: the matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection - is the fix, not a refinement. docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the images themselves are not redistributed. 175 tests pass. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
54 lines
2.6 KiB
Python
54 lines
2.6 KiB
Python
"""
|
|
Run the pipeline over store photos it has never seen.
|
|
|
|
python code/eval_external_images.py [~/dorayaki-data/shelf-images]
|
|
|
|
Reports detection and identity per image, and writes a JSON summary. See
|
|
docs/EXTERNAL-EVAL.md for what the numbers mean.
|
|
"""
|
|
import json, os, sys, time
|
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|
import cv2, numpy as np
|
|
from engine import build_default_engine, EngineSettings
|
|
from tags import TagService, TagSettings
|
|
from reconcile import attach_tags_to_boxes
|
|
|
|
ROOT = os.path.expanduser(sys.argv[1] if len(sys.argv) > 1 else "~/dorayaki-data/shelf-images")
|
|
manifest = json.load(open(os.path.join(ROOT, "manifest.json")))
|
|
|
|
engine = build_default_engine(EngineSettings(stock_dir="stock-lopia"))
|
|
catalog_by_sku = {p["sku_id"]: p for p in engine.catalog()}
|
|
service = TagService(engine.ocr_engine, settings=TagSettings())
|
|
|
|
rows = []
|
|
for entry in manifest["images"]:
|
|
path = os.path.join(ROOT, entry["file"])
|
|
image = cv2.imread(path)
|
|
if image is None:
|
|
continue
|
|
# keep the long side at 1600 so detection cost is comparable across images
|
|
height, width = image.shape[:2]
|
|
if max(height, width) > 1600:
|
|
scale = 1600.0 / max(height, width)
|
|
image = cv2.resize(image, (int(width*scale), int(height*scale)), interpolation=cv2.INTER_AREA)
|
|
started = time.perf_counter()
|
|
result = engine.recognize(image, classify=True, method="hybrid", with_empty_slots=True)
|
|
detections = result["detections"]
|
|
recognized = sum(1 for d in detections if (d.get("identity") or {}).get("state") == "recognized")
|
|
rejected = sum(1 for d in detections if (d.get("identity") or {}).get("state") == "rejected")
|
|
rows.append({
|
|
"file": entry["file"], "title": entry["title"], "license": entry["license"],
|
|
"size": f"{image.shape[1]}x{image.shape[0]}",
|
|
"detections": len(detections), "recognized": recognized, "rejected": rejected,
|
|
"ms": round((time.perf_counter()-started)*1000),
|
|
})
|
|
print(f" {entry['file'][:34]:<34} {image.shape[1]}x{image.shape[0]:<5} "
|
|
f"det={len(detections):>4} rec={recognized:>3} rej={rejected:>3} {rows[-1]['ms']:>5} ms")
|
|
|
|
total_det = sum(r["detections"] for r in rows)
|
|
total_rec = sum(r["recognized"] for r in rows)
|
|
with_det = sum(1 for r in rows if r["detections"] > 0)
|
|
print(f"\n{len(rows)} images, {total_det} detections, {total_rec} recognized "
|
|
f"({100*total_rec//max(1,total_det)}%), images with any detection: {with_det}")
|
|
json.dump(rows, open(os.path.join(ROOT, "eval.json"), "w"), ensure_ascii=False, indent=1)
|
|
print(f"summary: {os.path.join(ROOT, 'eval.json')}")
|