tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

feat: record a live session so it can become data

Browse source 
The deep review's first recommendation was real data, because every accuracy number
so far comes from one or two photographs - the identity benchmark even takes its
gallery and its queries from the same photo. This is the mechanism for replacing
that.

`--record-frames DIR` writes each processed frame, sampled every Nth, with a JSON
line describing what the server saw in it: the boxes, the identity channels and the
tag reading. The lines are the labelling input - `labelling_candidates()` returns
the facings the tag channel named with a corroborated price, which are seeds for the
confirmation screen rather than guesses. Nothing here is a training set on its own.

Sampling keeps each frame's own number in the index, so a sampled recording says so
instead of hiding it, and `/v1/health` reports how much has been written. Measured
on a 163-frame walk: 32 frames and their lines.

`docs/DATA.md` is the runbook: what to capture (a slow walk per rail, a close pass
for the barcode, and the same shelf on a second visit), how to turn it into labels,
and how to freeze a benchmark with a cross-visit split.

167 tests pass.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
kiyo9w 2026-09-25 01:34:15 +09:00
parent dc83e75a34
commit a718acdc15
4 changed files with 329 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

19
recognization/code/app.py
View file

@ -72,6 +72,7 @@ from engine import ( # noqa: E402
settings_from_env,
)
from demo import DemoCamera, DemoSourceError # noqa: E402
from record import FrameRecorder # noqa: E402
from live import LiveSession, LiveSettings # noqa: E402
LOGGER = logging.getLogger("dorayaki")
@ -99,6 +100,9 @@ class ServerSettings:
live_conf: float = 0.1
live_tags: bool = True
demo_walk: bool = False
# A live session can be recorded so it becomes data to label (record.py).
record_frames: Optional[str] = None
record_every: int = 3
log_level: str = "info"
ssl_certfile: Optional[str] = None
ssl_keyfile: Optional[str] = None
@ -278,6 +282,7 @@ def create_app(
"engine": getattr(engine.ocr_engine, "name", "none"),
"available": bool(getattr(engine.ocr_engine, "available", lambda: False)()),
},
"recording": recorder.stats() if recorder is not None else None,
"sessions_active": len(sessions),
"uptime_s": round(engine.uptime_s(), 1),
}
@ -322,6 +327,8 @@ def create_app(
# still recognition mapped into each walk frame, which is a film-making device,
# not a perception result. Every payload it produces is marked synthetic.
demo_recognition = None
recorder = FrameRecorder(settings.record_frames, every=settings.record_every) \
if settings.record_frames else None
@ -693,6 +700,10 @@ def create_app(
await send({"type": "error", "code": "INTERNAL", "frame_id": frame_id,
"message": "frame processing failed"})
continue
if recorder is not None:
# Record what the client sent and what we saw in it, so the walk
# can be labelled later instead of only measured on photographs.
recorder.write(session_id, frame_id, payload, result)
if demo_recognition is not None and demo_sample_ids:
# The walk's frames are served from the shelf's full recognition,
# so every product is already labelled with name and price.
@ -980,6 +991,10 @@ def main() -> None:
help="classifications in flight per session (demo walks want a larger budget)")
parser.add_argument("--live-attempts", type=int, default=None,
help="classification attempts per track")
parser.add_argument("--record-frames", type=str, default=None,
help="write processed frames and their results under this directory")
parser.add_argument("--record-every", type=int, default=None,
help="record one frame in every N (default 3)")
parser.add_argument("--demo-walk", action="store_true",
help="serve the demo walk's pre-recognised shelf as live tracks")
parser.add_argument("--no-tags", action="store_true", help="disable price-tag OCR")
@ -1017,6 +1032,10 @@ def main() -> None:
server.log_level = args.log_level
engine_settings = settings_from_env()
if args.record_frames is not None:
server.record_frames = args.record_frames
if args.record_every is not None:
server.record_every = args.record_every
if args.live_inflight is not None:
engine_settings.live_inflight = args.live_inflight
if args.live_attempts is not None:

137
recognization/code/record.py Normal file
View file

@ -0,0 +1,137 @@
"""
Record a live session so it can become data.
Everything measured so far comes from one or two photographs, which is the largest
weakness in the project. The fix is a store walk recorded from the device that will
do the scanning, with enough context to label it afterwards.
`--record-frames DIR` writes each processed frame plus a JSON line describing what
the server saw in it. The lines are the labelling input: a facing that the tag
channel named is a candidate label, and the confirmation screen is where a human
accepts or corrects it. Nothing here is a training set on its own - it is the raw
material for one.
Layout:
DIR/<session>/000123.jpg the frame, exactly as the client sent it
DIR/<session>/index.jsonl one line per frame:
{"frame": 123, "path": "...", "at": 1699..., "summary": {...},
"tracks": [{"box": [...], "identity": {...}, "tag": {...}}]}
Frames are written at most every `every` processed frames (default 3) so a long
walk does not fill a disk, and the index keeps the frame's own numbering so the
sampling is visible rather than silent.
"""
from __future__ import annotations
import json
import os
import threading
import time
from typing import Any, Dict, List, Optional
class FrameRecorder:
"""Write frames and their results to disk, for labelling later."""
def __init__(self, root: str, every: int = 3, max_frames_per_session: int = 4000) -> None:
self.root = root
self.every = max(1, int(every))
self.max_frames_per_session = int(max_frames_per_session)
self._lock = threading.Lock()
self._counts: Dict[str, int] = {}
self._written: Dict[str, int] = {}
os.makedirs(self.root, exist_ok=True)
def _session_dir(self, session_id: str) -> str:
path = os.path.join(self.root, session_id)
os.makedirs(path, exist_ok=True)
return path
def write(self, session_id: str, frame_id: int, payload: bytes, result: Dict[str, Any]) -> None:
"""Record one processed frame and what was seen in it."""
with self._lock:
seen = self._counts.get(session_id, 0) + 1
self._counts[session_id] = seen
if seen % self.every:
return
written = self._written.get(session_id, 0)
if written >= self.max_frames_per_session:
return
self._written[session_id] = written + 1
directory = self._session_dir(session_id)
name = f"{written:06d}.jpg"
try:
with open(os.path.join(directory, name), "wb") as handle:
handle.write(payload)
except OSError:
return
tracks: List[Dict[str, Any]] = []
for track in result.get("tracks") or []:
tracks.append(
{
"box": track.get("box"),
"identity": track.get("identity"),
"tag_identity": track.get("tag_identity"),
"tag": track.get("price_tag"),
}
)
line = {
"frame": frame_id,
"path": name,
"at": round(time.time(), 3),
"summary": result.get("summary"),
"tracks": tracks,
}
try:
with open(os.path.join(directory, "index.jsonl"), "a", encoding="utf-8") as handle:
handle.write(json.dumps(line, ensure_ascii=False) + "\n")
except OSError:
return
def stats(self) -> Dict[str, Any]:
with self._lock:
return {
"root": self.root,
"every": self.every,
"sessions": len(self._counts),
"frames_written": sum(self._written.values()),
}
def labelling_candidates(index_path: str) -> List[Dict[str, Any]]:
"""Turn a recorded index into candidate labels for a human to confirm.
A facing that the tag channel named carries a product name and a corroborated
price; those are the seeds. The rest are candidates grouped by position, to be
labelled in the confirmation screen rather than by hand from scratch.
"""
seeds: List[Dict[str, Any]] = []
with open(index_path, "r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line:
continue
entry = json.loads(line)
for track in entry.get("tracks") or []:
tag_identity = track.get("tag_identity") or {}
tag = track.get("tag") or {}
if not tag_identity.get("sku_id"):
continue
seeds.append(
{
"frame": entry["frame"],
"box": track.get("box"),
"sku_id": tag_identity["sku_id"],
"name_similarity": tag_identity.get("score"),
"tag_price": tag.get("price"),
"corroborated": bool(
{"tax_consistent", "tax_pair_consistent", "discount_consistent"}
& set(tag.get("flags") or [])
),
}
)
return seeds

81
recognization/code/test_record.py Normal file
View file

@ -0,0 +1,81 @@
"""Tests for recording a live session as data to label."""
from __future__ import annotations
import json
import os
import shutil
import sys
import tempfile
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from record import FrameRecorder, labelling_candidates # noqa: E402
def a_result(sku=None, flags=None):
return {
"summary": {"in_view": 1, "recognized": 1 if sku else 0},
"tracks": [
{
"box": [10, 100, 100, 300],
"identity": {"state": "recognized", "sku_id": sku} if sku else {"state": "unconfirmed"},
"tag_identity": {"sku_id": sku, "score": 0.7} if sku else {},
"price_tag": {"state": "readable", "price": 269, "flags": flags or []},
}
],
}
class RecorderTests(unittest.TestCase):
def setUp(self):
self.root = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.root, ignore_errors=True)
def test_frames_and_an_index_are_written(self):
recorder = FrameRecorder(self.root, every=1)
for frame_id in range(3):
recorder.write("session-a", frame_id, b"\xff\xd8jpeg", a_result("sku_l05"))
directory = os.path.join(self.root, "session-a")
self.assertEqual(sorted(os.listdir(directory)), ["000000.jpg", "000001.jpg", "000002.jpg", "index.jsonl"])
lines = [json.loads(line) for line in open(os.path.join(directory, "index.jsonl"), encoding="utf-8")]
self.assertEqual([line["frame"] for line in lines], [0, 1, 2])
self.assertEqual(lines[0]["tracks"][0]["tag_identity"]["sku_id"], "sku_l05")
def test_sampling_keeps_the_frame_numbering_visible(self):
recorder = FrameRecorder(self.root, every=3)
for frame_id in range(9):
recorder.write("session-b", frame_id, b"x", a_result())
lines = [json.loads(line) for line in open(os.path.join(self.root, "session-b", "index.jsonl"), encoding="utf-8")]
self.assertEqual([line["frame"] for line in lines], [2, 5, 8],
"every third frame, recorded under its own number")
def test_sessions_are_separate(self):
recorder = FrameRecorder(self.root, every=1)
recorder.write("one", 0, b"a", a_result())
recorder.write("two", 0, b"b", a_result())
self.assertTrue(os.path.exists(os.path.join(self.root, "one", "000000.jpg")))
self.assertTrue(os.path.exists(os.path.join(self.root, "two", "000000.jpg")))
def test_a_session_is_capped(self):
recorder = FrameRecorder(self.root, every=1, max_frames_per_session=2)
for frame_id in range(5):
recorder.write("capped", frame_id, b"x", a_result())
self.assertEqual(recorder.stats()["frames_written"], 2)
def test_labelling_candidates_come_from_the_tag_channel(self):
recorder = FrameRecorder(self.root, every=1)
recorder.write("session-c", 0, b"x", a_result("sku_l05", ["tax_consistent"]))
recorder.write("session-c", 1, b"x", a_result())
seeds = labelling_candidates(os.path.join(self.root, "session-c", "index.jsonl"))
self.assertEqual(len(seeds), 1, "only the tag-named facing is a seed")
self.assertEqual(seeds[0]["sku_id"], "sku_l05")
self.assertTrue(seeds[0]["corroborated"])
self.assertEqual(seeds[0]["tag_price"], 269)
if __name__ == "__main__":
unittest.main()

92
recognization/docs/DATA.md Normal file
View file

@ -0,0 +1,92 @@
# Capturing real data
Everything measured in this project so far comes from one or two photographs. That
is its largest weakness: the identity benchmark takes its gallery and its queries
from the same photo, so every accuracy number is an in-distribution number.
This is the procedure for replacing that with data from a real store, recorded on
the device that will do the scanning.
## 1. Record a walk
Start the server with recording on. Frames are written every Nth processed frame,
and each one is written with a line describing what the server saw in it.
```bash
cd recognization
YOLO_OFFLINE=1 .venv/bin/python code/app.py \
--port 8090 --host 0.0.0.0 \
--classify-workers 8 \
--record-frames ~/dorayaki-data/2026-09-25-storeA \
--record-every 3
```
Then run the app and scan. Each scan session becomes its own directory:
```
~/dorayaki-data/2026-09-25-storeA/<session>/
000000.jpg the frame, exactly as the client sent it
000001.jpg
index.jsonl one line per recorded frame:
{"frame": 123, "path": "000001.jpg", "at": 1699...,
"summary": {...},
"tracks": [{"box": [...], "identity": {...},
"tag_identity": {...}, "tag": {...}}]}
```
Frames are sampled, not continuous, and the index keeps each frame's own number so
the sampling is visible rather than silent. `/v1/health` reports how many frames
have been written.
## 2. What to capture, and why
Two visits to the same shelf are worth more than one long visit, because the
benchmark needs a **cross-visit split**: the gallery comes from visit 1 and the
test set from visit 2. That is the difference between "it works on this photo" and
"it works on this shelf".
Per shelf, capture:
- a **slow walk along each rail** at working distance (about 1 m) — the live loop's
real input;
- a **close pass** (about 30 cm) over a few tags — the only distance where the
barcode is decodable (measured: 0.63 px per module at 1 m needs ~3× more pixels);
- the same shelf again **on a second visit**, ideally with something changed: a gap,
a moved facing, a new product.
## 3. Turn a recording into labels
The tag channel seeds the labels. A facing whose tag named a product, with a price
the tag's own arithmetic corroborates, is a candidate label rather than a guess:
```bash
python -c "
import json, sys
sys.path.insert(0, 'code')
from record import labelling_candidates
seeds = labelling_candidates('~/dorayaki-data/2026-09-25-storeA/<session>/index.jsonl')
print(len(seeds), 'candidate labels')
print(json.dumps(seeds[:3], ensure_ascii=False, indent=1))
"
```
A human then confirms or corrects them in the app's confirmation screen — the same
screen that closes the flywheel. Nothing here is a training set on its own; it is
the raw material for one.
## 4. Freeze a benchmark
Write the confirmed labels into a manifest that the evaluation reads, and keep it in
the repository so results are reproducible:
```
benchmarks/<store>-<date>/
visit1/<sku_id>/<view>.jpg the gallery: 2-3 views per SKU
visit2/*.jpg the test frames
labels.json {"visit2/0007.jpg": [{"box": [...], "sku_id": "..."}]}
```
Then the identity benchmark (see `code/bench_identity.py`) can be run with a
cross-visit split, and every number it reports is a number worth acting on. Until
that exists, treat the current figures as what they are: measurements on a
photograph.