117 lines
5.2 KiB
Markdown
117 lines
5.2 KiB
Markdown
# Capturing real data
|
||
|
||
Everything measured in this project so far comes from one or two photographs. That
|
||
is its largest weakness: the identity benchmark takes its gallery and its queries
|
||
from the same photo, so every accuracy number is an in-distribution number.
|
||
|
||
This is the procedure for replacing that with data from a real store, recorded on
|
||
the device that will do the scanning.
|
||
|
||
## 1. Record a walk
|
||
|
||
From `recognization`, run `./run-demo-server.sh stock-lopia` once to provision
|
||
the ignored `config/local.env` token and the native apps' local LAN settings,
|
||
then stop that server. Rebuild the app after provisioning. The token is a secret:
|
||
keep `.demo-token`, `config/local.env`, and app-local settings out of commits and
|
||
shared logs. For a recorded walk, start the same gallery backend on the
|
||
launcher's port; `code/app.py` reads the token from `config/local.env`:
|
||
|
||
```bash
|
||
cd product-recognition-demo/recognization # from the app repository root
|
||
YOLO_OFFLINE=1 .venv/bin/python code/app.py \
|
||
--port 8099 --host 0.0.0.0 --stock-dir stock-lopia --identity gallery \
|
||
--live-conf 0.15 --classify-workers 8 \
|
||
--record-frames ~/dorayaki-data/storeA-visit1 \
|
||
--record-every 3
|
||
```
|
||
|
||
This requires the local pinned DINOv2 file described in `../README.md`
|
||
(`weights/dinov2_vits14_lvd142m.safetensors`, or
|
||
`DORAYAKI_DINOV2_WEIGHTS`). A LAN bind without a token is refused. Record the
|
||
second visit to a separate directory with the same catalog and model.
|
||
|
||
Then run the app and scan. Each scan session becomes its own directory:
|
||
|
||
```
|
||
~/dorayaki-data/storeA-visit1/<session>/
|
||
000000.jpg the frame, exactly as the client sent it
|
||
000001.jpg
|
||
index.jsonl one line per recorded frame:
|
||
{"frame": 123, "path": "000001.jpg", "at": 1699...,
|
||
"identity_backend": "gallery",
|
||
"summary": {...},
|
||
"tracks": [{"box": [...], "identity": {...},
|
||
"tag_identity": {...}, "tag": {...}}]}
|
||
```
|
||
|
||
Frames are sampled, not continuous, and the index keeps each frame's own number so
|
||
the sampling is visible rather than silent. `/v1/health` reports how many frames
|
||
have been written.
|
||
|
||
## 2. What to capture, and why
|
||
|
||
Two visits to the same shelf are worth more than one long visit, because the
|
||
benchmark needs a **cross-visit split**: the gallery comes from visit 1 and the
|
||
test set from visit 2. That is the difference between "it works on this photo" and
|
||
"it works on this shelf".
|
||
|
||
Per shelf, capture:
|
||
|
||
- a **slow walk along each rail** at working distance (about 1 m) — the live loop's
|
||
real input;
|
||
- a **close pass** (about 30 cm) over a few tags — the only distance where the
|
||
barcode is decodable (measured: 0.63 px per module at 1 m needs ~3× more pixels);
|
||
- the same shelf again **on a second visit**, ideally with something changed: a gap,
|
||
a moved facing, a new product.
|
||
|
||
## 3. Turn a recording into labels
|
||
|
||
The tag channel seeds the labels. A facing whose tag named a product, with a price
|
||
the tag's own arithmetic corroborates, is a candidate label rather than a guess:
|
||
|
||
```bash
|
||
python -c '
|
||
import json, sys
|
||
sys.path.insert(0, "code")
|
||
from record import labelling_candidates
|
||
seeds = labelling_candidates(sys.argv[1])
|
||
print(len(seeds), "candidate labels")
|
||
print(json.dumps(seeds[:3], ensure_ascii=False, indent=1))
|
||
' "$HOME/dorayaki-data/storeA-visit1/REPLACE_WITH_SESSION/index.jsonl"
|
||
```
|
||
|
||
A human then confirms or corrects them in the app's confirmation screen — the same
|
||
screen that closes the flywheel. Nothing here is a training set on its own; it is
|
||
the raw material for one.
|
||
|
||
## 4. Replay a cross-visit benchmark
|
||
|
||
Enroll the gallery from visit 1 only, then freeze that catalog. Confirm labels
|
||
for visit 2 independently; `labels.json` beside its `index.jsonl` (or supplied
|
||
with `--labels`) is a JSON list, or `{"labels": [...]}`. Each entry has `frame`
|
||
(the number in `index.jsonl`), `box` (pixel coordinates `[x1, y1, x2, y2]`),
|
||
and `sku_id` (a catalog SKU). These are confirmed annotations, not the server's
|
||
predictions. For the second visit, run from `recognization` with the same frozen
|
||
`stock-lopia` catalog:
|
||
|
||
```bash
|
||
SESSION="$HOME/dorayaki-data/storeA-visit2/REPLACE_WITH_SESSION"
|
||
YOLO_OFFLINE=1 .venv/bin/python code/eval_live.py "$SESSION" \
|
||
--stock-dir stock-lopia --labels "$SESSION/labels.json" \
|
||
--replay --device cpu
|
||
```
|
||
|
||
A recording written by the current server names `identity_backend` on each
|
||
index line, and the evaluator uses that. `--identity gallery` (or `matcher`)
|
||
is required when the index omits it, including recordings from before that
|
||
field existed, and must match the recorded value when both are present. It
|
||
does not silently score the matcher.
|
||
|
||
Replace `REPLACE_WITH_SESSION` with the actual session name. The evaluator replays
|
||
recorded JPEGs through `LiveSession`, writes `eval_live.json` in that directory,
|
||
and reports backend, loaded head, weight/catalog fingerprints, decision counts,
|
||
and held-out labelled top-1 when valid labels exist. It rejects missing or
|
||
unpinned DINOv2 weights rather than substituting matcher scores; labels copied
|
||
from catalog templates are excluded as leakage. Without confirmed held-out
|
||
visit-2 labels, counts are diagnostic only, **not gallery walk accuracy**. No
|
||
held-out cross-visit result is available here yet.
|