# Capturing real data

Everything measured in this project so far comes from one or two photographs. That
is its largest weakness: the identity benchmark takes its gallery and its queries
from the same photo, so every accuracy number is an in-distribution number.

This is the procedure for replacing that with data from a real store, recorded on
the device that will do the scanning.

## 1. Record a walk

From `recognization`, run `./run-demo-server.sh stock-lopia` once to provision
the ignored `config/local.env` token and the native apps' local LAN settings,
then stop that server. Rebuild the app after provisioning. The token is a secret:
keep `.demo-token`, `config/local.env`, and app-local settings out of commits and
shared logs. For a recorded walk, start the same gallery backend on the
launcher's port; `code/app.py` reads the token from `config/local.env`:

```bash
cd product-recognition-demo/recognization  # from the app repository root
YOLO_OFFLINE=1 .venv/bin/python code/app.py \
  --port 8099 --host 0.0.0.0 --stock-dir stock-lopia --identity gallery \
  --live-conf 0.15 --classify-workers 8 \
  --record-frames ~/dorayaki-data/storeA-visit1 \
  --record-every 3
```

This requires the local pinned DINOv2 file described in `../README.md`
(`weights/dinov2_vits14_lvd142m.safetensors`, or
`DORAYAKI_DINOV2_WEIGHTS`). A LAN bind without a token is refused. Record the
second visit to a separate directory with the same catalog and model.

Then run the app and scan. Each scan session becomes its own directory:

```
~/dorayaki-data/storeA-visit1/<session>/
    000000.jpg        the frame, exactly as the client sent it
    000001.jpg
    index.jsonl       one line per recorded frame:
                      {"frame": 123, "path": "000001.jpg", "at": 1699...,
                       "identity_backend": "gallery",
                       "summary": {...},
                       "tracks": [{"box": [...], "identity": {...},
                                   "tag_identity": {...}, "tag": {...}}]}
```

Frames are sampled, not continuous, and the index keeps each frame's own number so
the sampling is visible rather than silent. `/v1/health` reports how many frames
have been written.

## 2. What to capture, and why

Two visits to the same shelf are worth more than one long visit, because the
benchmark needs a **cross-visit split**: the gallery comes from visit 1 and the
test set from visit 2. That is the difference between "it works on this photo" and
"it works on this shelf".

Per shelf, capture:

- a **slow walk along each rail** at working distance (about 1 m) — the live loop's
  real input;
- a **close pass** (about 30 cm) over a few tags — the only distance where the
  barcode is decodable (measured: 0.63 px per module at 1 m needs ~3× more pixels);
- the same shelf again **on a second visit**, ideally with something changed: a gap,
  a moved facing, a new product.

## 3. Turn a recording into labels

The tag channel seeds the labels. A facing whose tag named a product, with a price
the tag's own arithmetic corroborates, is a candidate label rather than a guess:

```bash
python -c '
import json, sys
sys.path.insert(0, "code")
from record import labelling_candidates
seeds = labelling_candidates(sys.argv[1])
print(len(seeds), "candidate labels")
print(json.dumps(seeds[:3], ensure_ascii=False, indent=1))
' "$HOME/dorayaki-data/storeA-visit1/REPLACE_WITH_SESSION/index.jsonl"
```

A human then confirms or corrects them in the app's confirmation screen — the same
screen that closes the flywheel. Nothing here is a training set on its own; it is
the raw material for one.

## 4. Replay a cross-visit benchmark

Enroll the gallery from visit 1 only, then freeze that catalog. Confirm labels
for visit 2 independently; `labels.json` beside its `index.jsonl` (or supplied
with `--labels`) is a JSON list, or `{"labels": [...]}`. Each entry has `frame`
(the number in `index.jsonl`), `box` (pixel coordinates `[x1, y1, x2, y2]`),
and `sku_id` (a catalog SKU). These are confirmed annotations, not the server's
predictions. For the second visit, run from `recognization` with the same frozen
`stock-lopia` catalog:

```bash
SESSION="$HOME/dorayaki-data/storeA-visit2/REPLACE_WITH_SESSION"
YOLO_OFFLINE=1 .venv/bin/python code/eval_live.py "$SESSION" \
  --stock-dir stock-lopia --labels "$SESSION/labels.json" \
  --replay --device cpu
```

A recording written by the current server names `identity_backend` on each
index line, and the evaluator uses that. `--identity gallery` (or `matcher`)
is required when the index omits it, including recordings from before that
field existed, and must match the recorded value when both are present. It
does not silently score the matcher.

Replace `REPLACE_WITH_SESSION` with the actual session name. The evaluator replays
recorded JPEGs through `LiveSession`, writes `eval_live.json` in that directory,
and reports backend, loaded head, weight/catalog fingerprints, decision counts,
and held-out labelled top-1 when valid labels exist. It rejects missing or
unpinned DINOv2 weights rather than substituting matcher scores; labels copied
from catalog templates are excluded as leakage. Without confirmed held-out
visit-2 labels, counts are diagnostic only, **not gallery walk accuracy**. No
held-out cross-visit result is available here yet.