product-recognition-demo/recognization/docs/API.md

# Dorayaki recognition server API

The computer-vision model (YOLO26n product detector + the SURF/SIFT, ORB, BRISK and
Hybrid feature-matching classifier + the stock catalog) runs as a service. Capture
clients talk to it over REST and one WebSocket. Any number of clients can connect at
once; no client shares state with another.

- Web client (served by the server itself): `http://127.0.0.1:8080/`
- Reference non-browser client: `code/stream_client.py`
- Interactive API docs: `http://127.0.0.1:8080/docs`
- Planned extensions (price-tag OCR, reconciliation, compliance, mobile scan):
  `docs/design-price-tags-and-compliance.md`

## Running it

```bash
cd recognization
uv pip install --python .venv/bin/python -r requirements-dev.txt   # first time
.venv/bin/python code/app.py --port 8080
```

| Flag | Default | Meaning |
| --- | --- | --- |
| `--host` | `127.0.0.1` | Bind address. `0.0.0.0` exposes it on the LAN (set a token). |
| `--port` | `8080` | Port. |
| `--device` | `cpu` | `cpu`, `mps`, `cuda`, `cuda:0`. CPU is the measured default here. |
| `--imgsz` | `640` | Detector input size. |
| `--weights` | `runs/detect/train/weights/best.pt` | Model file. |
| `--stock-dir` | `stock` | Catalog directory (images + `stock.csv`). |
| `--classify-workers` | `5` | Classification thread pool size (measured default). |
| `--live-method` | `hybrid` | Default method for live sessions. |
| `--live-conf` | `0.1` | Detection threshold in live mode (the tracker gates). |
| `--no-tags` | off | Disable price-tag OCR entirely (`DORAYAKI_LIVE_TAGS=0`). |
| `DORAYAKI_OCR` | `auto` | OCR backend: `auto` (Vision on macOS), `vision`, `fake`, `none`. |
| `DORAYAKI_TORCH_THREADS` | `4` | torch CPU threads; 0 leaves torch's default. |
| `--max-sessions` | `4` | Concurrent live sessions. |
| `--ssl-certfile` / `--ssl-keyfile` | – | HTTPS (needed for a phone camera). |
| `--log-level` | `info` | Logging level. |

Environment: `DORAYAKI_*` equivalents (`DORAYAKI_API_TOKEN`, `DORAYAKI_DEVICE`,
`DORAYAKI_MAX_SESSIONS`, `DORAYAKI_CORS_ORIGINS`, `DORAYAKI_LIVE_INFLIGHT`, ...).
Command-line flags win over environment.

The model never phones home: the server sets `YOLO_OFFLINE=1` and
`YOLO_AUTOINSTALL=false` before importing ultralytics, and `lap` is pinned in
`requirements.txt` so the tracker never pip-installs anything mid-request.

## Authentication

When `DORAYAKI_API_TOKEN` is set, every `/v1` route except `GET /v1/health` requires
`Authorization: Bearer <token>`, and `/v1/stream` takes `?token=<token>` (browsers
cannot set WebSocket headers). The token is deliberately environment-only so it never
lands in shell history, and the access log is off because it would print the query
string. A token on a LAN bind is a speed bump, not security: prefer HTTPS.

## Coordinates

Boxes are `[x1, y1, x2, y2]` integers, origin top-left, in the pixel space of the
image or frame the client sent. Responses echo
`"coordinate_convention": "pixel-xyxy-origin-top-left"`.

## Identity states

One vocabulary for REST and WebSocket. **A product name is only ever attached to a
`recognized` result** - the model's uncertainty is never presented as certainty.

| `state` | Meaning | Has `sku_id`? |
| --- | --- | --- |
| `detected` | Product found, not classified yet (photo mode with `classify=false`) | no |
| `unconfirmed` | Live: tracked, not enough frames to classify yet | no |
| `identifying` | Live: classification in flight (reserved) | no |
| `recognized` | Accepted unique catalog match | **yes** |
| `needs_confirmation` | Ambiguous or weak match; `candidates` lists the top 3 | no |
| `unrecognized` | Not in the catalog (or no matching evidence) | no |

`identity_level` carries the raw decision: `accepted`, `ambiguous`, `unknown`,
`rejected`.

## Errors

Every failure uses one shape:

```json
{"error": {"code": "IMAGE_EXPIRED", "message": "image 'img_x' is no longer available"}}
```

| Code | HTTP | When |
| --- | --- | --- |
| `BAD_REQUEST` | 400 | Malformed request, unknown algorithm, empty body |
| `UNAUTHORIZED` | 401 | Missing or wrong bearer token |
| `NOT_FOUND` | 404 | Unknown SKU, sample, or route |
| `IMAGE_EXPIRED` | 404 | `image_id` left the still-image store (15 min / 16 images) |
| `PAYLOAD_TOO_LARGE` | 413 | Upload > 20 MB, live frame > 4 MB |
| `UNSUPPORTED_MEDIA_TYPE` | 415 | Body is not an image content type |
| `UNDECODABLE_IMAGE` | 422 | Bytes are not a readable image |
| `VALIDATION_ERROR` | 422 | Body failed validation |
| `INTERNAL` | 500 | Server fault (details only in the server log) |

## REST

### `GET /v1/health` (no auth)

```json
{"status": "ok",
 "model": {"version": "best.pt@ultralytics-8.4.153", "device": "cpu", "imgsz": 640, "warm": true},
 "catalog": {"version": "408a658680ed", "sku_count": 21},
 "sessions_active": 0, "uptime_s": 21.1}
```

### `GET /v1/catalog`

Products with stock and an `image_url` for the template photo. Local file paths are
never exposed.

### `GET /v1/catalog/{sku_id}/image`

Template image bytes. The path is resolved through the catalog map only, so a crafted
`sku_id` cannot reach the filesystem.

### `POST /v1/catalog/{sku_id}/stock`

```json
{"delta": -1}
```

`-100..100`, non-zero. Stock is in memory and resets on restart, as before.

### `GET /v1/samples` and `GET /v1/samples/{sample_id}`

The demo shelf images (`image.jpg`, `add/*`, the training mosaic) by id, from an
allowlist built at startup.

### `POST /v1/recognize?classify=true&method=hybrid&conf=0.25&empty_slots=true`

Body: raw image bytes (`Content-Type: image/jpeg|image/png|image/webp`) or multipart
with a `file` field.

```json
{"image_id": "img_11d072958c35bc35", "width": 2048, "height": 1365,
 "catalog_version": "408a658680ed", "model_version": "best.pt@ultralytics-8.4.153",
 "coordinate_convention": "pixel-xyxy-origin-top-left",
 "detections": [{"id": 0, "box": [1435, 383, 1493, 520], "confidence": 0.8258,
                 "identity": {"state": "recognized", "sku_id": "sku_005", "identity_level": "accepted",
                              "confidence": 0.8948, "score": 47.71,
                              "candidates": [{"sku_id": "sku_005", "score": 47.71}],
                              "attempts": 1, "method": "hybrid"}}],
 "empty_slots": [{"id": "empty_1", "box": [812, 404, 909, 549], "width": 97, "height": 145,
                  "missing_items_est": 2, "center": [860, 476], "row_index": 3}],
 "summary": {"in_view": 170, "recognized": 105, "needs_confirmation": 64, "unrecognized": 1,
             "detected": 0, "empty_slots": 4, "by_sku": [{"sku_id": "sku_003", "facings": 14}]},
 "timing_ms": {"detect": 24.77, "classify": 1498.88, "total": 1523.65}}
```

`classify=false` returns `detected` identities immediately so a client can paint boxes
before identities arrive. The `image_id` stays valid for 15 minutes for
`classify`/`inspect`.

### Price tags (OCR)

When an OCR backend is available (macOS Vision; see the design doc), a still can
carry tag evidence. `?tags=true` runs the tag pass synchronously (measured ~0.9 s
for a six-band shelf) and adds, per detection:

```json
"price_tag": {"id": "tag_3f2a", "box": [91, 657, 250, 744], "state": "readable",
              "price": 357, "regular_price": null, "discount": 0.24, "unit_price": 290.52,
              "name": "ゼロノンコレステロール", "flags": ["sale", "discounted"],
              "confidence": 0.82, "observations": 2, "settled": false},
"reconciliation": {"state": "confirmed | mismatch | unverified | no_tag | tag_only",
                   "signals": [{"kind": "price", "agrees": false, "item": 168, "tag": 269}],
                   "message": "tag says 269, catalog says 168"}
```

`price_tag.state` is `readable` (text parsed), `unreadable` (a tag was found but
its text was not read) or `absent` (no tag under this facing). The summary gains
`price_tags` counts and `compliance`:

```json
"price_tags": {"readable": 8, "unreadable": 18, "absent": 85},
"compliance": {"violations": 7, "by_rule": {"price_mismatch": 7},
               "rows": [{"row_index": 2, "rule": "price_mismatch", "box": [...], "missing": 1}],
               "hints": [{"rule": "unreadable_tags", "detail": "tags detected for 5 of 16 facings in this row"}]}
```

**Compliance fails closed on evidence quality.** A row is only judged when at
least 70% of its facings carry tag evidence (readable or unreadable); otherwise it
produces a hint such as `move closer: the price tags in this row could not be
read`, never a violation. Findings must also persist for 3 of the last 5 frames
before they publish. This is deliberate: the first implementation reported 106
missing-tag violations on a shelf where only 8 tags were readable, which was our
detection being incomplete, not the shelf.

`POST /v1/images/{image_id}/classify` re-reconciles against the cached tag bundle
and returns `reconciliation` (and `price_tag`) on each identity row, because in
photo mode the tags are read before the identities exist.

### `POST /v1/images/{image_id}/classify`

```json
{"method": "hybrid", "detection_ids": [0, 3]}
```

`detection_ids` optional. Returns identities for the whole image (or the selected
detections) plus a summary. A method change re-runs it.

### `POST /v1/images/{image_id}/inspect`

```json
{"box": [1435, 383, 1493, 520], "method": "hybrid"}
```

Returns the full decision (`identity`, per-method inlier `sub_details`, `product`), the
cropped product as a data URL, and a feature-match visualization. When the result is
not accepted, the visualization is drawn against the top candidate and
`is_candidate` is `true`.

## WebSocket `/v1/stream`

Live frames in, tracked products out. One session per connection.

**On connect**

```json
{"type": "hello", "protocol": 1, "session_id": "...", "catalog_version": "...",
 "model": {"version": "...", "device": "cpu", "imgsz": 640},
 "config": {"method": "hybrid", "conf": 0.1, "empty_slots": true},
 "limits": {"max_frame_bytes": 4194304, "max_crop_bytes": 1048576}}
```

**Frames (binary):** `4-byte big-endian frame id` + JPEG bytes. The id is echoed back
so the client can pair a result with the frame it sent.

**Control (text JSON)**

| Client sends | Server replies |
| --- | --- |
| `{"type":"config","method":"orb","conf":0.15,"empty_slots":false}` | `{"type":"config_ack","config":{...}}` |
| `{"type":"inspect","track_id":12,"method":"hybrid","request_id":"r1"}` | `{"type":"inspect_result","request_id":"r1",...}` or an error |
| `{"type":"capabilities","crop":true,"max_crop_bytes":131072}` | `{"type":"capabilities_ack","capabilities":{"crop":true,"max_crop_bytes":131072}}` |
| `{"type":"crop","request_id":"c1","track_id":12,"frame_id":41,"jpeg":"<base64 JPEG>"}` | `{"type":"crop_ack","request_id":"c1","track_id":12,"frame_id":41}` or `crop_error` |
| `{"type":"reset"}` | `{"type":"reset_ack"}` |
| `{"type":"ping","t":1}` | `{"type":"pong","t":1}` |

**Per processed frame**

```json
{"type": "result", "frame_id": 41, "seq": 41, "width": 1280, "height": 720,
 "tracks": [{"track_id": 12, "box": [812, 404, 909, 549], "raw_box": [...], "confidence": 0.83,
             "first_seen_seq": 30, "relabels": 0, "restarts": 0, "inherited": false,
             "identity": {"state": "recognized", "sku_id": "sku_005", "confidence": 0.89,
                          "score": 47.71, "candidates": [...], "attempts": 1, "method": "hybrid"},
             "product": {"sku_id": "sku_005", "product_name": "...", "price": 179.0}}],
 "empty_slots": [...], "quality": {"sharpness": 812.4, "blur": false, "glare": 0.004,
                                   "too_far": false, "hint": null},
 "summary": {"in_view": 61, "recognized": 8, "identifying": 0, "needs_confirmation": 1,
             "unrecognized": 0, "detected": 22, "unconfirmed": 30, "empty_slots": 0,
             "by_sku": [{"sku_id": "sku_005", "facings": 3}]},
 "timing_ms": {"decode": 3.1, "detect": 21.4, "track": 0.8, "total": 25.3},
 "stats": {"frames_received": 264, "frames_processed": 264, "frames_dropped": 0,
           "server_fps": 13.2, "classify_pending": 3, "classify_ms_p50": 88.9}}
```
The optional crop lane is additive to protocol `1`. Without `capabilities`, the
server sends no `crop_request` and existing clients keep their frame/control flow.
After opting in, a `result` may include `"crop_request":{"request_id":"c1",
"track_id":12,"frame_id":41,"box":[812,404,850,460]}` on an eligible
track. The box uses the **sent frame's** pixel coordinates (the raw detection),
not the camera's native resolution. Keep that exact frame's full-resolution
capture briefly, map its box into the capture, encode the crop as JPEG, and reply
in a text JSON `crop` message. Binary frames remain `4-byte id + JPEG`; the crop
does not take a frame id of its own. `crop_ack` means accepted for asynchronous
classification, not recognized; subsequent results carry the updated identity and
`identity.source` (`frame`, `crop`, or `inherited`). `stats.crops` has `requested`,
`accepted`, and `rejected` counts.

The hello limit is 1 MiB of **decoded crop bytes**. The `max_crop_bytes` in
`capabilities` states the client's limit and must be an integer in 1..1048576; a
statement that is not one is refused with `BAD_MESSAGE` and the lane stays where
it was, so both sides keep one limit (128 KiB when the key is `null`). Use the
acknowledged limit, not only hello's ceiling. At most four requests are outstanding
per session, and each track object is asked about at most `max_attempts` times
(the same budget the frame path spends, 2 by default); the budget refills when the
method changes or when a settled label drifts off the object. A request expires
after more than six processed frames; clients keep a bounded recent-frame buffer
and may miss a crop if its capture has already been evicted. A late response
cannot label a recycled track. Changing method, resetting, or dropping a drifted
label invalidates that object's outstanding requests.

Crop failures are recoverable text messages with `type: "crop_error"`, `code`,
and `message`. Pre-decode failures also echo `request_id`; session-level refusals
do not currently echo it. Codes: `CROP_BAD_MESSAGE` (missing/invalid base64 JPEG
or missing IDs), `CROP_TOO_LARGE` (empty or oversized decoded crop, excessive
encoded length, or a JPEG whose header claims more than 4 megapixels),
`CROP_UNKNOWN_REQUEST` (no outstanding request, including one already expired),
`CROP_MISMATCH` (track/frame differs), `CROP_STALE` (late or
invalidated track/session), `CROP_UNDECODABLE` (unreadable image), and
`CROP_UNAVAILABLE` (classifier shutting down). A request is single-use, including
when its response is rejected; continue streaming frames rather than retrying it.
The server may ask again for the same object while its budget lasts.

**Backpressure:** the server keeps one pending frame per session and processes the
newest, counting the ones it replaced in `frames_dropped`. A client should keep one
frame in flight (`stream_client.py --pace result` does exactly that); sending faster
just drops frames and breaks the pan's continuity for the tracker.

**Recoverable errors** (the socket stays open): `BAD_FRAME`, `UNDECODABLE_IMAGE`,
`PAYLOAD_TOO_LARGE`, `BAD_MESSAGE`, `TRACK_NOT_IN_VIEW`, `INTERNAL`.

**Closes:** `1008` wrong/missing token, `1013` session limit reached, `120 s` idle.

## What the client can rely on

- **A label is only shown for an accepted match.** Weak or ambiguous results are
  `needs_confirmation` with candidates; unknown products are `unrecognized`.
- **A label never changes silently.** A settled label is dropped only when the box has
  drifted off the product for two consecutive frames, or when the track id was
  restarted. Both increment `relabels`/`restarts`, so a client can audit every change.
- **One product is classified once.** While the camera pans, a fresh track that covers
  a recently recognized product inherits that label (`inherited: true`) instead of
  queueing for classification again.
- **Detection never waits for classification.** Frames come back at detection speed
  (~25-45 ms on this machine); labels fill in as the classification pool finishes.

## Design decisions

| Decision | Why | Rejected |
| --- | --- | --- |
| Server lives in this repo, entry point `python code/app.py` | It owns the weights, templates, and the team's workflow | A separate service repo (split the model from its data) |
| WebSocket carrying JPEG frames for live, REST for stills | Browsers and embedded clients both speak it; no media stack | WebRTC (heavy dependency, only wins on lossy WAN), gRPC (browsers), in-browser inference (the matcher is OpenCV Python) |
| Latest-frame-wins with one pending slot | Lowest latency and self-throttling on any network | Queueing every frame (adds latency, never catches up) |
| Tracking-by-detection (ultralytics ByteTrack) wrapped for per-session ids | Kalman prediction plus low-score second association is proven; ids must not be shared | No tracking (re-classify everything, far too slow), a hand-rolled tracker (no motion model) |
| Classify once per track, aggregate up to 3 attempts, lock on accept | Keeps detection real-time and labels stable | Classifying every box every frame (measured as impossible: ~44 ms per crop x 50-170 boxes) |
| Label inheritance by place | The same product gets several ids while panning; re-classifying it is waste | Re-classify on every id (measured 9/66 recognized vs 17-21/66 with inheritance) |
| `hybrid` as the live default | Most accurate (21/21 SKUs on `image.jpg`) | `hybrid_fast` (ORB shortlist): 3.4x faster but only 47% agreement with `hybrid` on 553 crops, so it is not the default |
| CPU as the default device | Measured: 26-29 ms p50 vs 112 ms on MPS, with byte-identical boxes | MPS (slower here), CUDA (no NVIDIA GPU on this machine) |
| Identity states instead of raw scores | "Never turn model uncertainty into business certainty" (project master) | Showing a best-guess SKU with a confidence number |
| Token via env var, access log off | Keeps credentials out of shell history and logs | A `--token` flag, default uvicorn access log (prints the query string) |

## Measured performance

Apple M5, 10 cores, 24 GB, CPU device, 2026-09-24 (`code/bench_engine.py`,
`code/stream_client.py --pace result`):

| Measurement | Value |
| --- | --- |
| Model load / first predict | 50 ms / 985 ms (startup warm-up covers this) |
| Detection, 1280x853 frame | p50 28.7 ms, p95 31.0 ms (169 boxes) |
| Detection, native 2048x1365 | p50 28.7 ms, p95 87.6 ms (170 boxes) |
| Detection during a live pan | p50 46-48 ms (classification pool running) |
| Classification per crop (hybrid) | p50 44 ms, p95 95 ms |
| Classification throughput | 92 crops/s at 8 threads |
| Still recognition, whole shelf | 170 detections, 105 recognized, ~2.0 s total |
| Live results, one client | 10.5-12.2 per second, round-trip p50 58-78 ms |
| Live results, two clients | 8.5 per second each, detection p50 75 ms |
| Time to first label on a pan | ~1.0-2.6 s (includes waiting for the product to enter) |

### Live accepted share after the cosine-gate fix (2026-09-28)

The live path used to re-gate the gallery's own scores at `COSINE_ACCEPT` (0.60)
and `COSINE_AMBIGUOUS` (0.50), lines above the fine-tuned head's own 0.40, so
matches the gallery had already accepted were refused again. Measured on the
same machine, store `stock-lopia`, `code/stream_client.py --source add/image6.jpg
--pan 12 --seconds 20 --pace result`:

| Gating | Tracks labelled | Final recognized / in view |
| --- | --- | --- |
| Pre-fix 0.60 / 0.50 | 16 of 261 | 2 of 30 |
| Gallery's own line | 83 of 255 | 6 of 31 |

5.2x the labelled tracks and 3x the settled facings on an identical walk. The
remaining `unconfirmed` tracks are products the gallery has no view for, not
refusals: `stock-lopia` holds 12 SKUs while the frame shows a full bay.

The live numbers use the shipped defaults: `--classify-workers 5` and torch capped at
4 threads (`DORAYAKI_TORCH_THREADS=4`). Uncapped torch and 8 workers measured slower
(11.1 results/s single-client) because detection and classification fight over the
CPU. Detection is the part that must stay real-time; classification fills in behind it.

The slide's "約28~30ms per image" matches the measured **detection** time (28.7 ms);
the full detect-and-classify of a whole 170-product shelf is ~2 s because each product
is matched against the catalog separately.

## Price-tag OCR: what it can and cannot do (measured)

- **Capture distance decides everything.** At `add/image6.jpg`'s ~1 m (tags ~300x80 px,
  digits ~22 px) macOS Vision reads a tag's printed fields reliably and the tag pass
  finds 8 readable tags with names and prices in ~0.9 s. At `image.jpg`'s ~2 m (digits
  ~14 px) it reads none of them - the pass correctly reports nothing readable.
- **Tag detection is the weak link, not the OCR.** On a shelf with ~100 facings the
  pass currently finds tags for a minority of them at this distance; the compliance
  hints say so instead of inventing violations.
- **A single read can misread a digit, and a consistent misread survives the vote.**
  The vote needs independent reads (live mode accumulates them across frames; stills
  read each band twice at different upscales) and a dead heat stays unread rather than
  guessing, but when one wrong reading wins most frames it settles as wrong - measured
  once in a 40-frame live run (`3609` for a `360` tag, 10 observations against 8). The
  safeguard is downstream: a wrong tag price shows as a `mismatch` with both values and
  is never applied to the product identity. Cross-checking the sale price against the
  regular price and discount is the recorded next improvement.
- **The demo catalog is fictional**, so tag prices legitimately disagree with it. That
  is shown as `mismatch` with both values; it is not an error.

## Known limitations

- **No session totals.** Facings are counted only for products currently in view. A
  product that leaves and re-enters gets a new track id, so cumulative counts would
  double count.
- **Identity thresholds are uncalibrated** closed-set defaults (`identity.py`), not
  measured accuracy.
- **`surf` is SIFT** in this OpenCV build (no nonfree SURF); the label stays for
  continuity with the paper.
- **Fast pans classify less.** A very fast pan creates more new tracks than the
  classification pool can settle; a slower, natural scan labels nearly everything.
- **Coverage guidance is partial**: glare/blur/too-far hints exist; "which part of the
  fixture was scanned" does not (that needs mosaicking).
- **Stock is in memory** and resets when the server restarts.
- **`hybrid_fast` is experimental** and off by default.

## Mobile client

The protocol is capture-source agnostic. The dorayaki app ships a native client
against it: `shared/.../data/shelf/LiveScanClient.kt` holds the Ktor WebSocket,
`platform/CameraFrames` supplies AVFoundation/CameraX frames, and ADR-002 in the
app repo governs the client side. What the client needs from this server:

- WebSocket client (Ktor `WebSockets` plugin), one frame in flight.
- Capture JPEG frames from `AVCaptureVideoDataOutput` (iOS) or CameraX
  `ImageAnalysis` (Android), sent upright; map server boxes into the view with the
  same object-fit transform the web client uses.
- iOS ATS: allow local networking (`NSAllowsLocalNetworking`) or use HTTPS; Android
  needs cleartext permission for plain `http://`.
- The crop lane is negotiated, not assumed: announce
  `{"type":"capabilities","crop":true,"max_crop_bytes":N}` after `hello`, wait for
  `capabilities_ack`, then answer `track.crop_request` with the exact
  `request_id`/`track_id`/`frame_id` and the crop's JPEG as base64. Silence is a
  valid answer for a frame the camera no longer holds; the request expires after
  `crop_ttl_frames`.

## Tests

```bash
cd recognization
YOLO_OFFLINE=1 YOLO_AUTOINSTALL=false python -m unittest discover -s code -p 'test_*.py' -v
```

`test_real_model` is discovered and skipped unless `DORAYAKI_REAL_MODEL=1`.
CI installs `requirements.lock` and runs the same command.