# Dorayaki recognition server API The computer-vision model (YOLO26n product detector + the SURF/SIFT, ORB, BRISK and Hybrid feature-matching classifier + the stock catalog) runs as a service. Capture clients talk to it over REST and one WebSocket. Any number of clients can connect at once; no client shares state with another. - Web client (served by the server itself): `http://127.0.0.1:8080/` - Reference non-browser client: `code/stream_client.py` - Interactive API docs: `http://127.0.0.1:8080/docs` - Planned extensions (price-tag OCR, reconciliation, compliance, mobile scan): `docs/design-price-tags-and-compliance.md` ## Running it ```bash cd recognization uv pip install --python .venv/bin/python -r requirements-dev.txt # first time .venv/bin/python code/app.py --port 8080 ``` | Flag | Default | Meaning | | --- | --- | --- | | `--host` | `127.0.0.1` | Bind address. `0.0.0.0` exposes it on the LAN (set a token). | | `--port` | `8080` | Port. | | `--device` | `cpu` | `cpu`, `mps`, `cuda`, `cuda:0`. CPU is the measured default here. | | `--imgsz` | `640` | Detector input size. | | `--weights` | `runs/detect/train/weights/best.pt` | Model file. | | `--stock-dir` | `stock` | Catalog directory (images + `stock.csv`). | | `--classify-workers` | `5` | Classification thread pool size (measured default). | | `--live-method` | `hybrid` | Default method for live sessions. | | `--live-conf` | `0.1` | Detection threshold in live mode (the tracker gates). | | `--no-tags` | off | Disable price-tag OCR entirely (`DORAYAKI_LIVE_TAGS=0`). | | `DORAYAKI_OCR` | `auto` | OCR backend: `auto` (Vision on macOS), `vision`, `fake`, `none`. | | `DORAYAKI_TORCH_THREADS` | `4` | torch CPU threads; 0 leaves torch's default. | | `--max-sessions` | `4` | Concurrent live sessions. | | `--ssl-certfile` / `--ssl-keyfile` | – | HTTPS (needed for a phone camera). | | `--log-level` | `info` | Logging level. | Environment: `DORAYAKI_*` equivalents (`DORAYAKI_API_TOKEN`, `DORAYAKI_DEVICE`, `DORAYAKI_MAX_SESSIONS`, `DORAYAKI_CORS_ORIGINS`, `DORAYAKI_LIVE_INFLIGHT`, ...). Command-line flags win over environment. The model never phones home: the server sets `YOLO_OFFLINE=1` and `YOLO_AUTOINSTALL=false` before importing ultralytics, and `lap` is pinned in `requirements.txt` so the tracker never pip-installs anything mid-request. ## Authentication When `DORAYAKI_API_TOKEN` is set, every `/v1` route except `GET /v1/health` requires `Authorization: Bearer `, and `/v1/stream` takes `?token=` (browsers cannot set WebSocket headers). The token is deliberately environment-only so it never lands in shell history, and the access log is off because it would print the query string. A token on a LAN bind is a speed bump, not security: prefer HTTPS. ## Coordinates Boxes are `[x1, y1, x2, y2]` integers, origin top-left, in the pixel space of the image or frame the client sent. Responses echo `"coordinate_convention": "pixel-xyxy-origin-top-left"`. ## Identity states One vocabulary for REST and WebSocket. **A product name is only ever attached to a `recognized` result** - the model's uncertainty is never presented as certainty. | `state` | Meaning | Has `sku_id`? | | --- | --- | --- | | `detected` | Product found, not classified yet (photo mode with `classify=false`) | no | | `unconfirmed` | Live: tracked, not enough frames to classify yet | no | | `identifying` | Live: classification in flight (reserved) | no | | `recognized` | Accepted unique catalog match | **yes** | | `needs_confirmation` | Ambiguous or weak match; `candidates` lists the top 3 | no | | `unrecognized` | Not in the catalog (or no matching evidence) | no | `identity_level` carries the raw decision: `accepted`, `ambiguous`, `unknown`, `rejected`. ## Errors Every failure uses one shape: ```json {"error": {"code": "IMAGE_EXPIRED", "message": "image 'img_x' is no longer available"}} ``` | Code | HTTP | When | | --- | --- | --- | | `BAD_REQUEST` | 400 | Malformed request, unknown algorithm, empty body | | `UNAUTHORIZED` | 401 | Missing or wrong bearer token | | `NOT_FOUND` | 404 | Unknown SKU, sample, or route | | `IMAGE_EXPIRED` | 404 | `image_id` left the still-image store (15 min / 16 images) | | `PAYLOAD_TOO_LARGE` | 413 | Upload > 20 MB, live frame > 4 MB | | `UNSUPPORTED_MEDIA_TYPE` | 415 | Body is not an image content type | | `UNDECODABLE_IMAGE` | 422 | Bytes are not a readable image | | `VALIDATION_ERROR` | 422 | Body failed validation | | `INTERNAL` | 500 | Server fault (details only in the server log) | ## REST ### `GET /v1/health` (no auth) ```json {"status": "ok", "model": {"version": "best.pt@ultralytics-8.4.153", "device": "cpu", "imgsz": 640, "warm": true}, "catalog": {"version": "408a658680ed", "sku_count": 21}, "sessions_active": 0, "uptime_s": 21.1} ``` ### `GET /v1/catalog` Products with stock and an `image_url` for the template photo. Local file paths are never exposed. ### `GET /v1/catalog/{sku_id}/image` Template image bytes. The path is resolved through the catalog map only, so a crafted `sku_id` cannot reach the filesystem. ### `POST /v1/catalog/{sku_id}/stock` ```json {"delta": -1} ``` `-100..100`, non-zero. Stock is in memory and resets on restart, as before. ### `GET /v1/samples` and `GET /v1/samples/{sample_id}` The demo shelf images (`image.jpg`, `add/*`, the training mosaic) by id, from an allowlist built at startup. ### `POST /v1/recognize?classify=true&method=hybrid&conf=0.25&empty_slots=true` Body: raw image bytes (`Content-Type: image/jpeg|image/png|image/webp`) or multipart with a `file` field. ```json {"image_id": "img_11d072958c35bc35", "width": 2048, "height": 1365, "catalog_version": "408a658680ed", "model_version": "best.pt@ultralytics-8.4.153", "coordinate_convention": "pixel-xyxy-origin-top-left", "detections": [{"id": 0, "box": [1435, 383, 1493, 520], "confidence": 0.8258, "identity": {"state": "recognized", "sku_id": "sku_005", "identity_level": "accepted", "confidence": 0.8948, "score": 47.71, "candidates": [{"sku_id": "sku_005", "score": 47.71}], "attempts": 1, "method": "hybrid"}}], "empty_slots": [{"id": "empty_1", "box": [812, 404, 909, 549], "width": 97, "height": 145, "missing_items_est": 2, "center": [860, 476], "row_index": 3}], "summary": {"in_view": 170, "recognized": 105, "needs_confirmation": 64, "unrecognized": 1, "detected": 0, "empty_slots": 4, "by_sku": [{"sku_id": "sku_003", "facings": 14}]}, "timing_ms": {"detect": 24.77, "classify": 1498.88, "total": 1523.65}} ``` `classify=false` returns `detected` identities immediately so a client can paint boxes before identities arrive. The `image_id` stays valid for 15 minutes for `classify`/`inspect`. ### Price tags (OCR) When an OCR backend is available (macOS Vision; see the design doc), a still can carry tag evidence. `?tags=true` runs the tag pass synchronously (measured ~0.9 s for a six-band shelf) and adds, per detection: ```json "price_tag": {"id": "tag_3f2a", "box": [91, 657, 250, 744], "state": "readable", "price": 357, "regular_price": null, "discount": 0.24, "unit_price": 290.52, "name": "ゼロノンコレステロール", "flags": ["sale", "discounted"], "confidence": 0.82, "observations": 2, "settled": false}, "reconciliation": {"state": "confirmed | mismatch | unverified | no_tag | tag_only", "signals": [{"kind": "price", "agrees": false, "item": 168, "tag": 269}], "message": "tag says 269, catalog says 168"} ``` `price_tag.state` is `readable` (text parsed), `unreadable` (a tag was found but its text was not read) or `absent` (no tag under this facing). The summary gains `price_tags` counts and `compliance`: ```json "price_tags": {"readable": 8, "unreadable": 18, "absent": 85}, "compliance": {"violations": 7, "by_rule": {"price_mismatch": 7}, "rows": [{"row_index": 2, "rule": "price_mismatch", "box": [...], "missing": 1}], "hints": [{"rule": "unreadable_tags", "detail": "tags detected for 5 of 16 facings in this row"}]} ``` **Compliance fails closed on evidence quality.** A row is only judged when at least 70% of its facings carry tag evidence (readable or unreadable); otherwise it produces a hint such as `move closer: the price tags in this row could not be read`, never a violation. Findings must also persist for 3 of the last 5 frames before they publish. This is deliberate: the first implementation reported 106 missing-tag violations on a shelf where only 8 tags were readable, which was our detection being incomplete, not the shelf. `POST /v1/images/{image_id}/classify` re-reconciles against the cached tag bundle and returns `reconciliation` (and `price_tag`) on each identity row, because in photo mode the tags are read before the identities exist. ### `POST /v1/images/{image_id}/classify` ```json {"method": "hybrid", "detection_ids": [0, 3]} ``` `detection_ids` optional. Returns identities for the whole image (or the selected detections) plus a summary. A method change re-runs it. ### `POST /v1/images/{image_id}/inspect` ```json {"box": [1435, 383, 1493, 520], "method": "hybrid"} ``` Returns the full decision (`identity`, per-method inlier `sub_details`, `product`), the cropped product as a data URL, and a feature-match visualization. When the result is not accepted, the visualization is drawn against the top candidate and `is_candidate` is `true`. ## WebSocket `/v1/stream` Live frames in, tracked products out. One session per connection. **On connect** ```json {"type": "hello", "protocol": 1, "session_id": "...", "catalog_version": "...", "model": {"version": "...", "device": "cpu", "imgsz": 640}, "config": {"method": "hybrid", "conf": 0.1, "empty_slots": true}, "limits": {"max_frame_bytes": 4194304}} ``` **Frames (binary):** `4-byte big-endian frame id` + JPEG bytes. The id is echoed back so the client can pair a result with the frame it sent. **Control (text JSON)** | Client sends | Server replies | | --- | --- | | `{"type":"config","method":"orb","conf":0.15,"empty_slots":false}` | `{"type":"config_ack","config":{...}}` | | `{"type":"inspect","track_id":12,"method":"hybrid","request_id":"r1"}` | `{"type":"inspect_result","request_id":"r1",...}` or an error | | `{"type":"reset"}` | `{"type":"reset_ack"}` | | `{"type":"ping","t":1}` | `{"type":"pong","t":1}` | **Per processed frame** ```json {"type": "result", "frame_id": 41, "seq": 41, "width": 1280, "height": 720, "tracks": [{"track_id": 12, "box": [812, 404, 909, 549], "raw_box": [...], "confidence": 0.83, "first_seen_seq": 30, "relabels": 0, "restarts": 0, "inherited": false, "identity": {"state": "recognized", "sku_id": "sku_005", "confidence": 0.89, "score": 47.71, "candidates": [...], "attempts": 1, "method": "hybrid"}, "product": {"sku_id": "sku_005", "product_name": "...", "price": 179.0}}], "empty_slots": [...], "quality": {"sharpness": 812.4, "blur": false, "glare": 0.004, "too_far": false, "hint": null}, "summary": {"in_view": 61, "recognized": 8, "identifying": 0, "needs_confirmation": 1, "unrecognized": 0, "detected": 22, "unconfirmed": 30, "empty_slots": 0, "by_sku": [{"sku_id": "sku_005", "facings": 3}]}, "timing_ms": {"decode": 3.1, "detect": 21.4, "track": 0.8, "total": 25.3}, "stats": {"frames_received": 264, "frames_processed": 264, "frames_dropped": 0, "server_fps": 13.2, "classify_pending": 3, "classify_ms_p50": 88.9}} ``` **Backpressure:** the server keeps one pending frame per session and processes the newest, counting the ones it replaced in `frames_dropped`. A client should keep one frame in flight (`stream_client.py --pace result` does exactly that); sending faster just drops frames and breaks the pan's continuity for the tracker. **Recoverable errors** (the socket stays open): `BAD_FRAME`, `UNDECODABLE_IMAGE`, `PAYLOAD_TOO_LARGE`, `BAD_MESSAGE`, `TRACK_NOT_IN_VIEW`, `INTERNAL`. **Closes:** `1008` wrong/missing token, `1013` session limit reached, `120 s` idle. ## What the client can rely on - **A label is only shown for an accepted match.** Weak or ambiguous results are `needs_confirmation` with candidates; unknown products are `unrecognized`. - **A label never changes silently.** A settled label is dropped only when the box has drifted off the product for two consecutive frames, or when the track id was restarted. Both increment `relabels`/`restarts`, so a client can audit every change. - **One product is classified once.** While the camera pans, a fresh track that covers a recently recognized product inherits that label (`inherited: true`) instead of queueing for classification again. - **Detection never waits for classification.** Frames come back at detection speed (~25-45 ms on this machine); labels fill in as the classification pool finishes. ## Design decisions | Decision | Why | Rejected | | --- | --- | --- | | Server lives in this repo, entry point `python code/app.py` | It owns the weights, templates, and the team's workflow | A separate service repo (split the model from its data) | | WebSocket carrying JPEG frames for live, REST for stills | Browsers and embedded clients both speak it; no media stack | WebRTC (heavy dependency, only wins on lossy WAN), gRPC (browsers), in-browser inference (the matcher is OpenCV Python) | | Latest-frame-wins with one pending slot | Lowest latency and self-throttling on any network | Queueing every frame (adds latency, never catches up) | | Tracking-by-detection (ultralytics ByteTrack) wrapped for per-session ids | Kalman prediction plus low-score second association is proven; ids must not be shared | No tracking (re-classify everything, far too slow), a hand-rolled tracker (no motion model) | | Classify once per track, aggregate up to 3 attempts, lock on accept | Keeps detection real-time and labels stable | Classifying every box every frame (measured as impossible: ~44 ms per crop x 50-170 boxes) | | Label inheritance by place | The same product gets several ids while panning; re-classifying it is waste | Re-classify on every id (measured 9/66 recognized vs 17-21/66 with inheritance) | | `hybrid` as the live default | Most accurate (21/21 SKUs on `image.jpg`) | `hybrid_fast` (ORB shortlist): 3.4x faster but only 47% agreement with `hybrid` on 553 crops, so it is not the default | | CPU as the default device | Measured: 26-29 ms p50 vs 112 ms on MPS, with byte-identical boxes | MPS (slower here), CUDA (no NVIDIA GPU on this machine) | | Identity states instead of raw scores | "Never turn model uncertainty into business certainty" (project master) | Showing a best-guess SKU with a confidence number | | Token via env var, access log off | Keeps credentials out of shell history and logs | A `--token` flag, default uvicorn access log (prints the query string) | | Legacy `/api/sessions*` kept, engine-backed | The earlier session model is still useful and was already tested | Deleting it (loses the work), leaving it on the fake adapter (silently returned nothing) | ## Measured performance Apple M5, 10 cores, 24 GB, CPU device, 2026-09-24 (`code/bench_engine.py`, `code/stream_client.py --pace result`): | Measurement | Value | | --- | --- | | Model load / first predict | 50 ms / 985 ms (startup warm-up covers this) | | Detection, 1280x853 frame | p50 28.7 ms, p95 31.0 ms (169 boxes) | | Detection, native 2048x1365 | p50 28.7 ms, p95 87.6 ms (170 boxes) | | Detection during a live pan | p50 46-48 ms (classification pool running) | | Classification per crop (hybrid) | p50 44 ms, p95 95 ms | | Classification throughput | 92 crops/s at 8 threads | | Still recognition, whole shelf | 170 detections, 105 recognized, ~2.0 s total | | Live results, one client | 10.5-12.2 per second, round-trip p50 58-78 ms | | Live results, two clients | 8.5 per second each, detection p50 75 ms | | Time to first label on a pan | ~1.0-2.6 s (includes waiting for the product to enter) | The live numbers use the shipped defaults: `--classify-workers 5` and torch capped at 4 threads (`DORAYAKI_TORCH_THREADS=4`). Uncapped torch and 8 workers measured slower (11.1 results/s single-client) because detection and classification fight over the CPU. Detection is the part that must stay real-time; classification fills in behind it. The slide's "約28~30ms per image" matches the measured **detection** time (28.7 ms); the full detect-and-classify of a whole 170-product shelf is ~2 s because each product is matched against the catalog separately. ## Price-tag OCR: what it can and cannot do (measured) - **Capture distance decides everything.** At `add/image6.jpg`'s ~1 m (tags ~300x80 px, digits ~22 px) macOS Vision reads a tag's printed fields reliably and the tag pass finds 8 readable tags with names and prices in ~0.9 s. At `image.jpg`'s ~2 m (digits ~14 px) it reads none of them - the pass correctly reports nothing readable. - **Tag detection is the weak link, not the OCR.** On a shelf with ~100 facings the pass currently finds tags for a minority of them at this distance; the compliance hints say so instead of inventing violations. - **A single read can misread a digit, and a consistent misread survives the vote.** The vote needs independent reads (live mode accumulates them across frames; stills read each band twice at different upscales) and a dead heat stays unread rather than guessing, but when one wrong reading wins most frames it settles as wrong - measured once in a 40-frame live run (`3609` for a `360` tag, 10 observations against 8). The safeguard is downstream: a wrong tag price shows as a `mismatch` with both values and is never applied to the product identity. Cross-checking the sale price against the regular price and discount is the recorded next improvement. - **The demo catalog is fictional**, so tag prices legitimately disagree with it. That is shown as `mismatch` with both values; it is not an error. ## Known limitations - **No session totals.** Facings are counted only for products currently in view. A product that leaves and re-enters gets a new track id, so cumulative counts would double count. - **Identity thresholds are uncalibrated** closed-set defaults (`identity.py`), not measured accuracy. - **`surf` is SIFT** in this OpenCV build (no nonfree SURF); the label stays for continuity with the paper. - **Fast pans classify less.** A very fast pan creates more new tracks than the classification pool can settle; a slower, natural scan labels nearly everything. - **Coverage guidance is partial**: glare/blur/too-far hints exist; "which part of the fixture was scanned" does not (that needs mosaicking). - **Stock is in memory** and resets when the server restarts. - **`hybrid_fast` is experimental** and off by default. ## Mobile client notes (future lane) The protocol is capture-source agnostic; an iPad/phone client needs no server changes. - WebSocket client (Ktor `WebSockets` plugin), one frame in flight. - Capture JPEG frames from `AVCaptureVideoDataOutput` (iOS) or CameraX `ImageAnalysis` (Android), sent upright; map server boxes into the view with the same object-fit transform the web client uses. - iOS ATS: allow local networking (`NSAllowsLocalNetworking`) or use HTTPS; Android needs cleartext permission for plain `http://`. - The dorayaki app repo requires an ADR before adding a second API base URL, and its scan screen is a timer-driven placeholder today - wiring it to this server is a separate lane. ## Tests ```bash cd recognization YOLO_OFFLINE=1 YOLO_AUTOINSTALL=false python -m unittest discover -s code -p 'test_*.py' -v ``` `test_real_model` is discovered and skipped unless `DORAYAKI_REAL_MODEL=1`. CI installs `requirements.lock` and runs the same command.