docs: API contract, README for the server and live mode, real-model test

docs/API.md is the full contract: run flags and environment, auth, coordinate convention, the identity vocabulary, every REST endpoint and WebSocket message with examples, the backpressure rule, what a client may rely on, the design decisions with their rejected alternatives, measured performance, known limitations, and notes for the future mobile client. README.md (Vietnamese) now documents the client-server shape, both modes, the flags, the API, the test commands, and the source layout. It also repairs the PowerShell escape damage in the original file (backtick-r had become a line break, backtick-b a backspace, backtick-a a bell), which had left the code fences and file paths mangled. test_real_model.py is the opt-in integration test (DORAYAKI_REAL_MODEL=1): it guards the measured baseline (170 detections, 105 recognized on image.jpg) and proves a pan keeps labels stable with no unexplained changes. run_app.sh / run_app.bat pass arguments through to the server. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-24 12:46:54 +09:00 · 2026-09-24 12:46:54 +09:00 · 4960553254
commit 4960553254
parent da2bed1c35
5 changed files with 568 additions and 77 deletions
--- a/recognization/docs/API.md
+++ b/recognization/docs/API.md
@ -0,0 +1,318 @@
+# Dorayaki recognition server API
+
+The computer-vision model (YOLO26n product detector + the SURF/SIFT, ORB, BRISK and
+Hybrid feature-matching classifier + the stock catalog) runs as a service. Capture
+clients talk to it over REST and one WebSocket. Any number of clients can connect at
+once; no client shares state with another.
+
+- Web client (served by the server itself): `http://127.0.0.1:8080/`
+- Reference non-browser client: `code/stream_client.py`
+- Interactive API docs: `http://127.0.0.1:8080/docs`
+
+## Running it
+
+```bash
+cd recognization
+uv pip install --python .venv/bin/python -r requirements-dev.txt   # first time
+.venv/bin/python code/app.py --port 8080
+```
+
+| Flag | Default | Meaning |
+| --- | --- | --- |
+| `--host` | `127.0.0.1` | Bind address. `0.0.0.0` exposes it on the LAN (set a token). |
+| `--port` | `8080` | Port. |
+| `--device` | `cpu` | `cpu`, `mps`, `cuda`, `cuda:0`. CPU is the measured default here. |
+| `--imgsz` | `640` | Detector input size. |
+| `--weights` | `runs/detect/train/weights/best.pt` | Model file. |
+| `--stock-dir` | `stock` | Catalog directory (images + `stock.csv`). |
+| `--classify-workers` | `cpu_count - 2` | Classification thread pool size. |
+| `--live-method` | `hybrid` | Default method for live sessions. |
+| `--live-conf` | `0.1` | Detection threshold in live mode (the tracker gates). |
+| `--max-sessions` | `4` | Concurrent live sessions. |
+| `--ssl-certfile` / `--ssl-keyfile` | – | HTTPS (needed for a phone camera). |
+| `--log-level` | `info` | Logging level. |
+
+Environment: `DORAYAKI_*` equivalents (`DORAYAKI_API_TOKEN`, `DORAYAKI_DEVICE`,
+`DORAYAKI_MAX_SESSIONS`, `DORAYAKI_CORS_ORIGINS`, `DORAYAKI_LIVE_INFLIGHT`, ...).
+Command-line flags win over environment.
+
+The model never phones home: the server sets `YOLO_OFFLINE=1` and
+`YOLO_AUTOINSTALL=false` before importing ultralytics, and `lap` is pinned in
+`requirements.txt` so the tracker never pip-installs anything mid-request.
+
+## Authentication
+
+When `DORAYAKI_API_TOKEN` is set, every `/v1` route except `GET /v1/health` requires
+`Authorization: Bearer <token>`, and `/v1/stream` takes `?token=<token>` (browsers
+cannot set WebSocket headers). The token is deliberately environment-only so it never
+lands in shell history, and the access log is off because it would print the query
+string. A token on a LAN bind is a speed bump, not security: prefer HTTPS.
+
+## Coordinates
+
+Boxes are `[x1, y1, x2, y2]` integers, origin top-left, in the pixel space of the
+image or frame the client sent. Responses echo
+`"coordinate_convention": "pixel-xyxy-origin-top-left"`.
+
+## Identity states
+
+One vocabulary for REST and WebSocket. **A product name is only ever attached to a
+`recognized` result** - the model's uncertainty is never presented as certainty.
+
+| `state` | Meaning | Has `sku_id`? |
+| --- | --- | --- |
+| `detected` | Product found, not classified yet (photo mode with `classify=false`) | no |
+| `unconfirmed` | Live: tracked, not enough frames to classify yet | no |
+| `identifying` | Live: classification in flight (reserved) | no |
+| `recognized` | Accepted unique catalog match | **yes** |
+| `needs_confirmation` | Ambiguous or weak match; `candidates` lists the top 3 | no |
+| `unrecognized` | Not in the catalog (or no matching evidence) | no |
+
+`identity_level` carries the raw decision: `accepted`, `ambiguous`, `unknown`,
+`rejected`.
+
+## Errors
+
+Every failure uses one shape:
+
+```json
+{"error": {"code": "IMAGE_EXPIRED", "message": "image 'img_x' is no longer available"}}
+```
+
+| Code | HTTP | When |
+| --- | --- | --- |
+| `BAD_REQUEST` | 400 | Malformed request, unknown algorithm, empty body |
+| `UNAUTHORIZED` | 401 | Missing or wrong bearer token |
+| `NOT_FOUND` | 404 | Unknown SKU, sample, or route |
+| `IMAGE_EXPIRED` | 404 | `image_id` left the still-image store (15 min / 16 images) |
+| `PAYLOAD_TOO_LARGE` | 413 | Upload > 20 MB, live frame > 4 MB |
+| `UNSUPPORTED_MEDIA_TYPE` | 415 | Body is not an image content type |
+| `UNDECODABLE_IMAGE` | 422 | Bytes are not a readable image |
+| `VALIDATION_ERROR` | 422 | Body failed validation |
+| `INTERNAL` | 500 | Server fault (details only in the server log) |
+
+## REST
+
+### `GET /v1/health` (no auth)
+
+```json
+{"status": "ok",
+ "model": {"version": "best.pt@ultralytics-8.4.153", "device": "cpu", "imgsz": 640, "warm": true},
+ "catalog": {"version": "408a658680ed", "sku_count": 21},
+ "sessions_active": 0, "uptime_s": 21.1}
+```
+
+### `GET /v1/catalog`
+
+Products with stock and an `image_url` for the template photo. Local file paths are
+never exposed.
+
+### `GET /v1/catalog/{sku_id}/image`
+
+Template image bytes. The path is resolved through the catalog map only, so a crafted
+`sku_id` cannot reach the filesystem.
+
+### `POST /v1/catalog/{sku_id}/stock`
+
+```json
+{"delta": -1}
+```
+
+`-100..100`, non-zero. Stock is in memory and resets on restart, as before.
+
+### `GET /v1/samples` and `GET /v1/samples/{sample_id}`
+
+The demo shelf images (`image.jpg`, `add/*`, the training mosaic) by id, from an
+allowlist built at startup.
+
+### `POST /v1/recognize?classify=true&method=hybrid&conf=0.25&empty_slots=true`
+
+Body: raw image bytes (`Content-Type: image/jpeg|image/png|image/webp`) or multipart
+with a `file` field.
+
+```json
+{"image_id": "img_11d072958c35bc35", "width": 2048, "height": 1365,
+ "catalog_version": "408a658680ed", "model_version": "best.pt@ultralytics-8.4.153",
+ "coordinate_convention": "pixel-xyxy-origin-top-left",
+ "detections": [{"id": 0, "box": [1435, 383, 1493, 520], "confidence": 0.8258,
+                 "identity": {"state": "recognized", "sku_id": "sku_005", "identity_level": "accepted",
+                              "confidence": 0.8948, "score": 47.71,
+                              "candidates": [{"sku_id": "sku_005", "score": 47.71}],
+                              "attempts": 1, "method": "hybrid"}}],
+ "empty_slots": [{"id": "empty_1", "box": [812, 404, 909, 549], "width": 97, "height": 145,
+                  "missing_items_est": 2, "center": [860, 476], "row_index": 3}],
+ "summary": {"in_view": 170, "recognized": 105, "needs_confirmation": 64, "unrecognized": 1,
+             "detected": 0, "empty_slots": 4, "by_sku": [{"sku_id": "sku_003", "facings": 14}]},
+ "timing_ms": {"detect": 24.77, "classify": 1498.88, "total": 1523.65}}
+```
+
+`classify=false` returns `detected` identities immediately so a client can paint boxes
+before identities arrive. The `image_id` stays valid for 15 minutes for
+`classify`/`inspect`.
+
+### `POST /v1/images/{image_id}/classify`
+
+```json
+{"method": "hybrid", "detection_ids": [0, 3]}
+```
+
+`detection_ids` optional. Returns identities for the whole image (or the selected
+detections) plus a summary. A method change re-runs it.
+
+### `POST /v1/images/{image_id}/inspect`
+
+```json
+{"box": [1435, 383, 1493, 520], "method": "hybrid"}
+```
+
+Returns the full decision (`identity`, per-method inlier `sub_details`, `product`), the
+cropped product as a data URL, and a feature-match visualization. When the result is
+not accepted, the visualization is drawn against the top candidate and
+`is_candidate` is `true`.
+
+## WebSocket `/v1/stream`
+
+Live frames in, tracked products out. One session per connection.
+
+**On connect**
+
+```json
+{"type": "hello", "protocol": 1, "session_id": "...", "catalog_version": "...",
+ "model": {"version": "...", "device": "cpu", "imgsz": 640},
+ "config": {"method": "hybrid", "conf": 0.1, "empty_slots": true},
+ "limits": {"max_frame_bytes": 4194304}}
+```
+
+**Frames (binary):** `4-byte big-endian frame id` + JPEG bytes. The id is echoed back
+so the client can pair a result with the frame it sent.
+
+**Control (text JSON)**
+
+| Client sends | Server replies |
+| --- | --- |
+| `{"type":"config","method":"orb","conf":0.15,"empty_slots":false}` | `{"type":"config_ack","config":{...}}` |
+| `{"type":"inspect","track_id":12,"method":"hybrid","request_id":"r1"}` | `{"type":"inspect_result","request_id":"r1",...}` or an error |
+| `{"type":"reset"}` | `{"type":"reset_ack"}` |
+| `{"type":"ping","t":1}` | `{"type":"pong","t":1}` |
+
+**Per processed frame**
+
+```json
+{"type": "result", "frame_id": 41, "seq": 41, "width": 1280, "height": 720,
+ "tracks": [{"track_id": 12, "box": [812, 404, 909, 549], "raw_box": [...], "confidence": 0.83,
+             "first_seen_seq": 30, "relabels": 0, "restarts": 0, "inherited": false,
+             "identity": {"state": "recognized", "sku_id": "sku_005", "confidence": 0.89,
+                          "score": 47.71, "candidates": [...], "attempts": 1, "method": "hybrid"},
+             "product": {"sku_id": "sku_005", "product_name": "...", "price": 179.0}}],
+ "empty_slots": [...], "quality": {"sharpness": 812.4, "blur": false, "glare": 0.004,
+                                   "too_far": false, "hint": null},
+ "summary": {"in_view": 61, "recognized": 8, "identifying": 0, "needs_confirmation": 1,
+             "unrecognized": 0, "detected": 22, "unconfirmed": 30, "empty_slots": 0,
+             "by_sku": [{"sku_id": "sku_005", "facings": 3}]},
+ "timing_ms": {"decode": 3.1, "detect": 21.4, "track": 0.8, "total": 25.3},
+ "stats": {"frames_received": 264, "frames_processed": 264, "frames_dropped": 0,
+           "server_fps": 13.2, "classify_pending": 3, "classify_ms_p50": 88.9}}
+```
+
+**Backpressure:** the server keeps one pending frame per session and processes the
+newest, counting the ones it replaced in `frames_dropped`. A client should keep one
+frame in flight (`stream_client.py --pace result` does exactly that); sending faster
+just drops frames and breaks the pan's continuity for the tracker.
+
+**Recoverable errors** (the socket stays open): `BAD_FRAME`, `UNDECODABLE_IMAGE`,
+`PAYLOAD_TOO_LARGE`, `BAD_MESSAGE`, `TRACK_NOT_IN_VIEW`, `INTERNAL`.
+
+**Closes:** `1008` wrong/missing token, `1013` session limit reached, `120 s` idle.
+
+## What the client can rely on
+
+- **A label is only shown for an accepted match.** Weak or ambiguous results are
+  `needs_confirmation` with candidates; unknown products are `unrecognized`.
+- **A label never changes silently.** A settled label is dropped only when the box has
+  drifted off the product for two consecutive frames, or when the track id was
+  restarted. Both increment `relabels`/`restarts`, so a client can audit every change.
+- **One product is classified once.** While the camera pans, a fresh track that covers
+  a recently recognized product inherits that label (`inherited: true`) instead of
+  queueing for classification again.
+- **Detection never waits for classification.** Frames come back at detection speed
+  (~25-45 ms on this machine); labels fill in as the classification pool finishes.
+
+## Design decisions
+
+| Decision | Why | Rejected |
+| --- | --- | --- |
+| Server lives in this repo, entry point `python code/app.py` | It owns the weights, templates, and the team's workflow | A separate service repo (split the model from its data) |
+| WebSocket carrying JPEG frames for live, REST for stills | Browsers and embedded clients both speak it; no media stack | WebRTC (heavy dependency, only wins on lossy WAN), gRPC (browsers), in-browser inference (the matcher is OpenCV Python) |
+| Latest-frame-wins with one pending slot | Lowest latency and self-throttling on any network | Queueing every frame (adds latency, never catches up) |
+| Tracking-by-detection (ultralytics ByteTrack) wrapped for per-session ids | Kalman prediction plus low-score second association is proven; ids must not be shared | No tracking (re-classify everything, far too slow), a hand-rolled tracker (no motion model) |
+| Classify once per track, aggregate up to 3 attempts, lock on accept | Keeps detection real-time and labels stable | Classifying every box every frame (measured as impossible: ~44 ms per crop x 50-170 boxes) |
+| Label inheritance by place | The same product gets several ids while panning; re-classifying it is waste | Re-classify on every id (measured 9/66 recognized vs 17-21/66 with inheritance) |
+| `hybrid` as the live default | Most accurate (21/21 SKUs on `image.jpg`) | `hybrid_fast` (ORB shortlist): 3.4x faster but only 47% agreement with `hybrid` on 553 crops, so it is not the default |
+| CPU as the default device | Measured: 26-29 ms p50 vs 112 ms on MPS, with byte-identical boxes | MPS (slower here), CUDA (no NVIDIA GPU on this machine) |
+| Identity states instead of raw scores | "Never turn model uncertainty into business certainty" (project master) | Showing a best-guess SKU with a confidence number |
+| Token via env var, access log off | Keeps credentials out of shell history and logs | A `--token` flag, default uvicorn access log (prints the query string) |
+| Legacy `/api/sessions*` kept, engine-backed | The earlier session model is still useful and was already tested | Deleting it (loses the work), leaving it on the fake adapter (silently returned nothing) |
+
+## Measured performance
+
+Apple M5, 10 cores, 24 GB, CPU device, 2026-09-24 (`code/bench_engine.py`,
+`code/stream_client.py --pace result`):
+
+| Measurement | Value |
+| --- | --- |
+| Model load / first predict | 50 ms / 985 ms (warm-up covers this at startup) |
+| Detection, 1280x853 frame | p50 28.7 ms, p95 31.0 ms (169 boxes) |
+| Detection, native 2048x1365 | p50 28.7 ms, p95 87.6 ms (170 boxes) |
+| Detection under live load | p50 43-66 ms (classification pool running) |
+| Classification per crop (hybrid) | p50 44 ms, p95 95 ms |
+| Classification throughput | 92 crops/s at 8 threads |
+| Still recognition, whole shelf | 170 detections, 105 recognized, ~2.0 s total |
+| Live results | 8-14 per second, round-trip p50 58-78 ms |
+| Time to first label on a pan | ~1.6-2.7 s (includes waiting for the product to enter) |
+
+The slide's "約28~30ms per image" matches the measured **detection** time (28.7 ms);
+the full detect-and-classify of a whole 170-product shelf is ~2 s because each product
+is matched against the catalog separately.
+
+## Known limitations
+
+- **No session totals.** Facings are counted only for products currently in view. A
+  product that leaves and re-enters gets a new track id, so cumulative counts would
+  double count.
+- **Identity thresholds are uncalibrated** closed-set defaults (`identity.py`), not
+  measured accuracy.
+- **`surf` is SIFT** in this OpenCV build (no nonfree SURF); the label stays for
+  continuity with the paper.
+- **Fast pans classify less.** A very fast pan creates more new tracks than the
+  classification pool can settle; a slower, natural scan labels nearly everything.
+- **Coverage guidance is partial**: glare/blur/too-far hints exist; "which part of the
+  fixture was scanned" does not (that needs mosaicking).
+- **Stock is in memory** and resets when the server restarts.
+- **`hybrid_fast` is experimental** and off by default.
+
+## Mobile client notes (future lane)
+
+The protocol is capture-source agnostic; an iPad/phone client needs no server changes.
+
+- WebSocket client (Ktor `WebSockets` plugin), one frame in flight.
+- Capture JPEG frames from `AVCaptureVideoDataOutput` (iOS) or CameraX
+  `ImageAnalysis` (Android), sent upright; map server boxes into the view with the
+  same object-fit transform the web client uses.
+- iOS ATS: allow local networking (`NSAllowsLocalNetworking`) or use HTTPS; Android
+  needs cleartext permission for plain `http://`.
+- The dorayaki app repo requires an ADR before adding a second API base URL, and its
+  scan screen is a timer-driven placeholder today - wiring it to this server is a
+  separate lane.
+
+## Tests
+
+```bash
+cd recognization/code
+YOLO_OFFLINE=1 ../.venv/bin/python -m unittest test_observe test_engine test_tracking test_live test_api -v
+DORAYAKI_REAL_MODEL=1 YOLO_OFFLINE=1 ../.venv/bin/python -m unittest test_real_model -v
+cd .. && YOLO_OFFLINE=1 .venv/bin/python code/test_suite.py     # the original CLI suite
+```
+
+Run tests with an explicit module list: `unittest discover` would import
+`test.py`/`fast_test.py`, which are scratch scripts with hardcoded paths from another
+machine and a blocking OpenCV window.