327 lines
16 KiB
Markdown
327 lines
16 KiB
Markdown
# Dorayaki recognition server API
|
||
|
||
The computer-vision model (YOLO26n product detector + the SURF/SIFT, ORB, BRISK and
|
||
Hybrid feature-matching classifier + the stock catalog) runs as a service. Capture
|
||
clients talk to it over REST and one WebSocket. Any number of clients can connect at
|
||
once; no client shares state with another.
|
||
|
||
- Web client (served by the server itself): `http://127.0.0.1:8080/`
|
||
- Reference non-browser client: `code/stream_client.py`
|
||
- Interactive API docs: `http://127.0.0.1:8080/docs`
|
||
- Planned extensions (price-tag OCR, reconciliation, compliance, mobile scan):
|
||
`docs/design-price-tags-and-compliance.md`
|
||
|
||
## Running it
|
||
|
||
```bash
|
||
cd recognization
|
||
uv pip install --python .venv/bin/python -r requirements-dev.txt # first time
|
||
.venv/bin/python code/app.py --port 8080
|
||
```
|
||
|
||
| Flag | Default | Meaning |
|
||
| --- | --- | --- |
|
||
| `--host` | `127.0.0.1` | Bind address. `0.0.0.0` exposes it on the LAN (set a token). |
|
||
| `--port` | `8080` | Port. |
|
||
| `--device` | `cpu` | `cpu`, `mps`, `cuda`, `cuda:0`. CPU is the measured default here. |
|
||
| `--imgsz` | `640` | Detector input size. |
|
||
| `--weights` | `runs/detect/train/weights/best.pt` | Model file. |
|
||
| `--stock-dir` | `stock` | Catalog directory (images + `stock.csv`). |
|
||
| `--classify-workers` | `5` | Classification thread pool size (measured default). |
|
||
| `--live-method` | `hybrid` | Default method for live sessions. |
|
||
| `--live-conf` | `0.1` | Detection threshold in live mode (the tracker gates). |
|
||
| `DORAYAKI_TORCH_THREADS` | `4` | torch CPU threads; 0 leaves torch's default. |
|
||
| `--max-sessions` | `4` | Concurrent live sessions. |
|
||
| `--ssl-certfile` / `--ssl-keyfile` | – | HTTPS (needed for a phone camera). |
|
||
| `--log-level` | `info` | Logging level. |
|
||
|
||
Environment: `DORAYAKI_*` equivalents (`DORAYAKI_API_TOKEN`, `DORAYAKI_DEVICE`,
|
||
`DORAYAKI_MAX_SESSIONS`, `DORAYAKI_CORS_ORIGINS`, `DORAYAKI_LIVE_INFLIGHT`, ...).
|
||
Command-line flags win over environment.
|
||
|
||
The model never phones home: the server sets `YOLO_OFFLINE=1` and
|
||
`YOLO_AUTOINSTALL=false` before importing ultralytics, and `lap` is pinned in
|
||
`requirements.txt` so the tracker never pip-installs anything mid-request.
|
||
|
||
## Authentication
|
||
|
||
When `DORAYAKI_API_TOKEN` is set, every `/v1` route except `GET /v1/health` requires
|
||
`Authorization: Bearer <token>`, and `/v1/stream` takes `?token=<token>` (browsers
|
||
cannot set WebSocket headers). The token is deliberately environment-only so it never
|
||
lands in shell history, and the access log is off because it would print the query
|
||
string. A token on a LAN bind is a speed bump, not security: prefer HTTPS.
|
||
|
||
## Coordinates
|
||
|
||
Boxes are `[x1, y1, x2, y2]` integers, origin top-left, in the pixel space of the
|
||
image or frame the client sent. Responses echo
|
||
`"coordinate_convention": "pixel-xyxy-origin-top-left"`.
|
||
|
||
## Identity states
|
||
|
||
One vocabulary for REST and WebSocket. **A product name is only ever attached to a
|
||
`recognized` result** - the model's uncertainty is never presented as certainty.
|
||
|
||
| `state` | Meaning | Has `sku_id`? |
|
||
| --- | --- | --- |
|
||
| `detected` | Product found, not classified yet (photo mode with `classify=false`) | no |
|
||
| `unconfirmed` | Live: tracked, not enough frames to classify yet | no |
|
||
| `identifying` | Live: classification in flight (reserved) | no |
|
||
| `recognized` | Accepted unique catalog match | **yes** |
|
||
| `needs_confirmation` | Ambiguous or weak match; `candidates` lists the top 3 | no |
|
||
| `unrecognized` | Not in the catalog (or no matching evidence) | no |
|
||
|
||
`identity_level` carries the raw decision: `accepted`, `ambiguous`, `unknown`,
|
||
`rejected`.
|
||
|
||
## Errors
|
||
|
||
Every failure uses one shape:
|
||
|
||
```json
|
||
{"error": {"code": "IMAGE_EXPIRED", "message": "image 'img_x' is no longer available"}}
|
||
```
|
||
|
||
| Code | HTTP | When |
|
||
| --- | --- | --- |
|
||
| `BAD_REQUEST` | 400 | Malformed request, unknown algorithm, empty body |
|
||
| `UNAUTHORIZED` | 401 | Missing or wrong bearer token |
|
||
| `NOT_FOUND` | 404 | Unknown SKU, sample, or route |
|
||
| `IMAGE_EXPIRED` | 404 | `image_id` left the still-image store (15 min / 16 images) |
|
||
| `PAYLOAD_TOO_LARGE` | 413 | Upload > 20 MB, live frame > 4 MB |
|
||
| `UNSUPPORTED_MEDIA_TYPE` | 415 | Body is not an image content type |
|
||
| `UNDECODABLE_IMAGE` | 422 | Bytes are not a readable image |
|
||
| `VALIDATION_ERROR` | 422 | Body failed validation |
|
||
| `INTERNAL` | 500 | Server fault (details only in the server log) |
|
||
|
||
## REST
|
||
|
||
### `GET /v1/health` (no auth)
|
||
|
||
```json
|
||
{"status": "ok",
|
||
"model": {"version": "best.pt@ultralytics-8.4.153", "device": "cpu", "imgsz": 640, "warm": true},
|
||
"catalog": {"version": "408a658680ed", "sku_count": 21},
|
||
"sessions_active": 0, "uptime_s": 21.1}
|
||
```
|
||
|
||
### `GET /v1/catalog`
|
||
|
||
Products with stock and an `image_url` for the template photo. Local file paths are
|
||
never exposed.
|
||
|
||
### `GET /v1/catalog/{sku_id}/image`
|
||
|
||
Template image bytes. The path is resolved through the catalog map only, so a crafted
|
||
`sku_id` cannot reach the filesystem.
|
||
|
||
### `POST /v1/catalog/{sku_id}/stock`
|
||
|
||
```json
|
||
{"delta": -1}
|
||
```
|
||
|
||
`-100..100`, non-zero. Stock is in memory and resets on restart, as before.
|
||
|
||
### `GET /v1/samples` and `GET /v1/samples/{sample_id}`
|
||
|
||
The demo shelf images (`image.jpg`, `add/*`, the training mosaic) by id, from an
|
||
allowlist built at startup.
|
||
|
||
### `POST /v1/recognize?classify=true&method=hybrid&conf=0.25&empty_slots=true`
|
||
|
||
Body: raw image bytes (`Content-Type: image/jpeg|image/png|image/webp`) or multipart
|
||
with a `file` field.
|
||
|
||
```json
|
||
{"image_id": "img_11d072958c35bc35", "width": 2048, "height": 1365,
|
||
"catalog_version": "408a658680ed", "model_version": "best.pt@ultralytics-8.4.153",
|
||
"coordinate_convention": "pixel-xyxy-origin-top-left",
|
||
"detections": [{"id": 0, "box": [1435, 383, 1493, 520], "confidence": 0.8258,
|
||
"identity": {"state": "recognized", "sku_id": "sku_005", "identity_level": "accepted",
|
||
"confidence": 0.8948, "score": 47.71,
|
||
"candidates": [{"sku_id": "sku_005", "score": 47.71}],
|
||
"attempts": 1, "method": "hybrid"}}],
|
||
"empty_slots": [{"id": "empty_1", "box": [812, 404, 909, 549], "width": 97, "height": 145,
|
||
"missing_items_est": 2, "center": [860, 476], "row_index": 3}],
|
||
"summary": {"in_view": 170, "recognized": 105, "needs_confirmation": 64, "unrecognized": 1,
|
||
"detected": 0, "empty_slots": 4, "by_sku": [{"sku_id": "sku_003", "facings": 14}]},
|
||
"timing_ms": {"detect": 24.77, "classify": 1498.88, "total": 1523.65}}
|
||
```
|
||
|
||
`classify=false` returns `detected` identities immediately so a client can paint boxes
|
||
before identities arrive. The `image_id` stays valid for 15 minutes for
|
||
`classify`/`inspect`.
|
||
|
||
### `POST /v1/images/{image_id}/classify`
|
||
|
||
```json
|
||
{"method": "hybrid", "detection_ids": [0, 3]}
|
||
```
|
||
|
||
`detection_ids` optional. Returns identities for the whole image (or the selected
|
||
detections) plus a summary. A method change re-runs it.
|
||
|
||
### `POST /v1/images/{image_id}/inspect`
|
||
|
||
```json
|
||
{"box": [1435, 383, 1493, 520], "method": "hybrid"}
|
||
```
|
||
|
||
Returns the full decision (`identity`, per-method inlier `sub_details`, `product`), the
|
||
cropped product as a data URL, and a feature-match visualization. When the result is
|
||
not accepted, the visualization is drawn against the top candidate and
|
||
`is_candidate` is `true`.
|
||
|
||
## WebSocket `/v1/stream`
|
||
|
||
Live frames in, tracked products out. One session per connection.
|
||
|
||
**On connect**
|
||
|
||
```json
|
||
{"type": "hello", "protocol": 1, "session_id": "...", "catalog_version": "...",
|
||
"model": {"version": "...", "device": "cpu", "imgsz": 640},
|
||
"config": {"method": "hybrid", "conf": 0.1, "empty_slots": true},
|
||
"limits": {"max_frame_bytes": 4194304}}
|
||
```
|
||
|
||
**Frames (binary):** `4-byte big-endian frame id` + JPEG bytes. The id is echoed back
|
||
so the client can pair a result with the frame it sent.
|
||
|
||
**Control (text JSON)**
|
||
|
||
| Client sends | Server replies |
|
||
| --- | --- |
|
||
| `{"type":"config","method":"orb","conf":0.15,"empty_slots":false}` | `{"type":"config_ack","config":{...}}` |
|
||
| `{"type":"inspect","track_id":12,"method":"hybrid","request_id":"r1"}` | `{"type":"inspect_result","request_id":"r1",...}` or an error |
|
||
| `{"type":"reset"}` | `{"type":"reset_ack"}` |
|
||
| `{"type":"ping","t":1}` | `{"type":"pong","t":1}` |
|
||
|
||
**Per processed frame**
|
||
|
||
```json
|
||
{"type": "result", "frame_id": 41, "seq": 41, "width": 1280, "height": 720,
|
||
"tracks": [{"track_id": 12, "box": [812, 404, 909, 549], "raw_box": [...], "confidence": 0.83,
|
||
"first_seen_seq": 30, "relabels": 0, "restarts": 0, "inherited": false,
|
||
"identity": {"state": "recognized", "sku_id": "sku_005", "confidence": 0.89,
|
||
"score": 47.71, "candidates": [...], "attempts": 1, "method": "hybrid"},
|
||
"product": {"sku_id": "sku_005", "product_name": "...", "price": 179.0}}],
|
||
"empty_slots": [...], "quality": {"sharpness": 812.4, "blur": false, "glare": 0.004,
|
||
"too_far": false, "hint": null},
|
||
"summary": {"in_view": 61, "recognized": 8, "identifying": 0, "needs_confirmation": 1,
|
||
"unrecognized": 0, "detected": 22, "unconfirmed": 30, "empty_slots": 0,
|
||
"by_sku": [{"sku_id": "sku_005", "facings": 3}]},
|
||
"timing_ms": {"decode": 3.1, "detect": 21.4, "track": 0.8, "total": 25.3},
|
||
"stats": {"frames_received": 264, "frames_processed": 264, "frames_dropped": 0,
|
||
"server_fps": 13.2, "classify_pending": 3, "classify_ms_p50": 88.9}}
|
||
```
|
||
|
||
**Backpressure:** the server keeps one pending frame per session and processes the
|
||
newest, counting the ones it replaced in `frames_dropped`. A client should keep one
|
||
frame in flight (`stream_client.py --pace result` does exactly that); sending faster
|
||
just drops frames and breaks the pan's continuity for the tracker.
|
||
|
||
**Recoverable errors** (the socket stays open): `BAD_FRAME`, `UNDECODABLE_IMAGE`,
|
||
`PAYLOAD_TOO_LARGE`, `BAD_MESSAGE`, `TRACK_NOT_IN_VIEW`, `INTERNAL`.
|
||
|
||
**Closes:** `1008` wrong/missing token, `1013` session limit reached, `120 s` idle.
|
||
|
||
## What the client can rely on
|
||
|
||
- **A label is only shown for an accepted match.** Weak or ambiguous results are
|
||
`needs_confirmation` with candidates; unknown products are `unrecognized`.
|
||
- **A label never changes silently.** A settled label is dropped only when the box has
|
||
drifted off the product for two consecutive frames, or when the track id was
|
||
restarted. Both increment `relabels`/`restarts`, so a client can audit every change.
|
||
- **One product is classified once.** While the camera pans, a fresh track that covers
|
||
a recently recognized product inherits that label (`inherited: true`) instead of
|
||
queueing for classification again.
|
||
- **Detection never waits for classification.** Frames come back at detection speed
|
||
(~25-45 ms on this machine); labels fill in as the classification pool finishes.
|
||
|
||
## Design decisions
|
||
|
||
| Decision | Why | Rejected |
|
||
| --- | --- | --- |
|
||
| Server lives in this repo, entry point `python code/app.py` | It owns the weights, templates, and the team's workflow | A separate service repo (split the model from its data) |
|
||
| WebSocket carrying JPEG frames for live, REST for stills | Browsers and embedded clients both speak it; no media stack | WebRTC (heavy dependency, only wins on lossy WAN), gRPC (browsers), in-browser inference (the matcher is OpenCV Python) |
|
||
| Latest-frame-wins with one pending slot | Lowest latency and self-throttling on any network | Queueing every frame (adds latency, never catches up) |
|
||
| Tracking-by-detection (ultralytics ByteTrack) wrapped for per-session ids | Kalman prediction plus low-score second association is proven; ids must not be shared | No tracking (re-classify everything, far too slow), a hand-rolled tracker (no motion model) |
|
||
| Classify once per track, aggregate up to 3 attempts, lock on accept | Keeps detection real-time and labels stable | Classifying every box every frame (measured as impossible: ~44 ms per crop x 50-170 boxes) |
|
||
| Label inheritance by place | The same product gets several ids while panning; re-classifying it is waste | Re-classify on every id (measured 9/66 recognized vs 17-21/66 with inheritance) |
|
||
| `hybrid` as the live default | Most accurate (21/21 SKUs on `image.jpg`) | `hybrid_fast` (ORB shortlist): 3.4x faster but only 47% agreement with `hybrid` on 553 crops, so it is not the default |
|
||
| CPU as the default device | Measured: 26-29 ms p50 vs 112 ms on MPS, with byte-identical boxes | MPS (slower here), CUDA (no NVIDIA GPU on this machine) |
|
||
| Identity states instead of raw scores | "Never turn model uncertainty into business certainty" (project master) | Showing a best-guess SKU with a confidence number |
|
||
| Token via env var, access log off | Keeps credentials out of shell history and logs | A `--token` flag, default uvicorn access log (prints the query string) |
|
||
| Legacy `/api/sessions*` kept, engine-backed | The earlier session model is still useful and was already tested | Deleting it (loses the work), leaving it on the fake adapter (silently returned nothing) |
|
||
|
||
## Measured performance
|
||
|
||
Apple M5, 10 cores, 24 GB, CPU device, 2026-09-24 (`code/bench_engine.py`,
|
||
`code/stream_client.py --pace result`):
|
||
|
||
| Measurement | Value |
|
||
| --- | --- |
|
||
| Model load / first predict | 50 ms / 985 ms (startup warm-up covers this) |
|
||
| Detection, 1280x853 frame | p50 28.7 ms, p95 31.0 ms (169 boxes) |
|
||
| Detection, native 2048x1365 | p50 28.7 ms, p95 87.6 ms (170 boxes) |
|
||
| Detection during a live pan | p50 46-48 ms (classification pool running) |
|
||
| Classification per crop (hybrid) | p50 44 ms, p95 95 ms |
|
||
| Classification throughput | 92 crops/s at 8 threads |
|
||
| Still recognition, whole shelf | 170 detections, 105 recognized, ~2.0 s total |
|
||
| Live results, one client | 10.5-12.2 per second, round-trip p50 58-78 ms |
|
||
| Live results, two clients | 8.5 per second each, detection p50 75 ms |
|
||
| Time to first label on a pan | ~1.0-2.6 s (includes waiting for the product to enter) |
|
||
|
||
The live numbers use the shipped defaults: `--classify-workers 5` and torch capped at
|
||
4 threads (`DORAYAKI_TORCH_THREADS=4`). Uncapped torch and 8 workers measured slower
|
||
(11.1 results/s single-client) because detection and classification fight over the
|
||
CPU. Detection is the part that must stay real-time; classification fills in behind it.
|
||
|
||
The slide's "約28~30ms per image" matches the measured **detection** time (28.7 ms);
|
||
the full detect-and-classify of a whole 170-product shelf is ~2 s because each product
|
||
is matched against the catalog separately.
|
||
|
||
## Known limitations
|
||
|
||
- **No session totals.** Facings are counted only for products currently in view. A
|
||
product that leaves and re-enters gets a new track id, so cumulative counts would
|
||
double count.
|
||
- **Identity thresholds are uncalibrated** closed-set defaults (`identity.py`), not
|
||
measured accuracy.
|
||
- **`surf` is SIFT** in this OpenCV build (no nonfree SURF); the label stays for
|
||
continuity with the paper.
|
||
- **Fast pans classify less.** A very fast pan creates more new tracks than the
|
||
classification pool can settle; a slower, natural scan labels nearly everything.
|
||
- **Coverage guidance is partial**: glare/blur/too-far hints exist; "which part of the
|
||
fixture was scanned" does not (that needs mosaicking).
|
||
- **Stock is in memory** and resets when the server restarts.
|
||
- **`hybrid_fast` is experimental** and off by default.
|
||
|
||
## Mobile client notes (future lane)
|
||
|
||
The protocol is capture-source agnostic; an iPad/phone client needs no server changes.
|
||
|
||
- WebSocket client (Ktor `WebSockets` plugin), one frame in flight.
|
||
- Capture JPEG frames from `AVCaptureVideoDataOutput` (iOS) or CameraX
|
||
`ImageAnalysis` (Android), sent upright; map server boxes into the view with the
|
||
same object-fit transform the web client uses.
|
||
- iOS ATS: allow local networking (`NSAllowsLocalNetworking`) or use HTTPS; Android
|
||
needs cleartext permission for plain `http://`.
|
||
- The dorayaki app repo requires an ADR before adding a second API base URL, and its
|
||
scan screen is a timer-driven placeholder today - wiring it to this server is a
|
||
separate lane.
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
cd recognization/code
|
||
YOLO_OFFLINE=1 ../.venv/bin/python -m unittest test_observe test_engine test_tracking test_live test_api -v
|
||
DORAYAKI_REAL_MODEL=1 YOLO_OFFLINE=1 ../.venv/bin/python -m unittest test_real_model -v
|
||
cd .. && YOLO_OFFLINE=1 .venv/bin/python code/test_suite.py # the original CLI suite
|
||
```
|
||
|
||
Run tests with an explicit module list: `unittest discover` would import
|
||
`test.py`/`fast_test.py`, which are scratch scripts with hardcoded paths from another
|
||
machine and a blocking OpenCV window.
|