The external evaluation showed the matcher cannot say "unknown": on 20 shelves whose
products are absent it claimed 47.8% of facings as recognised, and its raw inlier
score cannot be thresholded (foreign shelves' median 6.0 overlaps the real shelf's
5.2 p10, and the worst offenders score 50-190). So the gallery becomes a selectable
identity backend:
--identity gallery (or DORAYAKI_IDENTITY=gallery)
Measured on the same 20 foreign shelves: false accepts fall from 47.8% to 13.8% at
the default threshold and to **5%** at 0.60, with a smooth curve to pick an operating
point from. Detection is unchanged (the gallery only names boxes).
What it does NOT fix, measured on a milk shelf where 14 of 107 facings really are
明治 おいしい牛乳 (labelled by eye): precision is 28-45%, no better than the matcher's
39%, and enrolling the shelf from itself (1-3 views per product) is flat at ~25%
top-1. Off-the-shelf DINOv2 is not discriminative enough for fine-grained SKUs - the
same conclusion the literature reaches, now measured here.
The packaging's own text is a real channel and resolution-bound: 51% recall
single-scale, 60% with the tag reader's multi-scale + red-channel trick, 85% where
the text is large (MEGMILK) and 28% where it is small vertical Japanese.
docs/EXTERNAL-EVAL.md records all of it, including the identity architecture it
implies: reject by default, name from the tag when readable, from the packaging text
when legible, from a fine-tuned embedding otherwise, and hand the rest to the
operator.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The deep review's first recommendation was real data, because every accuracy number
so far comes from one or two photographs - the identity benchmark even takes its
gallery and its queries from the same photo. This is the mechanism for replacing
that.
`--record-frames DIR` writes each processed frame, sampled every Nth, with a JSON
line describing what the server saw in it: the boxes, the identity channels and the
tag reading. The lines are the labelling input - `labelling_candidates()` returns
the facings the tag channel named with a corroborated price, which are seeds for the
confirmation screen rather than guesses. Nothing here is a training set on its own.
Sampling keeps each frame's own number in the index, so a sampled recording says so
instead of hiding it, and `/v1/health` reports how much has been written. Measured
on a 163-frame walk: 32 frames and their lines.
`docs/DATA.md` is the runbook: what to capture (a slow walk per rail, a close pass
for the barcode, and the same shelf on a second visit), how to turn it into labels,
and how to freeze a benchmark with a cross-visit split.
167 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
From the deep review's hygiene list:
- The demo replay (a still recognition replayed as if it were live tracking) moves
out of app.py into demo_replay.py, and every payload it produces is marked
`synthetic: true`. It had been committed inside an unrelated feature commit, which
made a film-making device look like part of the server.
- `/v1/health` reports the OCR engine and whether it is available. Without it an
operator could not tell a broken OCR from a shelf with no readable tags.
- A failed OCR pass is counted and logged instead of swallowed: `TagService.stats()`
now carries `passes_failed` and `last_pass_error`.
- The live classification budget goes back to the measured production defaults
(16 in flight, 2 attempts); a demo walk over a nearly static shelf asks for more
through `--live-inflight` / `--live-attempts` rather than silently changing the
default for everyone.
162 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Stage 2 of the plan is "identity as retrieval", so the first thing measured was
whether retrieval can even work here. It cannot, on its own: an off-the-shelf
DINOv2 embedding does group identical products (two orange juices, two apple
juices, two almond drinks) but the fine-grained cases collapse - 32 cartons into
one cluster, and nine different bottled drinks grouped by *container shape*. That
matches the literature (RP2K, RPC all fine-tune; nobody ships off-the-shelf
features for SKU identity).
Meanwhile the shelf has been carrying the answer. Every product has a tag printed
under it, and the tag spells the product's name out in clean text, once per
product - the store's own label. This adds that as an identity channel:
- a tag with a readable name is matched against the catalog by name similarity;
- facings are assigned to the *nearest tag along their row* (the tags mark where
products sit), not by geometric grouping, which was measured to merge different
products into a 945 px "product" that then swallowed the wrong tag;
- rows are clustered first, because the same column on six shelves would
otherwise become one product.
Then the demo catalog itself had to be fixed: it listed another store's products
(senbei, Tropicana) while the shelf holds mayonnaise, so nothing could match. The
catalog for this shelf was read off the tags - 12 products with names and prices -
and the products were enrolled as packaging templates by cropping the facings the
tags had just named.
Measured through the real still path on add/image6.jpg (111 detections):
tag named the product 28 facings (all 28 verified correct by eye)
packaging named the product 74 facings
both spoke 23 facings, agreeing 16 (69 %)
facings with any identity 79
So packaging recognition makes a ~30 % error rate on the facings the tag can
check, and every disagreement carried a low packaging score. The tag therefore
wins a conflict, and the conflict is recorded (`identity_conflict`) instead of
hidden - a product whose tag says one thing and whose packaging says another is a
real finding, not noise to suppress.
144 tests pass; test_tag_identity.py adds 6.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
macOS Vision holds the GIL for the ~124 ms it takes to read a rail band, so the
tag pass decides the live frame rate: with 2 bands every 15 frames a walking
session fell to 1.7 results/s, and 1 band every 30 frames brings it back to 7.2
with tags still settling in a second or two. Measured, not guessed - the same
walk, same method (orb), only the OCR duty cycle changed.
The walk also gained a `pan` parameter (default 0.06, clamped 0.005-0.5) so a
demo can choose its pace: pan=0.02 sweeps a row over ~8 s, which reads like a
person rather than a time-lapse.
Demo configuration measured end to end on the walk (20 s, 1280x720, one client):
orb, pan=0.02, 1 band/30 frames -> 7.2 results/s, 178 labelled tracks, 21
distinct SKUs, first label 0.7 s, 3 readable price tags.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/v1/demo/plan + /v1/demo/frame give a demo the one thing a still image cannot:
motion. The virtual camera pans a shelf row left to right, steps up, and pans
back - a store employee sweeping a shelf - over a real shelf photo, ~1 ms per
frame, deterministic, and wrapping so an unattended demo loops. The client drives
only `step`, so the walk's speed follows whatever frame rate it can sustain.
Three real defects came out of measuring the walk rather than a still:
- Bands were keyed by on-screen position, so a walking camera minted a new key
every few pixels of pan and re-read the same shelf continuously. Keyed by row
now; OCR stops chasing the window.
- The classification budget was consumed by retries of products already leaving
the frame. Retries now skip anything near the edge, and a track gets 2 attempts.
- The pan advance was 0.22 of the window per step: a facing was in view for one
or two frames, so nothing could ever be classified (9 labelled tracks in 20 s).
0.06 keeps a product in view for about a second.
Measured on the walk (20 s, 1280x720 frames, one client): orb labels 296 tracks
across 21 SKUs with the first label at 0.2 s, while hybrid manages 13 tracks /
4 SKUs at 5.4 s - so the live default for a demo is the fast method, with hybrid
kept for tap-to-inspect.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Web client: price-tag boxes in a distinct style (teal, chip with the parsed
price), an OCR section in the product card showing the tag's price, name,
discount and the reconciliation verdict with both values, and a compliance strip
under the stage. The photo stage now draws the image into the canvas and uses the
same containment as the media, so boxes cannot drift from the product (they did:
the canvas stretched to the stage while the photo was contained).
Server fixes the browser runs forced out:
- Photo mode reconciled before identities existed, so every tag read "tag only".
The tag bundle is cached with the still and classify re-reconciles against the
identities it just produced, returning them per detection.
- The still path only ever read the first four bands (plan hands out two per
call); it now drains every band, twice at different upscales so the vote has
independent reads.
- Voting is confidence-weighted and a dead heat stays unread; the plausible
shelf-price range is 10-9999, which stops a misread from becoming a ¥26,300
price. The compliance coverage bar moved to 0.7 after 0.5 produced a false
"group without price tag" on a shelf where every facing has one.
Measured on add/image6.jpg: 8 readable tags with names and prices, 942 ms for
the whole still, 7 real price mismatches, no false violations, and hints that
say plainly when a row could not be read.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The engine owns the OCR backend, a shared OCR lock and a dedicated tag worker, so
a tag pass can never occupy a classification worker or slow a frame. A live
session plans rail bands on the frame loop (cheap), runs them on that worker, and
attaches whatever the cache already knows; a still runs one synchronous pass.
Tracks and detections now carry price_tag and reconciliation, and the summary
carries price_tags plus compliance. Config can turn tags off per session.
Compliance now fails closed on evidence quality. The first real run produced 106
"missing tag" violations on a shelf where only 8 tags were readable - our tag
*detection* was incomplete, not the shelf. A row is only judged when at least
half its facings carry tag evidence; otherwise it reports "tags detected for 14
of 91 facings in this row" as a hint. Missing-tag findings are one per row with a
count, not one per facing, and price_mismatch still reports every disagreement.
On add/image6.jpg the result is now 7 real price mismatches and no false
violations.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_live: a product is unconfirmed, then labelled, without blocking frames; a
weak match never names a SKU and ends as needs_confirmation; a restarted id is
re-verified rather than inheriting its old label; inherited labels are flagged;
latest-frame-wins accounting; payload shape; glare hint; per-session
classification cap; a method change clears labels and drops late results.
test_api: health, catalog shape (no local paths), template-image 404s including
traversal-looking ids, stock validation, recognize over raw bytes and multipart,
415/422/400 with one error shape, classify/inspect roundtrip, IMAGE_EXPIRED,
token auth on REST and the socket, the WebSocket protocol (hello, frames,
config/reset/ping, recoverable BAD_FRAME and UNDECODABLE_IMAGE, inspect and
TRACK_NOT_IN_VIEW), the session limit, and the legacy session route.
Server fixes the tests forced out: FastAPI validation errors now use the same
{"error": {...}} envelope as every other failure, and the legacy session routes
accept a missing JSON body instead of returning 500.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The client is a pure API consumer: live frames go out over /v1/stream, stills
over /v1/recognize, and it can be pointed at another server with
?server=&token=. It keeps the team's visual identity (green palette, logo, EN/JA
toggle) and adds the live surface: state-coloured boxes with name/price chips,
interpolated track boxes, a results/s + latency HUD, quality hints, and an
in-view list that highlights a SKU's boxes.
Photo mode recognises the whole shelf at once instead of one click per product:
boxes paint immediately, identities fill in, tapping a product inspects it
(crop + feature-match visualization), the algorithm selector re-classifies, and
dispense updates stock.
Fixes found by browser verification with a synthetic camera feed:
- The WebSocket handler referenced its processing loop before definition, so the
socket died with NameError and live mode never received a result. Helpers are
now defined inside the handler.
- The client never kept a reference to the video element, so frame capture
returned early and nothing was ever sent.
- /v1/images/{id}/classify passed detection dicts where the engine expected box
dicts (KeyError: 'width').
- Placeholder chrome (thumb, dashes, hidden groups) leaked on first paint.
Verified in Chrome with a fake camera: live 8-12 results/s, ~60 products in
view, product card and feature matching populated, photo 105/170 recognised on
image.jpg and 59/111 on add/image6.jpg, 0 console errors.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
app.py becomes the recognition server: /v1 REST (health, catalog + template
images, samples, recognize, classify, inspect, stock) plus /v1/stream over
WebSocket, with token auth, size limits, one consistent error shape, and the
legacy /api/sessions routes kept but now backed by the real engine instead of
the fake adapter. Default bind is 127.0.0.1; LAN and TLS are opt-in.
stream_client.py is the reference non-browser client (camera, video file, or a
pan across a still) and prints results/s, RTT, detect time and label stability.
Live quality work driven by measurements:
- Labels are remembered per place: a fresh track covering a recently recognized
product inherits the label instead of queueing for classification again. On a
continuous pan this raised recognized products in view from 9/66 to 17-21/66
and cut time-to-first-label to ~1.6 s.
- Classification priority is newest-first, because sorting by area spends the
budget on products that are already leaving the frame.
- Per-session classification concurrency raised to 12.
hybrid_fast (ORB shortlist then hybrid) is implemented but NOT the default: it
is 3.4x faster (13 ms vs 44 ms p50) yet agreed with hybrid on only 47% of
accepted crops across 553 crops, far below the 98% adoption rule. It stays
available for experiments; the numbers are in the commit and the benchmark.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
