The live session scored crops with the matcher regardless of the configured identity
backend, so a live scan used the old mechanism while the still path used the enrolled
gallery - measured, the matcher accepts look-alikes and cannot say "unknown", which is
exactly what a real shelf needs it to do. score_crop now routes to the gallery, and
the decision uses a similarity scale (accept at 0.60 with a 0.04 margin) rather than
inlier-count thresholds that mean nothing on a cosine.
The synthetic walk is gone: demo.py, demo_replay.py, the /v1/demo endpoints and the
walk benchmark are deleted, and the app's frame source is the camera on a device (the
single-image replay stays for the simulator, which has no camera).
stock-demo merges both demo shelves (27 products, 62 views, trained head) so one
server recognises either image, and run-demo-server.sh starts it bound to 0.0.0.0 and
prints the exact URL to put in Config.xcconfig.
Verified over the LAN with 1280x960 frames: 40 frames in 5.3s, 9 products recognised
on demo image 1.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two real store photographs, chosen for density and for products that look different
from each other, each with its own enrolled store:
demo-images/demo-image-1-mediya-dairy.jpg CC0 MEDIYA dairy case
21 products kept (13 price cards dropped), 16 recognised, all 14 products named
demo-images/demo-image-2-milk-case.jpg CC BY 2.0 supermarket milk case
56 detections, 49 recognised, all 13 products named
Both were enrolled the way a store would be: the products were read off a contact
sheet of the detections, their crops became the store's gallery (stock-mediya,
stock-milk150), and a head was trained on them. `build_store_from_shelf.py` is that
flow.
The filter came out of the first image. Every Japanese supermarket photo has the same
artifact: the detector boxes the shelf-edge price cards as products, because a card is
roughly the shape of a pack - 13 of 34 detections on image 1 were cards. Their text
separates them cleanly: a card's characters are 30-44% digits and it prints the
tax-excluded price (本体), while a product's are 3-19% digits. Cards are now dropped
before classification, and the count is reported as `price_cards_dropped` rather than
hidden.
demo-images/README.md records how to run both, what the demo shows, and the honest
caveat: both shelves are enrolled, so the recognition rate is in-distribution - what
the app does on a shelf it has been onboarded to, not on one it has never seen.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The head is now part of a store rather than a script: `finetune_identity.py --stock
stock-lopia` trains on that store's enrolled views (27 views over 7 products, 12
augmented copies each) and writes `<stock>/identity_head.pt`, which ProductGallery
loads automatically.
The first attempt exposed a trap worth recording. A discriminative head has no "none
of the above": trained with an angular margin to separate seven products, it maps
everything confidently onto those seven, including cartons it has never seen.
raw features, threshold 0.55 home shelf 78% accepted foreign shelves 5% accepted
trained head, threshold 0.40 home shelf 58% accepted foreign shelves 44% accepted
Rejection - the gallery's whole point - disappears. The fix needs no retraining: the
two mechanisms fail in opposite directions, so raw features decide *whether* to answer
(they generalise) and the head decides *what* to answer (it discriminates). Measured:
47% accepted on the home shelf, 5% on foreign ones. Lower coverage than raw-only,
which is the honest trade.
The head also needs its own threshold (0.40 against the raw 0.55): a trained head's
similarity scale is its own, and reusing the raw line rejected almost everything it
said.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Part 2 ended on the number that matters: enrolling a real shelf from itself leaves
off-the-shelf DINOv2 at ~25% top-1, because real milk cartons differ by small text
and a colour band. This trains a small projection head on frozen features - 384 -> 128,
cosine classifier, angular margin - with augmentation standing in for the second visit
we do not have, and measures it on 72 held-out facings over 8 products.
raw DINOv2 cosine top-1 2/72 (2%) balanced 1%
raw, rejecting below 0.55 top-1 2/72 (2%) balanced 1% (5% of answered right)
fine-tuned + margin top-1 28/72 (38%) balanced 73%
fine-tuned, rejecting below 0.4 top-1 17/72 (23%) balanced 58% (94% of answered right)
So fine-tuning is the difference between working and not (balanced 1% -> 73%), and
with a rejection threshold the model becomes precise - it answers 18 of 72 facings and
is right on 94% of them, which is the operating point a store wants. The raw model
cannot do that at any threshold.
Where it fails is where my labels are coarse: MEGMILK, which the contact sheet shows
is probably several products under one brand, scores 1/32 because the class is
internally inconsistent and the model correctly refuses most of it.
Honest limit, stated in the doc: enrolment and held-out facings come from one
photograph, so these are upper bounds; only a recorded second visit settles the real
number. What is settled is the mechanism - a head trained on labels the pipeline can
produce itself moves fine-grained identity from useless to usable.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The external evaluation showed the matcher cannot say "unknown": on 20 shelves whose
products are absent it claimed 47.8% of facings as recognised, and its raw inlier
score cannot be thresholded (foreign shelves' median 6.0 overlaps the real shelf's
5.2 p10, and the worst offenders score 50-190). So the gallery becomes a selectable
identity backend:
--identity gallery (or DORAYAKI_IDENTITY=gallery)
Measured on the same 20 foreign shelves: false accepts fall from 47.8% to 13.8% at
the default threshold and to **5%** at 0.60, with a smooth curve to pick an operating
point from. Detection is unchanged (the gallery only names boxes).
What it does NOT fix, measured on a milk shelf where 14 of 107 facings really are
明治 おいしい牛乳 (labelled by eye): precision is 28-45%, no better than the matcher's
39%, and enrolling the shelf from itself (1-3 views per product) is flat at ~25%
top-1. Off-the-shelf DINOv2 is not discriminative enough for fine-grained SKUs - the
same conclusion the literature reaches, now measured here.
The packaging's own text is a real channel and resolution-bound: 51% recall
single-scale, 60% with the tag reader's multi-scale + red-channel trick, 85% where
the text is large (MEGMILK) and 28% where it is small vertical Japanese.
docs/EXTERNAL-EVAL.md records all of it, including the identity architecture it
implies: reject by default, name from the tag when readable, from the packaging text
when legible, from a fine-tuned embedding otherwise, and hand the rest to the
operator.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Everything else here is measured on two photographs. This collects real store
shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY,
CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on
them. 46 of 54 images carry a shelf; the rest are book scans and a shop front.
What it found, and it is not what I expected:
- **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves,
spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong
grocery - tight boxes on images the model has never seen, and correctly nothing on
the book scans.
- **Identity transfers when the product is genuinely there.** On a 2011 milk-case
photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and
R-1 elsewhere. The matcher does reach new photographs of products it has templates
for.
- **Precision is the failure, and rejection is missing.** In that same photograph it
labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 -
they are all white milk cartons. Across 24 shelves whose products are not in the
catalog it accepted 451 identities, and the score cannot be thresholded to fix it:
the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median
is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and
189.9 (deer cookies). A threshold that catches the foreigners also throws away the
real matches.
That confirms on real data what the literature and the review predicted: the
matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the
failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection -
is the fix, not a refinement.
docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the
images themselves are not redistributed. 175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The camera moves and the shelf does not, but every measurement so far treats a
frame as the world: a facing is "a track id", and that id dies the moment the camera
pans away and comes back. Label inheritance, drift checks and restart counters were
each correct fixes for that, and all of them are symptoms.
`shelf_map.py` keeps a per-session map of the shelf plane. Frames are registered by
their content shift, detections are placed in shelf coordinates, and a detection that
lands where a facing already is merges into it - so identity votes and tag readings
pool on the physical facing.
Measured, and reported honestly:
- Registration is exact on the synthetic walk: 0.01 px median error against the demo
camera's own offsets, over 23 consecutive pairs, with both phase correlation and
ORB + RANSAC. That validates the harness, not the method - the walk is a pure
translation of one photograph. Real video needs step 1's footage.
- Pooling works: a facing seen 15 times across 2 track ids reports one identity at
1.00 agreement.
- De-duplication is NOT yet better than the tracker: 179 facings against 156 track
ids, where the shelf holds roughly 70. Both over-count, the map slightly more.
Two bugs were found by measuring, and both were silent: the content shift is the
negation of the camera's motion (getting it backwards turned ~60 facings into 857),
and matching purely by box overlap lets jitter split a facing (210 facings, 103 of
them seen once; matching by track continuity first brought it to 179 with 38).
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The deep review's first recommendation was real data, because every accuracy number
so far comes from one or two photographs - the identity benchmark even takes its
gallery and its queries from the same photo. This is the mechanism for replacing
that.
`--record-frames DIR` writes each processed frame, sampled every Nth, with a JSON
line describing what the server saw in it: the boxes, the identity channels and the
tag reading. The lines are the labelling input - `labelling_candidates()` returns
the facings the tag channel named with a corroborated price, which are seeds for the
confirmation screen rather than guesses. Nothing here is a training set on its own.
Sampling keeps each frame's own number in the index, so a sampled recording says so
instead of hiding it, and `/v1/health` reports how much has been written. Measured
on a 163-frame walk: 32 frames and their lines.
`docs/DATA.md` is the runbook: what to capture (a slow walk per rail, a close pass
for the barcode, and the same shelf on a second visit), how to turn it into labels,
and how to freeze a benchmark with a cross-visit split.
167 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A tag is only readable when its own arithmetic corroborates the price, so the
fixture needs the 162.00 that belongs with its 150 (150 x 1.08 = 162.00). Without
it the fixture asserted the old behaviour.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
From the deep review's hygiene list:
- The demo replay (a still recognition replayed as if it were live tracking) moves
out of app.py into demo_replay.py, and every payload it produces is marked
`synthetic: true`. It had been committed inside an unrelated feature commit, which
made a film-making device look like part of the server.
- `/v1/health` reports the OCR engine and whether it is available. Without it an
operator could not tell a broken OCR from a shelf with no readable tags.
- A failed OCR pass is counted and logged instead of swallowed: `TagService.stats()`
now carries `passes_failed` and `last_pass_error`.
- The live classification budget goes back to the measured production defaults
(16 in flight, 2 attempts); a demo walk over a nearly static shelf asks for more
through `--live-inflight` / `--live-attempts` rather than silently changing the
default for everyone.
162 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Three findings from the deep review, all about the tag channel claiming more than it
can support.
1. The exact signal is a barcode. The reconciliation compared `tag.jan` against the
catalog's JAN, but that field held the regular price - a number compared against
a product code. It now reads a `barcode` field (populated only by a close-up
decode) and the printed number is compared as a price.
2. A price-mismatch violation required "an anchor", which was `tag.jan` - a price.
The anchor is now the tag's own arithmetic agreeing, or a name that actually
resembles the product, and the name signal carries its score as a field instead
of being parsed back out of a human-readable note string.
3. Tag identity could reach anywhere along a row: a facing whose own tag was not
detected inherited whichever readable tag was nearest, which can be a different
product several columns away. Assignment is now capped at the product's own run
(2.5 facing widths, at least 200 px), and a tag may only overrule the packaging
channel when it is corroborated or its name matches at 0.6 rather than 0.4.
162 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two corrections, both found by the deep review.
1. The number printed large beside the sale price is NOT the JAN's last three
digits. Checked against the manufacturer, キユーピー ゼロ ノンコレステロール 310g
is JAN 4901577092381 - it ends in 381, not the 357 the tag prints. The number is
the regular (pre-discount) price, and all three tags on the demo shelf confirm it
against their printed discount: (357-269)/357 = 24.6 % -> 2.4割引,
(400-379)/400 = 5.25 % -> 0.5割引, (248-199)/248 = 19.8 % -> 1.9割引. The field
is renamed `regular_price`, the barcode keeps its own name, and the discount is
now a second arithmetic check alongside the tax-included price - so a tag can be
checked against itself twice.
2. A price is only readable when that arithmetic corroborates it. Measured on a live
walk before this: 90 tracks carried a tag marked readable, with prices of 20, 23,
36, 50, 72, 2002 and 4508. The gate has three strengths - an exact tax pair, an
agreeing printed discount, or both plus agreeing digits - and `TagRecord.state`
now consults it, so the wire cannot call an uncorroborated read readable.
Also fixed while measuring: voting picks each field independently, so a record could
publish a price from one frame beside a tax-included price from another (measured:
price 149 with unit 214.92, which never appeared together - 214.92/1.08 = 199). The
arithmetic is now re-run on the values actually being published, stale flags are
dropped, and an implausible regular price (3050 beside 349) is suppressed rather
than published with a discount derived from it.
Measured through the real still path: readable tags 9 -> 5, every published price
now satisfies its own tax-included price, and the live walk publishes none instead
of publishing garbage. 162 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The user's objection was right and it changed the design: the tag channel is
sparse in the real world (measured: 0 tags readable at ~2 m on the main shelf), so
packaging recognition has to carry identity. Measured on a 12-class, 57-facing
evaluation set read off the demo shelf, the packaging channel as shipped cannot:
views per SKU embedding top-1 shipped matcher top-1
1 57 % 33 %
2 72 % 27 %
3 75 % 20 %
4 76 % 5 %
Two things follow. First, two views per SKU is where most of the gain is - that is
the enrollment spec (a first scan already produces several facings per product).
Second, the matcher gets *worse* with more views, because its raw inlier score is
not calibrated to be aggregated and it cannot express two views of one SKU at all:
it keys templates by filename.
So identity becomes retrieval: `gallery.py` embeds every enrolled view with DINOv2
ViT-S/14, scores a SKU by its best view, and rejects when nothing is close enough
(an unknown product must come back unknown, not as the nearest stranger). Measured
with held-out views - the gallery never contains its own queries - 2 views per SKU
over 12 SKUs gives 69 % top-1 with rejection working, and the threshold moves the
rejection line.
It is affordable in the live loop only on the GPU: 159 ms per crop on CPU versus
3.6 ms on the M5's MPS, and a live frame only needs its new tracks embedded.
157 tests pass; test_gallery.py covers the retrieval contract (best view wins,
several views are one answer, strangers are rejected) with injected embeddings so
it needs no torch.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Stage 2 of the plan is "identity as retrieval", so the first thing measured was
whether retrieval can even work here. It cannot, on its own: an off-the-shelf
DINOv2 embedding does group identical products (two orange juices, two apple
juices, two almond drinks) but the fine-grained cases collapse - 32 cartons into
one cluster, and nine different bottled drinks grouped by *container shape*. That
matches the literature (RP2K, RPC all fine-tune; nobody ships off-the-shelf
features for SKU identity).
Meanwhile the shelf has been carrying the answer. Every product has a tag printed
under it, and the tag spells the product's name out in clean text, once per
product - the store's own label. This adds that as an identity channel:
- a tag with a readable name is matched against the catalog by name similarity;
- facings are assigned to the *nearest tag along their row* (the tags mark where
products sit), not by geometric grouping, which was measured to merge different
products into a 945 px "product" that then swallowed the wrong tag;
- rows are clustered first, because the same column on six shelves would
otherwise become one product.
Then the demo catalog itself had to be fixed: it listed another store's products
(senbei, Tropicana) while the shelf holds mayonnaise, so nothing could match. The
catalog for this shelf was read off the tags - 12 products with names and prices -
and the products were enrolled as packaging templates by cropping the facings the
tags had just named.
Measured through the real still path on add/image6.jpg (111 detections):
tag named the product 28 facings (all 28 verified correct by eye)
packaging named the product 74 facings
both spoke 23 facings, agreeing 16 (69 %)
facings with any identity 79
So packaging recognition makes a ~30 % error rate on the facings the tag can
check, and every disagreement carried a low packaging score. The tag therefore
wins a conflict, and the conflict is recorded (`identity_conflict`) instead of
hidden - a product whose tag says one thing and whose packaging says another is a
real finding, not noise to suppress.
144 tests pass; test_tag_identity.py adds 6.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two false positives came out of running the rules against a real shelf, and both
had the same shape: the rules were arguing with our own detector instead of with
the shelf.
1. One tag per product, not per facing. Measured: a row carries 16-18 facings but
only 4-5 tags, because four facings of a mayonnaise share a single tag. Counting
facings invents missing tags on a shelf that is perfectly labelled. The rules
now group a row's facings into products - by identity when the facings were
recognised, by size and spacing otherwise - and count products.
2. A product's tag may sit anywhere along its run. Measured: a run of
ピュアセレクト facings was split by geometry because one facing was detected
narrower than its neighbours (54 px beside 150 px), and the fragment was then
reported as missing the tag that was sitting further along its own run. A
product now counts as missing only when no tag exists for its SKU anywhere in
the row, and a fragment far narrower than the row's typical product is not
judged at all.
3. A mismatch needs one trustworthy anchor. Measured noise: a wrong identity plus
a garbled tag read (name match 0.11, no JAN) produced a confident "tag says 90,
catalog says 188" that was really two bad reads disagreeing. Without the tag's
JAN or a name that resembles the product, that is now a hint, not a violation.
Measured on add/image6.jpg through the real still path: 0 false violations without
identities (was 1), and with identities the only finding left is a genuine
tag-versus-catalog price difference. Coverage hints now read "tags detected for 2
of 4 products in this row". 143 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Until now a tag reached the rules only if its text could be read. A tag that was
there but unreadable was indistinguishable from a missing one, so a missing-tag
claim rested on the reader's failures rather than on the shelf - the false
violation that made an earlier run report 106 of them.
Two measurements shaped the detector:
- The tags are printed ON the rail: a red brick-patterned strip running the width
of the shelf. The rail is therefore findable without reading anything, and it is
where tags must be looked for.
- On the rail, a tag is a run of columns carrying paper (bright, unsaturated)
against the brick. Projecting onto x finds 27 tags on add/image6.jpg where
connected components found 9, because tag text fragments a component at every
printed line.
The pipeline now merges the two views: a tag whose text was read keeps its fields,
a tag that was only seen is reported unreadable with its box. Measured on
add/image6.jpg: unreadable tags 8 -> 17, "no tag" 94 -> 85, and the per-row hints
became specific - "tags detected for 8 of 16 facings in this row".
That last number is also a domain fact worth keeping: this store prints one tag
per product, not per facing, so facings are the wrong denominator for a
missing-tag rule. The coverage bar keeps the rules silent until that is fixed,
which is the honest behaviour in the meantime.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The card reader produces a JAN (the last three digits printed on the tag) and
says whether a price was measured from the big digits or derived from the
tax-included one; the session record and the wire payload dropped both, so the
strongest identity anchor the tag carries never left the reader.
Also reads each card at three scales rather than two. The tax-included price is
small print and only resolves at 3x; without it the price falls back to the
stylised digits, which misread (measured: 30 instead of 269, 20 instead of 199).
Measured through the real still path on add/image6.jpg, the tags now attached to
products read: 269/357, 379/400, 199/248, 209/500, 219/200, 349/150 - prices and
JANs, where the same run previously reported tags that were not in the readable
row at all and prices like 2630 from another tag's text.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The tag pass read a whole shelf rail and then guessed where one tag ended and
the next began. Measured on the demo shelf, that meant the four tags of the
readable row were not reported at all, and the tags that were reported carried
another tag's text (a 649/754.9 tag came back as price 2630, name
"コプサラダトレッシング"). The box it published was the band, so a tag box lined up
with nothing on the shelf.
Tags are now cards. The OCR engine's own line boxes define them: lines are
clustered into rows (a band's crop also catches the packaging text above and
below), the tag row is chosen by how much it looks like tags, and the cards are
split on the horizontal gap between them. A card's box is the union of its lines,
so it wraps the printed tag. Each card is then re-read at several scales and in
the red channel, and the reads are pooled for voting.
Two things make the fields trustworthy rather than merely plausible:
- Layout separates the price from the JAN. The price is the largest text on the
tag, so it comes from the tallest line; the JAN's last three digits come from a
small line. Getting this backwards made a 269 price look like a JAN and vanish.
- The tag checks itself. The small clean number is the tax-included price and the
big stylised one is the pre-tax price: measured, 269 x 1.08 = 290.52, 379 x 1.08
= 409.32, 199 x 1.08 = 214.92. When the two agree the price is "measured"; when
the big digits cannot be read the price is "derived" from the small ones and
flagged, because a tag that is 3.2x too small to read is still arithmetic.
The JAN's last three digits are now read (they never were before) - on a real
LOPIA tag that is the staff-facing identity anchor printed next to the discount.
Measured on the readable row of add/image6.jpg, against values read by eye:
15 of 15 fields correct across three tags (JAN, price, tax-included price,
discount, name), up from 0 tags reported. 141 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
macOS Vision holds the GIL for the ~124 ms it takes to read a rail band, so the
tag pass decides the live frame rate: with 2 bands every 15 frames a walking
session fell to 1.7 results/s, and 1 band every 30 frames brings it back to 7.2
with tags still settling in a second or two. Measured, not guessed - the same
walk, same method (orb), only the OCR duty cycle changed.
The walk also gained a `pan` parameter (default 0.06, clamped 0.005-0.5) so a
demo can choose its pace: pan=0.02 sweeps a row over ~8 s, which reads like a
person rather than a time-lapse.
Demo configuration measured end to end on the walk (20 s, 1280x720, one client):
orb, pan=0.02, 1 band/30 frames -> 7.2 results/s, 178 labelled tracks, 21
distinct SKUs, first label 0.7 s, 3 readable price tags.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/v1/demo/plan + /v1/demo/frame give a demo the one thing a still image cannot:
motion. The virtual camera pans a shelf row left to right, steps up, and pans
back - a store employee sweeping a shelf - over a real shelf photo, ~1 ms per
frame, deterministic, and wrapping so an unattended demo loops. The client drives
only `step`, so the walk's speed follows whatever frame rate it can sustain.
Three real defects came out of measuring the walk rather than a still:
- Bands were keyed by on-screen position, so a walking camera minted a new key
every few pixels of pan and re-read the same shelf continuously. Keyed by row
now; OCR stops chasing the window.
- The classification budget was consumed by retries of products already leaving
the frame. Retries now skip anything near the edge, and a track gets 2 attempts.
- The pan advance was 0.22 of the window per step: a facing was in view for one
or two frames, so nothing could ever be classified (9 labelled tracks in 20 s).
0.06 keeps a product in view for about a second.
Measured on the walk (20 s, 1280x720 frames, one client): orb labels 296 tracks
across 21 SKUs with the first label at 0.2 s, while hybrid manages 13 tracks /
4 SKUs at 5.4 s - so the live default for a demo is the fast method, with hybrid
kept for tap-to-inspect.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Web client: price-tag boxes in a distinct style (teal, chip with the parsed
price), an OCR section in the product card showing the tag's price, name,
discount and the reconciliation verdict with both values, and a compliance strip
under the stage. The photo stage now draws the image into the canvas and uses the
same containment as the media, so boxes cannot drift from the product (they did:
the canvas stretched to the stage while the photo was contained).
Server fixes the browser runs forced out:
- Photo mode reconciled before identities existed, so every tag read "tag only".
The tag bundle is cached with the still and classify re-reconciles against the
identities it just produced, returning them per detection.
- The still path only ever read the first four bands (plan hands out two per
call); it now drains every band, twice at different upscales so the vote has
independent reads.
- Voting is confidence-weighted and a dead heat stays unread; the plausible
shelf-price range is 10-9999, which stops a misread from becoming a ¥26,300
price. The compliance coverage bar moved to 0.7 after 0.5 produced a false
"group without price tag" on a shelf where every facing has one.
Measured on add/image6.jpg: 8 readable tags with names and prices, 942 ms for
the whole still, 7 real price mismatches, no false violations, and hints that
say plainly when a row could not be read.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The engine owns the OCR backend, a shared OCR lock and a dedicated tag worker, so
a tag pass can never occupy a classification worker or slow a frame. A live
session plans rail bands on the frame loop (cheap), runs them on that worker, and
attaches whatever the cache already knows; a still runs one synchronous pass.
Tracks and detections now carry price_tag and reconciliation, and the summary
carries price_tags plus compliance. Config can turn tags off per session.
Compliance now fails closed on evidence quality. The first real run produced 106
"missing tag" violations on a shelf where only 8 tags were readable - our tag
*detection* was incomplete, not the shelf. A row is only judged when at least
half its facings carry tag evidence; otherwise it reports "tags detected for 14
of 91 facings in this row" as a hint. Missing-tag findings are one per row with a
count, not one per facing, and price_mismatch still reports every disagreement.
On add/image6.jpg the result is now 7 real price mismatches and no false
violations.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
reconcile() compares one accepted identity with its tag and the catalog entry and
publishes a state plus the individual signals (JAN exact, name by longest common
substring, price against either the sale or the regular price). It never changes
the product identity: a disagreement is a second opinion shown next to the first,
and only a checksummed barcode outranks the matcher.
evaluate_compliance() is the first rule family: missing_price_tag, the owner's
missing_tag_group (>= 3 consecutive facings without a tag), orphan_tag,
price_mismatch, plus an unreadable_tags hint. Every rule fails closed on evidence
quality - a row whose tags could not be read produces "move closer", never a
violation. Findings must appear in 3 of the last 5 frames before they publish, so
the UI never flickers.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ocr.py: one interface, backends by availability - macOS Vision through pyobjc
(measured 182-246 ms per rail band), a fake engine for tests, and an explicit
"none". The owner chose the macOS backend; the interface keeps a cross-platform
engine a drop-in, and the module says so.
tags.py: rail bands from the detector's own row clustering (no new geometry
model), clustering of OCR lines into tag-shaped regions, deterministic Japanese
price-tag parsing (sale price by printed area, regular price, discount, unit
price, name, tax/size artifacts rejected), and TagService - a per-session cache
with multi-pass voting so a static tag becomes one settled reading instead of
thirty noisy ones, and so the frame loop never waits for OCR.
Measured on add/image6.jpg: 6 bands, 12 tags, 8 readable with real prices and
names (305 エッグケア, 357 ゼロノンコレステロール, 400 フィッテ), 39-254 ms per
band. On image.jpg (2 m) it correctly finds nothing readable - the tags are too
small there, which is the capture requirement recorded in the design.
Tests found four real bugs: unit prices leaking into the price field, packaging
sizes like <310g> read as prices, cluster_tags dropping the line boxes (so the
big sale price was never preferred), and brittle tag-to-product association when
a cluster spans facings.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The owner's brainstorm thought through to an implementable design, backed by
measurements taken against the real demo images rather than assumptions:
- Tag readability is a capture-distance problem first. At image.jpg's ~2 m the
tags are ~14 px digits and no engine reads them; at add/image6.jpg's ~1 m
(22 px digits) macOS Vision reads all five tag fields including the large red
sale price, and RapidOCR reads the printed fields at 0.97-1.00 confidence but
misses the stylised one. Tesseract jpn is unusable on tags.
- Engine choice is an interface with backends: macOS Vision (182 ms/rail band)
as the Mac fast path, RapidOCR (0.75-1.3 s/band, cross-platform) as the
default elsewhere, fake for tests, PaddleOCR held in reserve. Cloud OCR is
rejected: offline demo, and no owner decision to ship shelf images out.
- OCR must never sit in the frame loop: a TagService runs rail-band passes on a
cadence with a spatial cache and multi-pass voting (tags are static), which
also fixes digit misreads.
- Packaging text confirms 67% of the catalog templates (measured) but only
10%/3% on the demo shelves, because the demo catalog is fictional and the
photos are a real store. Absence is therefore neutral, never a contradiction,
and the demo needs one shelf whose tags and catalog describe the same store.
- Compliance rules fail closed on evidence quality: a row whose tags are merely
unreadable yields a capture hint, never a violation. The owner's "group of
items missing price tags" is the missing_tag_group rule (>= 3 consecutive
facings without a tag).
- Barcode/JAN is the exact-identity path, currently undecodable at demo
distance; the catalog needs optional jan and regular_price columns.
- The mobile lane can start now against protocol v1: tap-to-inspect works with
today's schema, and the tag fields are additive when they land.
Also records the five open questions for the owner (capture assets, engine
default, mobile order, catalog columns, parallel start).
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
STrack lives in ultralytics.trackers.byte_tracker and the YAML loader is
ultralytics.utils.YAML; the basetrack/yaml_load names this module used are gone
in the pinned version. Caught by running the tracker tests against the real
dependency.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Detection and classification share the CPU, and leaving both uncapped made them
fight: uncapped torch plus 8 workers measured 11.1 results/s single-client and
dropped to 6.3/s each with two clients. Capping torch at 4 threads and
classifying with 5 workers gives 12.2 results/s single-client (46 ms detection)
and 8.5/s each for two clients, with zero unexplained label changes.
Detection is the part that must stay real-time; classification fills in behind
it, so the defaults now protect detection's share of the CPU. Both remain
overridable (--classify-workers, DORAYAKI_TORCH_THREADS).
docs/API.md records the shipped numbers and says which configuration produced
them, including the honest gap: detection during a live pan is ~46-48 ms, not
the 28 ms of an idle measurement.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
docs/API.md is the full contract: run flags and environment, auth, coordinate
convention, the identity vocabulary, every REST endpoint and WebSocket message
with examples, the backpressure rule, what a client may rely on, the design
decisions with their rejected alternatives, measured performance, known
limitations, and notes for the future mobile client.
README.md (Vietnamese) now documents the client-server shape, both modes, the
flags, the API, the test commands, and the source layout. It also repairs the
PowerShell escape damage in the original file (backtick-r had become a line
break, backtick-b a backspace, backtick-a a bell), which had left the code
fences and file paths mangled.
test_real_model.py is the opt-in integration test (DORAYAKI_REAL_MODEL=1): it
guards the measured baseline (170 detections, 105 recognized on image.jpg) and
proves a pan keeps labels stable with no unexplained changes.
run_app.sh / run_app.bat pass arguments through to the server.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_live: a product is unconfirmed, then labelled, without blocking frames; a
weak match never names a SKU and ends as needs_confirmation; a restarted id is
re-verified rather than inheriting its old label; inherited labels are flagged;
latest-frame-wins accounting; payload shape; glare hint; per-session
classification cap; a method change clears labels and drops late results.
test_api: health, catalog shape (no local paths), template-image 404s including
traversal-looking ids, stock validation, recognize over raw bytes and multipart,
415/422/400 with one error shape, classify/inspect roundtrip, IMAGE_EXPIRED,
token auth on REST and the socket, the WebSocket protocol (hello, frames,
config/reset/ping, recoverable BAD_FRAME and UNDECODABLE_IMAGE, inspect and
TRACK_NOT_IN_VIEW), the session limit, and the legacy session route.
Server fixes the tests forced out: FastAPI validation errors now use the same
{"error": {...}} envelope as every other failure, and the legacy session routes
accept a missing JSON body instead of returning 500.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_engine: detection normalization, one-caller-at-a-time detection, method
validation, undecodable payloads, crop isolation, classify_boxes accepting the
detection dicts the API actually hands it, summary counting, empty-slot
normalization across frame sizes, the still-image store (roundtrip, TTL, bound),
catalog/stock behaviour, and template-image path resolution that refuses
anything outside the catalog.
test_tracking: per-session id numbering, the case that matters (constructing a
second tracker must not reset a running session's ids), independent sessions,
pan continuity, buffer expiry producing a new id, detection index mapping, and
reset scoping.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The client is a pure API consumer: live frames go out over /v1/stream, stills
over /v1/recognize, and it can be pointed at another server with
?server=&token=. It keeps the team's visual identity (green palette, logo, EN/JA
toggle) and adds the live surface: state-coloured boxes with name/price chips,
interpolated track boxes, a results/s + latency HUD, quality hints, and an
in-view list that highlights a SKU's boxes.
Photo mode recognises the whole shelf at once instead of one click per product:
boxes paint immediately, identities fill in, tapping a product inspects it
(crop + feature-match visualization), the algorithm selector re-classifies, and
dispense updates stock.
Fixes found by browser verification with a synthetic camera feed:
- The WebSocket handler referenced its processing loop before definition, so the
socket died with NameError and live mode never received a result. Helpers are
now defined inside the handler.
- The client never kept a reference to the video element, so frame capture
returned early and nothing was ever sent.
- /v1/images/{id}/classify passed detection dicts where the engine expected box
dicts (KeyError: 'width').
- Placeholder chrome (thumb, dashes, hidden groups) leaked on first paint.
Verified in Chrome with a fake camera: live 8-12 results/s, ~60 products in
view, product card and feature matching populated, photo 105/170 recognised on
image.jpg and 59/111 on add/image6.jpg, 0 console errors.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
app.py becomes the recognition server: /v1 REST (health, catalog + template
images, samples, recognize, classify, inspect, stock) plus /v1/stream over
WebSocket, with token auth, size limits, one consistent error shape, and the
legacy /api/sessions routes kept but now backed by the real engine instead of
the fake adapter. Default bind is 127.0.0.1; LAN and TLS are opt-in.
stream_client.py is the reference non-browser client (camera, video file, or a
pan across a still) and prints results/s, RTT, detect time and label stability.
Live quality work driven by measurements:
- Labels are remembered per place: a fresh track covering a recently recognized
product inherits the label instead of queueing for classification again. On a
continuous pan this raised recognized products in view from 9/66 to 17-21/66
and cut time-to-first-label to ~1.6 s.
- Classification priority is newest-first, because sorting by area spends the
budget on products that are already leaving the frame.
- Per-session classification concurrency raised to 12.
hybrid_fast (ORB shortlist then hybrid) is implemented but NOT the default: it
is 3.4x faster (13 ms vs 44 ms p50) yet agreed with hybrid on only 47% of
accepted crops across 553 crops, far below the 98% adoption rule. It stays
available for experiments; the numbers are in the commit and the benchmark.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
One LiveSession per streaming client. Each frame runs detection and tracking,
then classification is scheduled for products that are new, big enough, fully in
frame and not already settled; results are cached on the track so detection
never waits for classification. Empty-slot gaps are smoothed over frames and
quality hints (glare, blur, too far) come from a 320 px downscale.
Correctness work that the acceptance run forced out:
- Classification submissions happen OUTSIDE the session lock. add_done_callback
runs the callback inline when the future already completed, so attaching it
while holding the lock deadlocked the session on the second frame.
- A label is dropped only when its box has drifted off the product for two
consecutive frames (panning slides boxes onto neighbours), and the anchor is
the box the crop came from, not wherever the track moved by the time
classification finished.
- An id that disappears and comes back is a restart, not the same object: the
old label is cleared instead of silently labelling a neighbour. relabels and
restarts counters are monotonic per id so clients can tell a re-verification
from a silent flip.
- Track ids are tracked per session (tracking.py) and counters survive pruning.
Measured on the synthetic pan (30 frames, 248 label observations): 20
re-verifications, 0 unexplained label changes, detection 20 ms/frame.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
engine.py is the process-wide, thread-safe core the server is built on: one
YOLO instance behind a lock, thread-local matcher access, a classification pool
so detection is never starved, the stock catalog under its own lock, empty-slot
geometry normalized to a reference width (the detector's gap thresholds are
absolute pixels), and a small LRU of recent still images for later
classify/inspect calls. Detection output is normalized so every client sees the
same box shape.
tracking.py wraps ultralytics BYTETracker. The important part: ultralytics
mints track ids from a process-global counter and resets it when a tracker is
constructed, so with two live clients one client's ids would be re-issued while
the other still held cached labels. SessionTracker gives each tracker its own
counter and never touches the global one.
Measured defaults (bench_engine.py, M5): detection p50 26-29 ms on CPU vs
112 ms on MPS with byte-identical boxes, so device defaults to cpu; hybrid
classifies at ~51 ms/crop, 92 crops/s across 8 threads.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keeps the model's uncertainty out of the UI: decide_identity() separates raw
matcher scores into accepted / ambiguous / unknown / rejected, and only an
accepted result can name a SKU. Zero evidence is rejected, never a product.
observe.py adds frame-local observations and one-zone scan sessions (draft,
finalize, cancel, rescan) with replay, staleness and backpressure rules.
pipeline.process_click() now reports the identity level instead of failing
when the match is not accepted.
Incorporates earlier uncommitted session work in this repository
(identity.py, observe.py, yolo_adapter.py, test_observe.py, and the pipeline
hunk those needed); yolo_adapter keeps its decoded frame per thread so
concurrent sessions cannot swap images.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The recognition server will classify crops from several clients at once. Two
things blocked that:
- OpenCV Feature2D objects were shared across calls, so parallel
classify_crop() calls could corrupt each other. Each thread now gets its own
extractor set; the template cache is untouched.
- score_crop() is extracted from classify_crop() so callers that need every SKU
score (live track aggregation) do not have to re-derive it from the top-5
list. classify_crop() output is byte-identical to before (verified against the
pre-change module on 82 real crops x 4 methods).
- ShelfProductDetector accepts device/imgsz so the server can pin inference.
Dependencies: pin ultralytics to the version that wrote the checkpoint and that
the session tracker targets (8.4.153), add lap so ultralytics never
pip-installs it mid-request, and add websockets for the streaming endpoint.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
