Twenty independent-source images across eight SKUs, leak-tested (max template
match against add/image6.jpg is 0.675) and scored through the shipped lane:
13 correct, 7 abstain, 0 false accepts. The seven abstentions are all
unenrolled or weakly-read SKUs.
The result worth keeping is sku_l07: no enrolled image, both images accepted on
text alone. That is the text lane's coverage claim holding on data it had no
part in building.
F-12 is marked fixed with its measured effect, and the packshot-OCR check that
would have caught it is recorded as implemented but not yet trustworthy - it is
a false negative on l04 and l07.
Five blocks still printed pre-correction figures. Each was re-run through the
evaluator and replaced with what it actually outputs now:
- The lane comparison: local 75% at 91% against Van 64% at 38% (full), and
59% at 93% against 66% at 39% (live). At live resolution Van now answers more
facings correctly and is wrong on 67 of them to the local lane's 3 - stated
plainly rather than left as a win on both axes.
- The accept sweep: 75/67/67/62% end-to-end at 91/91/93/95% precision, with the
evaluator's own column definitions spelled out, since my first re-derivation
used the wrong denominator.
- The G1 stream claim: four pan replays give 63-70% coverage at 76-84%
precision, not 75% at 100%.
- The upscale paragraph: the guard is intact, but sku_l09.jpg is now a
1000x1000 packshot, so the current run no longer exercises it.
- catalog_hash and the pyramid floor: the hash covers view bytes, not just
names, and every view is now above the 800 px floor.
Also refreshes the server smoke against the renamed catalog: 53 named in 2.58 s,
no view-suffixed ids, no missing product info. l12 is not among them - it has
no enrolled view, which is the documented gap.
The reproduction block still printed the pre-enrollment numbers (catalog 69%,
decision 86%); it now prints what the command actually outputs today, with the
catalog figure's fall from 87% to 83% explained rather than left to look like a
regression.
Adds the three row corrections with their evidence, the near-unknown fix and
its measured effect, and the state of the four unenrolled SKUs - including that
l08's row name and size are themselves unverified, since no retailer listing
for a 30g スパイシーストーム ペリペリミックス exists.
cat_full's *_shelf.jpg views are crops of add/image6.jpg itself - template
matching puts each one back at 0.998-0.999 confidence - so committing them
reintroduced the evaluation photo as a catalog, which is the leak the audit
removed from stock-lopia. cat_pack is the same views without the shelf copies.
Both are regenerated by lfo_setup.py from stock-lopia and the labels, so the
evidence stays reproducible without the repository carrying the leak. Verified
after removal: lfo_setup.py rebuilds all 16 views.
E-3: the text lane adds 31 pp coverage at full resolution and 23 pp at live
while raising precision, against a +5 pp bar. E-2: LocalLane is RootSIFT-only,
so ORB/BRISK have nothing left to drop from the shipped path.
`classify_boxes` returned only the identity, so every accepted still-image
answer reached the client as a bare SKU id with no name or price. Measured
through the real server on add/image6.jpg: 41 of 41 accepted detections had no
`product`. The live path attached the row; the still path did not, which is
the F-02 finding the decision document recorded as still live in production.
The row is attached where the decision is made, so both paths carry it, and
`recognize` passes it through to the wire. Verified through the running server:
41 of 41 accepted answers now carry product_name, price, and stock counts.
Also corrects the decision document's full-resolution claim, which was
measured before the reference-upscaling and text-signature fixes: it said 77%
precision with 11 false accepts and a hub alarm, where the current lane reads
88% with 5 and no alarm. The accept line does not need to differ by input path
- correct facings sit at p25 2.07 (full) and 2.87 (live), wrong ones at p75
1.00 and 1.66, so 2.0 is the boundary in both - and the results table and the
stream G1 numbers are refreshed to the current measurements.
`catalog_hash` hashed the CSV and the view *names*, so replacing
`sku_l09.jpg` with a different product's packshot left the hash unchanged and
a report could claim the same catalog while every l09 decision in it had
moved. It hashes each view's bytes now.
`identity_lanes` carried its own `normalize_text` and `bigrams`, near-copies
of the ones in `rebuild_catalog` that had already drifted (the lane's table
folded ユ to itself, the catalog's did not). The lane imports the catalog's
now, so one read scores the same either way.
`evidence/image6_v2.json` was byte-identical to `eval/labels/image6.json`; the
evidence scripts and the decision document point at the canonical file and the
copy is gone. `score_crop`'s docstring now states the contract its callers
already depend on: the dict's insertion order is the lane's ranking, and
re-sorting by value names a different SKU than the lane chose.
Tests: the lane is shared by the classify pool, so a concurrent-ranking test
pins that one BFMatcher does not cross crops under eight threads.
The pacing paragraph described a defect that no longer exists. Replaced with
the measured before/after, and added the two review findings (the missing
always-argmax baseline lane, the size-token fill-in) with their test counts.
Two review findings, both confirmed by measurement.
1. The evaluator had no always-argmax lane, so every "vs Van" claim compared
against the *gated* matcher, which is not Van's contract - the original demo
answered every crop with its best-scoring SKU and had no accept line. The new
'van' lane is that contract: same scores, no gate. Against it the local lane
wins both axes at both resolutions (full 61%/77% vs 52%/29%; live 59%/95% vs
57%/32%), but the live coverage margin is one facing, not seven, and the 95%
figure is live-resolution only - at full resolution the same lane reads 77%
with 11 false accepts and a hub alarm.
2. A crop reading only "400g" scored 0.667 on three SKUs, above TEXT_ACCEPT, so
the text lane named a product from a size line. The fill-in job now requires
two non-digit bigrams; the veto is unchanged, so "65%" still contradicts a
local answer. The raw OCR read travels with the ranking through
score_crop_lanes for that check.
Tests: 3 for the size gate (policy + live) and the van lane's own row. Removing
the gate fails all three. Suite 276 -> 280.
The live label count is a function of frame pacing, not a fixed property: a
track needs two attempts to settle, so pushing frames faster than the classifier
drains them reports fewer labels (12/8/2/6 at full speed on the same clip, 14
paced, 38 of 30 frames). Precision stays 100% and 0 unknowns are accepted at
every pacing. Also records the local-scale and DecisionPolicy tests.
The record said '2 false accepts of 48 unknowns', which conflated two fields:
the 2 are 0 unknown accepts plus 1 unenrolled in-catalog facing and 1 enrolled
misidentification. Also restores the hybrid-verify line the lane vocabulary
edit dropped, and reframes the l09 note: its hubness ratio is 1.76 full /
1.36 live against G7's limit of 2.0, so the gate passes and the 30% internal
share heuristic is what raises the alarm.
Replaces the DINO gallery as the recognition path with the measured design
from docs/research/2026-09-28-recognition-architecture/DECISION.md: RootSIFT
over a per-SKU reference pyramid, similarity-constrained RANSAC and a hue
check, fused with an OCR text lane that vetoes a match whose package text
names a different catalog row and answers on its own when geometry finds
nothing.
- identity_lanes.py: LocalLane, TextLane, DecisionPolicy. The text lane
builds each row's signature from the catalog name plus its packshot OCR, so
a row with no enrolled image is still answerable.
- engine.py: every backend groups scores by canonical SKU, so no
view-suffixed id reaches the wire and a name and price always resolve.
decide_crop fuses the lanes; score_crop_lanes feeds the live path. Both
lanes are built in warm_up (non-fatally) so the first frame is not the slow
one, and the text lane shares the engine's OCR lock.
- live.py: decides per crop on the lane's own rank order. The lane's order
comes from its rank key and its value is only the gate, so re-sorting by
value named a different SKU than the lane chose; averaging in an empty
observation also diluted a good confidence below the accept line.
- identity.py: a `local` score scale whose gate is one absolute line, because
the value is already a margin over the runner-up.
- app.py, run-demo-server.sh: `--identity local` is the default; the demo
server no longer starts on the leaked `stock-lopia` catalog.
- eval_live.py: the replay vocabulary accepts `local`, so a recording from
the shipped server can be scored.
- README.md: documents the three backends and what each is for.
Measured on the corrected image6 labels at live resolution: 36/61 in-catalog
facings named at 95% precision, 0 of 48 unknowns accepted, 11 labels sourced
from the text lane alone. Through the real server: /v1/recognize 90%
precision, /v1/stream 100% precision, no view-suffixed ids on either.
Two things in API.md described a state that no longer exists.
The mobile section called the app's scan screen "a timer-driven placeholder
today" and said wiring it "is a separate lane". The app ships the real client
(`data/shelf/LiveScanClient.kt`, `platform/CameraFrames`) and ADR-002 governs
it, so the section now names what the client actually is and what it needs from
this server, including the negotiated crop lane.
The control-message table showed `{"type":"capabilities","max_crop_bytes":N}`
as the opt-in. `handle_capabilities` refuses exactly that message - it requires
`"crop": true` - so the documented example returned `BAD_MESSAGE` and left the
lane off. Verified against a running server: the corrected message answers
`capabilities_ack`, the old one answers `BAD_MESSAGE`.
The cosine-gate fix shipped without its end-to-end number. Measured on the same
walk (`stream_client.py --source add/image6.jpg --pan 12 --seconds 20 --pace
result`, store stock-lopia): 16 of 261 tracks labelled and 2 of 30 facings
recognized before, 83 of 255 and 6 of 31 after. The pre-fix figure was taken by
restoring the old 0.60/0.50 gate in place, measuring, and reverting.
The remaining unconfirmed tracks are products the 12-SKU catalog has no view
for, not refusals.
Two session models shared one process. `/v1/*` and `/v1/stream` are the live
one; `/api/sessions*` and `/api/compare_live_vs_still` were the earlier model,
still constructed on every `create_app()` through `build_legacy_scans` even
though nothing called them. The web client and `stream_client.py` speak only
`/v1`.
Deleted: `observe.py` (ScanService/PerceptionService/PerceptionAdapter),
`yolo_adapter.py` (imported only by the legacy builder), `pipeline.py` and
`test_suite.py` (the old single-image CLI, reachable only from each other),
`test_observe.py` (its Session* tests), the eight legacy route handlers,
`build_legacy_scans` with its two facades, `create_fastapi_app`, and the
`scans` parameter on `create_app`.
`IdentityRankingTests` moved to `test_identity.py` before `test_observe.py` was
deleted: those five ranking rules are the live path's contract, not the legacy
model's. `test_api.py` keeps every `/v1` and WebSocket test.
Verified: 246 tests OK (2 skipped), and a route dump of a freshly built app
shows no `/api/*` path with all nine `/v1` routes and the WebSocket intact.
The crop lane asked the phone for a full-resolution crop of a facing the
frame could not decide. Four defects made that unsafe or useless:
* A capability was half-applied. `crop: true` without a usable
`max_crop_bytes` enabled the lane on the client's say-so, leaving the two
sides believing different limits. A capability is now believed whole or
refused with BAD_MESSAGE, and the limit must be an integer in 1..1048576
(bool is an int in Python and is not a limit).
* A small JPEG could claim enormous dimensions. The header is now read
without decoding and anything over 4 megapixels is refused before the
decode allocates it.
* A request was single-use, so an expired one stranded a facing only the
crop lane could decide. The lane now spends the same max_attempts budget
the frame path spends, refilled on method change and on settled-label
drift, and a phone that never answers still cannot loop.
* A candidate could be asked about from a box the frame path would refuse:
clipped by the frame edge, carried over from an earlier frame, or measured
at a score the tracker does not trust. Both lanes now share one edge gate,
and only a track whose box this frame actually measured is asked about.
Drift and method change also cleared `identity` but left `confirmed = True`,
which permanently blocked both the crop lane and label inheritance for that
track; both now clear it, without which the budget reset was dead code.
Blur and glare are refused as whole-capture faults, while `too_far` is
exempt on purpose: a distant shelf is the one frame a native-resolution crop
exists to decide.
The head is now part of a store rather than a script: `finetune_identity.py --stock
stock-lopia` trains on that store's enrolled views (27 views over 7 products, 12
augmented copies each) and writes `<stock>/identity_head.pt`, which ProductGallery
loads automatically.
The first attempt exposed a trap worth recording. A discriminative head has no "none
of the above": trained with an angular margin to separate seven products, it maps
everything confidently onto those seven, including cartons it has never seen.
raw features, threshold 0.55 home shelf 78% accepted foreign shelves 5% accepted
trained head, threshold 0.40 home shelf 58% accepted foreign shelves 44% accepted
Rejection - the gallery's whole point - disappears. The fix needs no retraining: the
two mechanisms fail in opposite directions, so raw features decide *whether* to answer
(they generalise) and the head decides *what* to answer (it discriminates). Measured:
47% accepted on the home shelf, 5% on foreign ones. Lower coverage than raw-only,
which is the honest trade.
The head also needs its own threshold (0.40 against the raw 0.55): a trained head's
similarity scale is its own, and reusing the raw line rejected almost everything it
said.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Part 2 ended on the number that matters: enrolling a real shelf from itself leaves
off-the-shelf DINOv2 at ~25% top-1, because real milk cartons differ by small text
and a colour band. This trains a small projection head on frozen features - 384 -> 128,
cosine classifier, angular margin - with augmentation standing in for the second visit
we do not have, and measures it on 72 held-out facings over 8 products.
raw DINOv2 cosine top-1 2/72 (2%) balanced 1%
raw, rejecting below 0.55 top-1 2/72 (2%) balanced 1% (5% of answered right)
fine-tuned + margin top-1 28/72 (38%) balanced 73%
fine-tuned, rejecting below 0.4 top-1 17/72 (23%) balanced 58% (94% of answered right)
So fine-tuning is the difference between working and not (balanced 1% -> 73%), and
with a rejection threshold the model becomes precise - it answers 18 of 72 facings and
is right on 94% of them, which is the operating point a store wants. The raw model
cannot do that at any threshold.
Where it fails is where my labels are coarse: MEGMILK, which the contact sheet shows
is probably several products under one brand, scores 1/32 because the class is
internally inconsistent and the model correctly refuses most of it.
Honest limit, stated in the doc: enrolment and held-out facings come from one
photograph, so these are upper bounds; only a recorded second visit settles the real
number. What is settled is the mechanism - a head trained on labels the pipeline can
produce itself moves fine-grained identity from useless to usable.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The external evaluation showed the matcher cannot say "unknown": on 20 shelves whose
products are absent it claimed 47.8% of facings as recognised, and its raw inlier
score cannot be thresholded (foreign shelves' median 6.0 overlaps the real shelf's
5.2 p10, and the worst offenders score 50-190). So the gallery becomes a selectable
identity backend:
--identity gallery (or DORAYAKI_IDENTITY=gallery)
Measured on the same 20 foreign shelves: false accepts fall from 47.8% to 13.8% at
the default threshold and to **5%** at 0.60, with a smooth curve to pick an operating
point from. Detection is unchanged (the gallery only names boxes).
What it does NOT fix, measured on a milk shelf where 14 of 107 facings really are
明治 おいしい牛乳 (labelled by eye): precision is 28-45%, no better than the matcher's
39%, and enrolling the shelf from itself (1-3 views per product) is flat at ~25%
top-1. Off-the-shelf DINOv2 is not discriminative enough for fine-grained SKUs - the
same conclusion the literature reaches, now measured here.
The packaging's own text is a real channel and resolution-bound: 51% recall
single-scale, 60% with the tag reader's multi-scale + red-channel trick, 85% where
the text is large (MEGMILK) and 28% where it is small vertical Japanese.
docs/EXTERNAL-EVAL.md records all of it, including the identity architecture it
implies: reject by default, name from the tag when readable, from the packaging text
when legible, from a fine-tuned embedding otherwise, and hand the rest to the
operator.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Everything else here is measured on two photographs. This collects real store
shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY,
CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on
them. 46 of 54 images carry a shelf; the rest are book scans and a shop front.
What it found, and it is not what I expected:
- **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves,
spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong
grocery - tight boxes on images the model has never seen, and correctly nothing on
the book scans.
- **Identity transfers when the product is genuinely there.** On a 2011 milk-case
photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and
R-1 elsewhere. The matcher does reach new photographs of products it has templates
for.
- **Precision is the failure, and rejection is missing.** In that same photograph it
labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 -
they are all white milk cartons. Across 24 shelves whose products are not in the
catalog it accepted 451 identities, and the score cannot be thresholded to fix it:
the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median
is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and
189.9 (deer cookies). A threshold that catches the foreigners also throws away the
real matches.
That confirms on real data what the literature and the review predicted: the
matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the
failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection -
is the fix, not a refinement.
docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the
images themselves are not redistributed. 175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The deep review's first recommendation was real data, because every accuracy number
so far comes from one or two photographs - the identity benchmark even takes its
gallery and its queries from the same photo. This is the mechanism for replacing
that.
`--record-frames DIR` writes each processed frame, sampled every Nth, with a JSON
line describing what the server saw in it: the boxes, the identity channels and the
tag reading. The lines are the labelling input - `labelling_candidates()` returns
the facings the tag channel named with a corroborated price, which are seeds for the
confirmation screen rather than guesses. Nothing here is a training set on its own.
Sampling keeps each frame's own number in the index, so a sampled recording says so
instead of hiding it, and `/v1/health` reports how much has been written. Measured
on a 163-frame walk: 32 frames and their lines.
`docs/DATA.md` is the runbook: what to capture (a slow walk per rail, a close pass
for the barcode, and the same shelf on a second visit), how to turn it into labels,
and how to freeze a benchmark with a cross-visit split.
167 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The owner's brainstorm thought through to an implementable design, backed by
measurements taken against the real demo images rather than assumptions:
- Tag readability is a capture-distance problem first. At image.jpg's ~2 m the
tags are ~14 px digits and no engine reads them; at add/image6.jpg's ~1 m
(22 px digits) macOS Vision reads all five tag fields including the large red
sale price, and RapidOCR reads the printed fields at 0.97-1.00 confidence but
misses the stylised one. Tesseract jpn is unusable on tags.
- Engine choice is an interface with backends: macOS Vision (182 ms/rail band)
as the Mac fast path, RapidOCR (0.75-1.3 s/band, cross-platform) as the
default elsewhere, fake for tests, PaddleOCR held in reserve. Cloud OCR is
rejected: offline demo, and no owner decision to ship shelf images out.
- OCR must never sit in the frame loop: a TagService runs rail-band passes on a
cadence with a spatial cache and multi-pass voting (tags are static), which
also fixes digit misreads.
- Packaging text confirms 67% of the catalog templates (measured) but only
10%/3% on the demo shelves, because the demo catalog is fictional and the
photos are a real store. Absence is therefore neutral, never a contradiction,
and the demo needs one shelf whose tags and catalog describe the same store.
- Compliance rules fail closed on evidence quality: a row whose tags are merely
unreadable yields a capture hint, never a violation. The owner's "group of
items missing price tags" is the missing_tag_group rule (>= 3 consecutive
facings without a tag).
- Barcode/JAN is the exact-identity path, currently undecodable at demo
distance; the catalog needs optional jan and regular_price columns.
- The mobile lane can start now against protocol v1: tap-to-inspect works with
today's schema, and the tag fields are additive when they land.
Also records the five open questions for the owner (capture assets, engine
default, mobile order, catalog columns, parallel start).
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Detection and classification share the CPU, and leaving both uncapped made them
fight: uncapped torch plus 8 workers measured 11.1 results/s single-client and
dropped to 6.3/s each with two clients. Capping torch at 4 threads and
classifying with 5 workers gives 12.2 results/s single-client (46 ms detection)
and 8.5/s each for two clients, with zero unexplained label changes.
Detection is the part that must stay real-time; classification fills in behind
it, so the defaults now protect detection's share of the CPU. Both remain
overridable (--classify-workers, DORAYAKI_TORCH_THREADS).
docs/API.md records the shipped numbers and says which configuration produced
them, including the honest gap: detection during a live pan is ~46-48 ms, not
the 28 ms of an idle measurement.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
docs/API.md is the full contract: run flags and environment, auth, coordinate
convention, the identity vocabulary, every REST endpoint and WebSocket message
with examples, the backpressure rule, what a client may rely on, the design
decisions with their rejected alternatives, measured performance, known
limitations, and notes for the future mobile client.
README.md (Vietnamese) now documents the client-server shape, both modes, the
flags, the API, the test commands, and the source layout. It also repairs the
PowerShell escape damage in the original file (backtick-r had become a line
break, backtick-b a backspace, backtick-a a bell), which had left the code
fences and file paths mangled.
test_real_model.py is the opt-in integration test (DORAYAKI_REAL_MODEL=1): it
guards the measured baseline (170 detections, 105 recognized on image.jpg) and
proves a pan keeps labels stable with no unexplained changes.
run_app.sh / run_app.bat pass arguments through to the server.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
