feat(identity): local-feature matcher plus text lane as the default backend
Replaces the DINO gallery as the recognition path with the measured design from docs/research/2026-09-28-recognition-architecture/DECISION.md: RootSIFT over a per-SKU reference pyramid, similarity-constrained RANSAC and a hue check, fused with an OCR text lane that vetoes a match whose package text names a different catalog row and answers on its own when geometry finds nothing. - identity_lanes.py: LocalLane, TextLane, DecisionPolicy. The text lane builds each row's signature from the catalog name plus its packshot OCR, so a row with no enrolled image is still answerable. - engine.py: every backend groups scores by canonical SKU, so no view-suffixed id reaches the wire and a name and price always resolve. decide_crop fuses the lanes; score_crop_lanes feeds the live path. Both lanes are built in warm_up (non-fatally) so the first frame is not the slow one, and the text lane shares the engine's OCR lock. - live.py: decides per crop on the lane's own rank order. The lane's order comes from its rank key and its value is only the gate, so re-sorting by value named a different SKU than the lane chose; averaging in an empty observation also diluted a good confidence below the accept line. - identity.py: a `local` score scale whose gate is one absolute line, because the value is already a margin over the runner-up. - app.py, run-demo-server.sh: `--identity local` is the default; the demo server no longer starts on the leaked `stock-lopia` catalog. - eval_live.py: the replay vocabulary accepts `local`, so a recording from the shipped server can be scored. - README.md: documents the three backends and what each is for. Measured on the corrected image6 labels at live resolution: 36/61 in-catalog facings named at 95% precision, 0 of 48 unknowns accepted, 11 labels sourced from the text lane alone. Through the real server: /v1/recognize 90% precision, /v1/stream 100% precision, no view-suffixed ids on either.
This commit is contained in:
parent
25dbf132e7
commit
aaed5d388f
44 changed files with 7847 additions and 60 deletions
|
|
@ -0,0 +1,601 @@
|
|||
# Dorayaki product recognition: architecture decision
|
||||
|
||||
Audited 2026-09-28 · submodule `25dbf13` on `fix/live-recognition-quality` plus
|
||||
its uncommitted diff (snapshot: `git stash create` → `2785c07`) · auditor:
|
||||
Claude Opus 5.5
|
||||
Scope: identity pipeline (matcher, gallery, decision, evaluator, catalogs,
|
||||
labels), live track/label logic, the uncommitted diff. Not audited: detector
|
||||
quality, tag/planogram pipeline, iOS/Android rendering, on-device runtime.
|
||||
Evidence scripts and the corrected labels: [`evidence/`](evidence/README.md).
|
||||
|
||||
## Verdict
|
||||
|
||||
Build a **local-feature instance matcher plus a text lane, behind a calibrated
|
||||
open-set decision, with no global embedding in the critical path**. At the
|
||||
current scale the matcher runs exhaustively; past ~150 SKUs, a
|
||||
RootSIFT descriptor index replaces exhaustive matching. Generic DINOv2 is
|
||||
rejected for both classification and retrieval.
|
||||
|
||||
The biggest risk is none of the architectures. It is that **there is no
|
||||
trustworthy benchmark**. The only labelled data is one photo, `add/image6.jpg`,
|
||||
and 26 of its 111 labels (23 %) were wrong. Every number in the handoff was
|
||||
computed against those labels. Label and catalog errors are larger than any
|
||||
architecture difference measured so far. They also hide the real product
|
||||
failure: displayed-label precision is 29–43 % because same-brand siblings the
|
||||
catalog doesn't know about get labelled with confidence.
|
||||
|
||||
On corrected labels, with thresholds tuned on the same single photo (so read
|
||||
this as direction, not proof), the recommended design beats Van's contract on
|
||||
coverage and on precision together:
|
||||
|
||||
| Live resolution, clean 7-SKU catalog, 61 in-catalog + 48 unknown facings | Correct shown | Coverage (of 61) | Precision of shown labels | Unknowns labelled |
|
||||
|---|---|---|---|---|
|
||||
| Van's matcher, always argmax (the floor) | 35 | 57 % | 35 % | 48 / 48 |
|
||||
| Current server gate (`decide_identity` defaults) | 32 | 52 % | 43 % | 22 / 48 |
|
||||
| Improved matcher, geometric-confidence gate only (τ=2) | 26 | 43 % | 79 % | — |
|
||||
| **Improved matcher + OCR veto + OCR lane (τ=2)** | **36** | **59 %** | **95 %** | **0 / 48** |
|
||||
| Same, looser (τ=1) | 40 | 66 % | 87 % | 2 / 48 |
|
||||
|
||||
The one experiment that would most change this recommendation is a second,
|
||||
independently captured and labelled shelf visit. §7 defines it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem formulation
|
||||
|
||||
Dorayaki's identity task is **open-set instance recognition over a small,
|
||||
store-scoped, enrolled catalog, with live-video evidence accumulation**.
|
||||
|
||||
- **Instance, not category.** The classes are individual printed artworks
|
||||
(JAN-level SKUs). Siblings share maker, palette, lattice pattern, mascot and
|
||||
silhouette, and differ by a word ("コクうま 65%"), a colour (pink vs red
|
||||
lattice), or a small logo. That is the regime of planar instance matching
|
||||
(logo and text geometry), not of semantic embeddings. Measured: raw DINOv2 is
|
||||
12–21 % top-1 on 7 SKUs; the local matcher is 76–98 %.
|
||||
- **Retrieval-shaped at scale, closed-set-shaped today.** With tens of SKUs,
|
||||
every enrolled reference can be verified exhaustively. With thousands, it
|
||||
becomes retrieval plus verification (§6).
|
||||
- **Open set is the dominant difficulty, and it comes in two kinds.** *Far*
|
||||
unknowns (other categories) are rejected easily: 0.2–0.7 % false accepts on
|
||||
602 unrelated crops at a threshold that keeps most correct labels. *Near*
|
||||
unknowns (siblings of enrolled products: Kewpie 450 g next to Kewpie からし,
|
||||
Pure Select plain next to Pure Select コクうま) produce genuine,
|
||||
geometrically consistent evidence and can't be thresholded away. Near
|
||||
unknowns get fixed by **catalog completeness** and by **text**, not by a
|
||||
better score.
|
||||
- **Multimodal by nature.** Text is the designed discriminator between
|
||||
siblings; manufacturers print the variant name precisely so shoppers can
|
||||
tell them apart. OCR identifies the right SKU among all 12 catalog rows on
|
||||
41–42 of the 55 facings where it reads anything, including SKUs with no
|
||||
enrolled image, using only catalog names and packshot text.
|
||||
- **Temporal.** Live video gives many cheap, noisy looks at one physical
|
||||
facing. The right model is accumulated evidence per track, with a
|
||||
provisional → confirmed label contract. The current model is one accepted
|
||||
observation that locks the label forever.
|
||||
|
||||
So the hierarchy is: store/category scope → candidate generation (trivial
|
||||
now, descriptor index later) → local geometric verification + text
|
||||
verification → calibrated open-set decision → temporal accumulation.
|
||||
|
||||
## 2. Evidence audit
|
||||
|
||||
`R` reliable · `C` directionally useful but confounded · `X` invalidated ·
|
||||
`U` still unknown. "Reproduced" means re-run in this audit with the scripts in
|
||||
`evidence/`.
|
||||
|
||||
| # | Handoff claim | Verdict | What this audit found |
|
||||
|---|---|---|---|
|
||||
| 2 | Van's matcher is SIFT/ORB/BRISK + RANSAC, always argmax, no rejection | R | Scoring core in `code/feature_matcher.py` is byte-for-byte Van's (`git diff b443e3c`); only `decide_identity` was added. "SURF" is SIFT (SURF isn't in this OpenCV build). |
|
||||
| 2 | "170/170 named ≠ 170/170 correct" on image.jpg | R (logic) / U (numbers) | No labels exist for `image.jpg`; its accuracy was never measured. |
|
||||
| 3 | Current server collapsed onto hub SKUs (sku_003, sku_l09) | C | Hubness is real for DINO (50–60 % max-SKU share on the clean catalog, reproduced). The l09 collapse was mostly a wrong l09 reference plus shelf-cut templates. |
|
||||
| 3 | Live resolution degrades identity | X | Controlled rerun: with correctly scaled boxes, live is not worse than full (original matcher 35/42 live vs 32/42 full; improved 36 vs 40). |
|
||||
| 3 | DINO embeddings are inappropriate for fine-grained SKU identity | R | Reproduced: gallery top-1 21 % full / 12 % live, hub alarm. |
|
||||
| 4 | `stock-lopia` was corrupt, leaked, and incomplete | R | Worse than reported: its crops come from the eval photo, so scoring against it is memorization. It scores 94–97 % "identity", and the evaluator's leak check misses it because the re-cut crops aren't the same size. |
|
||||
| 7 | `eval_identity.py` is the right research direction | C | Right skeleton, four more defects (F-07). |
|
||||
| 8 | Bug #1, template IDs vs SKU IDs | R, and **still live in production** | Fixed only in the evaluator. The live path accepts `sku_l01_1`, then finds no catalog row for it (F-02). |
|
||||
| 9 | Hybrid 80 % > matcher 70 % | X | About one facing, on wrong labels. |
|
||||
| 10 | Bug #2, unscaled boxes caused the live collapse | R | Reproduced on *both* catalogs: unscaled boxes give 11–17 %, scaled boxes give 76–97 %. The confound is resolved. The coordinate fix, not the catalog cleanup, removed the collapse. |
|
||||
| 11 | Clean packshots are a dramatic improvement | R, with a caveat | True against the leak-free baseline. The l06 packshot shows a superseded package design (F-12); the l09 packshot is only 305 px tall. |
|
||||
| 12 | Bug #3, the reject gate inside the shortlist | R | Hybrid-49 %-vs-20 % result is invalid. |
|
||||
| 13 | DINO top-K sweep: recall 71 %/100 %, K doesn't hurt the matcher | C | On a 7-SKU catalog, top-5 is 71 % of the catalog; that sweep can't speak to retrieval. Reproduced on v2: top-3 recall 79–81 %, top-5 95–98 %. |
|
||||
| 14 | Matcher ~8 ms vs hybrid ~45 ms | R, with a correction | The hybrid lane scores the **full** catalog and then filters (`eval_identity.py` `HybridLane.rank`), so its latency is DINO + full matcher by construction. The measurement is honest; it doesn't describe a real retrieval design. |
|
||||
| 15 | 69 % full / 66 % live after the l09 fix | R as arithmetic, X as a measurement | Reproduced exactly (24/35, 23/35) **on v1 labels**. On corrected labels the same matcher and catalog give 71 %/76 % after the gate and 76 %/83 % ungated top-1 ranking. The 69/66 gap that justified `crop_first_attempt` doesn't exist. |
|
||||
| 16 | Local matcher is the strongest identity primitive | R | And improvable: multi-scale RootSIFT + colour re-rank 40/42 vs 32/42 at full (sign test p = 0.04); at live 36 vs 35 (noise). |
|
||||
| 16 | DINO as coarse retriever at scale | X (this setting) | Recall@10 falls 42 → 32 → 17 → 7 of 42 as the catalog grows 7 → 57 → 207 → 493 SKUs. A RootSIFT descriptor index holds 37–38/42 recall@5 over the same range. |
|
||||
| 17 | Threshold alone can't create correct identities | R | And the missing identities are mostly catalog gaps (§3). |
|
||||
| 17 | Argmax precision ~20 % | C | 29 % full / 35 % live on corrected labels; the old number counted mislabelled facings. |
|
||||
| 21 | Live tests: 49 tests, 1 failure, 5 errors | X (stale) | `python -m unittest test_live` now passes 49/49 on both the dirty tree and the committed baseline. One test feeds a `str` result that raises inside a future callback and is swallowed, on both trees (F-13). |
|
||||
| 22 | "High confidence" list items 1–8 | R | All hold. Item 4 is now measured with corrected labels. |
|
||||
| 22 | Medium confidence 1 (69 %/66 %) | X | See row 15. |
|
||||
| 22 | Medium confidence 3 (more views help) | R direction | Leave-facing-out proxy (§3, E-5) supports it strongly, with a same-photo caveat. |
|
||||
| 22 | Not established 1–9 | U | 1 and 4 are now partly answered (§6); 2 partly (RootSIFT/multi-scale beat Van's composite); 3, 5, 6, 7, 8, 9 remain open. Item 9 (ground truth) was the most consequential open item: it was wrong. |
|
||||
|
||||
### The four ceilings, separated
|
||||
|
||||
On corrected labels (`evidence/image6_v2.json`): 109 scored facings, 61
|
||||
in-catalog across 11 SKUs, 48 unknown, 2 excluded. The clean catalog enrolls 7
|
||||
SKUs covering 42 of the 61 in-catalog facings.
|
||||
|
||||
| Ceiling | Value (live, improved matcher) | What limits it |
|
||||
|---|---|---|
|
||||
| A. Catalog | 42 / 61 = 69 % of in-catalog facings are answerable by image | l10 (11 facings), l07 (6), l04, l08 unenrolled; l12 has **zero** real facings on the shelf |
|
||||
| B. Retrieval | n/a at 7 SKUs; RootSIFT index recall@5 = 38/42 at 493 SKUs | the 4 misses are catalog-quality cases (l06 package version) |
|
||||
| C. Identity (ungated top-1 on enrolled) | 36 / 42 (86 %, Wilson 72–93 %) | l06 at live (1/4), two l09 facings that are mostly occluded |
|
||||
| D. Decision | 26 correct at 79 % precision (geometry only); 36 at 95 % (+ text) | near-unknown siblings: 9 of 11 unenrolled l10 facings → l09, Kewpie 450 g → l03 |
|
||||
|
||||
## 3. Root-cause model
|
||||
|
||||
Ordered by how much of the gap to a good product each class explains.
|
||||
|
||||
**Evaluator and ground-truth problems** (they decided every conclusion so far)
|
||||
- 23 % of v1 labels were wrong: 14 in-catalog facings marked unknown (Egg Care
|
||||
#4–5, Fitte #22–23, Lucky #27–28, Kokuuma #40–45, Pure Select #59); 9
|
||||
non-catalog items labelled as catalog SKUs (demi-glace pouches #31–32 as l09,
|
||||
a garlic jar #16 as l05, a Heinz can #64 as l10, dressing bottles #76–79 as
|
||||
l12, Kewpie 450 g #85 as l03); 2 neighbour shifts (#29 l08→l07, #75
|
||||
l12→l11). The pattern (runs shifted along a shelf row, a `match` IoU field on
|
||||
every row) looks like labels propagated from shelf-tag or box matching rather
|
||||
than read from the packages.
|
||||
- The evaluator reports `top1` and `coverage` as the same quantity (accepted
|
||||
correct / enrolled), so the identity ceiling and the decision ceiling were
|
||||
never separated.
|
||||
- Precision excludes unenrolled in-catalog facings, which are the most
|
||||
dangerous false accepts.
|
||||
- The gallery's candidate set comes from ground-truth labels.
|
||||
- Leak detection only catches same-size crops.
|
||||
|
||||
**Catalog and data problems** (the largest recognition lever)
|
||||
- Incomplete category enrollment is what creates near unknowns. In the
|
||||
leave-facing-out proxy (E-5), adding one shelf view for each of the 11 SKUs
|
||||
on the shelf takes argmax from 34/52 to 47/52. The number that survives at
|
||||
95 % precision goes from 14–16 to 32–39.
|
||||
- Wrong or stale references: the l09 history; the l06 package is a superseded
|
||||
design (the match holds at full resolution, fails at live: 1/4); the l09
|
||||
packshot is 305 px, so its scores dwarf others under Van's formula.
|
||||
- Metadata integrity: `stock-lopia-clean/stock.csv` has no `sku_l09` row for
|
||||
`sku_l09.jpg`. Its `sku_l10` row carries l09's stock counts with l10's name
|
||||
and price. The catalog name for l12 ("焼肉のたれ 中辛", ¥299) probably
|
||||
names the ユーラス 旨たれ bottles on the shelf (tag ¥299). Several catalog
|
||||
sizes disagree with the shelf tags (l10 tag says 400 g).
|
||||
|
||||
**Identity-model problems** (real but second-order on this data)
|
||||
- Scale mismatch: ~1000 px packshots vs 110–250 px crops is a 4–8× ratio.
|
||||
ORB's 8×1.2 pyramid tops out at 3.6×, so ORB contributes **zero** inliers at
|
||||
live resolution, and BRISK nearly none. Van's weighted composite is
|
||||
effectively SIFT-only there, carrying dead weight.
|
||||
- Van's score, `inliers·1.5 + inliers/min(kp)·80`, isn't comparable across
|
||||
templates of different resolution. With fewer than 4 matches it counts raw
|
||||
ratio-test matches as "inliers" without any geometric check (`compute_geometric_inliers`).
|
||||
- Grayscale-only: pink (ゼロ) vs red (ハーフ) lattice is invisible to SIFT. The
|
||||
hue re-rank fixes 2–3 facings in every configuration.
|
||||
|
||||
**Open-set decision problems**
|
||||
- `decide_identity` for the inlier scale accepts a score ≥ 1.0 with a relative
|
||||
margin ≥ 0.15. A single unverified match clears the floor, and the margin
|
||||
can't reject a sibling with a strong partial match. It accepts 22–28 of 48
|
||||
unknowns.
|
||||
|
||||
**Live and tracking problems** (code-path evidence, unmeasured on device)
|
||||
- A confirmed track is never re-evaluated (`_on_classified` returns early when
|
||||
`track.confirmed`), and with `vote_agree = 1` the first accepted observation
|
||||
locks it. A wrong first label is permanent. A later, sharper, correct view
|
||||
can't correct it.
|
||||
- Aggregation is the mean over the last three observations, with empty
|
||||
observations counted as zeros. That's a heuristic, not evidence
|
||||
accumulation.
|
||||
- Uncommitted diff: the crop blur gate returns without consuming an attempt,
|
||||
so a permanently soft crop (a distant facing) is retried every `retry_gap`
|
||||
frames for as long as it stays in view.
|
||||
|
||||
**UX latency problems**
|
||||
- No provisional label exists on the wire or in the clients, so any multi-look
|
||||
confirmation becomes dead time on screen. That's why `vote_agree = 2` was
|
||||
(correctly) backed out.
|
||||
|
||||
## 4. Architecture candidates
|
||||
|
||||
"Recognition" = does it improve correct shown labels at fixed precision?
|
||||
"Scalability" = does it keep latency and recall flat as the catalog grows?
|
||||
|
||||
| ID | Candidate | Tested here? | Recognition | Scalability | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| A | Van's matcher, full reference set, argmax | yes | 32–35 / 42 identity; 29–35 % precision | linear, ~0.1–0.25 ms per reference | Floor. Not shippable (no open set). |
|
||||
| B | Matcher + engineering: RootSIFT, per-SKU reference pyramid (200/400 px tall), SKU grouping, similarity-constrained RANSAC, hue feature, calibrated gate | yes | 40/42 full (p = 0.04 vs A), 36/42 live; gate alone reaches 79–85 % precision at 36–43 % coverage | same as A | **Adopt** as identity core. |
|
||||
| C | Learned local features: ALIKED/DISK/SuperPoint + LightGlue; dense LoFTR/RoMa | no (weights not installed; downloading needs your approval) | Unknown. Headroom on enrolled facings is ≤ 6/42 at live, mostly catalog-caused. Likely strongest on small or blurred crops and package-version drift. | LightGlue is per-pair; tens of ms per pair on GPU and much more on CPU, so only viable as a top-K verifier | Postpone to experiment E-C. Adopt only if it clears the gate. |
|
||||
| D | Retrieval → Van's matcher | yes (DINO) | DINO recall collapses with scale (7/42 @10 at 493 SKUs) | DINO adds ~41 ms per crop | **Reject DINO retrieval.** |
|
||||
| D′ | RootSIFT descriptor index (FLANN) → vote → verify top-10 | yes | 35–37/42, equal to or slightly better than exhaustive | 1.7–3.9 ms per crop flat from 7 to 493 SKUs; build 171 ms for 64k descriptors | **Adopt past ~150 SKUs.** |
|
||||
| E | Retrieval → learned verifier | no | — | — | Only if C wins. |
|
||||
| F | OCR text lane against catalog text signatures | yes (macOS Vision) | Veto: precision 62 → 82 % at τ=0.5 for −1 correct. As a lane: +10 correct at live, including unenrolled SKUs. | ~18–26 ms per crop on Mac; on-device on iOS (Vision) and Android (ML Kit) | **Adopt.** |
|
||||
| G | Fusion (geometry + colour + text + temporal + tag prior) | partly (geometry + colour + text) | Largest measured gain | — | **Adopt**; the text and geometry errors are complementary (text reads siblings apart; geometry covers text-poor faces). |
|
||||
| H | Store-trained identity head on DINO | no (it was trained on the eval shelf) | Leaked; no valid measurement exists | — | Retire with DINO. |
|
||||
| I | Barcode/JAN | — | Front faces rarely show JANs; shelf tags carry product text, not scannable codes, at scan distance | — | Only for the enrollment tool (scan a JAN, pull the packshot). |
|
||||
| J | Shelf-tag / planogram prior | no (`tags.py` exists) | Every facing run on image6 sits above a readable tag with name and price | — | Strong hypothesis, H-02. |
|
||||
|
||||
## 5. Recommended architecture (current scale: tens of SKUs)
|
||||
|
||||
```
|
||||
frame / native crop
|
||||
│
|
||||
├─ local lane: RootSIFT on crop ─► verify against EVERY reference view
|
||||
│ (exhaustive while refs ≲ 300), similarity-RANSAC,
|
||||
│ per-SKU evidence = max over views:
|
||||
│ {inliers, 2nd-best ratio, inlier hull cover, hue agreement}
|
||||
│
|
||||
├─ text lane: OCR (upsampled ×2 when crop height < 200 px)
|
||||
│ ─► bigram similarity to every catalog row's text signature
|
||||
│ (name + OCR of its packshot), including rows with no image
|
||||
│
|
||||
▼
|
||||
decision: P(correct | local, text) from a small calibrated model
|
||||
accepted if P ≥ p_confirm (target precision ≥ 95 %)
|
||||
provisional if P ≥ p_provisional (shown as tentative)
|
||||
veto if text names a different catalog row by margin
|
||||
unknown otherwise
|
||||
│
|
||||
▼
|
||||
track: log-odds accumulation per track across looks; confirm on
|
||||
threshold or text agreement; re-open on strong contradiction
|
||||
(hysteresis), never on a single weak frame
|
||||
```
|
||||
|
||||
Why each piece earns its place:
|
||||
- **Local lane.** It's the only primitive that ranks fine-grained siblings
|
||||
correctly: 36–40 of 42 enrolled facings, against 5–9 for DINO. It costs
|
||||
~4–10 ms per crop at 7 SKUs.
|
||||
- **Reference pyramid (200/400 px) and RootSIFT.** Together they fix the scale
|
||||
mismatch that kills ORB/BRISK and makes scores comparable across references.
|
||||
Drop ORB/BRISK unless ablation H-08 finds a case they win.
|
||||
- **Hue feature.** It's the cheapest fix for colour-coded variants (+2–3
|
||||
facings everywhere).
|
||||
- **Text lane.** It's the only measured mechanism that rejects near-unknown
|
||||
siblings, and it can name catalog SKUs that have no image yet. It
|
||||
contributed +10 correct labels and 0 unknown accepts at live, τ=2.
|
||||
- **Calibrated decision.** It replaces a hand-set floor that accepts one
|
||||
unverified match. It's calibrated on Tier-2 data, never on the photo it's
|
||||
evaluated on.
|
||||
- **No DINO, no retrieval stage.** Neither adds recognition at this scale, and
|
||||
DINO actively fails at larger scale. Exhaustive verification is cheaper than
|
||||
one DINO forward pass until ~300–400 references.
|
||||
|
||||
Placement: keep the matcher server-side for now; it's small and deterministic.
|
||||
Put OCR on the phone when tag and label OCR move there. Apple Vision and ML
|
||||
Kit are the official platform paths, matching the repo's `platform/` rule.
|
||||
Send the text with the crop so the server fuses both. Until then, run OCR on
|
||||
the server (macOS Vision is already wired in `code/ocr.py`).
|
||||
|
||||
## 6. Scaling architecture
|
||||
|
||||
Measured cost and recall to ~500 SKUs; beyond that, extrapolated and marked.
|
||||
|
||||
| Catalog | Candidate generation | Verification | Evidence |
|
||||
|---|---|---|---|
|
||||
| 10–150 SKUs | none (exhaustive) | local + text | 3.7 ms at 7 SKUs, 7.7 ms at 57, 21 ms at 207 (small refs). Packshot pyramids cost ~2–3× per reference. |
|
||||
| 150–5,000 | RootSIFT descriptor index (FLANN now; FAISS IVF-PQ when memory matters), vote per SKU, top-10 | local + text on the shortlist | Measured flat 2–4 ms and recall@5 38/42 up to 493 SKUs. Descriptor memory at 5k SKUs × 2 views × ~500 descriptors ≈ 5 M × 128 B (uint8) ≈ 640 MB before PQ. **Extrapolated.** |
|
||||
| 5,000–100,000 | Scope first: store assortment → aisle/bay (tag or planogram prior) cuts candidates to hundreds. Then an aggregated-local-feature index (ASMK/VLAD-style) or a product-instance embedding fine-tuned on packshot↔shelf pairs. | local + text on ≤ 20 | **Hypothesis.** No store stocks 100k SKUs at once. Scoping by store and bay is the real scaling lever. Generic DINO is ruled out; a fine-tuned embedding needs paired training data that doesn't exist yet. |
|
||||
|
||||
Exhaustive matching stops being sensible when its p95 exceeds the per-crop
|
||||
budget. At ~0.1–0.25 ms per reference and a 30 ms budget, that's roughly
|
||||
150–300 references (≈ 75–150 SKUs at 2 views). **Descriptor indexing removes
|
||||
the need for a global DINO pass**: at 493 SKUs it retrieves better than DINO
|
||||
at a tenth of the cost. The open question is same-category growth: hundreds of
|
||||
mayonnaises instead of other-category distractors. That's experiment E-6.
|
||||
|
||||
## 7. Experimental plan
|
||||
|
||||
Rule for every experiment: change one variable, run on fixed crops and a fixed
|
||||
catalog hash, log per facing (truth, enrolled?, per-SKU evidence vector, truth
|
||||
rank, decision, lane latencies), and report the four ceilings separately.
|
||||
|
||||
### Benchmark tiers (build these first; they block every other decision)
|
||||
|
||||
- **Tier 1, diagnostic:** `evidence/image6_v2.json` (109 facings). Dev only.
|
||||
Free to tune on. Its numbers never ship.
|
||||
- **Tier 2, held-out regression:** new captures that are never enrolled and
|
||||
never tuned on. At least 3 visits across ≥ 2 LOPIA locations or days, same
|
||||
aisle, both still photos and the phone's own 1920×1080 frames. Target ≥ 300
|
||||
in-catalog facings over ≥ 20 SKUs, ≥ 300 near unknowns from the same aisle,
|
||||
and the 602-crop far-unknown set from `add/`.
|
||||
- Labels need provenance: shelf-tag correspondence plus readable package
|
||||
text, two independent passes, disagreements marked `exclude`.
|
||||
- Every label row carries `source`, `confidence`, and `labeler`. A label
|
||||
file without them fails a schema check.
|
||||
- **Tier 3, live replay:** FrameRecorder walks (`code/record.py`) of the same
|
||||
aisles with Tier-2-style labels, replayed through `code/eval_live.py`.
|
||||
|
||||
### Experiments
|
||||
|
||||
| ID | Question | Variable | Metric | Acceptance / falsifier |
|
||||
|---|---|---|---|---|
|
||||
| E-0 | Are the labels right? | none | Double-labelling agreement on Tier 2 | κ ≥ 0.9 on SKU identity. Otherwise fix labels before anything else. |
|
||||
| E-1 | Does B beat A on held-out data? | matcher variant | Paired identity top-1 on enrolled facings, McNemar | B ≥ A with p < 0.05 at native-crop resolution. If A ≥ B, keep Van's scoring and add only SKU grouping and the decision layer. |
|
||||
| E-2 | Is ORB/BRISK dead weight? | drop ORB/BRISK from B | top-1, latency | Drop them if top-1 changes by ≤ 1 facing per 100. |
|
||||
| E-3 | Does text move the frontier? | ± text lane | Correct shown at precision ≥ 95 %; near-unknown FAR | Text lane adds ≥ 5 pp coverage at 95 % precision. Otherwise text becomes veto only, or is removed. |
|
||||
| E-4 | Calibrated decision vs hand threshold | decision policy | Coverage at 95 % precision, calibration (ECE) on Tier 2 | Model ECE ≤ 0.05 and coverage ≥ hand threshold's. |
|
||||
| E-5 | How much is catalog completeness worth? | enroll the aisle's remaining SKUs with packshot + one shelf view from a *different* visit | Coverage at 95 % precision; near-unknown FAR | ≥ 70 % of the same-photo proxy gain (14→32 full, 16→39 live) survives cross-visit. If not, shelf views are overfitting to capture conditions. |
|
||||
| E-6 | Same-category scaling | grow the catalog with other sauce/mayo SKUs (packshots) | top-1, recall@K of descriptor index vs exhaustive; latency p50/p95 | Index recall@10 ≥ 98 % of exhaustive top-1 at ≥ 300 SKUs. Otherwise stay exhaustive and parallelize. |
|
||||
| E-C | Learned matcher | ALIKED+LightGlue (or DISK+LightGlue) as verifier over B's top-5 | top-1, coverage at 95 % precision, latency | Adopt only if +3 pp coverage at 95 % precision **and** p95 identity latency ≤ 80 ms server-side. |
|
||||
| E-7 | Live temporal model | log-odds accumulation + provisional labels vs current single-shot lock | Tier 3: time to provisional, time to correct confirmed, flips, correct labels lost | Time-to-first-correct p50 ≤ current single-shot; flips ≤ 2 %; correct-then-lost ≤ 1 %. |
|
||||
| E-8 | Does first-attempt native crop help? | `crop_first_attempt` on/off | Tier 3 correct confirmed labels, crop traffic | Keep only if +3 pp correct at ≤ 1.5× crop traffic. On Tier 1 it has no basis: live ≥ full. |
|
||||
| E-9 | Tag prior | ± shelf-tag evidence from `tags.py` | Coverage at 95 % precision | Adopt if +5 pp; also use it for catalog-gap discovery. |
|
||||
|
||||
## 8. Migration strategy
|
||||
|
||||
Snapshot first: the uncommitted work is saved as stash object `2785c07` (and as
|
||||
`tracked.diff` / `untracked.tgz` in the audit scratchpad). Nothing has been
|
||||
reverted.
|
||||
|
||||
| Item | Class | Action |
|
||||
|---|---|---|
|
||||
| `code/eval_identity.py` (untracked) | evaluation infrastructure | **Keep and fix** (F-07). |
|
||||
| `gallery.py` `rank()` (diff) | evaluation infrastructure | Keep, eval-only; no production caller. |
|
||||
| `run-demo-server.sh` `--identity matcher` (diff) | clearly correct production fix | **Keep.** Also change the default `STOCK` away from `stock-lopia` (F-03). |
|
||||
| `live.py` `crop_first_attempt`, `max_attempts 2→4` | experimental | **Revert.** Its stated rationale (69 vs 66) is invalidated. Re-test as E-8. |
|
||||
| `live.py` blur gate early return, `sharpen_ratio`, `best_sharpness` | experimental, with a retry hazard | **Revert.** Re-introduce inside the E-7 evidence model, where a soft crop is a low-weight observation, not a free retry. |
|
||||
| `live.py` `vote_agree` + `_agreed_observations` | experimental | **Revert** (default 1 makes it dead code). E-7 replaces it. |
|
||||
| `test_live.py` change | encodes the reverted budget | Revert with it. |
|
||||
| `code/rebuild_catalog.py` (untracked) | catalog tooling | Keep. Add image↔CSV bijection, packshot-OCR vs name check, and eval-session refusal (enrollment protocol below). |
|
||||
| `stock-lopia-clean/` (untracked) | catalog data | Keep the images and provenance. Fix `stock.csv` (F-04) and the README's "Six SKUs" (it lists five). Replace the l06 packshot with the current package. |
|
||||
| `eval/labels/image6.json` (untracked) | ground truth | **Replace** with `evidence/image6_v2.json`, schema: `label`, `source`, `note`, `v1_label`. |
|
||||
| `identity.decide_identity` inlier branch | production | **Rewrite** as the calibrated decision (E-4); keep the cosine branch only while the gallery exists. |
|
||||
| Production score path (`engine.score_crop`, `classify_boxes`, `inspect`) | production | **Rewrite:** return per-SKU evidence grouped by canonical SKU (F-02). |
|
||||
| `feature_matcher.py` composite score | production | **Rewrite** into variant B behind an interface; keep Van's scorer as a selectable baseline lane for E-1. |
|
||||
| Identity interface | — | **Isolate:** `IdentityLane.evidence(crop) -> {sku: Evidence}` with `LocalLane`, `TextLane`, and `GalleryLane` (postponed); `DecisionPolicy` separate; `TrackEvidence` in live. |
|
||||
| DINO gallery, identity head, `finetune_identity.py` | production code | **Postpone and fence off:** not in the live path, not the default; delete after E-6 if nothing depends on it. |
|
||||
| Descriptor index | — | **Postpone** until the catalog passes ~150 SKUs (E-6). |
|
||||
| Provisional label on the wire + clients | protocol + native UI | **Postponed dependency** of E-7. Shared model gets a `provisional` identity state; Compose and SwiftUI render it as tentative. |
|
||||
|
||||
### Enrollment protocol
|
||||
|
||||
- **Views per SKU:** at least one current-package front packshot, stored as a
|
||||
200/400/800 px-tall pyramid (source ≥ 800 px tall), plus at least one real
|
||||
shelf capture at scan distance from a store visit that is **not** an
|
||||
evaluation session. Add side or back views only for SKUs displayed rotated.
|
||||
- **Minimum usable resolution:** reject any reference view under 160 px tall
|
||||
at its largest pyramid level. Crops that small carry no text geometry
|
||||
(observed: facings under 60 px wide fail every lane).
|
||||
- **Metadata:** JAN, maker, product name as printed, size, package version or
|
||||
date, source URL, capture session id, SHA-256.
|
||||
- The enrollment tool validates a one-to-one match between images and CSV
|
||||
rows.
|
||||
- It OCRs each packshot and requires the catalog name's distinctive tokens
|
||||
to appear (this check would have caught the l09/l10 mix-up and flagged the
|
||||
l06 package version).
|
||||
- It flags cross-SKU near-duplicates as "text-discriminated siblings".
|
||||
- **Leakage:** enrollment refuses any image whose capture session id appears
|
||||
in an evaluation manifest. Pixel equality is not a leak test; re-cut crops
|
||||
pass it.
|
||||
- **Versioning:** the catalog manifest has a content hash; every eval report
|
||||
records catalog hash, label-file hash, and code SHA.
|
||||
- **Completeness rule:** a bay is "live-ready" when every SKU on its shelf tags
|
||||
is enrolled or explicitly listed as text-only. Near-unknown precision
|
||||
depends on this more than on any threshold.
|
||||
|
||||
## 9. Completion gate
|
||||
|
||||
Measured on **Tier 2 held-out** (and Tier 3 for time-to-label). Same crops,
|
||||
same catalog hash, same detector for both systems. "Van" means Van's scorer
|
||||
with always-argmax, run on that same catalog.
|
||||
|
||||
| Gate | Definition | Pass |
|
||||
|---|---|---|
|
||||
| G1 coverage | correct shown labels ÷ all in-catalog facings (enrolled or not) | new ≥ Van; lower 95 % bound of the paired-bootstrap difference ≥ −2 pp |
|
||||
| G2 correctness | correct shown ÷ all shown | ≥ 95 % (Van: 29–35 % on Tier 1) |
|
||||
| G3 unknown handling | near-unknown false-accept rate; far-unknown false-accept rate | ≤ 5 %; ≤ 1 % (Van: 100 %) |
|
||||
| G4 time to label | Tier 3: p50 / p95 seconds from a facing entering view to a correct label shown (provisional counts only if later confirmed as the same SKU) | p50 ≤ 1.0 s and p95 ≤ 2.5 s, and p50 no worse than Van's click-to-label on a still (tap + ~50–100 ms) |
|
||||
| G5 stability | confirmed labels that later change; correct labels that disappear | ≤ 2 %; ≤ 1 % |
|
||||
| G6 per-SKU floor | recall of every enrolled SKU with ≥ 5 Tier-2 facings | ≥ 60 % (catches l06-type catalog faults) |
|
||||
| G7 hubness | max predicted-SKU share ÷ that SKU's true share | ≤ 2.0 |
|
||||
| G8 runtime | server identity latency p95 per crop; server memory | ≤ 60 ms; within the demo host's budget |
|
||||
|
||||
Not complete until G1–G7 pass on the device path (Tier 3), not just offline.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
| ID | Finding | Severity | Evidence | Priority | Source |
|
||||
|---|---|---|---|---|---|
|
||||
| F-01 | image6 ground truth 23 % wrong | blocking | VERIFIED | P0 | open-world |
|
||||
| F-11 | No held-out benchmark exists (one photo) | blocking | VERIFIED | P0 | open-world |
|
||||
| F-02 | Production scores are per template, not per SKU → self-ambiguity and nameless labels | blocking | VERIFIED | P0 | open-world |
|
||||
| F-03 | Demo server defaults to the leaked, corrupt `stock-lopia` | blocking | VERIFIED | P0 | open-world |
|
||||
| F-04 | Clean catalog CSV ↔ image mismatch (l09/l10) | blocking | VERIFIED | P0 | open-world |
|
||||
| F-05 | Near-unknown siblings accepted; shown-label precision 29–43 % | blocking | VERIFIED | P0 | open-world |
|
||||
| F-07 | Evaluator defects beyond bugs #1–#3 | major | VERIFIED | P1 | open-world |
|
||||
| F-06 | Matcher scoring defects (scale, dead ORB/BRISK, unverified "inliers", grayscale) | major | VERIFIED | P1 | open-world |
|
||||
| F-08 | DINO unfit as classifier and retriever here | major | VERIFIED | P1 | open-world |
|
||||
| F-09 | Confirmed labels never re-evaluated; first accept locks | major | HIGH CONFIDENCE | P1 | open-world |
|
||||
| F-10 | Uncommitted live diff: unbounded blur retries; rationale invalidated | major | HIGH CONFIDENCE | P1 | open-world |
|
||||
| F-12 | l06 packshot is a superseded package design | major | VERIFIED | P2 | open-world |
|
||||
| F-13 | Stale handoff test status; swallowed callback exception in a test fixture | polish | VERIFIED | P3 | open-world |
|
||||
|
||||
Counts: 11 VERIFIED / 2 HIGH CONFIDENCE / 0 HYPOTHESIS (hypotheses listed
|
||||
separately) · 13 open-world. No reference baseline was used: this audit
|
||||
covers a recognition backend. The product-level floor is Van's system (G1–G4).
|
||||
|
||||
### Workstreams
|
||||
|
||||
1. **Evidence (single owner, first):** F-01, F-11, F-07. Blocks every
|
||||
architecture change.
|
||||
2. **Catalog (parallel with 1):** F-03, F-04, F-12, enrollment tool checks.
|
||||
3. **Identity core (single owner, after 1):** F-02, F-06, text lane, decision
|
||||
layer, E-1–E-4.
|
||||
4. **Live (after 3's interface exists):** F-10 revert now; F-09 and the
|
||||
provisional label (E-7) later. The protocol plus native-UI part touches the
|
||||
KMP app and needs its own owner.
|
||||
5. **Scaling (after 3):** E-6, and the descriptor index only when it's needed.
|
||||
|
||||
### F-01 — image6 labels are 23 % wrong — P0 · blocking · VERIFIED
|
||||
- **Root cause:** labels appear propagated from tag/box matching (every row has
|
||||
a `match` field) rather than read from packages.
|
||||
- **Evidence:** `evidence/image6-v1-labels-overlay.jpg`; the per-index list in §3.
|
||||
Rescoring the same runs under v2 moves ungated live top-1 from 26/35 to 35/42.
|
||||
- **Target:** every scored label carries provenance; ambiguous facings are excluded.
|
||||
- **Change:** adopt `evidence/image6_v2.json`; schema-check `label/source/note`.
|
||||
- **Validation:** E-0 double-labelling on Tier 2.
|
||||
- **Acceptance:** no architecture claim cites v1.
|
||||
|
||||
### F-02 — template-level scores in production — P0 · blocking · VERIFIED
|
||||
- **Root cause:** `ProductFeatureMatcher.score_crop` keys scores by file stem;
|
||||
`engine.score_crop` (`code/engine.py:424-448`), `classify_boxes`, and `inspect` pass
|
||||
them straight to `decide_identity`. The live payload looks up
|
||||
`identity.sku_id` in the catalog, which has only canonical ids.
|
||||
- **Evidence:** on `stock-lopia` (the demo default), 22/109 facings at full
|
||||
and 29/109 at live are "ambiguous" only because the top two are views of the
|
||||
same SKU. 68 of 69 (full) and 56 of 58 (live) accepted ids are
|
||||
view-suffixed (`sku_l01_1`), which have no catalog row, so the payload
|
||||
carries no `product` block.
|
||||
- **Target:** scores grouped by canonical SKU (max over views) before any
|
||||
decision; the id shown is always a catalog row.
|
||||
- **Change:** group at the lane boundary (the future `IdentityLane.evidence`).
|
||||
Share one canonicalizer with `rebuild_catalog.sku_of`.
|
||||
- **Validation:** unit test: two strong views of one SKU → accepted, canonical id.
|
||||
- **Acceptance:** zero view-suffixed ids on the wire.
|
||||
|
||||
### F-03 — demo default catalog is the leaked one — P0 · blocking · VERIFIED
|
||||
- **Evidence:** `run-demo-server.sh:11` `STOCK="${1:-stock-lopia}"`. That
|
||||
catalog is shelf crops cut from the eval photo, and it scores 94–97 %
|
||||
"identity" on its own exam.
|
||||
- **Change:** default to the clean catalog once F-04 is fixed; refuse catalogs
|
||||
without a provenance manifest.
|
||||
|
||||
### F-04 — clean catalog metadata mismatch — P0 · blocking · VERIFIED
|
||||
- **Evidence:** `stock-lopia-clean/stock.csv` has no `sku_l09` row for
|
||||
`sku_l09.jpg`; its `sku_l10` row has l09's counts (30/13/17) with l10's name
|
||||
and price. Every l09 answer ships without product info; l10 has a row but no
|
||||
image. The README says "Six SKUs … have no official pack shot" and lists five.
|
||||
- **Change:** restore the l09 row from `stock-lopia/stock.csv`, drop the
|
||||
mangled l10 counts, fix the README, and add the bijection check to
|
||||
`rebuild_catalog.py`.
|
||||
|
||||
### F-05 — near-unknown siblings accepted — P0 · blocking · VERIFIED
|
||||
- **Evidence:** the current gate accepts 22–28/48 unknowns; 9/11 unenrolled
|
||||
l10 facings go to l09 with margin 0.67–0.93 (higher than many correct l11
|
||||
matches); Kewpie 450 g → l03 with 17–28 SIFT inliers.
|
||||
- **Target / change:** the decision layer + text veto (§5) + enrollment
|
||||
completeness (E-5).
|
||||
- **Acceptance:** G2, G3.
|
||||
|
||||
### F-06 — matcher scoring defects — P1 · major · VERIFIED
|
||||
- **Evidence:** ORB inliers are 0 on nearly every live crop (per-facing log
|
||||
from `diag.py`). `compute_geometric_inliers` returns `len(good_matches)`
|
||||
unverified when there are fewer than 4. The l09 reference (305 px) scores
|
||||
85–129 while others score 10–40.
|
||||
- **Change:** variant B (§5), behind the lane interface, with Van's scorer kept
|
||||
as a baseline lane.
|
||||
- **Acceptance:** E-1, E-2.
|
||||
|
||||
### F-07 — evaluator defects — P1 · major · VERIFIED
|
||||
- `top1` ≡ `coverage` (both are accepted-correct ÷ enrolled), so the identity
|
||||
and decision ceilings were never separated.
|
||||
- Precision's denominator excludes unenrolled in-catalog facings
|
||||
(`covered` in `main`): it reports 50–53 % where the honest figure is 39–43 %.
|
||||
- The gallery's SKU set comes from ground-truth labels (`sku_ids` from
|
||||
`held`).
|
||||
- Leak check requires the same shape.
|
||||
- `HybridLane` scores the full catalog, so its latency isn't a retrieval
|
||||
design.
|
||||
- `group_by_sku` only handles the `sku_l` prefix.
|
||||
- **Change:** report the four ceilings explicitly (the per-facing log in
|
||||
`diag.py`/`variants.py` is the template). Use the catalog, not labels, for
|
||||
candidate sets. Refuse leakage by session manifest.
|
||||
|
||||
### F-08 — DINO — P1 · major · VERIFIED
|
||||
- **Evidence:** gallery top-1 21 %/12 % with 50–60 % hub share; recall@10 of
|
||||
7/42 at 493 SKUs (`dino_scale.py`); 41 ms per embedding.
|
||||
- **Change:** fence off (§8). The uncommitted `--identity matcher` default is
|
||||
the right call.
|
||||
|
||||
### F-09 — no identity re-evaluation — P1 · major · HIGH CONFIDENCE
|
||||
- **Evidence:** `code/live.py` `_on_classified`: `if track.state !=
|
||||
STATE_DETECTED or track.confirmed: return`. With `vote_agree = 1` the first
|
||||
accepted observation confirms.
|
||||
- **Target:** E-7's evidence model with provisional labels and hysteresis.
|
||||
- **Validation:** Tier 3 flips and correct-then-lost metrics.
|
||||
|
||||
### F-10 — uncommitted live experiments — P1 · major · HIGH CONFIDENCE
|
||||
- **Evidence:** the blur gate `return`s before `track.attempts += 1`, so
|
||||
eligibility never runs out for a soft crop. `crop_first_attempt` cites the
|
||||
invalidated 69/66 numbers.
|
||||
- **Change:** revert per §8.
|
||||
|
||||
### F-12 — l06 package version — P2 · major · VERIFIED
|
||||
- **Evidence:** the packshot shows "内臓脂肪ケア フィッテ"; the shelf shows
|
||||
"内臓脂肪を減らす キユーピーフィッテ". Improved matcher 4/4 at full, 1/4 at
|
||||
live; 3/3 with a shelf reference.
|
||||
- **Change:** enroll the current package; adopt the packshot-OCR check.
|
||||
|
||||
### F-13 — stale test status — P3 · polish · VERIFIED
|
||||
- **Evidence:** 49/49 pass on both trees. An `AttributeError: 'str' object has
|
||||
no attribute 'items'` is raised and swallowed inside a future callback in one
|
||||
test, on both trees.
|
||||
- **Change:** make that fixture return a dict, or assert the callback path.
|
||||
|
||||
## Hypotheses
|
||||
|
||||
| ID | Suspected | Why it matters | The check that settles it |
|
||||
|---|---|---|---|
|
||||
| H-01 | Learned matchers (ALIKED/DISK + LightGlue) beat B on small, blurred, or re-designed packages | Could lift the identity ceiling | E-C (needs weight download approval) |
|
||||
| H-02 | Shelf-tag OCR prior raises coverage and precision | Tags are readable above every run on image6 | E-9 using `code/tags.py` |
|
||||
| H-03 | Cross-visit shelf references retain most of the leave-facing-out gain | Decides the enrollment protocol | E-5 |
|
||||
| H-04 | Evidence accumulation shortens time-to-correct-label vs single-shot lock | G4/G5 | E-7 on Tier 3 |
|
||||
| H-05 | On-device OCR is fast enough (< 30 ms per crop) and as accurate at native crop resolution | Where the text lane runs | Benchmark Vision on iPhone and ML Kit on Android over Tier-2 crops |
|
||||
| H-06 | Stream JPEG quality costs SIFT inliers at live | Cheap fix if true | Rerun `variants.py` on re-encoded crops at the client's quality |
|
||||
| H-07 | The hue feature is robust to store lighting | The colour gain could be lighting-specific | Tier 2 across stores and times |
|
||||
| H-08 | ORB/BRISK removal costs nothing | Simplifies and speeds the matcher | E-2 at native-crop resolution, where ORB helped the small l09 reference |
|
||||
| H-09 | Full-resolution OCR with ×2 upsampling matches live's text-lane gain | Full-res fusion was weaker (75 % precision) with ×1 OCR | Rerun `ocr_exp.py` + `fuse_eval.py` with full ×2 |
|
||||
|
||||
## Preserved decisions
|
||||
|
||||
- The matcher kept as a selectable baseline lane. The shipped default is now
|
||||
`local` (variant B plus the text lane), which is what the measurements below
|
||||
support; `--identity matcher` reproduces Van's scorer for E-1.
|
||||
- Bounded, drift-checked crop lane protocol (`aa25f99`, `13a4e0f`); its tests
|
||||
pass.
|
||||
- `eval_identity.py`'s architecture-neutral lane framing and explicit
|
||||
enrollment ceiling.
|
||||
- Clean-catalog provenance table and the `rebuild_catalog.py` refusal rules.
|
||||
- `vote_agree = 1` without provisional labels. Backing out `2` was right.
|
||||
|
||||
## Implementation record (2026-09-28)
|
||||
|
||||
What §8 asked for, and where it landed. Verified by the suite (251 tests) and by
|
||||
running the real server against `add/image6.jpg`.
|
||||
|
||||
| §8 item | Status |
|
||||
|---|---|
|
||||
| F-02 per-SKU scores in `score_crop`, `classify_boxes`, `inspect` | Done. `group_by_sku` on every backend; no view-suffixed id reaches the wire. |
|
||||
| F-03 demo default catalog | Done. `run-demo-server.sh` defaults to `stock-lopia-clean`. |
|
||||
| F-04 clean catalog CSV | Done. All 12 rows; `sku_l09` added, `sku_l10` corrected. |
|
||||
| F-05 near-unknown siblings | Done via the text lane. 0 of 48 unknowns accepted at live resolution. |
|
||||
| F-06 variant B behind a lane interface | Done. `identity_lanes.LocalLane`; Van's scorer stays selectable as `--identity matcher`. |
|
||||
| F-07 evaluator defects | Done. Four separate ceilings, honest precision denominator, catalog-derived candidate set, `sku_of` grouping, leak signature. |
|
||||
| F-08 DINO fenced off | Done. Not the default, not in the live path, refuses to start without weights. |
|
||||
| F-10 uncommitted live experiments | Reverted. `crop_first_attempt`, the blur early return, `vote_agree` and `max_attempts 4` are gone. |
|
||||
| F-12 l06 current package | **Blocked on a capture.** Kewpie serves the superseded design for JAN 4901577083600, so the current package cannot be enrolled from the official URL. The text lane ranks l06 first on all 4 shelf facings but abstains below the accept line, so no wrong label is shown. |
|
||||
| F-13 stale status / swallowed exception | Done. The fixture returns the matcher's `(scores, details)` shape; zero swallowed callback exceptions. |
|
||||
| F-09 re-evaluation with provisional labels | **Postponed as §8 specifies** — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet. |
|
||||
|
||||
Two defects found while wiring the lanes into the live path, both fixed:
|
||||
`live` re-sorted the local lane's scores by value, which discards the lane's own
|
||||
rank order (order comes from `rank_key`, the value is only the gate), and the
|
||||
early return on an empty local observation made the text lane's "answer on its
|
||||
own" job unreachable. Live now decides per crop on the lane's own order.
|
||||
|
||||
Measured after the change, through the real server: `/v1/recognize` names 41 of
|
||||
112 detections at 90% precision with no view-suffixed ids; the live WebSocket
|
||||
names 12 at 100% precision with 0 unknown accepts, first label on frame 3.
|
||||
Offline on the corrected labels: 36/61 (59%) at 95% precision, 0/48 unknowns,
|
||||
11 of those labels sourced from the text lane alone.
|
||||
|
||||
Still open, and not implementable from here: Tier 2 held-out labels (F-11), the
|
||||
E-1..E-9 experiments, and the completion gate G1-G7, all of which need a second
|
||||
independent shelf visit.
|
||||
|
||||
## Out of scope
|
||||
|
||||
Detector recall and box quality (all experiments use the same boxes). Tag and
|
||||
planogram pipeline internals. Native UI. On-device performance. Learned local
|
||||
features (not installed; downloading them needs your approval). `image.jpg` /
|
||||
`stock/` (no labels exist). Timings from parallel runs are relative;
|
||||
`scale.py` timings are single-process on this Mac.
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Evidence scripts (2026-09-28 audit)
|
||||
|
||||
Read-only against product code: every script imports from `code/` and writes
|
||||
only into this folder. Run from `recognization/` with `.venv/bin/python`.
|
||||
|
||||
`image6_v2.json` is the corrected image6 label set (26 of 111 v1 labels
|
||||
changed; every row carries `v1_label`, `source`, and a `note` where changed).
|
||||
`image6-v1-labels-overlay.jpg` shows the v1 labels on the shelf; green = catalog
|
||||
label, red = unknown. `sheet.py` regenerates per-SKU contact sheets.
|
||||
|
||||
| Step | Command (prefix `E=docs/research/2026-09-28-recognition-architecture/evidence`) | Produces |
|
||||
|---|---|---|
|
||||
| Original matcher, both catalogs, full/live/live-with-unscaled-boxes | `.venv/bin/python $E/diag.py --stock stock-lopia-clean --labels $E/image6_v2.json [--live] [--bug] --out $E/run_stock-lopia-clean_<full\|live\|livebug>.pkl` (repeat with `--stock stock-lopia`) | `run_*.pkl` |
|
||||
| Score them under v1 or v2 labels | `.venv/bin/python $E/analyze.py <labels.json> $E/run_*.pkl` | identity top-1, argmax, current gate |
|
||||
| Matcher variants | `.venv/bin/python $E/variants.py --stock stock-lopia-clean --labels $E/image6_v2.json [--live] --scales 200,400 --root --geom affine --out $E/v_ms_root_aff_<full\|live>.pkl` (also `--tmpl-h 250/400`, no flags = raw) | `v_*.pkl` |
|
||||
| Variant ranking + open-set frontier | `.venv/bin/python $E/an2.py $E $(basename -a $E/v_*.pkl)` (`LABELS=image6_v2_lfo.json` for the leave-facing-out runs) | correct at 100/95/90 % precision |
|
||||
| Leave-facing-out catalog | `.venv/bin/python $E/lfo_setup.py`, then `variants.py --stock $E/cat_<pack\|full> --labels $E/image6_v2_lfo.json --scales 200,400 --root --geom affine [--live] --out $E/lfo_cat_<pack\|full>_<full\|live>.pkl` | `lfo_*.pkl` |
|
||||
| Unrelated-shelf distractors | `YOLO_OFFLINE=1 .venv/bin/python $E/distract.py stock-lopia-clean $E/cat_full` | `distract.pkl` |
|
||||
| Catalog growth, exhaustive vs descriptor index | `YOLO_OFFLINE=1 .venv/bin/python $E/scale.py` | stdout |
|
||||
| DINO retrieval under growth | `YOLO_OFFLINE=1 .venv/bin/python $E/dino_scale.py` | stdout |
|
||||
| OCR read-rate | `.venv/bin/python $E/ocr_exp.py` | `ocr.pkl` |
|
||||
| OCR signatures, then fusion | `.venv/bin/python $E/ocr_fuse.py && .venv/bin/python $E/fuse_eval.py` | `sig.pkl`, stdout |
|
||||
| Per-SKU contact sheets + the labelled shelf | `.venv/bin/python $E/sheet.py <outdir>` | `gt_<label>.jpg`, `shelf_labels.jpg` |
|
||||
|
||||
Timings in DECISION.md come from `scale.py` (single process). The other
|
||||
scripts ran up to ten processes at once, so their ms figures are relative only.
|
||||
|
||||
## Verified 2026-09-28
|
||||
|
||||
Every command above was run from `recognization/` and produced its listed
|
||||
output. Two rows were wrong as written:
|
||||
|
||||
- The table said `python`; the `python` on this machine's PATH has no `cv2`, so
|
||||
every row failed on import. It is `.venv/bin/python`.
|
||||
- `an2.py` takes the folder and then file names *inside* it, so `v_*.pkl`
|
||||
expanded in the working directory, where those files do not live. It is
|
||||
`$(basename -a $E/v_*.pkl)`.
|
||||
|
||||
`sheet.py` was missing from the table. It takes its output directory as a
|
||||
positional argument, not a flag: `sheet.py --help` reads `--help` as a
|
||||
directory name, writes nothing, and still exits 0. `analyze.py` and `an2.py`
|
||||
with no arguments raise `IndexError`; neither validates its own usage.
|
||||
|
||||
v1 labels are no longer a file on disk — `eval/labels/image6.json` is the v2
|
||||
set. To score a run under v1, rebuild it from the `v1_label` field:
|
||||
|
||||
.venv/bin/python -c "import json;d=json.load(open('$E/image6_v2.json'));\
|
||||
[f.update(label=f.pop('v1_label')) for f in d['facings']];\
|
||||
json.dump(d,open('/tmp/image6_v1.json','w'),ensure_ascii=False)"
|
||||
|
||||
That reconstruction reproduces DECISION.md's v1 numbers exactly: `analyze.py`
|
||||
gives ungated top-1 26/35 at both resolutions and the current gate 24/35 full,
|
||||
23/35 live (§2 row 15, F-01).
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
import pickle, json, sys, numpy as np, collections
|
||||
import os; S = sys.argv[1]; L = {f['index']: f['label'] for f in json.load(open(f'{S}/' + os.environ.get('LABELS','image6_v2.json')))['facings']}
|
||||
RANK = {
|
||||
'inl': lambda e: e['inliers'],
|
||||
'inl*col': lambda e: e['inliers'] * (0.25 + e['color']),
|
||||
}
|
||||
def conf_feats(ranked):
|
||||
(k1, e1), (k2, e2) = ranked[0], ranked[1]
|
||||
i1, i2 = e1['inliers'], e2['inliers']
|
||||
return {
|
||||
'inl': i1,
|
||||
'margin': i1 - i2,
|
||||
'ratio': 1 - i2 / max(i1, 1),
|
||||
'inl*ratio': i1 * (1 - i2 / max(i1, 1)),
|
||||
'cover': e1['cover'],
|
||||
'inl*ratio*qcov': i1 * (1 - i2 / max(i1, 1)) * np.sqrt(e1.get('qcover', 0)),
|
||||
'inl*ratio*col': i1 * (1 - i2 / max(i1, 1)) * e1['color'],
|
||||
}
|
||||
def evaluate(path, rank='inl'):
|
||||
r = pickle.load(open(path, 'rb')); E = set(r['enrolled'])
|
||||
rows = [x for x in r['rows'] if L[x['index']] != 'exclude']
|
||||
n_in = sum(L[x['index']] != 'unknown' for x in rows); n_en = sum(L[x['index']] in E for x in rows)
|
||||
recs = []; top1 = 0
|
||||
for x in rows:
|
||||
t = L[x['index']]
|
||||
if not x['ev']: recs.append((t, None, {k: 0 for k in conf_feats([('a', dict(inliers=0,cover=0,color=0)),('b',dict(inliers=0,cover=0,color=0))])})); continue
|
||||
ranked = sorted(x['ev'].items(), key=lambda kv: -RANK[rank](kv[1]))
|
||||
b = ranked[0][0] if ranked[0][1]['inliers'] > 0 else None
|
||||
if t in E and b == t: top1 += 1
|
||||
recs.append((t, b, conf_feats(ranked)))
|
||||
out = dict(top1=f'{top1}/{n_en}', argmax_e2e=f'{top1}/{n_in}')
|
||||
for f in ['inl', 'margin', 'ratio', 'inl*ratio', 'inl*ratio*qcov', 'inl*ratio*col']:
|
||||
pts = sorted(((c[f], t == b and b is not None) for t, b, c in recs if b), key=lambda p: -p[0])
|
||||
best = {0.9: 0, 0.95: 0, 1.0: 0}; c = a = 0; i = 0
|
||||
while i < len(pts):
|
||||
j = i
|
||||
while j < len(pts) and pts[j][0] == pts[i][0]: c += pts[j][1]; a += 1; j += 1
|
||||
for p in best:
|
||||
if c / a >= p: best[p] = max(best[p], c)
|
||||
i = j
|
||||
out[f] = '/'.join(str(best[p]) for p in (1.0, 0.95, 0.9))
|
||||
return out
|
||||
if __name__ == '__main__':
|
||||
print('feature cells = correct accepted at precision 100%/95%/90% (enrolled in-catalog n=42, all in-catalog n=61, unknown n=48)')
|
||||
for p in sys.argv[2:]:
|
||||
for rk in ['inl', 'inl*col']:
|
||||
print(p.split('/')[-1].replace('.pkl',''), rk, json.dumps(evaluate(f'{S}/{p}', rk)))
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
import pickle, json, sys, collections, numpy as np
|
||||
sys.path.insert(0,'code')
|
||||
from identity import decide_identity
|
||||
def load_labels(p): return {f['index']: f['label'] for f in json.load(open(p))['facings']}
|
||||
def feats(sc):
|
||||
r = sorted(sc.items(), key=lambda kv: -kv[1])
|
||||
if not r: return None, 0.0, 0.0
|
||||
top = r[0][1]; run = r[1][1] if len(r) > 1 else 0.0
|
||||
return r[0][0], top, (top-run)/max(top,1.0)
|
||||
def analyze(run, labels, verbose=False):
|
||||
E = set(run['enrolled']); rows = [x for x in run['rows'] if labels[x['index']] != 'exclude']
|
||||
T = lambda x: labels[x['index']]
|
||||
inc = [x for x in rows if T(x) != 'unknown']; ince = [x for x in inc if T(x) in E]
|
||||
unk = [x for x in rows if T(x) == 'unknown']
|
||||
# identity ceiling: ungated argmax on enrolled in-catalog
|
||||
ranks = []
|
||||
for x in ince:
|
||||
order = [k for k, _ in sorted(x['scores'].items(), key=lambda kv: -kv[1])]
|
||||
ranks.append(order.index(T(x))+1 if T(x) in order and x['scores'].get(T(x),0)>0 else 99)
|
||||
top1 = sum(r == 1 for r in ranks)
|
||||
# Van contract: argmax always
|
||||
arg_correct = top1; answered = sum(1 for x in rows if x['scores'] and max(x['scores'].values())>0)
|
||||
# current decision rule
|
||||
acc = collections.Counter(); c = w = wu = wi = 0
|
||||
for x in rows:
|
||||
d = decide_identity(x['scores'], method='hybrid')
|
||||
b = d['best_sku'] if d['identity_level']=='accepted' else None
|
||||
if not b: continue
|
||||
acc[b]+=1
|
||||
if b == T(x): c += 1
|
||||
else:
|
||||
w += 1
|
||||
if T(x)=='unknown': wu+=1
|
||||
else: wi+=1
|
||||
hub = max(acc.values())/sum(acc.values()) if acc else 0
|
||||
out = dict(n=len(rows), in_catalog=len(inc), enrolled_in=len(ince), unknown=len(unk),
|
||||
identity_top1=f"{top1}/{len(ince)}={top1/max(1,len(ince)):.0%}",
|
||||
argmax_e2e=f"{arg_correct}/{len(inc)}={arg_correct/max(1,len(inc)):.0%}",
|
||||
argmax_precision=f"{arg_correct}/{answered}={arg_correct/max(1,answered):.0%}",
|
||||
gated=f"correct {c} wrong {w} (unk {wu}, in-cat {wi}) cov_enr {c/max(1,len(ince)):.0%} prec {c/max(1,c+w):.0%} hub {hub:.0%}",
|
||||
ms=round(float(np.median([x['ms'] for x in run['rows'] if x['ms']>0] or [0])),1))
|
||||
return out, ranks
|
||||
def sweep(run, labels, feature='margin'):
|
||||
"""Pareto of (correct accepted, precision) over a single threshold on a feature."""
|
||||
E=set(run['enrolled']); rows=[x for x in run['rows'] if labels[x['index']]!='exclude']
|
||||
pts=[]
|
||||
for x in rows:
|
||||
b,top,mar=feats(x['scores']); t=labels[x['index']]
|
||||
v={'margin':mar,'top':top,'both':min(mar/0.3, top/40)}[feature]
|
||||
pts.append((v, b==t and b is not None, b))
|
||||
pts.sort(key=lambda p:-p[0]); res=[]; c=a=0
|
||||
for v,ok,b in pts:
|
||||
a+=1; c+=ok; res.append((v,c,a,c/a))
|
||||
return res
|
||||
if __name__=='__main__':
|
||||
labels=load_labels(sys.argv[1])
|
||||
for p in sys.argv[2:]:
|
||||
r=pickle.load(open(p,'rb')); o,ranks=analyze(r,labels)
|
||||
print(p.split('/')[-1], json.dumps(o, ensure_ascii=False))
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
"""Diagnostic identity harness. Logs full per-facing score vectors so every
|
||||
ceiling (catalog, identity rank, decision) is separable. Read-only on product code."""
|
||||
import sys, os, json, time, argparse, pickle
|
||||
sys.path.insert(0, 'code')
|
||||
import cv2, numpy as np
|
||||
from feature_matcher import ProductFeatureMatcher
|
||||
|
||||
def sku_of(name):
|
||||
p = name.split('_'); return '_'.join(p[:2])
|
||||
|
||||
def crops(image_path, labels, live, bug=False, long_side=1280):
|
||||
im = cv2.imread(image_path); f = 1.0
|
||||
if live:
|
||||
h, w = im.shape[:2]; f = long_side / max(h, w)
|
||||
im = cv2.resize(im, (int(w*f), int(h*f)), interpolation=cv2.INTER_AREA)
|
||||
out = []
|
||||
for fc in labels['facings']:
|
||||
b = fc['box'] if bug else [int(round(v*f)) for v in fc['box']]
|
||||
x1, y1, x2, y2 = b
|
||||
out.append((fc, im[max(0,y1):y2, max(0,x1):x2]))
|
||||
return out
|
||||
|
||||
def run(stock, image, labels, live, bug, method):
|
||||
m = ProductFeatureMatcher(stock_dir=stock)
|
||||
rows = []
|
||||
for fc, c in crops(image, labels, live, bug):
|
||||
if c.size == 0 or min(c.shape[:2]) < 8: rows.append(dict(index=fc["index"], truth=fc["label"], shape=c.shape[:2], scores={}, templ={}, sub={}, ms=0.0)); continue
|
||||
t = time.perf_counter(); s, det = m.score_crop(c, method=method); dt = time.perf_counter()-t
|
||||
g = {}
|
||||
for k, v in s.items(): g[sku_of(k)] = max(g.get(sku_of(k), 0.0), v)
|
||||
sub = {k: det[k].get('sub_details') for k in det}
|
||||
rows.append(dict(index=fc['index'], truth=fc['label'], shape=c.shape[:2], scores=g, templ=s, sub=sub, ms=dt*1000))
|
||||
return dict(stock=stock, live=live, bug=bug, method=method, enrolled=sorted({sku_of(k) for k in m.db.templates}), rows=rows)
|
||||
|
||||
if __name__ == '__main__':
|
||||
ap = argparse.ArgumentParser(); ap.add_argument('--stock'); ap.add_argument('--labels'); ap.add_argument('--live', action='store_true')
|
||||
ap.add_argument('--bug', action='store_true'); ap.add_argument('--method', default='hybrid'); ap.add_argument('--out')
|
||||
a = ap.parse_args(); L = json.load(open(a.labels))
|
||||
r = run(a.stock, 'add/image6.jpg', L, a.live, a.bug, a.method)
|
||||
pickle.dump(r, open(a.out, 'wb'))
|
||||
print(a.out, len(r['rows']), 'rows', 'median ms', round(float(np.median([x['ms'] for x in r['rows']])),1))
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
import sys,os,glob,json,time,numpy as np,cv2,random
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__)); os.environ['YOLO_OFFLINE']='1'
|
||||
from gallery import ProductGallery
|
||||
from detector import ShelfProductDetector
|
||||
from diag import crops, sku_of
|
||||
from PIL import Image
|
||||
S=os.path.dirname(__file__); random.seed(0)
|
||||
g=ProductGallery('stock-lopia-clean',['sku_l01','sku_l02','sku_l03','sku_l05','sku_l06','sku_l09','sku_l11']); assert g.load()
|
||||
refs=[(sku_of(os.path.basename(p)[:-4]),g.embed(cv2.imread(p))) for p in sorted(glob.glob('stock-lopia-clean/*.jpg'))]
|
||||
det=ShelfProductDetector(); dis=[]
|
||||
for p in sorted(glob.glob('add/*')):
|
||||
if 'image6' in p: continue
|
||||
im=cv2.cvtColor(np.array(Image.open(p).convert('RGB')),cv2.COLOR_RGB2BGR)
|
||||
for j,b in enumerate(det.detect(im).get('boxes',[])):
|
||||
x1,y1,x2,y2=[int(v) for v in b['box']]; c=im[max(0,y1):y2,max(0,x1):x2]
|
||||
if min(c.shape[:2])>=40: dis.append((f'd{j}_{os.path.basename(p)}',c))
|
||||
random.shuffle(dis)
|
||||
D=[(k,g.embed(c)) for k,c in dis]
|
||||
L=json.load(open(f'{S}/image6_v2.json')); E={k for k,_ in refs}
|
||||
Q=[(fc['label'],g.embed(c)) for fc,c in crops('add/image6.jpg',L,True) if fc['label'] in E]
|
||||
t=time.perf_counter(); _=g.embed(Q and dis[0][1]); print('embed ms',round((time.perf_counter()-t)*1000,1))
|
||||
for N in [0,50,200,len(D)]:
|
||||
R=refs+D[:N]; M=np.stack([e for _,e in R]); names=[k for k,_ in R]
|
||||
rec={1:0,5:0,10:0,20:0}
|
||||
for truth,q in Q:
|
||||
s=M@q; order=[]
|
||||
for i in np.argsort(-s):
|
||||
if names[i] not in order: order.append(names[i])
|
||||
for K in rec: rec[K]+= truth in order[:K]
|
||||
print(f'skus={len(set(names)):4d} DINO recall@1/5/10/20 '+'/'.join(str(rec[K]) for K in rec)+f' of {len(Q)}')
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
import sys,os,glob,pickle,numpy as np,cv2,json
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__))
|
||||
os.environ['YOLO_OFFLINE']='1'
|
||||
from detector import ShelfProductDetector
|
||||
from variants import Matcher
|
||||
from PIL import Image
|
||||
det=ShelfProductDetector()
|
||||
res={}
|
||||
for cat in sys.argv[1:]:
|
||||
name=os.path.basename(cat); m=Matcher(cat,root=True,geom='affine',scales=(200,400)); rows=[]
|
||||
for p in sorted(glob.glob('add/*')):
|
||||
if 'image6' in p: continue
|
||||
im=cv2.cvtColor(np.array(Image.open(p).convert('RGB')),cv2.COLOR_RGB2BGR)
|
||||
for b in det.detect(im).get('boxes',[]):
|
||||
x1,y1,x2,y2=[int(v) for v in b['box']]; c=im[max(0,y1):y2,max(0,x1):x2]
|
||||
if c.size==0 or min(c.shape[:2])<20: continue
|
||||
rows.append(dict(img=p,ev=m.score(c)))
|
||||
res[name]=rows; print(name,len(rows),'distractor crops')
|
||||
pickle.dump(res,open(f'{os.path.dirname(__file__)}/distract.pkl','wb'))
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
import sys,os,json,pickle,csv,numpy as np
|
||||
S=os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0,S)
|
||||
from ocr_fuse import norm,sim # noqa (module runs OCR of packshots on import; cheap)
|
||||
sig=pickle.load(open(f'{S}/sig.pkl','rb')); O=pickle.load(open(f'{S}/ocr.pkl','rb'))
|
||||
L={f['index']:f['label'] for f in json.load(open(f'{S}/image6_v2.json'))['facings']}
|
||||
def conf(ev):
|
||||
r=sorted(ev.items(),key=lambda kv:-kv[1]['inliers']*(0.25+kv[1]['color']))
|
||||
(k1,e1),(k2,e2)=r[0],r[1]; i1,i2=e1['inliers'],e2['inliers']
|
||||
return k1, i1*(1-i2/max(i1,1))*e1['color']
|
||||
for res,up,run in [('full',1,'v_ms_root_aff_full.pkl'),('live',2,'v_ms_root_aff_live.pkl')]:
|
||||
ocr={r['index']:r['text'] for r in O[res] if r['up']==up}
|
||||
M={x['index']:x for x in pickle.load(open(f'{S}/{run}','rb'))['rows']}
|
||||
n_in=sum(1 for t in L.values() if t not in ('unknown','exclude'))
|
||||
# OCR-alone identity among ALL catalog SKUs
|
||||
ok=tot=0
|
||||
for i,t in L.items():
|
||||
if t in ('unknown','exclude') or i not in ocr: continue
|
||||
tx=ocr[i];
|
||||
if len(norm(tx))<4: continue
|
||||
s={k:sim(tx,v) for k,v in sig.items()}; tot+=1; ok+= max(s,key=s.get)==t
|
||||
print(f'== {res}: OCR-alone top-1 over all 12 catalog SKUs, on facings with text: {ok}/{tot}')
|
||||
for delta in [None,0.0,0.1,0.2]:
|
||||
line=[]
|
||||
for tau in [0.5,1,2,3]:
|
||||
c=a=0
|
||||
for i,t in L.items():
|
||||
if t=='exclude' or i not in M or not M[i]['ev']: continue
|
||||
k,cf=conf(M[i]['ev'])
|
||||
if cf<tau: continue
|
||||
if delta is not None and len(norm(ocr.get(i,'')))>=4:
|
||||
s={kk:sim(ocr[i],v) for kk,v in sig.items()}; j=max(s,key=s.get)
|
||||
if j!=k and s[j]-s[k]>delta: continue # text names a different catalog product: veto
|
||||
a+=1; c+= k==t
|
||||
line.append(f'tau{tau}: {c}/{n_in} p={c/max(a,1):.0%}')
|
||||
print(' veto' if delta is not None else ' matcher only', delta, ' | '.join(line))
|
||||
print('== union: matcher (veto) OR OCR-only identity')
|
||||
for res,up,run in [('full',1,'v_ms_root_aff_full.pkl'),('live',2,'v_ms_root_aff_live.pkl')]:
|
||||
ocr={r['index']:r['text'] for r in O[res] if r['up']==up}
|
||||
M={x['index']:x for x in pickle.load(open(f'{S}/{run}','rb'))['rows']}
|
||||
n_in=sum(1 for t in L.values() if t not in ('unknown','exclude'))
|
||||
for tau in [0.5,1,2]:
|
||||
for sg,mu in [(0.5,0.15),(0.6,0.2),(0.7,0.25)]:
|
||||
c=a=0; fa_unk=0; bysrc={'m':0,'o':0}
|
||||
for i,t in L.items():
|
||||
if t=='exclude' or i not in M: continue
|
||||
tx=ocr.get(i,''); s={kk:sim(tx,v) for kk,v in sig.items()} if len(norm(tx))>=4 else {}
|
||||
j=max(s,key=s.get) if s else None
|
||||
ans=None
|
||||
if M[i]['ev']:
|
||||
k,cf=conf(M[i]['ev'])
|
||||
if cf>=tau and not (j and j!=k and s[j]-s[k]>0): ans=k; src='m'
|
||||
if ans is None and s:
|
||||
r=sorted(s.values(),reverse=True)
|
||||
if r[0]>=sg and r[0]-r[1]>=mu: ans=j; src='o'
|
||||
if ans:
|
||||
a+=1; c+= ans==t; bysrc[src]+= ans==t; fa_unk+= t=='unknown'
|
||||
print(f' {res} tau{tau} ocr>={sg} m{mu}: correct {c}/{n_in} ({c/n_in:.0%}) precision {c/max(a,1):.0%} accepted-unknown {fa_unk}/48 from {bysrc}')
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
File diff suppressed because it is too large
Load diff
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,22 @@
|
|||
"""Build the leave-facing-out catalogs: cat_pack (packshots only) and cat_full
|
||||
(packshots + the widest facing of each in-catalog SKU as a shelf view), and
|
||||
image6_v2_lfo.json with those reference facings excluded from scoring.
|
||||
Same-photo references make this an optimistic proxy, not a held-out result."""
|
||||
import json, cv2, os, shutil, collections
|
||||
E = os.path.dirname(os.path.abspath(__file__))
|
||||
L = json.load(open(f'{E}/image6_v2.json')); im = cv2.imread('add/image6.jpg')
|
||||
by = collections.defaultdict(list)
|
||||
for f in L['facings']:
|
||||
if f['label'] not in ('unknown', 'exclude'): by[f['label']].append(f)
|
||||
refs = {k: max(fs, key=lambda f: f['box'][2] - f['box'][0])['index'] for k, fs in by.items() if len(fs) >= 2}
|
||||
for cat in ('cat_pack', 'cat_full'):
|
||||
d = f'{E}/{cat}'; shutil.rmtree(d, ignore_errors=True); os.makedirs(d)
|
||||
for p in os.listdir('stock-lopia-clean'):
|
||||
if p.endswith('.jpg'): shutil.copy(f'stock-lopia-clean/{p}', d)
|
||||
if cat == 'cat_full':
|
||||
for k, i in refs.items():
|
||||
x1, y1, x2, y2 = L['facings'][i]['box']; cv2.imwrite(f'{d}/{k}_shelf.jpg', im[y1:y2, x1:x2])
|
||||
for f in L['facings']:
|
||||
if f['index'] in refs.values(): f['label'] = 'exclude'
|
||||
json.dump(L, open(f'{E}/image6_v2_lfo.json', 'w'), ensure_ascii=False)
|
||||
print('shelf references', refs)
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
import sys,os,json,pickle,numpy as np,cv2,time
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__))
|
||||
from ocr import VisionOcr
|
||||
from diag import crops
|
||||
S=os.path.dirname(__file__)
|
||||
L=json.load(open(f'{S}/image6_v2.json'))
|
||||
# distinguishing tokens per SKU (any hit counts); includes siblings for near-unknowns
|
||||
KEY={'sku_l01':['エッグケア','卵を使って'],'sku_l02':['アマニ'],'sku_l03':['からし','Spicy'],'sku_l04':['めんたい'],
|
||||
'sku_l05':['ゼロ','ノンコレ'],'sku_l06':['フィッテ','内臓脂肪'],'sku_l07':['ラッキー'],'sku_l08':['PERI','ペリ'],
|
||||
'sku_l09':['コクうま','65'],'sku_l10':['ピュアセレクト','マヨネーズ'],'sku_l11':['ハーフ','1/2'],'kewpie450':['KEWPIE','マヨネーズ']}
|
||||
o=VisionOcr(languages=['ja-JP','en-US']); out={}
|
||||
for res in ['full','live']:
|
||||
rows=[]
|
||||
for fc,c in crops('add/image6.jpg',L,res=='live'):
|
||||
if fc['label']=='exclude' or c.size==0 or min(c.shape[:2])<8: continue
|
||||
for up in [1,2]:
|
||||
cc=cv2.resize(c,None,fx=up,fy=up,interpolation=cv2.INTER_CUBIC) if up>1 else c
|
||||
t=time.perf_counter(); r=o.read(cc); ms=(time.perf_counter()-t)*1000
|
||||
txt=' '.join(l.text for l in r.lines)
|
||||
rows.append(dict(index=fc['index'],truth=fc['label'],up=up,text=txt,ms=ms))
|
||||
out[res]=rows
|
||||
pickle.dump(out,open(f'{S}/ocr.pkl','wb'))
|
||||
for res,rows in out.items():
|
||||
for up in [1,2]:
|
||||
rr=[r for r in rows if r['up']==up]; tr=[r for r in rr if r['truth'] in KEY]
|
||||
hit=sum(any(k in r['text'] for k in KEY[r['truth']]) for r in tr)
|
||||
print(res,'up',up,f'in-catalog key-token hit {hit}/{len(tr)}','median ms',round(np.median([r['ms'] for r in rr]),1))
|
||||
for r in out['live']:
|
||||
if r['up']==2 and r['truth']!='unknown': print(r['index'],r['truth'],'|',r['text'][:80])
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
import sys,os,json,pickle,csv,unicodedata,numpy as np,cv2,glob
|
||||
sys.path.insert(0,'code'); from ocr import VisionOcr
|
||||
S=os.path.dirname(__file__)
|
||||
def norm(s):
|
||||
s=unicodedata.normalize('NFKC',s).upper()
|
||||
tr=str.maketrans('ッャュョァィゥェォヮヵヶユ','ツヤユヨアイウエオワカケユ'); s=s.translate(tr)
|
||||
s=s.replace('キユーピー','キユピ').replace('キューピー','キユピ')
|
||||
return ''.join(ch for ch in s if ch.isalnum() or ch=='%')
|
||||
def grams(s): return {s[i:i+2] for i in range(len(s)-1)}
|
||||
def sim(a,b):
|
||||
A,B=grams(norm(a)),grams(norm(b))
|
||||
return len(A&B)/max(1,len(A)) if A else 0.0 # share of OCR bigrams explained by signature
|
||||
# leak-free signatures: catalog names (all SKUs) + OCR of enrolled packshots
|
||||
names={r['SKU_ID']:r['product_name'] for r in csv.DictReader(open('stock-lopia/stock.csv'))}
|
||||
if __name__=='__main__':
|
||||
o=VisionOcr(languages=['ja-JP','en-US']); sig={}
|
||||
for k,n in names.items():
|
||||
sig[k]=n
|
||||
p=f'stock-lopia-clean/{k}.jpg'
|
||||
if os.path.exists(p): sig[k]+=' '+' '.join(l.text for l in o.read(cv2.imread(p)).lines)
|
||||
for k,v in sig.items(): print(k, v[:90])
|
||||
pickle.dump(sig,open(f'{S}/sig.pkl','wb'))
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
"""Catalog growth: packshots + N distractor references from unrelated shelves.
|
||||
Compares exhaustive local matching vs a global RootSIFT descriptor index
|
||||
(vote -> shortlist K -> geometric verification). No DINO pass."""
|
||||
import sys,os,glob,json,time,numpy as np,cv2,random
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__)); os.environ['YOLO_OFFLINE']='1'
|
||||
from detector import ShelfProductDetector
|
||||
from diag import crops, sku_of
|
||||
from PIL import Image
|
||||
S=os.path.dirname(__file__); random.seed(0)
|
||||
sift=cv2.SIFT_create(nfeatures=2000)
|
||||
def root(d):
|
||||
if d is None: return None
|
||||
d=d/(np.abs(d).sum(1,keepdims=True)+1e-7); return np.sqrt(d).astype(np.float32)
|
||||
def feat(img,h):
|
||||
if h and img.shape[0]>h: img=cv2.resize(img,(max(1,int(img.shape[1]*h/img.shape[0])),h),interpolation=cv2.INTER_AREA)
|
||||
g=cv2.cvtColor(img,cv2.COLOR_BGR2GRAY); kp,d=sift.detectAndCompute(g,None); return kp,root(d),g.shape
|
||||
# references
|
||||
refs=[]
|
||||
for p in sorted(glob.glob('stock-lopia-clean/*.jpg')):
|
||||
img=cv2.imread(p); k=sku_of(os.path.basename(p)[:-4])
|
||||
for h in (200,400): kp,d,sh=feat(img,h); refs.append((k,kp,d,sh))
|
||||
det=ShelfProductDetector(); dis=[]
|
||||
for p in sorted(glob.glob('add/*')):
|
||||
if 'image6' in p: continue
|
||||
im=cv2.cvtColor(np.array(Image.open(p).convert('RGB')),cv2.COLOR_RGB2BGR)
|
||||
for j,b in enumerate(det.detect(im).get('boxes',[])):
|
||||
x1,y1,x2,y2=[int(v) for v in b['box']]; c=im[max(0,y1):y2,max(0,x1):x2]
|
||||
if min(c.shape[:2])>=40: dis.append((f'd_{os.path.basename(p)}_{j}',c))
|
||||
random.shuffle(dis); print('distractor refs available',len(dis))
|
||||
L=json.load(open(f'{S}/image6_v2.json')); Q=[(fc,c) for fc,c in crops('add/image6.jpg',L,True) if fc['label']!='exclude' and min(c.shape[:2])>=8]
|
||||
Qf=[(fc['label'],)+feat(c,None) for fc,c in Q]
|
||||
bf=cv2.BFMatcher(cv2.NORM_L2)
|
||||
def verify(kq,dq,r):
|
||||
k,kp,d,sh=r
|
||||
if dq is None or d is None or len(d)<2 or len(dq)<2: return 0
|
||||
good=[m for m,n in (p for p in bf.knnMatch(dq,d,k=2) if len(p)==2) if m.distance<0.75*n.distance]
|
||||
if len(good)<4: return 0
|
||||
src=np.float32([kq[m.queryIdx].pt for m in good]); dst=np.float32([kp[m.trainIdx].pt for m in good])
|
||||
H,mask=cv2.estimateAffinePartial2D(src,dst,method=cv2.RANSAC,ransacReprojThreshold=0.02*max(sh))
|
||||
return int(mask.sum()) if mask is not None else 0
|
||||
E={r[0] for r in refs}
|
||||
for N in [0,50,200,len(dis)]:
|
||||
R=refs+[(k,)+feat(c,None) for k,c in dis[:N]]
|
||||
# exhaustive
|
||||
t=time.perf_counter(); ok=0; fa=0; n=0
|
||||
best_all=[]
|
||||
for truth,kq,dq,_ in Qf:
|
||||
sc={}
|
||||
for r in R: sc[r[0]]=max(sc.get(r[0],0),verify(kq,dq,r))
|
||||
b=max(sc,key=sc.get); best_all.append(sc)
|
||||
if truth in E: n+=1; ok+= b==truth
|
||||
ex_ms=(time.perf_counter()-t)*1000/len(Qf)
|
||||
# descriptor index
|
||||
D=np.vstack([r[2] for r in R if r[2] is not None]); owner=np.concatenate([[i]*len(r[2]) for i,r in enumerate(R) if r[2] is not None])
|
||||
t0=time.perf_counter(); fl=cv2.FlannBasedMatcher(dict(algorithm=1,trees=4),dict(checks=64)); fl.add([D]); fl.train(); build=time.perf_counter()-t0
|
||||
rec={5:0,10:0,20:0}; t=time.perf_counter(); ok_idx=0
|
||||
for truth,kq,dq,_ in Qf:
|
||||
votes={}
|
||||
if dq is not None and len(dq)>=2:
|
||||
for m,nn in (p for p in fl.knnMatch(dq,k=2) if len(p)==2):
|
||||
if m.distance<0.8*nn.distance:
|
||||
k=R[owner[m.trainIdx]][0]; votes[k]=votes.get(k,0)+1
|
||||
order=sorted(votes,key=lambda k:-votes[k])
|
||||
if truth in E:
|
||||
for K in rec: rec[K]+= truth in order[:K]
|
||||
short=set(order[:10]); sc={}
|
||||
for r in R:
|
||||
if r[0] in short: sc[r[0]]=max(sc.get(r[0],0),verify(kq,dq,r))
|
||||
if truth in E and sc: ok_idx+= max(sc,key=sc.get)==truth
|
||||
ix_ms=(time.perf_counter()-t)*1000/len(Qf)
|
||||
print(f'refs={len(R):4d} skus={len({r[0] for r in R}):4d} | exhaustive top1 {ok}/{n} {ex_ms:6.1f} ms/crop | index recall@5/10/20 {rec[5]}/{rec[10]}/{rec[20]} of {n}, index+verify(K=10) top1 {ok_idx}/{n} {ix_ms:6.1f} ms/crop, build {build*1000:.0f} ms, {len(D)} descriptors')
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
import json,cv2,numpy as np,sys,glob,os
|
||||
d=json.load(open('eval/labels/image6.json'));im=cv2.imread('add/image6.jpg')
|
||||
S=sys.argv[1]
|
||||
H=300
|
||||
def tile(img,txt,col=(0,0,0)):
|
||||
h,w=img.shape[:2];s=H/h;img=cv2.resize(img,(max(1,int(w*s)),H))
|
||||
pad=np.full((H+30,img.shape[1]+6,3),255,np.uint8);pad[30:,3:3+img.shape[1]]=img
|
||||
cv2.putText(pad,txt,(2,20),cv2.FONT_HERSHEY_SIMPLEX,0.5,col,1);return pad
|
||||
groups={}
|
||||
for f in d['facings']:
|
||||
x1,y1,x2,y2=f['box'];c=im[y1:y2,x1:x2]
|
||||
groups.setdefault(f['label'],[]).append(tile(c,f"#{f['index']}"))
|
||||
for lab,ts in groups.items():
|
||||
ref=glob.glob(f'stock-lopia-clean/{lab}.jpg')
|
||||
row=([tile(cv2.imread(ref[0]),'REF',(0,0,255))] if ref else [])+ts
|
||||
rows=[row[i:i+12] for i in range(0,len(row),12)]
|
||||
out=[]
|
||||
for r in rows:
|
||||
w=sum(t.shape[1] for t in r);canvas=np.full((H+30,max(w,10),3),255,np.uint8);x=0
|
||||
for t in r: canvas[:,x:x+t.shape[1]]=t;x+=t.shape[1]
|
||||
out.append(canvas)
|
||||
W=max(o.shape[1] for o in out);out=[np.pad(o,((0,0),(0,W-o.shape[1]),(0,0)),constant_values=255) for o in out]
|
||||
cv2.imwrite(f'{S}/gt_{lab}.jpg',np.vstack(out))
|
||||
vis=im.copy()
|
||||
for f in d['facings']:
|
||||
x1,y1,x2,y2=f['box'];c=(0,0,255) if f['label']=='unknown' else (0,200,0)
|
||||
cv2.rectangle(vis,(x1,y1),(x2,y2),c,2);cv2.putText(vis,f"{f['index']}:{f['label'].replace('sku_','')}",(x1+2,y1+15),cv2.FONT_HERSHEY_SIMPLEX,0.45,c,1)
|
||||
cv2.imwrite(f'{S}/shelf_labels.jpg',vis)
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
"""Matcher variants. Each variant returns per-SKU evidence dicts so ranking and
|
||||
open-set features can be analysed separately."""
|
||||
import sys, os, json, time, glob, pickle, argparse
|
||||
import cv2, numpy as np
|
||||
sys.path.insert(0, os.path.dirname(__file__)); from diag import crops, sku_of
|
||||
|
||||
class Matcher:
|
||||
def __init__(self, stock, tmpl_h=None, root=False, geom='homog', nfeat=2000, scales=(None,)):
|
||||
self.sift = cv2.SIFT_create(nfeatures=nfeat); self.root = root; self.geom = geom
|
||||
self.T = []
|
||||
for p in sorted(glob.glob(f'{stock}/*.jp*g')+glob.glob(f'{stock}/*.png')):
|
||||
img = cv2.imread(p); name = os.path.splitext(os.path.basename(p))[0]
|
||||
for h in scales:
|
||||
im = img if h is None else cv2.resize(img, (max(1,int(img.shape[1]*h/img.shape[0])), h), interpolation=cv2.INTER_AREA)
|
||||
g = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY); kp, des = self.sift.detectAndCompute(g, None)
|
||||
des = self._root(des)
|
||||
hsv = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)
|
||||
self.T.append(dict(name=name, sku=sku_of(name), kp=kp, des=des, shape=g.shape, hist=self._hist(hsv)))
|
||||
self.bf = cv2.BFMatcher(cv2.NORM_L2)
|
||||
def _root(self, des):
|
||||
if des is None or not self.root: return des
|
||||
des = des / (np.abs(des).sum(1, keepdims=True) + 1e-7); return np.sqrt(des).astype(np.float32)
|
||||
@staticmethod
|
||||
def _hist(hsv):
|
||||
m = (hsv[...,1] > 60) & (hsv[...,2] > 50) # chromatic pixels only
|
||||
h = cv2.calcHist([hsv], [0], m.astype(np.uint8), [18], [0, 180]).ravel()
|
||||
return h / (h.sum() + 1e-6)
|
||||
def score(self, crop):
|
||||
g = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY); kq, dq = self.sift.detectAndCompute(g, None); dq = self._root(dq)
|
||||
hq = self._hist(cv2.cvtColor(crop, cv2.COLOR_BGR2HSV))
|
||||
ev = {}
|
||||
for t in self.T:
|
||||
e = dict(inliers=0, cover=0.0, color=float(np.minimum(hq, t['hist']).sum()), nq=len(kq))
|
||||
if dq is not None and t['des'] is not None and len(dq) >= 2 and len(t['des']) >= 2:
|
||||
good = [m for m, n in (p for p in self.bf.knnMatch(dq, t['des'], k=2) if len(p) == 2) if m.distance < 0.75*n.distance]
|
||||
if len(good) >= 4:
|
||||
src = np.float32([kq[m.queryIdx].pt for m in good]); dst = np.float32([t['kp'][m.trainIdx].pt for m in good])
|
||||
if self.geom == 'homog':
|
||||
H, mask = cv2.findHomography(src, dst, cv2.RANSAC, 0.02*max(t['shape']))
|
||||
else:
|
||||
H, mask = cv2.estimateAffinePartial2D(src, dst, method=cv2.RANSAC, ransacReprojThreshold=0.02*max(t['shape']))
|
||||
if mask is not None:
|
||||
inl = mask.ravel().astype(bool); n = int(inl.sum())
|
||||
if n >= 3:
|
||||
pts = dst[inl]; hull = cv2.convexHull(pts)
|
||||
e['inliers'] = n; e['cover'] = float(cv2.contourArea(hull) / (t['shape'][0]*t['shape'][1]))
|
||||
# crop-side coverage too: does the match span the query?
|
||||
qh = cv2.convexHull(src[inl]); e['qcover'] = float(cv2.contourArea(qh)/(g.shape[0]*g.shape[1]))
|
||||
k = t['sku']
|
||||
if k not in ev or e['inliers'] > ev[k]['inliers']: ev[k] = e
|
||||
return ev
|
||||
|
||||
if __name__ == '__main__':
|
||||
ap = argparse.ArgumentParser(); ap.add_argument('--stock'); ap.add_argument('--labels'); ap.add_argument('--live', action='store_true')
|
||||
ap.add_argument('--tmpl-h', type=int, default=None); ap.add_argument('--root', action='store_true'); ap.add_argument('--geom', default='homog')
|
||||
ap.add_argument('--scales', default=''); ap.add_argument('--out')
|
||||
a = ap.parse_args(); L = json.load(open(a.labels))
|
||||
scales = tuple(int(s) for s in a.scales.split(',')) if a.scales else ((a.tmpl_h,) if a.tmpl_h else (None,))
|
||||
t0 = time.perf_counter(); m = Matcher(a.stock, root=a.root, geom=a.geom, scales=scales); init = time.perf_counter()-t0
|
||||
rows = []
|
||||
for fc, c in crops('add/image6.jpg', L, a.live):
|
||||
if c.size == 0 or min(c.shape[:2]) < 8: rows.append(dict(index=fc['index'], ev={}, ms=0)); continue
|
||||
t = time.perf_counter(); ev = m.score(c); rows.append(dict(index=fc['index'], ev=ev, ms=(time.perf_counter()-t)*1000, shape=c.shape[:2]))
|
||||
pickle.dump(dict(rows=rows, enrolled=sorted({t['sku'] for t in m.T}), init_s=init, args=vars(a)), open(a.out, 'wb'))
|
||||
print(a.out, 'init', round(init,2), 'median ms', round(float(np.median([r['ms'] for r in rows if r['ms']])),1))
|
||||
Loading…
Add table
Add a link
Reference in a new issue