diff --git a/recognization/docs/design-price-tags-and-compliance.md b/recognization/docs/design-price-tags-and-compliance.md new file mode 100644 index 0000000..2027192 --- /dev/null +++ b/recognization/docs/design-price-tags-and-compliance.md @@ -0,0 +1,581 @@ +# Design: price-tag OCR, reconciliation, compliance, and the mobile client + +Status: proposal, evidence-backed, not implemented. +Written 2026-09-24 after measuring OCR engines against the real demo images. +Read `docs/API.md` first for the system this extends. + +The owner's brainstorm, restated as requirements: + +1. Detect price tags in the image, transcribe them to text. +2. Run that **in parallel** with product recognition, so both boxes are shown. +3. **Reconcile** the recognized item with what the tag says; show both results. +4. Tap a product (web **and** mobile live scan) and see the product card **plus the + OCR information from the tag below it**. +5. **Price-tag compliance**: detect a group of items missing price tags. + +Everything below is designed so that none of it slows down detection, and so that no +compliance claim is ever made from unreadable evidence. + +--- + +## 1. What was measured before designing + +All numbers from this machine (Apple M5, 10 cores), 2026-09-24. + +### 1.1 Tag readability depends almost entirely on capture distance + +| Image | Distance | Tag size | Digit height | macOS Vision on tag | RapidOCR on tag | +| --- | --- | --- | --- | --- | --- | +| `image.jpg` (2048 px) | ~2 m | ~110x35 px | ~14 px | **nothing** | not attempted (same scale) | +| `add/image6.jpg` (1500 px) | ~1 m | ~300x80 px | ~22 px | **reads 5/5 fields** | reads printed text, misses the big red digits | + +Vision on a single `image6` tag (`300x80` crop, 191 ms) returned: + +``` +0.30 キュービーゼロ ノンコレステロール<310g> product name +1.00 357 regular price +0.30 2.4割引 discount +0.30 269 sale price (large, stylised) +1.00 290.52 unit price +``` + +RapidOCR on the same crop (263 ms) returned `357`, `2.4割引`, `290.52`, `<310g>` at +0.97-1.00 confidence but **missed `269`** (large red outlined digits). + +**Conclusion:** a real Japanese price tag carries a JAN barcode, the product name, +the sale price, the regular price, the discount, and a unit price - and at the +demo's `image.jpg` distance none of it is readable by any engine. Tag OCR is a +capture-quality problem before it is a model problem. + +### 1.2 Engine comparison + +| Engine | Platform | Rail band 1500x120 | Full 2048 px image | Small printed text | Large stylised digits | Verdict | +| --- | --- | --- | --- | --- | --- | --- | +| macOS Vision (accurate) | macOS only | 182 ms, 20 regions | 510 ms, 58 regions | good | **reads them** | best here, not portable | +| RapidOCR (PP-OCR ONNX) | cross-platform | 752-1290 ms, 23 regions | ~3.4 s at 1280 px, 143 regions | good | misses them | cross-platform default | +| Tesseract `jpn` | cross-platform | ~300 ms | garbage on tags (`と1拓5 MI HMI`) | unusable | unusable | rejected | +| PaddleOCR (full) | cross-platform | heavier sibling of RapidOCR | - | - | - | candidate if RapidOCR's accuracy is not enough | + +### 1.3 The same OCR pass reads packaging text - but confirmation depends on catalog coherence + +Measured by OCR-ing an image and checking, for every accepted detection, whether a text +region inside its box matches the catalog name (normalized, longest-common-substring +ratio >= 0.4): + +| Image | Accepted | Confirmed by packaging text | Note | +| --- | --- | --- | --- | +| the 21 catalog **template** photos | 21 | **14 (67%)** | catalog and photo describe the same product, by construction | +| `image.jpg` (2 m shelf) | 105 | 11 (10%) | only the products our fictional catalog happens to share with the photo (Tropicana, 野菜生活) | +| `add/image6.jpg` (1 m shelf) | 59 | 2 (3%) | the shelf is a different store's assortment (キューピー); our catalog does not describe it | + +The template row is the honest measure of the *mechanism* (67%, with exact matches at +`野菜生活100オリジナル`, `野菜生活100ベリーサラダ`, `野菜生活100アップルサラダ`, +`歌舞伎揚`). The shelf rows measure the *data problem*: our demo catalog is fictional +and the photos are a real store, so most products have nothing to match against. + +Failure modes seen in the 7 template misses, all explainable: + +- **Romanized reads** - `亀田のまがりせんべい` came back as `Magari Senbei`; character + LCS scores 0.08 while the semantics match. Matching must compare kana/kanji *and* + a romaji transliteration of the catalog name. +- **Brand-only reads** - `三幸製菓`, `岩塚製菓` (the maker) without the product name. +- **Name not on the visible face** - the 明治 milk carton's name sat outside the + detected region. + +Consequence for the design: packaging text is a **bonus confirmation**, never a gate. +Its absence must be neutral (never a contradiction), and its evaluation needs a +coherent catalog-photo pair - which is the same coherence requirement as price +reconciliation (§6.1). + +So OCR has **two** uses: + +- **Tag OCR** - price, name, JAN. Needs capture distance; blocked on demo data. +- **Packaging-text identity** - a second opinion on what the matcher decided; works + today on coherent data (67% on templates), informative but sparse on the demo + shelves. + +### 1.4 The barcode is the exact-identity path, and it is currently too small + +`image6`'s tag carries an EAN-13-style barcode; at this distance (~150x25 px, blurred) +`cv2.barcode.BarcodeDetector` fails even at 4x upscale. An EAN-13 needs roughly 3 px +per module (~300 px width) to decode. A phone at 40-60 cm decodes it trivially. +**Design consequence:** barcode decoding is the strongest reconciliation signal +available, and it is gated by the same capture requirement as the text. + +### 1.5 The catalog has no JAN and no price semantics + +`stock.csv` has `SKU_ID, product_name, receipt_date, expiration_date, number_in, +number_out, number_on, price`. There is no JAN, no regular-vs-sale price, and the demo +prices (¥168 for Tropicana) belong to a fictional store, not the store in the photo. +Exact reconciliation needs a catalog change, and an honest demo needs the catalog and +the tags to describe the same store. + +--- + +## 2. Design principles this system inherits + +1. **The UI never turns uncertainty into certainty** (project master). A tag read is + evidence with a confidence, never a fact. +2. **Fail closed on evidence quality.** "No tag detected" and "tag detected, text + unreadable" are different states. Only the first may become a compliance + violation; the second is a capture-quality hint. +3. **Detection stays real-time.** OCR is the slowest stage (0.2-3.4 s); it must never + sit in the frame's critical path. +4. **One product, one classification, one tag read.** Tags are static; the same + physical tag seen in 30 frames is one tag with one vote history, not 30 reads. +5. **Additive protocol changes only.** The live protocol is version 1; new fields are + optional, old clients keep working. + +--- + +## 3. Architecture: a second, parallel perception stage + +``` + ┌───────────────────────── frame loop (must stay ~25-45 ms) ─────┐ + client frames ──▶ decode ──▶ detect ──▶ track ──▶ schedule classification ──▶ result + │ │ ▲ + │ (same decoded frame, read-only) │ + ▼ ▼ │ + ┌──────────────────────────────────────────┐ │ + │ TagService (own thread, 1-2 workers) │ cached tag state ────┘ + │ │ merged into the result + │ rail bands from row geometry │ + │ → crop → OCR (engine backend) │ + │ → parse → vote over passes │ + │ → associate tag ↔ product │ + └──────────────────────────────────────────┘ + │ + ▼ + Reconciliation + compliance rules + (pure functions over settled evidence) +``` + +Why a separate service and not another stage in `process_frame`: + +- **OCR is 10-100x slower than detection.** 0.75-1.3 s per rail band (RapidOCR) means + it cannot run per frame; it runs on a cadence and its answers are cached. +- **Tags are static.** A tag read stays valid for the whole pass over that shelf. +- **The frame loop already has a budget.** Detection (46 ms under live load) plus + tracking plus classification scheduling fills it. + +### 3.1 The tag pass (cadence and caching) + +A *tag pass* processes one rail band and updates the tag cache. + +| Trigger | When | +| --- | --- | +| Periodic | every N frames (default 15, ~1-1.5 s at live rates) while tags are unsettled | +| Movement | the band's screen position moved more than ~2% of frame width since the last pass | +| Settle | a band whose tags are all `settled` (see below) is skipped entirely | +| On demand | photo mode, or the client asks (`inspect`) | + +Band extraction reuses the geometry we already have: `empty_shelf_detector.cluster_rows()` +groups product boxes into rows, and a rail band is the strip below a row +(`row_bottom` to `row_bottom + ~8% of row height`, clipped to the frame). No new +geometry model is needed, and it is the same code the empty-slot detector trusts. + +### 3.2 Voting: the accuracy lever + +Every pass produces a `TagObservation` (box, text lines, parsed fields, confidence). +Observations that overlap the same tag (IoU ≥ 0.5 against the cached tag) accumulate +into that tag's history, and the *published* value is the vote, not the latest read: + +- **Price**: majority vote over parsed integers; ties break to the highest-confidence + read. (A misread `26円` vs `269` is resolved by repetition - the single strongest + cheap accuracy improvement available, and the reason caching is a feature.) +- **Name**: longest common substring across observations, or the highest-confidence + read when observations disagree entirely. +- **Barcode**: first successful decode wins permanently (it either checksums or it + does not). +- A tag is `settled` after 3 agreeing observations or 8 passes, whichever comes first; + settled tags stop costing OCR time. + +### 3.3 Latency budget + +| Stage | Cost | Cadence | Contribution to frame loop | +| --- | --- | --- | --- | +| Rail band crop | <1 ms | per pass | 0 (outside the loop) | +| OCR, Vision backend | ~180 ms/band | 1-2 Hz | 0 | +| OCR, RapidOCR backend | 0.75-1.3 s/band | 0.3-0.5 Hz, band-parallel | 0 | +| Parse + vote + associate | <1 ms | per frame | ~0.2 ms | +| Merge into result | <0.5 ms | per frame | ~0.2 ms | + +Worst realistic case: 4 bands x RapidOCR at 1.3 s = 5.2 s of work per full shelf +sweep, spread over 1-2 workers while detection keeps running. On macOS with Vision it +is ~0.7 s for the whole shelf. The frame loop is unaffected either way. + +--- + +## 4. Price-tag detection + +Four approaches were considered. + +| Approach | Needs training data | Robust to blur | Works on demo images | Verdict | +| --- | --- | --- | --- | --- | +| A. Geometry only (bright quadrilateral in the rail band) | no | partly | no (tags too small to segment reliably) | fallback | +| B. Trained tag detector (YOLO-nano on tags) | **yes, none exists** | yes | after training | target state | +| C. OCR-native text regions clustered into tags | no | it is the same failure as OCR | at readable distance | **start here** | +| D. Hybrid: C now, B later, A as fallback | no → yes | yes | at readable distance | **recommended path** | + +Recommended sequence: **C → B**. + +**C (now).** The OCR engine's own detector returns text-region boxes (RapidOCR found +23 in a rail band; Vision found 20). Cluster regions inside a rail band into tag +candidates: + +1. Take regions whose box height is 8-40% of the band height and whose center is + inside the band. +2. Cluster by horizontal proximity (gap ≤ 1.5 x median region height) into groups. +3. A group becomes a tag if its bounding box has aspect 2.5:1 to 8:1 and width + ≥ 4% of frame width - the measured tag shape is ~300x80 px (3.75:1) at 1 m. +4. Reject groups that are pure noise: no parsed price AND no kana/kanji AND fewer + than 2 regions. + +**B (later).** Bootstrap a tag dataset from C's output (auto-labelled tag boxes, +human-verified once), train a YOLO-nano (same ultralytics toolchain as the product +detector, same deployment path), and switch detection to it. A trained detector makes +tag detection independent of OCR quality - the tag box is found even when its text is +unreadable, which is exactly what compliance needs. + +**A (fallback)** stays for the case where OCR returns nothing: a brightness/saturation +rectangle test inside the band, used only to answer "is there *something* tag-shaped +here", never to read text. + +### 4.1 Associating a tag with a product + +A tag belongs to the product whose facing sits directly above it on the same rail: + +1. For each product box, the candidate band is `[row_bottom, row_bottom + 8% height]`. +2. Candidate tags are those with horizontal overlap ≥ 50% of the *tag* width, or, + failing that, nearest tag center within 0.5 x product width. +3. Each tag belongs to at most one product; each product gets at most one tag + (the closest by center distance). Unclaimed tags are `orphan_tag` candidates. +4. The association is recomputed per frame from cached tags - cheap, and it follows + the camera as the track moves. + +--- + +## 5. OCR engine abstraction + +Same pattern as the app's `platform/` bridges: one interface, several implementations, +picked by availability and configuration. + +```python +class OcrEngine(Protocol): + name: str + def read(self, image: np.ndarray, *, hints: OcrHints) -> list[OcrLine]: ... + def available(self) -> bool: ... + +@dataclass +class OcrLine: + text: str + confidence: float + box: list[int] # xyxy in the input image's pixels + +@dataclass +class OcrHints: + languages: tuple[str, ...] = ("ja", "en") + mode: str = "sparse" # "sparse" = rail bands/tags; "dense" = full page + max_regions: int | None = None +``` + +Backends: + +| Backend | Class | Selection rule | +| --- | --- | --- | +| `vision` | `VisionOcr` (macOS, `pyobjc` `VNRecognizeTextRequest`) | default on macOS when importable; measured 182 ms/band | +| `rapidocr` | `RapidOcr` (`rapidocr-onnxruntime`) | default elsewhere (Windows/Linux); measured 0.75-1.3 s/band | +| `fake` | `FakeOcr` | tests, and the deterministic demo | +| `paddle` | `PaddleOcr` | only if RapidOCR's Japanese accuracy proves insufficient | + +Tesseract is rejected on measurement (§1.2). Cloud OCR (Google/Azure) is rejected: the +demo must run offline, and shipping shelf images to a third party is a privacy decision +the owner has not made. + +Two preprocessing rules, both from the measurements: + +- **Never OCR the whole frame in live mode.** Crop the rail band (and later the tag + box) - a full 1280 px frame costs 3.4 s in RapidOCR for 143 regions we do not want. +- **Upscale small tags 2-3x with INTER_CUBIC before OCR** when the tag's digit height + is below ~20 px; beyond 3x the blur wins and accuracy falls. + +### 5.1 Parsing a Japanese price tag + +Field extraction is deterministic parsing over the OCR lines, not another model: + +| Field | Rule | Example | +| --- | --- | --- | +| `price` | largest integer 2-5 digits inside a red/black large region, or the integer adjacent to `円` | `269` | +| `regular_price` | integer near `割引` / on the left with smaller size | `357` | +| `discount` | `(\d+(\.\d+)?)\s*割` | `2.4割引` → 24% | +| `unit_price` | `\d+\.\d{1,2}` (per 100 g / per 10 mL convention) | `290.52` | +| `name` | longest kana/kanji run with confidence ≥ 0.3, excluding prices and sale words (`毎日お買得`) | `キュービーゼロ ノンコレステロール<310g>` | +| `jan` | barcode decode of the tag's barcode region (separate step) | `4902105...` | +| `flags` | `sale` when a discount or 割引 present, `tax_included` on 税込, `new` on 新発売 | | + +Sanity rules that reject a parse outright: price outside 1-99999; discount > 90%; +more than one candidate price with no size ordering. A rejected parse publishes +`price: null` with `quality: "unparsed"` - never a guess. + +--- + +## 6. Reconciliation: item vs tag + +Inputs per facing: the identity decision (accepted / ambiguous / unknown / rejected, +with candidates), the associated tag (if any), and the catalog entry. + +Signals, strongest first: + +| Signal | Strength | Requires | +| --- | --- | --- | +| JAN decoded from the tag's barcode == catalog JAN | **exact** | catalog JAN column, capture quality | +| OCR name ≈ catalog name (normalized kana/kanji + romaji form, LCS ratio ≥ 0.6 for tags, ≥ 0.4 for packaging) | strong | readable name; catalog describes the store | +| OCR price == catalog price (regular or sale) | medium | coherent catalog/tag data | +| Position (tag directly below, same rail slot) | association only | geometry | + +Matching must compare three forms of the catalog name - normalized kana/kanji, +NFKC-folded, and a romaji transliteration - because the packaging sometimes prints the +name in Latin script (`Magari Senbei` for 亀田のまがりせんべい, measured). + +Reconciliation states (published per facing, alongside - never replacing - the +identity state): + +| State | Condition | UI | +| --- | --- | --- | +| `confirmed` | accepted identity AND a strong/medium signal agrees | green chip "tag matches" | +| `mismatch` | accepted identity AND a signal disagrees (price differs, name unrelated, JAN differs) | amber chip, both values shown | +| `unverified` | accepted identity, tag present but unreadable or unparsed | grey chip "tag unread" | +| `no_tag` | accepted identity, no tag found under the facing | red chip (compliance input) | +| `tag_only` | tag read, no accepted identity above it | tag shown on its own | + +Hard rule: `mismatch` never silently changes the product identity. It is a *second +opinion* displayed next to the first; only a JAN match may promote an `ambiguous` +identity to `accepted` (that is the one case where the tag is stronger evidence than +the matcher, because a checksummed barcode is not a guess). + +### 6.1 Catalog changes required + +```csv +SKU_ID,product_name,receipt_date,expiration_date,number_in,number_out,number_on,image_link,price,jan,regular_price +sku_003,Tropicana エッセンシャルズ マルチビタミン,...,168,4902105xxxxxx,198 +``` + +- `jan` (optional): enables exact reconciliation. +- `regular_price` (optional): lets the tag's regular/sale pair be checked separately. +- Both are additive; `InventoryManager` already ignores unknown columns, so nothing + breaks until the columns appear. +- **Demo coherence:** for a believable reconciliation demo, the tags photographed must + belong to the same store as the catalog. Either shoot/curate shelf photos whose tags + match the demo catalog, or render synthetic tags for the demo catalog and place them + on the shelf (a small, honest demo asset - labelled as synthetic). Real-store photos + with our fictional catalog will legitimately show `mismatch` everywhere, which + demonstrates the compliance path but not the reconciliation path. + +--- + +## 7. Price-tag compliance + +First rule family of the (future) rules engine. Each rule is a pure function over +settled evidence, returning findings with boxes and counts - no side effects, no +network. + +| Rule id | Fires when | Evidence | Guard | +| --- | --- | --- | --- | +| `missing_price_tag` | an accepted facing has no associated tag | facing box, row, SKU | only on rows where tag detection is `readable` (≥1 tag parsed in that row) | +| `missing_tag_group` | **≥ 3 consecutive facings in a row** have no tag | row index, span box, count | same guard; this is the owner's "group of items missing price tags" | +| `orphan_tag` | a tag has no facing above it | tag box, row | tag must be parsed | +| `price_mismatch` | `mismatch` state from a price signal | facing box, tag price, catalog price | catalog price must exist | +| `duplicate_tag` | one tag associated with two facings | tag box, both facing boxes | - | +| `unreadable_tags` | a row where tags were found but none parsed | row box, count | informational, never a violation | + +The guard in the first two rows is the whole design: a row whose tags are simply +unreadable (every row in `image.jpg`) must produce a **capture-quality hint**, not a +compliance violation. That is principle 2, implemented. + +Aggregation: findings roll up per row, then per frame/session, into +`compliance: {rows: [...], violations: N, by_rule: {...}}`. The UI shows a row-level +strip ("Row 3: 4 facings without price tag") with the offending boxes highlighted - +the same red-box language the empty-slot detector already uses. + +Evaluation cadence: per frame from settled evidence only (cheap), with a stability +filter (a finding must hold for 3 consecutive frames) so nothing flickers. + +--- + +## 8. Protocol changes (additive, protocol stays 1) + +Per track/detection: + +```json +"price_tag": { + "box": [812, 404, 909, 441], + "state": "readable | unreadable | absent", + "price": 269, "regular_price": 357, "discount": 0.24, "unit_price": 290.52, + "name": "キュービーゼロ ノンコレステロール<310g>", + "jan": null, + "confidence": 0.82, + "observations": 3, + "settled": true +}, +"reconciliation": { + "state": "confirmed | mismatch | unverified | no_tag | tag_only", + "signals": [{"kind": "price", "agrees": true, "item": 269, "tag": 269}], + "message": null +} +``` + +Frame summary gains: + +```json +"summary": { + "price_tags": {"readable": 12, "unreadable": 3, "absent": 4}, + "compliance": {"violations": 1, "by_rule": {"missing_tag_group": 1}, + "rows": [{"row_index": 3, "span": [x1,y1,x2,y2], "missing": 4, + "skus": ["sku_003", "sku_009"]}]} +} +``` + +New WebSocket control message (optional, for tap-to-inspect freshness): + +```json +{"type": "read_tags", "request_id": "t1", "boxes": [[x1,y1,x2,y2]]} +→ {"type": "read_tags_result", "request_id": "t1", "tags": [...]} +``` + +REST `recognize` gains the same two fields per detection, plus +`?tags=true` to request the tag pass on a still (photo mode default on). +`inspect` gains `tag` (the tag associated with the tapped box) so the mobile/web card +can show OCR text without a second round trip. + +--- + +## 9. Client changes (web now, mobile after) + +Web (`code/web/`): + +- Tag boxes drawn in a distinct style (thin teal outline, small price chip when + parsed) so they never read as product boxes. +- Product card gains an **OCR section**: price read vs catalog price, name read, + discount, and the reconciliation chip (`confirmed` / `mismatch` / `unreadable`). +- A **compliance strip** under the stage: rule, row, count; clicking highlights the + boxes. +- Quality hint gains `move_closer_for_tags` when a row's tags are unreadable - the + measured failure mode from §1.1. + +--- + +## 10. Mobile live scan (the second ask) + +Target: the dorayaki app's scan screen does what the web demo does - live boxes, +tap a product, see the product card, and see the tag/OCR information under it. + +The app repo (`/Users/ngotrung/Documents/Projects/dorayaki`) requires an **ADR** +before a second API base URL is added (its `AGENTS.md`), and its `ScanCaptureView` is +currently a timer-driven placeholder (`ScanViewModel.tick()`), not a camera client. +So this lane is: one ADR, then a feature slice copied from the kit's own recipes. + +| Piece | Where | What | +| --- | --- | --- | +| ADR | `docs/decisions/ADR-002-recognition-server.md` | second base URL, why, flavor mapping (Debug → `http://:8080`, Staging/Prod later) | +| Camera bridge | `shared/.../platform/CameraFrames.kt` + `iosMain` (AVFoundation) + `androidMain` (CameraX) | `Flow` at a chosen cadence; JPEG encode on the platform side (official APIs only) | +| Live client | `shared/.../data/shelf/LiveScanClient.kt` (Ktor WebSockets) | connect, send frames, receive results, expose `Flow` | +| Domain | `shared/.../domain/shelf/LiveScan.kt` | `LiveTrack`, `LiveIdentity`, `PriceTag`, `ComplianceFinding`, `LiveScanFrame` - mirroring the wire schema, with defaults for every optional field (kit rule) | +| ViewModel | `shared/.../feature/shelf/LiveScanViewModel.kt` | `@NativeCoroutinesState` flows; tap → `selectTrack(id)`; inspect via the socket | +| iOS UI | `iosApp/iosApp/ScanCaptureView.swift` | overlay boxes (SwiftUI `Canvas`), tap hit-test, product sheet with the OCR section | +| Android UI | `androidApp/.../screens/ScanScreen.kt` | same, Compose | +| Tests | `commonTest` | protocol parsing with a fake socket; identity/price-tag state mapping | + +Sequencing inside the lane: iOS first (the owner demos on iPad), Android immediately +after, because the kit's contract is two native UIs. + +Protocol subset the app needs: `hello`, binary frames, `result`, `inspect` / +`inspect_result`. Everything else is optional. Tap-to-inspect maps to the existing +`inspect` message, so the card shows crop + feature matching + tag in one reply. + +Constraint to respect: phone cameras are only available over HTTPS or localhost, so +the demo path is either `--ssl-certfile` (mkcert) or a tailnet-only HTTPS edge. The +server already supports the flags; the ADR should record the chosen option. + +--- + +## 11. Data and capture plan + +1. **Capture requirement (measured):** tag digit height ≥ 20 px, tag width ≥ 150 px. + At a typical Japanese store tag (60 mm wide, digits ~8 mm), that is a phone at + 40-80 cm from the rail, or a 4K frame at ~1.5 m. The live client should show + `move_closer_for_tags` below that. +2. **Tag images for the demo:** shoot one shelf rail (or three) at ~50 cm with a phone + in the existing demo store style; 10-20 tags is enough to validate the whole path. + Alternatively render synthetic tags for the demo catalog and composite them onto a + shelf photo - faster, honest if labelled, and it makes reconciliation demonstrable + without a real store trip. +3. **Tag detector dataset (later):** auto-label with the §4 pipeline, human-verify + once, train YOLO-nano. Target ~300 tag instances across 30-50 rail images. +4. **Catalog:** add `jan` and `regular_price` columns (optional, additive). +5. **Barcode decode:** `zxing-cpp` (pip, cross-platform) or `pyzbar` (needs the zbar + system library); decode the tag's barcode region only, at 2-3x upscale. Adopt once + capture quality allows - a decoded JAN is worth more than every other signal + combined. + +--- + +## 12. Risks and failure modes + +| Risk | Impact | Mitigation | +| --- | --- | --- | +| Demo images cannot validate tag OCR | the feature looks unproven at demo time | capture plan §11.1-2 before implementation lands; validate on the catalog templates meanwhile (67% measured) | +| Catalog does not describe the photographed store | text/price confirmation is sparse (measured 10% / 3%) and every "mismatch" is noise | treat absence as neutral; validate on coherent data (templates, or a re-shot demo shelf); §6.1 | +| OCR misreads a digit and a mismatch is reported | false compliance alarm | multi-pass voting (§3.2); mismatch is displayed, never auto-applied; compliance rules require the readable guard | +| OCR load starves detection | frame rate drops | separate worker, band crops only, settle-and-skip cache, measured budget §3.3 | +| Tag detector without training data | no tag boxes on unreadable rows | hybrid C→B; geometry fallback marks "something tag-shaped" without reading | +| Catalog/tag data incoherent in the demo | reconciliation demo shows only mismatches | §6.1 demo coherence choice | +| Japanese text normalization (kana variants, 半角/全角, size suffix) | name matching noise | normalize NFKC + strip size/`<310g>` before LCS | +| Mobile lane needs an ADR and touches the kit's architecture | schedule risk in the other repo | ADR first, then a slice copied from `add-kmp-feature`; no kit changes | + +--- + +## 13. Phased plan + +| Phase | Deliverable | Acceptance | +| --- | --- | --- | +| **0. Capture assets** | one rail shot at ≤80 cm (or synthetic tags composited) | a tag crop where Vision and RapidOCR both read price + name | +| **1. OCR service + packaging-text identity** | `OcrEngine` interface, Vision + RapidOCR + fake backends, band cropping, cache, vote; packaging text joins the identity decision as a confirmation signal | on the 21 catalog templates, packaging text confirms ≥ 60% of identities (measured baseline 67%) with **zero false contradictions**; on `image.jpg` the signal stays informational (measured 10%); frame loop timing unchanged (detect p50 within 10% of baseline) | +| **2. Tag detection + OCR + association** | rail-band geometry, tag clustering, parsing, tag↔product association; API fields; web tag boxes + OCR card section | on the Phase-0 rail shot: ≥ 80% of tags detected, ≥ 70% of prices parsed correctly after voting; tap shows tag text in the web card | +| **3. Reconciliation + compliance** | signal table, states, five rules, compliance strip, stability filter | price-mismatch and missing-tag-group findings reproduce on a seeded shelf; unreadable rows produce hints, never violations | +| **4. Mobile lane** | ADR, camera bridge, Ktor WS client, domain models, iOS scan overlay + tap card, Android parity | on device: live boxes from the server, tap opens the product card with tag/OCR info | +| **5. Data upgrades** | JAN + regular_price columns, barcode decode, tag detector training | JAN from a tag promotes an ambiguous identity to accepted in a test | + +Phases 1-3 are server + web (this repo). Phase 5 is independent. + +**Phase 4 does not have to wait.** The mobile lane can start against protocol v1 +exactly as it is today: the app gets live boxes, tap-to-inspect, and the product card +with crop and feature matching. Tag/OCR fields are additive, so when Phase 2 lands the +card gains the OCR section without a protocol change and without touching the app's +transport layer. Starting it early is the parallel workstream; freezing Phase 2's +schema first only removes the option of renaming a field later. + +Recommended parallel split: + +| Track | Owner | Files | +| --- | --- | --- | +| Server: OCR service, tag pass, reconciliation, compliance | this repo | `code/ocr.py`, `code/tags.py`, `code/live.py`, `code/engine.py`, `code/app.py` | +| Web client: tag boxes, OCR card section, compliance strip | this repo | `code/web/**` (disjoint from the server files) | +| Mobile: ADR, camera bridge, WS client, tap card | dorayaki app repo | `shared/**`, `iosApp/**`, `androidApp/**` | + +--- + +## 14. Open questions for the owner + +1. **Capture**: do we shoot real tags (a store trip, or the existing photos at closer + range), or composite synthetic tags for the demo catalog? The second is faster and + makes reconciliation demonstrable; the first is more honest as a product claim. + Either way the demo needs one shelf whose tags and catalog describe the same store. +2. **Engine default**: accept RapidOCR as the cross-platform default, with macOS + Vision as the fast path on Mac? (PaddleOCR stays a candidate if Japanese accuracy + proves insufficient.) +3. **Mobile order**: iOS first then Android, or both at once? (The kit's contract is + both; the demo is an iPad.) +4. **Catalog**: add `jan` and `regular_price` now (needed for exact reconciliation), + or keep price-only reconciliation for the demo? +5. **Parallel start**: should the mobile lane start now against protocol v1, in + parallel with the server work?