tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

docs: design for price-tag OCR, reconciliation, compliance, and mobile scan

Browse source 
The owner's brainstorm thought through to an implementable design, backed by
measurements taken against the real demo images rather than assumptions:

- Tag readability is a capture-distance problem first. At image.jpg's ~2 m the
  tags are ~14 px digits and no engine reads them; at add/image6.jpg's ~1 m
  (22 px digits) macOS Vision reads all five tag fields including the large red
  sale price, and RapidOCR reads the printed fields at 0.97-1.00 confidence but
  misses the stylised one. Tesseract jpn is unusable on tags.
- Engine choice is an interface with backends: macOS Vision (182 ms/rail band)
  as the Mac fast path, RapidOCR (0.75-1.3 s/band, cross-platform) as the
  default elsewhere, fake for tests, PaddleOCR held in reserve. Cloud OCR is
  rejected: offline demo, and no owner decision to ship shelf images out.
- OCR must never sit in the frame loop: a TagService runs rail-band passes on a
  cadence with a spatial cache and multi-pass voting (tags are static), which
  also fixes digit misreads.
- Packaging text confirms 67% of the catalog templates (measured) but only
  10%/3% on the demo shelves, because the demo catalog is fictional and the
  photos are a real store. Absence is therefore neutral, never a contradiction,
  and the demo needs one shelf whose tags and catalog describe the same store.
- Compliance rules fail closed on evidence quality: a row whose tags are merely
  unreadable yields a capture hint, never a violation. The owner's "group of
  items missing price tags" is the missing_tag_group rule (>= 3 consecutive
  facings without a tag).
- Barcode/JAN is the exact-identity path, currently undecodable at demo
  distance; the catalog needs optional jan and regular_price columns.
- The mobile lane can start now against protocol v1: tap-to-inspect works with
  today's schema, and the tag fields are additive when they land.

Also records the five open questions for the owner (capture assets, engine
default, mobile order, catalog columns, parallel start).

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
kiyo9w 2026-09-24 13:45:54 +09:00
parent 5f93f6e278
commit 7c2a6f7061
1 changed files with 581 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

581
recognization/docs/design-price-tags-and-compliance.md Normal file
View file

@ -0,0 +1,581 @@
# Design: price-tag OCR, reconciliation, compliance, and the mobile client
Status: proposal, evidence-backed, not implemented.
Written 2026-09-24 after measuring OCR engines against the real demo images.
Read `docs/API.md` first for the system this extends.
The owner's brainstorm, restated as requirements:
1. Detect price tags in the image, transcribe them to text.
2. Run that **in parallel** with product recognition, so both boxes are shown.
3. **Reconcile** the recognized item with what the tag says; show both results.
4. Tap a product (web **and** mobile live scan) and see the product card **plus the
OCR information from the tag below it**.
5. **Price-tag compliance**: detect a group of items missing price tags.
Everything below is designed so that none of it slows down detection, and so that no
compliance claim is ever made from unreadable evidence.
---
## 1. What was measured before designing
All numbers from this machine (Apple M5, 10 cores), 2026-09-24.
### 1.1 Tag readability depends almost entirely on capture distance
| Image | Distance | Tag size | Digit height | macOS Vision on tag | RapidOCR on tag |
| --- | --- | --- | --- | --- | --- |
| `image.jpg` (2048 px) | ~2 m | ~110x35 px | ~14 px | **nothing** | not attempted (same scale) |
| `add/image6.jpg` (1500 px) | ~1 m | ~300x80 px | ~22 px | **reads 5/5 fields** | reads printed text, misses the big red digits |
Vision on a single `image6` tag (`300x80` crop, 191 ms) returned:
```
0.30 キュービーゼロ ノンコレステロール<310g> product name
1.00 357 regular price
0.30 2.4割引 discount
0.30 269 sale price (large, stylised)
1.00 290.52 unit price
```
RapidOCR on the same crop (263 ms) returned `357`, `2.4割引`, `290.52`, `<310g>` at
0.97-1.00 confidence but **missed `269`** (large red outlined digits).
**Conclusion:** a real Japanese price tag carries a JAN barcode, the product name,
the sale price, the regular price, the discount, and a unit price - and at the
demo's `image.jpg` distance none of it is readable by any engine. Tag OCR is a
capture-quality problem before it is a model problem.
### 1.2 Engine comparison
| Engine | Platform | Rail band 1500x120 | Full 2048 px image | Small printed text | Large stylised digits | Verdict |
| --- | --- | --- | --- | --- | --- | --- |
| macOS Vision (accurate) | macOS only | 182 ms, 20 regions | 510 ms, 58 regions | good | **reads them** | best here, not portable |
| RapidOCR (PP-OCR ONNX) | cross-platform | 752-1290 ms, 23 regions | ~3.4 s at 1280 px, 143 regions | good | misses them | cross-platform default |
| Tesseract `jpn` | cross-platform | ~300 ms | garbage on tags (`と1拓5 MI HMI`) | unusable | unusable | rejected |
| PaddleOCR (full) | cross-platform | heavier sibling of RapidOCR | - | - | - | candidate if RapidOCR's accuracy is not enough |
### 1.3 The same OCR pass reads packaging text - but confirmation depends on catalog coherence
Measured by OCR-ing an image and checking, for every accepted detection, whether a text
region inside its box matches the catalog name (normalized, longest-common-substring
ratio >= 0.4):
| Image | Accepted | Confirmed by packaging text | Note |
| --- | --- | --- | --- |
| the 21 catalog **template** photos | 21 | **14 (67%)** | catalog and photo describe the same product, by construction |
| `image.jpg` (2 m shelf) | 105 | 11 (10%) | only the products our fictional catalog happens to share with the photo (Tropicana, 野菜生活) |
| `add/image6.jpg` (1 m shelf) | 59 | 2 (3%) | the shelf is a different store's assortment (キューピー); our catalog does not describe it |
The template row is the honest measure of the *mechanism* (67%, with exact matches at
`野菜生活100オリジナル`, `野菜生活100ベリーサラダ`, `野菜生活100アップルサラダ`,
`歌舞伎揚`). The shelf rows measure the *data problem*: our demo catalog is fictional
and the photos are a real store, so most products have nothing to match against.
Failure modes seen in the 7 template misses, all explainable:
- **Romanized reads** - `亀田のまがりせんべい` came back as `Magari Senbei`; character
LCS scores 0.08 while the semantics match. Matching must compare kana/kanji *and*
a romaji transliteration of the catalog name.
- **Brand-only reads** - `三幸製菓`, `岩塚製菓` (the maker) without the product name.
- **Name not on the visible face** - the 明治 milk carton's name sat outside the
detected region.
Consequence for the design: packaging text is a **bonus confirmation**, never a gate.
Its absence must be neutral (never a contradiction), and its evaluation needs a
coherent catalog-photo pair - which is the same coherence requirement as price
reconciliation (§6.1).
So OCR has **two** uses:
- **Tag OCR** - price, name, JAN. Needs capture distance; blocked on demo data.
- **Packaging-text identity** - a second opinion on what the matcher decided; works
today on coherent data (67% on templates), informative but sparse on the demo
shelves.
### 1.4 The barcode is the exact-identity path, and it is currently too small
`image6`'s tag carries an EAN-13-style barcode; at this distance (~150x25 px, blurred)
`cv2.barcode.BarcodeDetector` fails even at 4x upscale. An EAN-13 needs roughly 3 px
per module (~300 px width) to decode. A phone at 40-60 cm decodes it trivially.
**Design consequence:** barcode decoding is the strongest reconciliation signal
available, and it is gated by the same capture requirement as the text.
### 1.5 The catalog has no JAN and no price semantics
`stock.csv` has `SKU_ID, product_name, receipt_date, expiration_date, number_in,
number_out, number_on, price`. There is no JAN, no regular-vs-sale price, and the demo
prices (¥168 for Tropicana) belong to a fictional store, not the store in the photo.
Exact reconciliation needs a catalog change, and an honest demo needs the catalog and
the tags to describe the same store.
---
## 2. Design principles this system inherits
1. **The UI never turns uncertainty into certainty** (project master). A tag read is
evidence with a confidence, never a fact.
2. **Fail closed on evidence quality.** "No tag detected" and "tag detected, text
unreadable" are different states. Only the first may become a compliance
violation; the second is a capture-quality hint.
3. **Detection stays real-time.** OCR is the slowest stage (0.2-3.4 s); it must never
sit in the frame's critical path.
4. **One product, one classification, one tag read.** Tags are static; the same
physical tag seen in 30 frames is one tag with one vote history, not 30 reads.
5. **Additive protocol changes only.** The live protocol is version 1; new fields are
optional, old clients keep working.
---
## 3. Architecture: a second, parallel perception stage
```
┌───────────────────────── frame loop (must stay ~25-45 ms) ─────┐
client frames ──▶ decode ──▶ detect ──▶ track ──▶ schedule classification ──▶ result
│ │ ▲
│ (same decoded frame, read-only) │
▼ ▼ │
┌──────────────────────────────────────────┐ │
│ TagService (own thread, 1-2 workers) │ cached tag state ────┘
│ │ merged into the result
│ rail bands from row geometry │
│ → crop → OCR (engine backend) │
│ → parse → vote over passes │
│ → associate tag ↔ product │
└──────────────────────────────────────────┘
│
▼
Reconciliation + compliance rules
(pure functions over settled evidence)
```
Why a separate service and not another stage in `process_frame`:
- **OCR is 10-100x slower than detection.** 0.75-1.3 s per rail band (RapidOCR) means
it cannot run per frame; it runs on a cadence and its answers are cached.
- **Tags are static.** A tag read stays valid for the whole pass over that shelf.
- **The frame loop already has a budget.** Detection (46 ms under live load) plus
tracking plus classification scheduling fills it.
### 3.1 The tag pass (cadence and caching)
A *tag pass* processes one rail band and updates the tag cache.
| Trigger | When |
| --- | --- |
| Periodic | every N frames (default 15, ~1-1.5 s at live rates) while tags are unsettled |
| Movement | the band's screen position moved more than ~2% of frame width since the last pass |
| Settle | a band whose tags are all `settled` (see below) is skipped entirely |
| On demand | photo mode, or the client asks (`inspect`) |
Band extraction reuses the geometry we already have: `empty_shelf_detector.cluster_rows()`
groups product boxes into rows, and a rail band is the strip below a row
(`row_bottom` to `row_bottom + ~8% of row height`, clipped to the frame). No new
geometry model is needed, and it is the same code the empty-slot detector trusts.
### 3.2 Voting: the accuracy lever
Every pass produces a `TagObservation` (box, text lines, parsed fields, confidence).
Observations that overlap the same tag (IoU ≥ 0.5 against the cached tag) accumulate
into that tag's history, and the *published* value is the vote, not the latest read:
- **Price**: majority vote over parsed integers; ties break to the highest-confidence
read. (A misread `26円` vs `269` is resolved by repetition - the single strongest
cheap accuracy improvement available, and the reason caching is a feature.)
- **Name**: longest common substring across observations, or the highest-confidence
read when observations disagree entirely.
- **Barcode**: first successful decode wins permanently (it either checksums or it
does not).
- A tag is `settled` after 3 agreeing observations or 8 passes, whichever comes first;
settled tags stop costing OCR time.
### 3.3 Latency budget
| Stage | Cost | Cadence | Contribution to frame loop |
| --- | --- | --- | --- |
| Rail band crop | <1 ms | per pass | 0 (outside the loop) |
| OCR, Vision backend | ~180 ms/band | 1-2 Hz | 0 |
| OCR, RapidOCR backend | 0.75-1.3 s/band | 0.3-0.5 Hz, band-parallel | 0 |
| Parse + vote + associate | <1 ms | per frame | ~0.2 ms |
| Merge into result | <0.5 ms | per frame | ~0.2 ms |
Worst realistic case: 4 bands x RapidOCR at 1.3 s = 5.2 s of work per full shelf
sweep, spread over 1-2 workers while detection keeps running. On macOS with Vision it
is ~0.7 s for the whole shelf. The frame loop is unaffected either way.
---
## 4. Price-tag detection
Four approaches were considered.
| Approach | Needs training data | Robust to blur | Works on demo images | Verdict |
| --- | --- | --- | --- | --- |
| A. Geometry only (bright quadrilateral in the rail band) | no | partly | no (tags too small to segment reliably) | fallback |
| B. Trained tag detector (YOLO-nano on tags) | **yes, none exists** | yes | after training | target state |
| C. OCR-native text regions clustered into tags | no | it is the same failure as OCR | at readable distance | **start here** |
| D. Hybrid: C now, B later, A as fallback | no → yes | yes | at readable distance | **recommended path** |
Recommended sequence: **C → B**.
**C (now).** The OCR engine's own detector returns text-region boxes (RapidOCR found
23 in a rail band; Vision found 20). Cluster regions inside a rail band into tag
candidates:
1. Take regions whose box height is 8-40% of the band height and whose center is
inside the band.
2. Cluster by horizontal proximity (gap ≤ 1.5 x median region height) into groups.
3. A group becomes a tag if its bounding box has aspect 2.5:1 to 8:1 and width
≥ 4% of frame width - the measured tag shape is ~300x80 px (3.75:1) at 1 m.
4. Reject groups that are pure noise: no parsed price AND no kana/kanji AND fewer
than 2 regions.
**B (later).** Bootstrap a tag dataset from C's output (auto-labelled tag boxes,
human-verified once), train a YOLO-nano (same ultralytics toolchain as the product
detector, same deployment path), and switch detection to it. A trained detector makes
tag detection independent of OCR quality - the tag box is found even when its text is
unreadable, which is exactly what compliance needs.
**A (fallback)** stays for the case where OCR returns nothing: a brightness/saturation
rectangle test inside the band, used only to answer "is there *something* tag-shaped
here", never to read text.
### 4.1 Associating a tag with a product
A tag belongs to the product whose facing sits directly above it on the same rail:
1. For each product box, the candidate band is `[row_bottom, row_bottom + 8% height]`.
2. Candidate tags are those with horizontal overlap ≥ 50% of the *tag* width, or,
failing that, nearest tag center within 0.5 x product width.
3. Each tag belongs to at most one product; each product gets at most one tag
(the closest by center distance). Unclaimed tags are `orphan_tag` candidates.
4. The association is recomputed per frame from cached tags - cheap, and it follows
the camera as the track moves.
---
## 5. OCR engine abstraction
Same pattern as the app's `platform/` bridges: one interface, several implementations,
picked by availability and configuration.
```python
class OcrEngine(Protocol):
name: str
def read(self, image: np.ndarray, *, hints: OcrHints) -> list[OcrLine]: ...
def available(self) -> bool: ...
@dataclass
class OcrLine:
text: str
confidence: float
box: list[int] # xyxy in the input image's pixels
@dataclass
class OcrHints:
languages: tuple[str, ...] = ("ja", "en")
mode: str = "sparse" # "sparse" = rail bands/tags; "dense" = full page
max_regions: int | None = None
```
Backends:
| Backend | Class | Selection rule |
| --- | --- | --- |
| `vision` | `VisionOcr` (macOS, `pyobjc` `VNRecognizeTextRequest`) | default on macOS when importable; measured 182 ms/band |
| `rapidocr` | `RapidOcr` (`rapidocr-onnxruntime`) | default elsewhere (Windows/Linux); measured 0.75-1.3 s/band |
| `fake` | `FakeOcr` | tests, and the deterministic demo |
| `paddle` | `PaddleOcr` | only if RapidOCR's Japanese accuracy proves insufficient |
Tesseract is rejected on measurement (§1.2). Cloud OCR (Google/Azure) is rejected: the
demo must run offline, and shipping shelf images to a third party is a privacy decision
the owner has not made.
Two preprocessing rules, both from the measurements:
- **Never OCR the whole frame in live mode.** Crop the rail band (and later the tag
box) - a full 1280 px frame costs 3.4 s in RapidOCR for 143 regions we do not want.
- **Upscale small tags 2-3x with INTER_CUBIC before OCR** when the tag's digit height
is below ~20 px; beyond 3x the blur wins and accuracy falls.
### 5.1 Parsing a Japanese price tag
Field extraction is deterministic parsing over the OCR lines, not another model:
| Field | Rule | Example |
| --- | --- | --- |
| `price` | largest integer 2-5 digits inside a red/black large region, or the integer adjacent to `円` | `269` |
| `regular_price` | integer near `割引` / on the left with smaller size | `357` |
| `discount` | `(\d+(\.\d+)?)\s*割` | `2.4割引` → 24% |
| `unit_price` | `\d+\.\d{1,2}` (per 100 g / per 10 mL convention) | `290.52` |
| `name` | longest kana/kanji run with confidence ≥ 0.3, excluding prices and sale words (`毎日お買得`) | `キュービーゼロ ノンコレステロール<310g>` |
| `jan` | barcode decode of the tag's barcode region (separate step) | `4902105...` |
| `flags` | `sale` when a discount or 割引 present, `tax_included` on 税込, `new` on 新発売 | |
Sanity rules that reject a parse outright: price outside 1-99999; discount > 90%;
more than one candidate price with no size ordering. A rejected parse publishes
`price: null` with `quality: "unparsed"` - never a guess.
---
## 6. Reconciliation: item vs tag
Inputs per facing: the identity decision (accepted / ambiguous / unknown / rejected,
with candidates), the associated tag (if any), and the catalog entry.
Signals, strongest first:
| Signal | Strength | Requires |
| --- | --- | --- |
| JAN decoded from the tag's barcode == catalog JAN | **exact** | catalog JAN column, capture quality |
| OCR name ≈ catalog name (normalized kana/kanji + romaji form, LCS ratio ≥ 0.6 for tags, ≥ 0.4 for packaging) | strong | readable name; catalog describes the store |
| OCR price == catalog price (regular or sale) | medium | coherent catalog/tag data |
| Position (tag directly below, same rail slot) | association only | geometry |
Matching must compare three forms of the catalog name - normalized kana/kanji,
NFKC-folded, and a romaji transliteration - because the packaging sometimes prints the
name in Latin script (`Magari Senbei` for 亀田のまがりせんべい, measured).
Reconciliation states (published per facing, alongside - never replacing - the
identity state):
| State | Condition | UI |
| --- | --- | --- |
| `confirmed` | accepted identity AND a strong/medium signal agrees | green chip "tag matches" |
| `mismatch` | accepted identity AND a signal disagrees (price differs, name unrelated, JAN differs) | amber chip, both values shown |
| `unverified` | accepted identity, tag present but unreadable or unparsed | grey chip "tag unread" |
| `no_tag` | accepted identity, no tag found under the facing | red chip (compliance input) |
| `tag_only` | tag read, no accepted identity above it | tag shown on its own |
Hard rule: `mismatch` never silently changes the product identity. It is a *second
opinion* displayed next to the first; only a JAN match may promote an `ambiguous`
identity to `accepted` (that is the one case where the tag is stronger evidence than
the matcher, because a checksummed barcode is not a guess).
### 6.1 Catalog changes required
```csv
SKU_ID,product_name,receipt_date,expiration_date,number_in,number_out,number_on,image_link,price,jan,regular_price
sku_003,Tropicana エッセンシャルズ マルチビタミン,...,168,4902105xxxxxx,198
```
- `jan` (optional): enables exact reconciliation.
- `regular_price` (optional): lets the tag's regular/sale pair be checked separately.
- Both are additive; `InventoryManager` already ignores unknown columns, so nothing
breaks until the columns appear.
- **Demo coherence:** for a believable reconciliation demo, the tags photographed must
belong to the same store as the catalog. Either shoot/curate shelf photos whose tags
match the demo catalog, or render synthetic tags for the demo catalog and place them
on the shelf (a small, honest demo asset - labelled as synthetic). Real-store photos
with our fictional catalog will legitimately show `mismatch` everywhere, which
demonstrates the compliance path but not the reconciliation path.
---
## 7. Price-tag compliance
First rule family of the (future) rules engine. Each rule is a pure function over
settled evidence, returning findings with boxes and counts - no side effects, no
network.
| Rule id | Fires when | Evidence | Guard |
| --- | --- | --- | --- |
| `missing_price_tag` | an accepted facing has no associated tag | facing box, row, SKU | only on rows where tag detection is `readable` (≥1 tag parsed in that row) |
| `missing_tag_group` | **≥ 3 consecutive facings in a row** have no tag | row index, span box, count | same guard; this is the owner's "group of items missing price tags" |
| `orphan_tag` | a tag has no facing above it | tag box, row | tag must be parsed |
| `price_mismatch` | `mismatch` state from a price signal | facing box, tag price, catalog price | catalog price must exist |
| `duplicate_tag` | one tag associated with two facings | tag box, both facing boxes | - |
| `unreadable_tags` | a row where tags were found but none parsed | row box, count | informational, never a violation |
The guard in the first two rows is the whole design: a row whose tags are simply
unreadable (every row in `image.jpg`) must produce a **capture-quality hint**, not a
compliance violation. That is principle 2, implemented.
Aggregation: findings roll up per row, then per frame/session, into
`compliance: {rows: [...], violations: N, by_rule: {...}}`. The UI shows a row-level
strip ("Row 3: 4 facings without price tag") with the offending boxes highlighted -
the same red-box language the empty-slot detector already uses.
Evaluation cadence: per frame from settled evidence only (cheap), with a stability
filter (a finding must hold for 3 consecutive frames) so nothing flickers.
---
## 8. Protocol changes (additive, protocol stays 1)
Per track/detection:
```json
"price_tag": {
"box": [812, 404, 909, 441],
"state": "readable | unreadable | absent",
"price": 269, "regular_price": 357, "discount": 0.24, "unit_price": 290.52,
"name": "キュービーゼロ ノンコレステロール<310g>",
"jan": null,
"confidence": 0.82,
"observations": 3,
"settled": true
},
"reconciliation": {
"state": "confirmed | mismatch | unverified | no_tag | tag_only",
"signals": [{"kind": "price", "agrees": true, "item": 269, "tag": 269}],
"message": null
}
```
Frame summary gains:
```json
"summary": {
"price_tags": {"readable": 12, "unreadable": 3, "absent": 4},
"compliance": {"violations": 1, "by_rule": {"missing_tag_group": 1},
"rows": [{"row_index": 3, "span": [x1,y1,x2,y2], "missing": 4,
"skus": ["sku_003", "sku_009"]}]}
}
```
New WebSocket control message (optional, for tap-to-inspect freshness):
```json
{"type": "read_tags", "request_id": "t1", "boxes": [[x1,y1,x2,y2]]}
→ {"type": "read_tags_result", "request_id": "t1", "tags": [...]}
```
REST `recognize` gains the same two fields per detection, plus
`?tags=true` to request the tag pass on a still (photo mode default on).
`inspect` gains `tag` (the tag associated with the tapped box) so the mobile/web card
can show OCR text without a second round trip.
---
## 9. Client changes (web now, mobile after)
Web (`code/web/`):
- Tag boxes drawn in a distinct style (thin teal outline, small price chip when
parsed) so they never read as product boxes.
- Product card gains an **OCR section**: price read vs catalog price, name read,
discount, and the reconciliation chip (`confirmed` / `mismatch` / `unreadable`).
- A **compliance strip** under the stage: rule, row, count; clicking highlights the
boxes.
- Quality hint gains `move_closer_for_tags` when a row's tags are unreadable - the
measured failure mode from §1.1.
---
## 10. Mobile live scan (the second ask)
Target: the dorayaki app's scan screen does what the web demo does - live boxes,
tap a product, see the product card, and see the tag/OCR information under it.
The app repo (`/Users/ngotrung/Documents/Projects/dorayaki`) requires an **ADR**
before a second API base URL is added (its `AGENTS.md`), and its `ScanCaptureView` is
currently a timer-driven placeholder (`ScanViewModel.tick()`), not a camera client.
So this lane is: one ADR, then a feature slice copied from the kit's own recipes.
| Piece | Where | What |
| --- | --- | --- |
| ADR | `docs/decisions/ADR-002-recognition-server.md` | second base URL, why, flavor mapping (Debug → `http://<mac>:8080`, Staging/Prod later) |
| Camera bridge | `shared/.../platform/CameraFrames.kt` + `iosMain` (AVFoundation) + `androidMain` (CameraX) | `Flow<FrameBytes>` at a chosen cadence; JPEG encode on the platform side (official APIs only) |
| Live client | `shared/.../data/shelf/LiveScanClient.kt` (Ktor WebSockets) | connect, send frames, receive results, expose `Flow<LiveScanFrame>` |
| Domain | `shared/.../domain/shelf/LiveScan.kt` | `LiveTrack`, `LiveIdentity`, `PriceTag`, `ComplianceFinding`, `LiveScanFrame` - mirroring the wire schema, with defaults for every optional field (kit rule) |
| ViewModel | `shared/.../feature/shelf/LiveScanViewModel.kt` | `@NativeCoroutinesState` flows; tap → `selectTrack(id)`; inspect via the socket |
| iOS UI | `iosApp/iosApp/ScanCaptureView.swift` | overlay boxes (SwiftUI `Canvas`), tap hit-test, product sheet with the OCR section |
| Android UI | `androidApp/.../screens/ScanScreen.kt` | same, Compose |
| Tests | `commonTest` | protocol parsing with a fake socket; identity/price-tag state mapping |
Sequencing inside the lane: iOS first (the owner demos on iPad), Android immediately
after, because the kit's contract is two native UIs.
Protocol subset the app needs: `hello`, binary frames, `result`, `inspect` /
`inspect_result`. Everything else is optional. Tap-to-inspect maps to the existing
`inspect` message, so the card shows crop + feature matching + tag in one reply.
Constraint to respect: phone cameras are only available over HTTPS or localhost, so
the demo path is either `--ssl-certfile` (mkcert) or a tailnet-only HTTPS edge. The
server already supports the flags; the ADR should record the chosen option.
---
## 11. Data and capture plan
1. **Capture requirement (measured):** tag digit height ≥ 20 px, tag width ≥ 150 px.
At a typical Japanese store tag (60 mm wide, digits ~8 mm), that is a phone at
40-80 cm from the rail, or a 4K frame at ~1.5 m. The live client should show
`move_closer_for_tags` below that.
2. **Tag images for the demo:** shoot one shelf rail (or three) at ~50 cm with a phone
in the existing demo store style; 10-20 tags is enough to validate the whole path.
Alternatively render synthetic tags for the demo catalog and composite them onto a
shelf photo - faster, honest if labelled, and it makes reconciliation demonstrable
without a real store trip.
3. **Tag detector dataset (later):** auto-label with the §4 pipeline, human-verify
once, train YOLO-nano. Target ~300 tag instances across 30-50 rail images.
4. **Catalog:** add `jan` and `regular_price` columns (optional, additive).
5. **Barcode decode:** `zxing-cpp` (pip, cross-platform) or `pyzbar` (needs the zbar
system library); decode the tag's barcode region only, at 2-3x upscale. Adopt once
capture quality allows - a decoded JAN is worth more than every other signal
combined.
---
## 12. Risks and failure modes
| Risk | Impact | Mitigation |
| --- | --- | --- |
| Demo images cannot validate tag OCR | the feature looks unproven at demo time | capture plan §11.1-2 before implementation lands; validate on the catalog templates meanwhile (67% measured) |
| Catalog does not describe the photographed store | text/price confirmation is sparse (measured 10% / 3%) and every "mismatch" is noise | treat absence as neutral; validate on coherent data (templates, or a re-shot demo shelf); §6.1 |
| OCR misreads a digit and a mismatch is reported | false compliance alarm | multi-pass voting (§3.2); mismatch is displayed, never auto-applied; compliance rules require the readable guard |
| OCR load starves detection | frame rate drops | separate worker, band crops only, settle-and-skip cache, measured budget §3.3 |
| Tag detector without training data | no tag boxes on unreadable rows | hybrid C→B; geometry fallback marks "something tag-shaped" without reading |
| Catalog/tag data incoherent in the demo | reconciliation demo shows only mismatches | §6.1 demo coherence choice |
| Japanese text normalization (kana variants, 半角/全角, size suffix) | name matching noise | normalize NFKC + strip size/`<310g>` before LCS |
| Mobile lane needs an ADR and touches the kit's architecture | schedule risk in the other repo | ADR first, then a slice copied from `add-kmp-feature`; no kit changes |
---
## 13. Phased plan
| Phase | Deliverable | Acceptance |
| --- | --- | --- |
| **0. Capture assets** | one rail shot at ≤80 cm (or synthetic tags composited) | a tag crop where Vision and RapidOCR both read price + name |
| **1. OCR service + packaging-text identity** | `OcrEngine` interface, Vision + RapidOCR + fake backends, band cropping, cache, vote; packaging text joins the identity decision as a confirmation signal | on the 21 catalog templates, packaging text confirms ≥ 60% of identities (measured baseline 67%) with **zero false contradictions**; on `image.jpg` the signal stays informational (measured 10%); frame loop timing unchanged (detect p50 within 10% of baseline) |
| **2. Tag detection + OCR + association** | rail-band geometry, tag clustering, parsing, tag↔product association; API fields; web tag boxes + OCR card section | on the Phase-0 rail shot: ≥ 80% of tags detected, ≥ 70% of prices parsed correctly after voting; tap shows tag text in the web card |
| **3. Reconciliation + compliance** | signal table, states, five rules, compliance strip, stability filter | price-mismatch and missing-tag-group findings reproduce on a seeded shelf; unreadable rows produce hints, never violations |
| **4. Mobile lane** | ADR, camera bridge, Ktor WS client, domain models, iOS scan overlay + tap card, Android parity | on device: live boxes from the server, tap opens the product card with tag/OCR info |
| **5. Data upgrades** | JAN + regular_price columns, barcode decode, tag detector training | JAN from a tag promotes an ambiguous identity to accepted in a test |
Phases 1-3 are server + web (this repo). Phase 5 is independent.
**Phase 4 does not have to wait.** The mobile lane can start against protocol v1
exactly as it is today: the app gets live boxes, tap-to-inspect, and the product card
with crop and feature matching. Tag/OCR fields are additive, so when Phase 2 lands the
card gains the OCR section without a protocol change and without touching the app's
transport layer. Starting it early is the parallel workstream; freezing Phase 2's
schema first only removes the option of renaming a field later.
Recommended parallel split:
| Track | Owner | Files |
| --- | --- | --- |
| Server: OCR service, tag pass, reconciliation, compliance | this repo | `code/ocr.py`, `code/tags.py`, `code/live.py`, `code/engine.py`, `code/app.py` |
| Web client: tag boxes, OCR card section, compliance strip | this repo | `code/web/**` (disjoint from the server files) |
| Mobile: ADR, camera bridge, WS client, tap card | dorayaki app repo | `shared/**`, `iosApp/**`, `androidApp/**` |
---
## 14. Open questions for the owner
1. **Capture**: do we shoot real tags (a store trip, or the existing photos at closer
range), or composite synthetic tags for the demo catalog? The second is faster and
makes reconciliation demonstrable; the first is more honest as a product claim.
Either way the demo needs one shelf whose tags and catalog describe the same store.
2. **Engine default**: accept RapidOCR as the cross-platform default, with macOS
Vision as the fast path on Mac? (PaddleOCR stays a candidate if Japanese accuracy
proves insufficient.)
3. **Mobile order**: iOS first then Android, or both at once? (The kit's contract is
both; the demo is an iPad.)
4. **Catalog**: add `jan` and `regular_price` now (needed for exact reconciliation),
or keep price-only reconciliation for the demo?
5. **Parallel start**: should the mobile lane start now against protocol v1, in
parallel with the server work?