tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

Compare commits

base: tngo:fix/live-recognition-quality
Branches Tags
tngo:main
tngo:fix/live-recognition-quality
vnguyen:main
..
compare: vnguyen:main
Branches Tags
vnguyen:main
tngo:main
tngo:fix/live-recognition-quality

No commits in common. "fix/live-recognition-quality" and "main" have entirely different histories.
fix/live-r ... main

595 changed files with 2016 additions and 673742 deletions
Download patch file Download diff file Expand all files Collapse all files

24
.github/workflows/test.yml vendored
View file

@ -1,24 +0,0 @@
name: tests
on:
push:
pull_request:
jobs:
model-free:
runs-on: ubuntu-latest
defaults:
run:
working-directory: recognization
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install pinned dependencies
run: python -m pip install -r requirements.lock
- name: Model-free suite
env:
YOLO_OFFLINE: "1"
YOLO_AUTOINSTALL: "false"
run: python -m unittest discover -s code -p 'test_*.py' -v

19
recognization/.gitignore vendored
View file

@ -19,23 +19,4 @@ ENV/
# Temporary weights
runs/detect/train/weights/last.pt
weights/dinov2_vits14_lvd142m.safetensors
*.log
stock-*/identity_head.pt
.gallery_cache_*.npz
.demo-token
config/local.env
# Fetched images. Every set keeps its manifest.json (source URL, licence,
# artist) and its results in git; the pixels are re-fetchable from those URLs
# and are 59 MB for the Wikimedia shelves alone.
data/external-shelves/*/*.jpg
data/external-shelves/*/*.jpeg
data/external-shelves/*/*.JPG
data/external-shelves/*/*.png
data/packshots/research/*/images/
data/packshots/research/*/pages/
data/packshots/research/*/*.jpg
# Rebuildable sweep cache (code/sweep_accept_lines.py --cache).
*.pkl

81
recognization/DEMO.md
View file

@ -1,81 +0,0 @@
# Running the demo
Two surfaces: the **web client** (open it in a browser) and the **iOS/Android app**
(scan screen). Both are clients of the same recognition server.
## 1. Start the server
```bash
cd recognization
YOLO_OFFLINE=1 .venv/bin/python code/app.py --port 8080 --classify-workers 8
```
Ready when `curl -s http://127.0.0.1:8080/v1/health` answers (about 25 s: the model
and the catalog load, then the OCR engine reports itself).
Flags that matter for a demo:
| Flag | Why |
| --- | --- |
| `--classify-workers 8` | More parallel classification; the demo's bottleneck. |
| `--host 0.0.0.0` | Only if a phone or iPad on the LAN must reach it (set `DORAYAKI_API_TOKEN` too). |
| `--no-tags` | Disables price-tag OCR entirely if it is fighting for CPU. |
| `DORAYAKI_OCR=none` | Same, via environment. |
## 2. The web client
Open <http://127.0.0.1:8080>.
- **Live** — pick a camera (or “Video file…”), press Start. Boxes appear as
products are recognised, with name and price; teal boxes mark the shelf-edge
price tags; amber means “check this”; red marks empty shelf space.
- **Photo** — a still shelf image: boxes paint immediately, identities fill in,
the algorithm selector re-classifies, and tapping a product opens the card with
the catalog photo, stock, and the tag reading compared against the catalog.
## 3. The app (iOS simulator, no camera needed)
The app can drive a **synthetic store walk**: the server renders a virtual camera
sweeping a real shelf photo — pan a row left to right, step up, pan back — so the
scan screen behaves like an employee walking the shelf.
Start the server on the port the app expects (8090 is the Debug default; 8097 is
what the verification scripts used — pass the override if you change it):
```bash
YOLO_OFFLINE=1 .venv/bin/python code/app.py --port 8090 --host 127.0.0.1 --classify-workers 8
```
Then run the app in the simulator and open **Scan**. Press the scan control: the
stage shows the walk moving, green boxes with names appear, teal tag boxes carry
prices, and tapping a product opens the card.
On a real device, replace the walk with the camera: the app uses the device camera
and the same server (a LAN IP needs HTTPS or an ATS exception — see
`docs/API.md`).
## 4. The demo configuration (measured)
Everything below was measured on this machine, 2026-09-24, one client, 1280×720
frames, the synthetic walk:
| Setting | Result |
| --- | --- |
| live method `orb` | **7.2 results/s, 178 labelled tracks, 21 distinct SKUs, first label 0.7 s** |
| live method `hybrid` | 1.4–9.9/s depending on load, 4–13 tracks, first label 3–5 s |
| walk `pan=0.02` | a row sweeps in ~8 s — reads like a person |
| walk `pan=0.06` | ~3 s per row — a time-lapse |
| OCR cadence (live) | 1 rail band every 30 frames; more than that drops the frame rate |
So: **ORB for live labelling, Hybrid for “Show matching”** (tap-to-inspect), and
the slower walk pace. The method menu in both clients switches between them live.
## 5. What to say, honestly
Working today: live detection and tracking, product identity with names and
prices, price-tag reading and the tag-vs-catalog comparison, empty-shelf gaps,
tap-to-inspect with the catalog photo and stock, and the walk.
Not built yet: configurable shelf rules (facing counts, brand blocks), the
fix → rescan → verify loop, and a head-office dashboard. Do not claim those on
camera.

251
recognization/README.md
View file

@ -1,208 +1,99 @@
# 🍡 Dorayaki - Smart Supermarket Shelf AI & Inventory Recognition
Hệ thống nhận diện sản phẩm trên kệ siêu thị và quản lý tồn kho theo kiến trúc **client - server**: mô hình thị giác máy tính (YOLO + Local Feature Matching: SURF/SIFT, ORB, BRISK, Hybrid theo paper *Electronics 2023, 12, 3640*) chạy trong một **recognition server**, còn các client (web, thiết bị chụp ảnh) chỉ gửi khung hình và nhận kết quả.
Server phục vụ **2 chế độ**:
1. **Live (nhận diện thời gian thực)** - camera hoặc video gửi liên tục qua WebSocket, sản phẩm được phát hiện trên từng khung hình, theo dõi (tracking) xuyên khung hình, phân loại một lần cho mỗi sản phẩm và gắn nhãn ngay trên màn hình.
2. **Photo (ảnh tĩnh)** - chụp/upload/chọn ảnh kệ, nhận diện toàn bộ kệ trong một lần, bấm vào từng sản phẩm để xem chi tiết, tồn kho và điểm matching.
Hệ thống nhận diện sản phẩm trên kệ siêu thị và quản lý tồn kho thời gian thực dựa trên kiến trúc kết hợp **YOLO** (Object Detection) và **Local Feature Matching** (SURF / SIFT, ORB, BRISK, Hybrid) theo paper quốc tế (*Electronics 2023, 12, 3640*).
---
## 📋 Yêu cầu hệ thống (System Requirements)
- **Hệ điều hành:** Windows 10/11, macOS hoặc Linux.
- **Python:** 3.10 - 3.12 (khuyên dùng 3.11/3.12).
- **Phần cứng:** CPU 4 lõi trở lên, RAM 8GB. GPU không bắt buộc: trên máy M5 đo được CPU nhanh hơn MPS (28ms so với 112ms mỗi khung hình).
- **Hệ điều hành:** Windows 10/11, macOS hoặc Linux (Ubuntu/Debian).
- **Python:** Python 3.9, 3.10 hoặc 3.11 (khuyên dùng Python 3.10).
- **Phần cứng:** CPU tối thiểu 4 lõi, RAM 8GB (có GPU NVIDIA là lợi thế nhưng CPU chạy hoàn toàn mượt mà).
---
## 🚀 Cài đặt và chạy
## 🚀 Hướng dẫn cài đặt và chạy Step-by-Step
### Bước 1: Chuẩn bị thư mục dự án
### Bước 1: Sao chép toàn bộ thư mục dự án sang máy mới
Đảm bảo bạn copy toàn bộ thư mục dự án, đặc biệt là các thư mục cốt lõi:
- code/: Toàn bộ mã nguồn backend, frontend, thuật toán nhận diện.
-
uns/: Chứa file trọng số mô hình YOLO đã train (
uns/detect/train/weights/best.pt).
- stock/: Chứa file catalog stock.csv và ảnh mẫu sản phẩm chuẩn sku_001 -> sku_016.
- dd/ & image.jpg: Các ảnh kệ hàng mẫu để test và demo.
-
equirements.txt: Danh sách thư viện cần thiết.
Cần có đầy đủ:
### Bước 2: Tạo môi trường ảo Python (Khuyên dùng)
Mở Terminal / PowerShell / Command Prompt tại thư mục dự án:
- `code/` - mã nguồn server, client web, client mẫu.
- `runs/` - trọng số YOLO đã train (`runs/detect/train/weights/best.pt`).
- `weights/dinov2_vits14_lvd142m.safetensors` - DINOv2 backbone for the identity gallery. Not in Git. See "DINOv2 weights" below.
- `stock/` - catalog `stock.csv` và ảnh mẫu sản phẩm `sku_001` -> `sku_022`.
- `add/` và `image.jpg` - ảnh kệ mẫu để test/demo.
- `requirements.txt`, `requirements-dev.txt`.
### Bước 2: Tạo môi trường ảo
```bash
conda create -n dorayaki python=3.11 -y
#### Cách 1: Sử dụng Conda (Khuyên dùng nếu đã cài Anaconda/Miniconda)
`ash
conda create -n dorayaki python=3.10 -y
conda activate dorayaki
```
`
hoặc venv:
#### Cách 2: Sử dụng venv chuẩn của Python
- **Trên Windows:**
`ash
python -m venv venv
venv\Scripts\activate
`
- **Trên macOS / Linux:**
`ash
python3 -m venv venv
source venv/bin/activate
`
```bash
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
```
### Bước 3: Cài đặt các thư viện phụ thuộc
Chạy lệnh sau để cài đặt toàn bộ package:
`ash
pip install -r requirements.txt
`
### Bước 3: Cài đặt thư viện
*(Ghi chú nếu máy có GPU NVIDIA và muốn tăng tốc độ YOLO, có thể cài PyTorch CUDA theo hướng dẫn tại pytorch.org).*
```bash
pip install -r requirements-dev.txt
```
### Bước 4: Khởi chạy ứng dụng Dorayaki Web App
### DINOv2 weights
The identity gallery embeds crops with DINOv2 ViT-S/14 (LVD-142M), Apache-2.0. It loads only a local file and does not contact Hugging Face, including when `YOLO_OFFLINE` is unset. Without the file, gallery identity fails and the server log says the weights are missing.
The pinned file is `model.safetensors` from `timm/vit_small_patch14_dinov2.lvd142m`, sha256 `04d27f3400d059fc0cfd7d17dd1909a75bf3ea8fb3eeb48b97cb99e57ee20081` (~84 MB). Copy a cache you already have; do not fetch one from an unknown URL:
```bash
mkdir -p weights
cp "$HOME/.cache/huggingface/hub/models--timm--vit_small_patch14_dinov2.lvd142m/snapshots/"*/model.safetensors \
weights/dinov2_vits14_lvd142m.safetensors
```
`DORAYAKI_DINOV2_WEIGHTS` overrides that path. A file with any other sha256 is refused.
With the gallery identity backend, enrolled views are embedded into an ignored
`stock-dir/.gallery_cache_<image_size>.npz` feature cache. The cache is reused
only when its key matches the preprocessing version, image size, pinned DINOv2
SHA-256, catalog CSV, enrolled SKU/view order and image bytes; its matrix shape,
numeric finite values and row identities are also checked. A stale or malformed
cache is rebuilt. A cache hit still checks the local model file's SHA-256, and
gallery startup refuses readiness if the pinned weights or usable gallery are
missing. OCR continues on the separate tag worker (`tag_executor`).
(`requirements-dev.txt` = `requirements.txt` + `httpx` cho test.)
### Bước 4: Khởi chạy recognition server
```bash
#### Cách 1: Chạy bằng lệnh Python
`ash
python code/app.py --port 8080
```
`
*(Nếu dùng macOS/Linux có thể dùng python3 code/app.py --port 8080)*
Script có sẵn: `run_app.bat` (Windows) hoặc `./run_app.sh` (Linux/macOS).
#### Cách 2: Chạy nhanh bằng script có sẵn
- **Trên Windows:** Click đúp vào file
un_app.bat
- **Trên Linux/macOS:** Chạy ./run_app.sh
Các tham số thường dùng:
| Tham số | Mặc định | Ý nghĩa |
| --- | --- | --- |
| `--host` | `127.0.0.1` | Đổi thành `0.0.0.0` để mở cho máy khác trong LAN (nên đặt `DORAYAKI_API_TOKEN`). |
| `--device` | `cpu` | `cpu`, `mps`, `cuda`, `cuda:0`. |
| `--imgsz` | `640` | Kích thước đầu vào của detector. |
| `--live-method` | `hybrid` | Thuật toán mặc định cho chế độ live. |
| `--max-sessions` | `4` | Số client live đồng thời. |
| `--ssl-certfile` / `--ssl-keyfile` | - | Bật HTTPS (cần cho camera trên điện thoại). |
### Bước 5: Mở ứng dụng
Mở trình duyệt: **http://127.0.0.1:8080** (hoặc `http://127.0.0.1:8080`)
- Tab **Live**: chọn camera (hoặc "Video file…") rồi bấm **Start**. Sản phẩm hiện khung xanh kèm tên và giá khi đã nhận diện chắc chắn; khung vàng "要確認" là trường hợp chưa chắc chắn.
- Tab **Photo**: chọn ảnh mẫu, chụp ảnh hoặc upload ảnh kệ. Nhấn vào sản phẩm để xem chi tiết và bấm **Show matching** để xem các điểm matching.
Camera trên điện thoại chỉ hoạt động khi trang mở bằng **https** hoặc **localhost** (yêu cầu bảo mật của trình duyệt).
### Bước 5: Mở ứng dụng trên trình duyệt
Mở trình duyệt (Chrome, Edge, Safari, Firefox) và truy cập:
👉 **http://localhost:8080** (hoặc **http://127.0.0.1:8080**)
---
## 🔌 API (dùng cho client khác)
## 🛠️ Các lệnh kiểm tra hệ thống (Test Scripts)
Server có REST + WebSocket đầy đủ, tài liệu chi tiết trong [`docs/API.md`](docs/API.md). Swagger UI: **http://127.0.0.1:8080/docs**.
Live WebSocket protocol `1` also supports an optional full-resolution crop lane:
clients advertise `capabilities`, then answer per-track `crop_request` messages
from their buffered capture. Clients without the capability are unchanged; wire
examples, byte limits, expiry and errors are in [`docs/API.md`](docs/API.md#websocket-v1stream).
```bash
# Nhận diện một ảnh kệ
curl -X POST --data-binary @image.jpg -H 'Content-Type: image/jpeg' \
'http://127.0.0.1:8080/v1/recognize?classify=true'
# Client live mẫu (camera, video, hoặc pan qua ảnh tĩnh)
python code/stream_client.py --source image.jpg --pan 12 --seconds 20
python code/stream_client.py --source 0 # webcam
```
Bất kỳ client nào gửi được ảnh JPEG đều dùng được: điện thoại, iPad, camera gắn xe đẩy, camera đeo cổ, robot. Giao thức không phụ thuộc trình duyệt.
For the native app demo, `./run-demo-server.sh` uses `stock-lopia-clean` by
default: `stock-lopia` is shelf crops cut from the `image6` evaluation photo, so
scoring against it is memorization. Pass another stock directory explicitly to
change stores; the script writes ignored local app endpoints and a local token.
### Identity backends
`--identity` picks what names a product. The default is `local`:
- `local` — RootSIFT over a per-SKU reference pyramid, similarity-constrained
RANSAC, a hue check, then a text lane (OCR of the crop against each catalog
row's name plus its packshot OCR) that vetoes a match whose package text names
a different row and answers on its own when geometry finds nothing. Measured
on the corrected `image6` labels at live resolution: 59% of in-catalog facings
named at 95% precision, 0 of 48 unknowns accepted. Rows with no enrolled image
are still answerable, because the text lane builds their signature from the
catalog name.
- `matcher` — Van's original SURF/ORB/BRISK composite, kept as the E-1 baseline
lane for comparison. Always answers, so it accepts near-unknown siblings.
- `gallery` — the DINOv2 embedding gallery. Fenced off: it is not the default
and it refuses to start without the pinned weights. Its recall collapses as
the catalog grows (7/42 recall@10 at 493 SKUs), so it is not the recognition
path.
The text lane needs OCR. On macOS that is Vision, which is already wired in
`code/ocr.py`; `DORAYAKI_OCR=none` disables it and the local lane then stands
alone (43% coverage at 79% precision on the same labels).
1. **Test toàn diện hệ thống (Test Suite):**
`ash
python code/test_suite.py
`
2. **Test nhận diện sản phẩm trên ảnh image.jpg:**
`ash
python code/test_image.py
`
3. **Test nhận diện các chỗ trống trên kệ (Empty Shelf Slots):**
`ash
python code/test_empty.py
`
*(Kết quả trực quan sẽ được xuất ra file detected_empty_slots.jpg với viền đỏ bao quanh các vị trí hết hàng).*
---
## 🛠️ Kiểm tra hệ thống (Tests)
From `recognization/`, after `pip install -r requirements.lock` (or `pip install -r requirements-dev.txt`):
```bash
YOLO_OFFLINE=1 YOLO_AUTOINSTALL=false python -m unittest discover -s code -p 'test_*.py' -v
```
That discovers every `code/test_*.py` module. `test_real_model` is included and skipped unless `DORAYAKI_REAL_MODEL=1` (it loads the YOLO weights). CI runs the same command from `.github/workflows/test.yml`.
```bash
YOLO_OFFLINE=1 python code/bench_engine.py # speed measurement, not part of the suite
```
---
## 🌟 Tính năng chính
1. **Nhận diện thời gian thực (Live)**: phát hiện sản phẩm trên từng khung hình, theo dõi xuyên khung hình, phân loại một lần cho mỗi sản phẩm, gắn nhãn tên/giá ngay trên khung hình; hiển thị số khung/giây và độ trễ.
2. **Phát hiện chỗ trống trên kệ (Out-of-Stock)**: khoanh đỏ các vị trí hết hàng kèm ước lượng số lượng thiếu; có bộ lọc ổn định để khung không nhấp nháy.
3. **Nhận diện ảnh tĩnh (Photo)**: nhận diện toàn bộ kệ trong một lần thay vì bấm từng sản phẩm.
4. **Phân loại đa thuật toán**: Hybrid (Paper), SURF/SIFT, ORB, BRISK - đổi được ngay trong lúc chạy.
5. **Trạng thái trung thực**: chỉ hiện tên sản phẩm khi kết quả được chấp nhận; trường hợp mơ hồ hiện "要確認" kèm các ứng viên, không đoán bừa.
6. **Tương tác chi tiết**: bấm vào sản phẩm để xem tên, giá, ngày nhập, hạn sử dụng, tồn kho và điểm matching.
7. **Đa client**: nhiều client kết nối đồng thời, mỗi client có session và tracking riêng.
8. **Đa ngôn ngữ**: Tiếng Anh và Tiếng Nhật.
9. **Gợi ý chất lượng ảnh**: cảnh báo lóa (glare), rung (blur), chụp quá xa.
---
## 📁 Cấu trúc mã nguồn
```
code/
app.py # server: REST /v1 + WebSocket /v1/stream + phục vụ web client
engine.py # RecognitionEngine: detector + matcher + catalog, thread-safe
live.py # LiveSession: tracking, phân loại nền, gắn nhãn, chất lượng ảnh
tracking.py # SessionTracker: ByteTrack với ID riêng cho từng session
detector.py # YOLO detector
feature_matcher.py # SURF/SIFT, ORB, BRISK, Hybrid
identity.py # accepted / ambiguous / unknown / rejected
empty_shelf_detector.py, inventory.py, gallery.py, tags.py, tag_cards.py
stream_client.py # client mẫu (camera / video / pan ảnh tĩnh)
bench_engine.py # đo tốc độ
test_*.py # unittest modules (discover -s code -p 'test_*.py')
web/ # web client (index.html, styles.css, app.js)
docs/API.md # tài liệu API đầy đủ
docs/design-price-tags-and-compliance.md # thiết kế OCR giá, đối chiếu, compliance, mobile
```
## 🌟 Các tính năng chính của ứng dụng
1. **Phát hiện sản phẩm (YOLO):** Tự động detect hơn 170+ vật phẩm trên kệ trong 1 lượt scan.
2. **Phát hiện chỗ trống trên kệ (Out-of-Stock Detection):** Tự động phát hiện các vị trí hết hàng và khoanh đỏ, ước lượng số lượng sản phẩm đang thiếu.
3. **Phân loại đa thuật toán:** Cho phép chọn giữa **Hybrid (Paper)**, **SURF / SIFT**, **ORB**, **BRISK**.
4. **Tương tác trực quan:** Click vào bất kỳ sản phẩm nào trên ảnh kệ để xem tên, giá, ngày nhập, hạn sử dụng, số lượng tồn kho và xem các điểm matching inliers.
5. **Đa ngôn ngữ:** Hỗ trợ chuyển đổi nhanh Tiếng Anh và Tiếng Nhật.
6. **Tải ảnh linh hoạt:** Chọn ảnh từ danh mục mẫu, chụp ảnh trực tiếp từ webcam/camera điện thoại, hoặc upload ảnh từ máy tính.

2333
recognization/code/app.py
View file

File diff suppressed because it is too large Load diff

337
recognization/code/bench_engine.py
View file

@ -1,337 +0,0 @@
"""
Component benchmark for the recognition engine.
Measures the two things that decide the live-mode defaults: how long detection
takes per frame (per device) and how long classification takes per product crop
(per method), including parallel throughput.
Usage (from recognization/):
.venv/bin/python code/bench_engine.py
.venv/bin/python code/bench_engine.py --devices cpu mps --image image.jpg
.venv/bin/python code/bench_engine.py --compare hybrid hybrid_fast
"""
from __future__ import annotations
import argparse
import json
import os
import statistics
import sys
import time
from concurrent.futures import ThreadPoolExecutor
from typing import Any, Dict, List, Optional
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(current_dir)
if os.path.exists(os.path.join(project_root, "stock")):
os.chdir(project_root)
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
import numpy as np # noqa: E402
def percentile(values: List[float], pct: float) -> float:
if not values:
return 0.0
ordered = sorted(values)
index = min(len(ordered) - 1, max(0, int(round((pct / 100.0) * (len(ordered) - 1)))))
return ordered[index]
def format_table(rows: List[List[str]], headers: List[str]) -> str:
widths = [len(h) for h in headers]
for row in rows:
for i, cell in enumerate(row):
widths[i] = max(widths[i], len(str(cell)))
lines = [" | ".join(h.ljust(widths[i]) for i, h in enumerate(headers)),
"-+-".join("-" * w for w in widths)]
for row in rows:
lines.append(" | ".join(str(cell).ljust(widths[i]) for i, cell in enumerate(row)))
return "\n".join(lines)
def load_image(path: str, width: Optional[int] = None) -> Any:
image = cv2.imread(path)
if image is None:
raise SystemExit(f"cannot read {path}")
if width:
scale = width / float(image.shape[1])
image = cv2.resize(image, (int(round(image.shape[1] * scale)), int(round(image.shape[0] * scale))))
return image
def bench_detection(image_path: str, devices: List[str], runs: int, imgsz: int) -> List[Dict[str, Any]]:
from detector import ShelfProductDetector
weights = "runs/detect/train/weights/best.pt"
results: List[Dict[str, Any]] = []
native = load_image(image_path)
frame = load_image(image_path, width=1280)
for device in devices:
try:
detector = ShelfProductDetector(weights, device=None if device == "default" else device, imgsz=imgsz)
except Exception as exc:
results.append({"device": device, "error": str(exc)})
continue
for label, sample in (("native", native), ("1280x720-ish", frame)):
latencies: List[float] = []
boxes = 0
for _ in range(runs):
started = time.perf_counter()
det = detector.detect(sample)
latencies.append((time.perf_counter() - started) * 1000.0)
boxes = len(det.get("boxes", []))
results.append(
{
"device": device,
"size": label,
"shape": f"{sample.shape[1]}x{sample.shape[0]}",
"boxes": boxes,
"p50_ms": round(statistics.median(latencies), 2),
"p95_ms": round(percentile(latencies, 95), 2),
"min_ms": round(min(latencies), 2),
}
)
return results
def box_iou(a: List[int], b: List[int]) -> float:
ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
iw, ih = max(0, ix2 - ix1), max(0, iy2 - iy1)
inter = iw * ih
if inter <= 0:
return 0.0
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
return inter / max(1, area_a + area_b - inter)
def detection_agreement(a: List[Dict[str, Any]], b: List[Dict[str, Any]]) -> Dict[str, Any]:
if not a or not b:
return {"matched": 0, "mean_best_iou": 0.0, "count_ratio": 0.0}
ious = []
for box_a in a:
best = max((box_iou(box_a["box"], box_b["box"]) for box_b in b), default=0.0)
ious.append(best)
return {
"matched": sum(1 for i in ious if i >= 0.5),
"mean_best_iou": round(float(np.mean(ious)), 3),
"count_ratio": round(len(a) / float(max(1, len(b))), 3),
}
def bench_crops(image_path: str, methods: List[str]) -> Dict[str, Any]:
from detector import ShelfProductDetector
from feature_matcher import ProductFeatureMatcher
detector = ShelfProductDetector("runs/detect/train/weights/best.pt")
matcher = ProductFeatureMatcher(stock_dir="stock")
image = load_image(image_path)
det = detector.detect(image)
crops = []
for box in det.get("boxes", []):
x1, y1, x2, y2 = box["box"]
crops.append(image[y1:y2, x1:x2].copy())
report: Dict[str, Any] = {"crop_count": len(crops), "methods": {}}
for method in methods:
latencies: List[float] = []
levels: Dict[str, int] = {}
skus: List[str] = []
for crop in crops:
started = time.perf_counter()
result = matcher.classify_crop(crop, method=method)
latencies.append((time.perf_counter() - started) * 1000.0)
level = result.get("identity_level", "error")
levels[level] = levels.get(level, 0) + 1
if result.get("best_sku"):
skus.append(result["best_sku"])
report["methods"][method] = {
"p50_ms": round(statistics.median(latencies), 2) if latencies else 0.0,
"p95_ms": round(percentile(latencies, 95), 2),
"mean_ms": round(float(np.mean(latencies)), 2) if latencies else 0.0,
"levels": levels,
"accepted_skus": len(set(skus)),
}
return report
def bench_parallel(crops: List[Any], matcher: Any, method: str, pools: List[int], num_threads: List[int]) -> List[Dict[str, Any]]:
rows = []
for threads in num_threads:
previous = cv2.getNumThreads()
cv2.setNumThreads(threads)
for pool in pools:
started = time.perf_counter()
with ThreadPoolExecutor(max_workers=pool) as executor:
list(executor.map(lambda c: matcher.classify_crop(c, method=method), crops))
elapsed = time.perf_counter() - started
rows.append(
{
"cv2_threads": threads,
"pool": pool,
"crops": len(crops),
"seconds": round(elapsed, 2),
"crops_per_s": round(len(crops) / elapsed, 1),
}
)
cv2.setNumThreads(previous)
return rows
def compare_methods(image_path: str, methods: List[str], extra_images: Optional[List[str]] = None) -> Dict[str, Any]:
"""Agreement between two methods on the same crops (adoption gate)."""
from detector import ShelfProductDetector
from feature_matcher import ProductFeatureMatcher
if len(methods) != 2:
raise SystemExit("--compare needs exactly two method names")
reference, candidate = methods
detector = ShelfProductDetector("runs/detect/train/weights/best.pt")
matcher = ProductFeatureMatcher(stock_dir="stock")
images = [image_path] + (extra_images or [])
accepted = 0
agree = 0
disagree_accepts = 0
total = 0
for path in images:
image = load_image(path)
for box in detector.detect(image).get("boxes", []):
x1, y1, x2, y2 = box["box"]
crop = image[y1:y2, x1:x2].copy()
if min(crop.shape[:2]) < 16:
continue
a = matcher.classify_crop(crop, method=reference)
b = matcher.classify_crop(crop, method=candidate)
total += 1
if a.get("identity_level") == "accepted":
accepted += 1
if a.get("best_sku") == b.get("best_sku"):
agree += 1
elif b.get("identity_level") == "accepted":
disagree_accepts += 1
return {
"reference": reference,
"candidate": candidate,
"crops": total,
"reference_accepted": accepted,
"agreement_on_accepted": round(agree / accepted, 4) if accepted else None,
"conflicting_accepts": disagree_accepts,
}
def main() -> None:
parser = argparse.ArgumentParser(description="Dorayaki engine component benchmark")
parser.add_argument("--image", default="image.jpg")
parser.add_argument("--devices", nargs="*", default=["default", "cpu", "mps"])
parser.add_argument("--runs", type=int, default=15)
parser.add_argument("--imgsz", type=int, default=640)
parser.add_argument("--methods", nargs="*", default=["surf", "orb", "brisk", "hybrid"])
parser.add_argument("--pools", nargs="*", type=int, default=[1, 4, 8])
parser.add_argument("--cv2-threads", nargs="*", type=int, default=[0, 1])
parser.add_argument("--compare", nargs="*", default=None)
parser.add_argument("--compare-images", nargs="*", default=None)
parser.add_argument("--json", default=None)
args = parser.parse_args()
print("=" * 78)
print("DORAYAKI ENGINE BENCHMARK")
print("=" * 78)
from detector import ShelfProductDetector
started = time.perf_counter()
detector = ShelfProductDetector("runs/detect/train/weights/best.pt")
load_ms = (time.perf_counter() - started) * 1000.0
print(f"model load: {load_ms:.0f} ms")
warm_image = load_image(args.image, width=1280)
started = time.perf_counter()
detector.detect(warm_image)
print(f"first predict (cold): {(time.perf_counter() - started) * 1000.0:.0f} ms")
started = time.perf_counter()
detector.detect(warm_image)
print(f"second predict (warm): {(time.perf_counter() - started) * 1000.0:.0f} ms")
print("\n--- detection latency ---")
detection = bench_detection(args.image, args.devices, args.runs, args.imgsz)
rows = []
for row in detection:
if "error" in row:
rows.append([row["device"], "-", "-", "-", f"ERROR: {row['error'][:40]}"])
continue
rows.append([row["device"], row["size"], row["shape"], row["boxes"],
f"{row['p50_ms']} / {row['p95_ms']}"])
print(format_table(rows, ["device", "size", "shape", "boxes", "p50/p95 ms"]))
# device agreement: do mps boxes match cpu boxes?
by_device = {}
for device in args.devices:
if device in ("default", "cpu"):
continue
try:
detector_d = ShelfProductDetector("runs/detect/train/weights/best.pt",
device=None if device == "default" else device)
except Exception:
continue
by_device[device] = detector_d.detect(warm_image).get("boxes", [])
cpu_boxes = None
try:
cpu_detector = ShelfProductDetector("runs/detect/train/weights/best.pt", device="cpu")
cpu_boxes = cpu_detector.detect(warm_image).get("boxes", [])
except Exception:
pass
if cpu_boxes is not None:
for device, boxes in by_device.items():
print(f"agreement cpu vs {device}: {json.dumps(detection_agreement(cpu_boxes, boxes))}")
print("\n--- classification latency (real crops from", args.image, ") ---")
crop_report = bench_crops(args.image, args.methods)
print(f"crops: {crop_report['crop_count']}")
rows = []
for method, stats in crop_report["methods"].items():
levels = stats["levels"]
rows.append([method, f"{stats['p50_ms']} / {stats['p95_ms']}",
f"acc={levels.get('accepted', 0)}",
f"amb={levels.get('ambiguous', 0)}",
f"unk={levels.get('unknown', 0)}",
f"rej={levels.get('rejected', 0)}",
f"skus={stats['accepted_skus']}"])
print(format_table(rows, ["method", "p50/p95 ms", "accepted", "ambiguous", "unknown", "rejected", "distinct skus"]))
print("\n--- parallel throughput (hybrid) ---")
from feature_matcher import ProductFeatureMatcher
matcher = ProductFeatureMatcher(stock_dir="stock")
image = load_image(args.image)
boxes = detector.detect(image).get("boxes", [])
crops = [image[b["box"][1]:b["box"][3], b["box"][0]:b["box"][2]].copy() for b in boxes]
rows = []
for row in bench_parallel(crops, matcher, "hybrid", args.pools, args.cv2_threads):
rows.append([row["cv2_threads"], row["pool"], row["crops"], row["seconds"], row["crops_per_s"]])
print(format_table(rows, ["cv2_threads", "pool", "crops", "seconds", "crops/s"]))
if args.compare:
print("\n--- method agreement ---")
comparison = compare_methods(args.image, args.compare, args.compare_images)
print(json.dumps(comparison, indent=2))
if args.json:
with open(args.json, "w") as handle:
json.dump(
{
"model_load_ms": round(load_ms, 1),
"detection": detection,
"classification": crop_report,
},
handle,
indent=2,
)
print(f"\nwrote {args.json}")
if __name__ == "__main__":
main()

68
recognization/code/bench_identity.py
View file

@ -1,68 +0,0 @@
"""
How many views per SKU does enrollment need?
The number that decides the capture spec: one enrolled view identifies 57 % of
facings, two lift it to 72 %, three to 75 %. The shipped matcher goes the other
way (33 % -> 5 %) because its raw inlier score is not calibrated to be aggregated
across views, and it cannot even express two views of one SKU.
python code/bench_identity.py # expects the eval crops described below
"""
import glob, os, sys, random
sys.path.insert(0, "code")
import cv2, numpy as np, torch, timm
from feature_matcher import ProductFeatureMatcher
# Crops extracted from add/image6.jpg and labelled by shelf position; the labels
# are read off the shelf, not off the matcher being measured.
CROPS = sorted(glob.glob(os.environ.get("DORAYAKI_EVAL_CROPS", "/tmp/dorayaki-verify/eval") + "/*.png"))
items = [(p, os.path.basename(p).replace(".png","").split("_",1)[1]) for p in CROPS]
by_class = {}
for path, label in items:
by_class.setdefault(label, []).append(path)
matcher = ProductFeatureMatcher(stock_dir="stock-lopia")
model = timm.create_model("vit_small_patch14_dinov2.lvd142m", pretrained=True, num_classes=0); model.eval()
MEAN = np.array([0.485,0.456,0.406], np.float32); STD = np.array([0.229,0.224,0.225], np.float32)
def embed(paths, size=518):
batch = []
for path in paths:
image = cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)
h, w = image.shape[:2]; side = max(h, w)
canvas = np.zeros((side, side, 3), np.uint8)
canvas[(side-h)//2:(side-h)//2+h, (side-w)//2:(side-w)//2+w] = image
image = cv2.resize(canvas, (size,size), interpolation=cv2.INTER_AREA).astype(np.float32)/255.0
batch.append(((image-MEAN)/STD).transpose(2,0,1))
with torch.no_grad():
features = model(torch.from_numpy(np.stack(batch)))
return torch.nn.functional.normalize(features, dim=1).numpy()
def matcher_best(crop):
"""Scores grouped by SKU, so several views of one SKU count as one answer."""
scores, _ = matcher.score_crop(crop)
grouped = {}
for name, value in scores.items():
sku = name.split("_")[0] + "_" + name.split("_")[1] if name.startswith("sku_l") else name
grouped[sku] = max(grouped.get(sku, 0.0), float(value))
return max(grouped.items(), key=lambda kv: kv[1])[0] if grouped else None
random.seed(7)
print(f"{len(items)} crops, {len(by_class)} classes")
print(f" {'views/SKU':>9} {'embedding top-1':>15} {'matcher top-1':>14}")
for views in (1, 2, 3, 4):
gallery, test = [], []
for label, paths in by_class.items():
shuffled = paths[:]; random.shuffle(shuffled)
gallery += [(p, label) for p in shuffled[:views]]
test += [(p, label) for p in shuffled[views:]]
if not test:
continue
features_gallery = embed([p for p, _ in gallery])
features_test = embed([p for p, _ in test])
gallery_labels = [l for _, l in gallery]
similarity = features_test @ features_gallery.T
correct = sum(1 for i, (_, truth) in enumerate(test) if gallery_labels[int(similarity[i].argmax())] == truth)
matcher_correct = sum(1 for p, truth in test if matcher_best(cv2.imread(p)) == truth)
print(f" {views:>9} {correct:>6}/{len(test):<6} {100*correct//len(test):>3}% "
f"{matcher_correct:>6}/{len(test):<6} {100*matcher_correct//len(test):>3}%")

42
recognization/code/bench_ocr.py
View file

@ -1,42 +0,0 @@
"""OCR cost per crop, on production-shaped crops.
The decision record claims ~18-26 ms per crop on Mac and re-measured 18.0 ms on
40 production crops. This is that measurement, so the number is reproducible
rather than remembered.
.venv/bin/python code/bench_ocr.py
"""
import json, statistics, sys, time
sys.path.insert(0, 'code')
import cv2
from eval_identity import LocalLaneAdapter, crop_of
lane = LocalLaneAdapter('stock-lopia-clean')
labels = json.load(open('eval/labels/image6.json'))
image = cv2.imread('add/image6.jpg')
crops = []
for facing in labels['facings']:
if facing.get('label') in (None, 'exclude'):
continue
crop = crop_of(image, facing['box'])
if crop is not None and crop.size:
crops.append(crop)
if len(crops) >= 40:
break
# Warm the recognizer: the first call loads the Vision model.
lane.text_lane.read(crops[0])
times = []
for crop in crops:
started = time.perf_counter()
lane.text_lane.read(crop)
times.append((time.perf_counter() - started) * 1000)
times.sort()
print(f'{len(times)} crops')
print(f' p50 {statistics.median(times):.1f} ms')
print(f' mean {statistics.mean(times):.1f} ms')
print(f' p95 {times[int(len(times) * 0.95)]:.1f} ms')
print(f' min {times[0]:.1f} max {times[-1]:.1f}')

103
recognization/code/benchmark.py Normal file
View file

@ -0,0 +1,103 @@
"""
Benchmark Suite for Supermarket Product Classification
Compares SURF, ORB, BRISK, and Hybrid methods following
MDPI Electronics 2023, 12(17), 3640.
"""
import os
import sys
import time
import cv2
import numpy as np
from typing import Dict, List, Any
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from feature_matcher import ProductFeatureMatcher
def run_benchmark(stock_dir: str = "stock", num_runs: int = 3):
print("=" * 70)
print("RETAIL SHELF PRODUCT RECOGNITION - ALGORITHM BENCHMARK")
print("Methods Evaluated: SURF | ORB | BRISK | HYBRID (Paper Method)")
print("=" * 70)
matcher = ProductFeatureMatcher(stock_dir=stock_dir)
templates = matcher.db.templates
if not templates:
print("[Error] No templates found in stock directory.")
return
methods = ["surf", "orb", "brisk", "hybrid"]
results = {m: {"latencies": [], "inliers": [], "keypoints_query": []} for m in methods}
# Test images: We will evaluate using crops synthesized with slight transformations
# (scale, rotation, noise) as well as the template images themselves
print(f"\nEvaluating across {len(templates)} catalog SKUs (repeat count: {num_runs})...\n")
for sku_id, tmpl in templates.items():
original_img = tmpl["image"]
h, w = original_img.shape[:2]
# Apply synthetic shelf condition (rotation by 10 deg, resize by 0.85)
M = cv2.getRotationMatrix2D((w / 2, h / 2), 10, 0.85)
query_img = cv2.warpAffine(original_img, M, (w, h))
for method in methods:
times = []
for _ in range(num_runs):
t0 = time.perf_counter()
res = matcher.classify_crop(query_img, method=method)
t1 = time.perf_counter()
times.append((t1 - t0) * 1000.0) # ms
avg_latency = np.mean(times)
results[method]["latencies"].append(avg_latency)
# Record inliers/score
if res.get("status") == "success":
top_sku = res["best_sku"]
score = res["score"]
results[method]["inliers"].append(score)
# Summarize metrics
summary_rows = []
for method in methods:
latencies = results[method]["latencies"]
scores = results[method]["inliers"]
avg_lat = np.mean(latencies) if latencies else 0.0
p95_lat = np.percentile(latencies, 95) if latencies else 0.0
avg_score = np.mean(scores) if scores else 0.0
# Paper characteristics notes
notes = {
"surf": "Scale & rotation robust; Hessian gradient features",
"orb": "Ultra fast binary descriptor; FAST + BRIEF",
"brisk": "Scale-space AGAST; circular pattern descriptor",
"hybrid": "Paper fusion (SURF+ORB+BRISK); optimal accuracy/speed"
}[method]
summary_rows.append([
method.upper(),
f"{avg_lat:.2f} ms",
f"{p95_lat:.2f} ms",
f"{avg_score:.1f}",
notes
])
headers = ["Method", "Avg Latency (ms)", "P95 Latency (ms)", "Avg Match Score", "Characteristics / Paper Findings"]
print("\n" + "-" * 105)
print(f"{headers[0]:<10} | {headers[1]:<18} | {headers[2]:<18} | {headers[3]:<16} | {headers[4]}")
print("-" * 105)
for row in summary_rows:
print(f"{row[0]:<10} | {row[1]:<18} | {row[2]:<18} | {row[3]:<16} | {row[4]}")
print("-" * 105)
print("\n[Benchmark Complete] All methods operational.")
if __name__ == "__main__":
run_benchmark()

79
recognization/code/build_stock_all.py
View file

@ -1,79 +0,0 @@
"""Merge every stock folder into one catalog the server can match against.
The demo had five stock folders and the server could only read one, so a shelf
holding anything outside the active folder came back blank. SKU ids are unique
across the folders (`sku_0xx`, `sku_lxx`, `sku_mxx`, `sku_kxx`), so the union is
a plain copy plus a CSV concat - no id rewriting, no dedup heuristics.
`stock-demo` is byte-identical to `stock-mediya` + `stock-milk150`, so it is not
a source: including it would only duplicate 62 images.
Run: .venv/bin/python code/build_stock_all.py
"""
from __future__ import annotations
import csv
import os
import shutil
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
SOURCES = ("stock", "stock-lopia", "stock-mediya", "stock-milk150")
DEST = os.path.join(ROOT, "stock-all")
IMAGE_SUFFIXES = (".jpg", ".jpeg", ".png")
FIELDS = ("SKU_ID", "product_name", "receipt_date", "expiration_date",
"number_in", "number_out", "number_on", "image_link", "price")
def main() -> int:
if os.path.isdir(DEST):
shutil.rmtree(DEST)
os.makedirs(DEST)
rows: list[dict] = []
seen_sku: dict[str, str] = {}
copied = 0
for source in SOURCES:
directory = os.path.join(ROOT, source)
if not os.path.isdir(directory):
print(f"skip {source}: not a directory", file=sys.stderr)
continue
for name in sorted(os.listdir(directory)):
if not name.lower().endswith(IMAGE_SUFFIXES):
continue
target = os.path.join(DEST, name)
if os.path.exists(target):
# Same filename in two folders is the same product photographed
# once; the first copy wins and the duplicate is reported.
print(f"duplicate image {name} from {source}", file=sys.stderr)
continue
shutil.copy2(os.path.join(directory, name), target)
copied += 1
csv_path = os.path.join(directory, "stock.csv")
if not os.path.exists(csv_path):
continue
with open(csv_path, newline="", encoding="utf-8") as handle:
for row in csv.DictReader(handle):
sku = (row.get("SKU_ID") or "").strip()
if not sku:
continue
if sku in seen_sku:
print(f"duplicate SKU {sku} in {source} (kept {seen_sku[sku]})",
file=sys.stderr)
continue
seen_sku[sku] = source
rows.append({field: (row.get(field) or "").strip() for field in FIELDS})
with open(os.path.join(DEST, "stock.csv"), "w", newline="", encoding="utf-8") as handle:
writer = csv.DictWriter(handle, fieldnames=FIELDS)
writer.writeheader()
writer.writerows(rows)
print(f"{DEST}: {copied} images, {len(rows)} SKUs")
return 0
if __name__ == "__main__":
raise SystemExit(main())

253
recognization/code/build_store_from_shelf.py
View file

@ -1,253 +0,0 @@
"""
Crop a labelled shelf photo into a stock directory.
This is a one-shot offline helper. SKU names and prices are supplied by a human
who read a contact sheet; this script does not enroll a phone recording and
does not talk to the recognition server.
The Meidiya dairy labels that built stock-mediya are kept as --example mediya.
Point --image, --boxes, and --output at real paths. There is no default home
directory or /tmp path.
python code/build_store_from_shelf.py \\
--image shelf.jpg --boxes boxes.json --output /path/to/new-stock \\
--labels labels.json
boxes.json is {"boxes": [[x1, y1, x2, y2], ...]} in the image's own pixels;
this script does not rescale the photo. labels.json is
{"products": [{"sku", "name", "price", "boxes": [index, ...]}]}.
An existing output directory is left untouched unless --replace is passed,
and even then the old directory is renamed to "<output>.previous", not deleted.
"""
from __future__ import annotations
import argparse
import csv
import json
import os
import shutil
import sys
import tempfile
# ponytail: demo counts/dates are the Meidiya sheet's bookkeeping, not a catalog API.
EXAMPLE_RECEIPT = "2026-09-25"
EXAMPLE_EXPIRATION = "2026-10-20"
EXAMPLE_COUNTS = (24, 8, 16)
# Named example only. These are the labels read off the Meidiya contact sheet,
# not an enrollment of a phone walk.
MEDIYA_PRODUCTS = [
{"sku": "sku_m01", "name": "小岩井 低脂肪牛乳", "price": 186, "boxes": [9, 23, 31]},
{"sku": "sku_m02", "name": "荻山酪農 カフェ・オ・レ", "price": 397, "boxes": [10]},
{"sku": "sku_m03", "name": "丹那 3.6牛乳", "price": 214, "boxes": [11]},
{"sku": "sku_m04", "name": "木次 ミルクコーヒー", "price": 195, "boxes": [12]},
{"sku": "sku_m05", "name": "木次 パスチャライズ牛乳", "price": 240, "boxes": [13]},
{"sku": "sku_m06", "name": "キスキ ブラウン・スイス牛乳", "price": 254, "boxes": [14]},
{"sku": "sku_m07", "name": "丹那 飲んでやさしいヨーグルト", "price": 225, "boxes": [15]},
{"sku": "sku_m08", "name": "農場3.7牛乳 特選", "price": 216, "boxes": [24]},
{"sku": "sku_m09", "name": "MINOR FIGURES OAT", "price": 300, "boxes": [26]},
{"sku": "sku_m10", "name": "北海道 低脂肪乳", "price": 214, "boxes": [27]},
{"sku": "sku_m11", "name": "タカナシ 低温殺菌牛乳", "price": 396, "boxes": [28]},
{"sku": "sku_m12", "name": "RUDE HEALTH Oat Organic", "price": 400, "boxes": [29]},
{"sku": "sku_m13", "name": "小岩井 無脂肪牛乳", "price": 178, "boxes": [30]},
{"sku": "sku_m14", "name": "きすき ノンホモ牛乳", "price": 198, "boxes": [8]},
]
class BuildStoreError(Exception):
"""Refused input. The destination, if it already existed, is unchanged."""
def _is_within(path: str, root: str) -> bool:
path = os.path.realpath(path)
root = os.path.realpath(root)
return path == root or path.startswith(root + os.sep)
def _load_json(path: str):
try:
with open(path, encoding="utf-8") as handle:
return json.load(handle)
except (OSError, json.JSONDecodeError) as exc:
raise BuildStoreError(f"{path}: {exc}") from exc
def load_boxes(path: str) -> list:
payload = _load_json(path)
boxes = payload.get("boxes") if isinstance(payload, dict) else None
if not isinstance(boxes, list) or not boxes:
raise BuildStoreError(f"{path}: expected a non-empty {{\"boxes\": [...]}} list")
parsed = []
for index, box in enumerate(boxes):
if not isinstance(box, (list, tuple)) or len(box) != 4:
raise BuildStoreError(f"{path}: box {index} is not [x1, y1, x2, y2]")
try:
coords = [int(value) for value in box]
except (TypeError, ValueError) as exc:
raise BuildStoreError(f"{path}: box {index} is not four numbers") from exc
x1, y1, x2, y2 = coords
if x2 <= x1 or y2 <= y1:
raise BuildStoreError(f"{path}: box {index} has no area")
parsed.append(coords)
return parsed
def load_labels(path: str | None, example: str | None) -> list:
if example and path:
raise BuildStoreError("pass either --labels or --example, not both")
if example == "mediya":
return [dict(item) for item in MEDIYA_PRODUCTS]
if example:
raise BuildStoreError(f"unknown example {example!r}; the only named example is mediya")
if not path:
raise BuildStoreError("pass --labels, or --example mediya for the demo sheet")
payload = _load_json(path)
products = payload.get("products") if isinstance(payload, dict) else None
if not isinstance(products, list) or not products:
raise BuildStoreError(f"{path}: expected a non-empty {{\"products\": [...]}} list")
return products
def _sku_name(value) -> str:
text = str(value).strip()
if not text or text in {".", ".."} or "/" in text or "\\" in text or "\x00" in text:
raise BuildStoreError(f"refusing sku {value!r}: it is not a single file name")
return text
def _products(labels: list) -> list:
seen = set()
products = []
for item in labels:
if not isinstance(item, dict):
raise BuildStoreError("each product must be an object with sku, name, price, boxes")
sku = _sku_name(item.get("sku", ""))
if sku in seen:
raise BuildStoreError(f"duplicate sku {sku}")
seen.add(sku)
name = str(item.get("name", "")).strip()
if not name or any(char in name for char in ",\n\r"):
raise BuildStoreError(f"{sku}: name must be one CSV field")
try:
price = int(item["price"])
indices = [int(index) for index in item["boxes"]]
except (KeyError, TypeError, ValueError) as exc:
raise BuildStoreError(f"{sku}: price and boxes must be integers") from exc
if price < 0 or not indices:
raise BuildStoreError(f"{sku}: price and boxes must be present")
products.append({"sku": sku, "name": name, "price": price, "boxes": indices})
return products
def check_destination(output: str, image: str, boxes_path: str, labels_path: str | None, replace: bool) -> str:
output = os.path.abspath(output)
real_output = os.path.realpath(output)
if real_output == os.path.realpath(os.sep) or real_output == os.path.realpath(os.getcwd()):
raise BuildStoreError("refusing to use the filesystem root or the current directory as the store")
parent = os.path.dirname(output)
if not parent or not os.path.isdir(parent):
raise BuildStoreError(f"{output}: parent directory does not exist")
for source in (image, boxes_path, labels_path):
if source and (_is_within(source, real_output) or _is_within(real_output, os.path.realpath(source))):
raise BuildStoreError(f"refusing to write {output}: it aliases input {source}")
if os.path.lexists(output):
if not replace:
raise BuildStoreError(f"{output}: already exists; pass --replace to move it aside")
if os.path.islink(output) or not os.path.isdir(output):
raise BuildStoreError(f"{output}: --replace only moves aside a real directory")
previous = output + ".previous"
if os.path.lexists(previous):
raise BuildStoreError(f"{previous}: already exists; move it away before --replace")
return output
def build_store(image_path: str, boxes: list, products: list, output: str, replace: bool,
boxes_path: str, labels_path: str | None) -> int:
output = check_destination(output, image_path, boxes_path, labels_path, replace)
import cv2
image = cv2.imread(image_path)
if image is None or getattr(image, "size", 0) == 0:
raise BuildStoreError(f"{image_path}: not a readable image")
height, width = image.shape[:2]
parent = os.path.dirname(output)
staging = tempfile.mkdtemp(prefix=".build-store-", dir=parent)
try:
rows = [[
"SKU_ID", "product_name", "receipt_date", "expiration_date",
"number_in", "number_out", "number_on", "image_link", "price",
]]
written = []
for product in products:
saved = 0
for index in product["boxes"]:
if index < 0 or index >= len(boxes):
raise BuildStoreError(f"{product['sku']}: box {index} is outside the boxes file")
x1, y1, x2, y2 = boxes[index]
x1, y1 = max(0, x1), max(0, y1)
x2, y2 = min(width, x2), min(height, y2)
crop = image[y1:y2, x1:x2] if x2 > x1 and y2 > y1 else image[0:0, 0:0]
if crop.size == 0:
raise BuildStoreError(f"{product['sku']}: box {index} does not crop the image")
suffix = "" if saved == 0 else f"_{saved}"
target = os.path.join(staging, f"{product['sku']}{suffix}.jpeg")
if not cv2.imwrite(target, crop):
raise BuildStoreError(f"{product['sku']}: failed to write crop {index}")
saved += 1
written.append(product["sku"])
number_in, number_out, number_on = EXAMPLE_COUNTS
rows.append([
product["sku"], product["name"], EXAMPLE_RECEIPT, EXAMPLE_EXPIRATION,
number_in, number_out, number_on, "", product["price"],
])
with open(os.path.join(staging, "stock.csv"), "w", encoding="utf-8", newline="") as handle:
csv.writer(handle).writerows(rows)
if not os.path.lexists(output):
os.rename(staging, output)
staging = None
else:
previous = output + ".previous"
os.rename(output, previous)
try:
os.rename(staging, output)
except OSError:
os.rename(previous, output)
raise
staging = None
print(f"previous store kept at {previous}", file=sys.stderr)
return len(written)
finally:
if staging and os.path.isdir(staging):
shutil.rmtree(staging)
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--image", required=True, help="shelf photo whose pixels match the box coordinates")
parser.add_argument("--boxes", required=True, help='JSON file {"boxes": [[x1, y1, x2, y2], ...]}')
parser.add_argument("--output", required=True, help="new stock directory to create")
parser.add_argument("--labels", help='JSON file {"products": [{"sku", "name", "price", "boxes"}]}')
parser.add_argument("--example", choices=["mediya"], help="use the named demo contact-sheet labels")
parser.add_argument("--replace", action="store_true", help="replace an existing output directory")
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
try:
products = _products(load_labels(args.labels, args.example))
count = build_store(
args.image, load_boxes(args.boxes), products, args.output, args.replace,
args.boxes, args.labels,
)
except BuildStoreError as exc:
print(f"build_store_from_shelf: {exc}", file=sys.stderr)
return 1
print(f"{os.path.abspath(args.output)}: {count} products, csv written")
print("phone recordings are not enrolled by this script; labels stay a manual contact sheet")
return 0
if __name__ == "__main__":
raise SystemExit(main())

202
recognization/code/collect_shelf_images.py
View file

@ -1,202 +0,0 @@
"""
Collect openly licensed store-shelf photographs for evaluation.
Everything measured so far comes from two photographs. This pulls real store photos
from Wikimedia Commons - scriptable, licensed, and no authentication - so the
pipeline can be measured on images it has never seen.
Only permissively licensed files are kept (CC0, CC BY, CC BY-SA, public domain), and
each one is written to a manifest with its author, licence and source URL, so an
attribution page can be produced from it.
python code/collect_shelf_images.py [output_dir]
"""
from __future__ import annotations
import json
import os
import ssl
import sys
import time
import urllib.parse
import urllib.request
import certifi
CONTEXT = ssl.create_default_context(cafile=certifi.where())
API = "https://commons.wikimedia.org/w/api.php"
USER_AGENT = "dorayaki-eval/0.1 (shelf recognition research; contact: repo owner)"
# Commons asks for a polite pace and offers thumbnails instead of originals. Both
# matter: a burst of original downloads got rate limited (HTTP 429) on the first run.
THUMBNAIL_WIDTH = 1600
REQUEST_DELAY = 1.2
MAX_RETRIES = 4
# Shelf photos of the categories the catalog covers: drinks and dairy, rice
# crackers and snacks, sauces and condiments.
QUERIES = [
# the catalog's own products first: drinks, dairy, rice crackers
"野菜生活100",
"伊藤園 1日分の野菜",
"Japanese vegetable juice shelf",
"Japanese milk cartons supermarket",
"meiji oishii milk",
"せんべい 棚",
"Japanese rice cracker shop display",
"Japanese confectionery shelf",
"Japanese supermarket beverage aisle",
"Japanese convenience store refrigerator drinks",
"Japanese supermarket dairy case",
"スーパー 飲料 陳列",
"お菓子 売り場",
"Japanese snack aisle supermarket",
"Japanese supermarket shelves drinks bottles",
"Japanese supermarket interior shelves",
]
PERMISSIVE = ("cc0", "cc-by", "cc by", "public domain", "pd-", "attribution")
def fetch(url: str, timeout: int = 60) -> bytes:
"""GET with backoff: Commons answers 429 when a client is too eager."""
for attempt in range(MAX_RETRIES):
try:
request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(request, timeout=timeout, context=CONTEXT) as response:
return response.read()
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == MAX_RETRIES - 1:
raise
time.sleep(REQUEST_DELAY * (2 ** attempt))
raise RuntimeError("unreachable")
def api(params: dict) -> dict:
url = API + "?" + urllib.parse.urlencode(params)
time.sleep(REQUEST_DELAY)
return json.loads(fetch(url, timeout=30).decode("utf-8"))
def search_files() -> dict:
"""Candidate files, with how many queries found each one."""
found: dict = {}
for query in QUERIES:
try:
result = api({"action": "query", "list": "search", "srsearch": query,
"srnamespace": "6", "srlimit": "25", "format": "json"})
except Exception as error:
print(f" search failed for {query!r}: {error}")
continue
for hit in result.get("query", {}).get("search", []):
title = hit["title"]
if title.lower().endswith((".svg", ".pdf", ".ogv", ".webm")):
continue
found[title] = found.get(title, 0) + 1
time.sleep(0.2)
return found
def image_info(titles: list) -> dict:
"""URL, size and licence for each file, in batches of 50."""
info: dict = {}
for start in range(0, len(titles), 50):
batch = titles[start:start + 50]
try:
result = api({"action": "query", "titles": "|".join(batch), "prop": "imageinfo",
"iiprop": "url|size|extmetadata", "iiurlwidth": str(THUMBNAIL_WIDTH),
"format": "json"})
except Exception as error:
print(f" info failed for batch {start}: {error}")
continue
for page in (result.get("query", {}).get("pages") or {}).values():
entries = page.get("imageinfo") or []
if not entries:
continue
entry = entries[0]
meta = entry.get("extmetadata") or {}
info[page["title"]] = {
"url": entry.get("thumburl") or entry.get("url"),
"width": entry.get("width"),
"height": entry.get("height"),
"license": (meta.get("LicenseShortName") or {}).get("value", ""),
"artist": (meta.get("Artist") or {}).get("value", ""),
"description": (meta.get("ImageDescription") or {}).get("value", "")[:200],
"page": f"https://commons.wikimedia.org/wiki/{urllib.parse.quote(page['title'].replace(' ', '_'))}",
}
time.sleep(0.2)
return info
def main() -> None:
default = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"data", "external-shelves", "wikimedia-commons")
root = sys.argv[1] if len(sys.argv) > 1 else default
os.makedirs(root, exist_ok=True)
print("searching Commons...")
found = search_files()
print(f" {len(found)} distinct candidate files")
info = image_info(sorted(found))
print(f" metadata for {len(info)} files")
kept, skipped = [], {"license": 0, "size": 0, "missing": 0, "not_a_photo": 0}
for title, entry in sorted(info.items()):
if not entry.get("url"):
skipped["missing"] += 1
continue
licence = (entry.get("license") or "").lower()
if not any(token in licence for token in PERMISSIVE):
skipped["license"] += 1
continue
# A book scan is not a shelf. The search terms pull in .djvu volumes
# whose page 1 is a title page, and Commons serves those as JPEG
# thumbnails, so the extension check has to be on the source title.
if any(title.lower().endswith(ext) for ext in (".djvu", ".pdf", ".svg", ".tif", ".tiff")):
skipped["not_a_photo"] += 1
continue
if (entry.get("width") or 0) < 1000 or (entry.get("height") or 0) < 700:
skipped["size"] += 1
continue
kept.append((title, entry))
print(f" keeping {len(kept)} (skipped: {skipped})")
manifest = []
for index, (title, entry) in enumerate(kept):
# Truncate the stem, never the extension: a 60-character cut on the
# whole basename drops it, and the file then has no type for cv2 or
# for anyone reading the directory.
stem, extension = os.path.splitext(
os.path.basename(urllib.parse.urlparse(entry["url"]).path).replace(" ", "_")
)
name = f"{index:03d}_{urllib.parse.unquote(stem)[:60]}{extension.lower()}"
path = os.path.join(root, name)
if not os.path.exists(path):
try:
payload = fetch(entry["url"])
with open(path, "wb") as handle:
handle.write(payload)
except Exception as error:
print(f" download failed for {title}: {error}")
continue
manifest.append({
"file": name,
"title": title,
"width": entry["width"],
"height": entry["height"],
"license": entry["license"],
"artist": entry["artist"],
"source": entry["page"],
"queries": found.get(title, 0),
})
print(f" [{index:>3}] {entry['width']}x{entry['height']} {entry['license']:<14} {title}")
manifest_path = os.path.join(root, "manifest.json")
with open(manifest_path, "w", encoding="utf-8") as handle:
json.dump({"source": "Wikimedia Commons", "images": manifest}, handle, ensure_ascii=False, indent=1)
print(f"\n{len(manifest)} images in {root}\nmanifest: {manifest_path}")
if __name__ == "__main__":
main()

20
recognization/code/detector.py
View file

@ -99,17 +99,8 @@ from ultralytics import YOLO
class ShelfProductDetector:
"""Wraps the YOLO detector for supermarket shelf products."""
def __init__(
self,
weights_path: str = "runs/detect/train/weights/best.pt",
device: Optional[str] = None,
imgsz: int = 640,
half: bool = False
):
def __init__(self, weights_path: str = "runs/detect/train/weights/best.pt"):
self.weights_path = weights_path
self.device = device
self.imgsz = imgsz
self.half = half
if not os.path.exists(weights_path):
raise FileNotFoundError(f"YOLO weights not found at: {weights_path}")
@ -138,14 +129,7 @@ class ShelfProductDetector:
return {"status": "error", "message": "Failed to read image", "boxes": []}
h, w = image_bgr.shape[:2]
predict_kwargs: Dict[str, Any] = dict(
conf=conf_threshold, iou=iou_threshold, imgsz=self.imgsz, verbose=False
)
if self.device:
predict_kwargs["device"] = self.device
if self.half:
predict_kwargs["half"] = True
results = self.model(image_bgr, **predict_kwargs)
results = self.model(image_bgr, conf=conf_threshold, iou=iou_threshold, verbose=False)
boxes_list = []
if results and len(results) > 0:

41
recognization/code/download_stock.py Normal file
View file

@ -0,0 +1,41 @@
import os
import sys
import io
import urllib.request
from PIL import Image
# Ensure stdout handles utf-8 cleanly
if sys.platform.startswith('win'):
sys.stdout.reconfigure(encoding='utf-8')
urls = {
'sku_011': ('https://www.kagome.co.jp/products/brand/ys100/regular/img/img_products_apple@2x_v2.png', '野菜生活100 アップルサラダ'),
'sku_012': ('https://www.kagome.co.jp/products/brand/ys100/regular/img/img_products_lemon@2x_v2.png', '野菜生活100 レモンサラダ'),
'sku_013': ('https://www.kagome.co.jp/products/brand/ys100/yssmoothie/img/green-product-pic_2.png', '野菜生活100 Smoothie グリーンスムージー'),
'sku_014': ('https://www.kagome.co.jp/products/brand/ys100/yssmoothie/img/orange-product-pic_2.png', '野菜生活100 Smoothie ビタミンスムージー'),
'sku_015': ('https://www.kagome.co.jp/products/brand/ys100/yssmoothie/img/yellow-product-pic_2.png', '野菜生活100 Smoothie バナナ＆アーモンドmix'),
'sku_016': ('https://www.kagome.co.jp/products/brand/ys100/yssmoothie/img/red-product-pic_2.png', '野菜生活100 Smoothie アサイー＆ベリーmix')
}
os.makedirs('stock', exist_ok=True)
for sku, (url, name) in urls.items():
print(f'Fetching {sku}: {name} from {url}...')
req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'})
try:
with urllib.request.urlopen(req, timeout=15) as resp:
data = resp.read()
img = Image.open(io.BytesIO(data))
if img.mode in ('RGBA', 'LA') or (img.mode == 'P' and 'transparency' in img.info):
bg = Image.new('RGB', img.size, (255, 255, 255))
if img.mode != 'RGBA':
img = img.convert('RGBA')
bg.paste(img, mask=img.split()[3])
img = bg
else:
img = img.convert('RGB')
out_path = f'stock/{sku}.jpeg'
img.save(out_path, 'JPEG', quality=95)
print(f'Successfully saved {sku}: {name} -> {out_path} ({img.size})')
except Exception as e:
print(f'Error downloading {sku} ({url}): {e}')

1101
recognization/code/engine.py
View file

File diff suppressed because it is too large Load diff

136
recognization/code/eval_distractors.py
View file

@ -1,136 +0,0 @@
"""Open-world false-accept probe: unrelated shelf photos through the shipped lane.
data/external-shelves/ holds Wikimedia Commons supermarket photos of shelves
that share nothing with the LOPIA catalog. Every accept here is a false accept
by construction, which is what makes the set useful: the image6 shelf can only
measure false accepts on the products it happens to contain.
Measured the way the server measures: a 1280-long-side frame detected at
`LiveSettings.conf`. The sources are 4032-4512 px photos, and detecting them at
full resolution and at the default `detect_conf` instead measures a
configuration the phone never runs, which is how a threshold change reads as a
pass on evidence the product does not produce.
.venv/bin/python code/eval_distractors.py --stock stock-lopia-clean \
data/external-shelves/wikimedia-commons
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import sys
import time
from typing import Any, Dict, List
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
from eval_identity import ( # noqa: E402
LIVE_LONG_SIDE,
LOCAL_ACCEPT,
LocalLaneAdapter,
at_live_resolution,
)
from identity_lanes import TEXT_MIN_SHARED # noqa: E402
def main(argv: List[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("directories", nargs="+")
parser.add_argument("--stock", default="stock-lopia-clean")
parser.add_argument("--write", action="store_true")
parser.add_argument("--limit", type=int, default=0, help="images per set (0 = all)")
args = parser.parse_args(argv)
from engine import build_default_engine, EngineSettings
from live import LiveSettings
engine = build_default_engine(EngineSettings(stock_dir=args.stock))
lane = LocalLaneAdapter(args.stock)
conf = LiveSettings().conf
for directory in args.directories:
manifest_path = os.path.join(directory, "manifest.json")
try:
with open(manifest_path) as handle:
manifest = json.load(handle)
except (OSError, ValueError):
manifest = {"images": []}
meta = {row["file"]: row for row in manifest.get("images", [])}
rows: List[Dict[str, Any]] = []
paths = sorted(
path for path in glob.glob(os.path.join(directory, "*"))
if os.path.splitext(path)[1].lower() in (".jpg", ".jpeg", ".png")
)
if args.limit:
paths = paths[: args.limit]
for path in paths:
name = os.path.basename(path)
image = cv2.imread(path)
if image is None:
continue
started = time.perf_counter()
frame = at_live_resolution(image)
detections = engine.detect(frame, conf=conf)
accepted = 0
for detection in detections:
box = detection.get("box") or detection.get("bbox")
if not box:
continue
x1, y1, x2, y2 = [int(v) for v in box]
crop = frame[max(0, y1):y2, max(0, x1):x2]
if crop.size == 0:
continue
decision = lane.decide(crop, lane.rank(crop), accept=LOCAL_ACCEPT,
margin=0.0, scale="full")
if decision.get("best_sku"):
accepted += 1
rows.append({
"file": name,
"title": meta.get(name, {}).get("title", ""),
"detections": len(detections),
"false_accepts": accepted,
"ms": round((time.perf_counter() - started) * 1000),
})
detections = sum(r["detections"] for r in rows)
accepts = sum(r["false_accepts"] for r in rows)
report = {
"dataset": os.path.basename(directory.rstrip("/")),
"catalog": args.stock,
"note": "Unrelated shelves: every accept is a false accept by construction.",
# The configuration the numbers above were measured under, so the
# stored evidence is checkable against the constants it claims.
"config": {
"frame_long_side": LIVE_LONG_SIDE,
"detect_conf": conf,
"local_accept": LOCAL_ACCEPT,
"text_min_shared": TEXT_MIN_SHARED,
},
"images": len(rows),
"detections": detections,
"false_accepts": accepts,
"false_accept_rate": round(accepts / detections, 4) if detections else None,
"rows": rows,
}
print(f"{report['dataset']}: {len(rows)} images, {detections} detections, "
f"{accepts} false accepts ({report['false_accept_rate'] or 0:.2%})")
for row in sorted(rows, key=lambda r: -r["false_accepts"])[:5]:
if row["false_accepts"]:
print(f" {row['false_accepts']:>3} {row['file'][:55]}")
if args.write:
out = os.path.join(directory, "eval_result.json")
with open(out, "w") as handle:
json.dump(report, handle, ensure_ascii=False, indent=1)
print(f" wrote {out}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

127
recognization/code/eval_external.py
View file

@ -1,127 +0,0 @@
"""Score external-proxy image sets through the shipped identity lane.
These sets are not Tier 2 (see data/external-eval/README.md). They exist to test
the pipeline on data it had no part in building, so the one thing this script
must never do is score an image that has since been enrolled - that is
training-set accuracy wearing a held-out label.
`leaked()` is the same check the shelf evaluator runs, and it is applied here
for that reason: enrolling a packshot from an external set is a normal thing to
do (it is how F-12 was closed), and it silently converts one of these rows into
a leak unless something refuses it.
.venv/bin/python code/eval_external.py --stock stock-lopia-clean \
data/external-eval/kewpie data/external-eval/other-mayo
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import sys
from typing import Any, Dict, List
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
from eval_identity import LOCAL_ACCEPT, LocalLaneAdapter, leaked # noqa: E402
def score_set(lane: Any, directory: str, stock_dir: str) -> Dict[str, Any]:
rows: List[Dict[str, Any]] = []
for path in sorted(glob.glob(os.path.join(directory, "*.jpg"))):
name = os.path.basename(path)
truth = "sku_" + name.split("_")[1]
image = cv2.imread(path)
if image is None:
rows.append({"file": name, "truth": truth, "verdict": "unreadable"})
continue
# The whole image is the crop: an external packshot is not a shelf facing.
box = (0, 0, image.shape[1], image.shape[0])
leak = leaked(image, box, stock_dir)
if leak:
rows.append({
"file": name, "truth": truth, "verdict": "leaked",
"leaked_into": leak,
"note": "enrolled in the catalog; scoring it would be training accuracy",
})
continue
decision = lane.decide(
image, lane.rank(image), accept=LOCAL_ACCEPT, margin=0.0, scale="full"
)
best = decision.get("best_sku")
verdict = "correct" if best == truth else ("abstain" if not best else "wrong")
rows.append({
"file": name, "truth": truth, "best": best,
"confidence": decision.get("confidence"),
"status": decision.get("status"),
"source": decision.get("source"),
"verdict": verdict,
})
scored = [r for r in rows if r["verdict"] in ("correct", "abstain", "wrong")]
counts = {v: sum(1 for r in scored if r["verdict"] == v)
for v in ("correct", "abstain", "wrong")}
# Per SKU, because a set total hides which products the lane can and cannot
# name. Every SKU here has images from two independent source domains, so a
# SKU that scores the same on both is not one lucky photograph.
by_sku: Dict[str, Dict[str, int]] = {}
for row in scored:
bucket = by_sku.setdefault(row["truth"], {"correct": 0, "abstain": 0, "wrong": 0})
bucket[row["verdict"]] += 1
return {
"dataset": os.path.basename(directory.rstrip("/")),
"catalog": stock_dir,
"note": "External proxy images, not Tier 2. Scored through the shipped "
"LocalLaneAdapter + DecisionPolicy.",
"rows": rows,
"scored": len(scored),
"leaked": sum(1 for r in rows if r["verdict"] == "leaked"),
"by_sku": by_sku,
**counts,
}
def main(argv: List[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("directories", nargs="+", help="external set directories")
parser.add_argument("--stock", default="stock-lopia-clean")
parser.add_argument("--write", action="store_true",
help="write eval_result.json into each set directory")
args = parser.parse_args(argv)
lane = LocalLaneAdapter(args.stock)
totals = {"correct": 0, "abstain": 0, "wrong": 0, "leaked": 0, "scored": 0}
for directory in args.directories:
report = score_set(lane, directory, args.stock)
for key in totals:
totals[key] += report[key]
print(f"{report['dataset']}: correct {report['correct']} | "
f"abstain {report['abstain']} | wrong {report['wrong']} | "
f"leaked {report['leaked']} of {report['scored']} scored")
for sku, bucket in sorted(report["by_sku"].items()):
print(f" {sku:<10} correct {bucket['correct']} "
f"abstain {bucket['abstain']} wrong {bucket['wrong']}")
for row in report["rows"]:
if row["verdict"] in ("wrong", "leaked"):
print(f" {row['verdict'].upper():<7} {row['file']} "
f"truth={row['truth']} best={row.get('best')} "
f"{row.get('leaked_into', '')}")
if args.write:
out = os.path.join(directory, "eval_result.json")
with open(out, "w") as handle:
json.dump(report, handle, ensure_ascii=False, indent=1)
print(f" wrote {out}")
print(f"\nTOTAL: {totals['scored']} scored, correct {totals['correct']}, "
f"abstain {totals['abstain']}, FALSE ACCEPTS {totals['wrong']}, "
f"leaked {totals['leaked']}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

59
recognization/code/eval_external_images.py
View file

@ -1,59 +0,0 @@
"""
Run the pipeline over store photos it has never seen.
python code/eval_external_images.py [data/external-shelves/wikimedia-commons]
Reports detection and identity per image, and writes a JSON summary. See
docs/EXTERNAL-EVAL.md for what the numbers mean.
The images live in the repository (`data/external-shelves/`), not in a home
directory: they are evaluation evidence, and evidence that only exists on one
machine cannot be re-checked.
"""
import json, os, sys, time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import cv2, numpy as np
from engine import build_default_engine, EngineSettings
from tags import TagService, TagSettings
from reconcile import attach_tags_to_boxes
DEFAULT_ROOT = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"data", "external-shelves", "wikimedia-commons")
ROOT = os.path.expanduser(sys.argv[1] if len(sys.argv) > 1 else DEFAULT_ROOT)
engine = build_default_engine(EngineSettings(stock_dir="stock-lopia"))
catalog_by_sku = {p["sku_id"]: p for p in engine.catalog()}
service = TagService(engine.ocr_engine, settings=TagSettings())
rows = []
for entry in manifest["images"]:
path = os.path.join(ROOT, entry["file"])
image = cv2.imread(path)
if image is None:
continue
# keep the long side at 1600 so detection cost is comparable across images
height, width = image.shape[:2]
if max(height, width) > 1600:
scale = 1600.0 / max(height, width)
image = cv2.resize(image, (int(width*scale), int(height*scale)), interpolation=cv2.INTER_AREA)
started = time.perf_counter()
result = engine.recognize(image, classify=True, method="hybrid", with_empty_slots=True)
detections = result["detections"]
recognized = sum(1 for d in detections if (d.get("identity") or {}).get("state") == "recognized")
rejected = sum(1 for d in detections if (d.get("identity") or {}).get("state") == "rejected")
rows.append({
"file": entry["file"], "title": entry["title"], "license": entry["license"],
"size": f"{image.shape[1]}x{image.shape[0]}",
"detections": len(detections), "recognized": recognized, "rejected": rejected,
"ms": round((time.perf_counter()-started)*1000),
})
print(f" {entry['file'][:34]:<34} {image.shape[1]}x{image.shape[0]:<5} "
f"det={len(detections):>4} rec={recognized:>3} rej={rejected:>3} {rows[-1]['ms']:>5} ms")
total_det = sum(r["detections"] for r in rows)
total_rec = sum(r["recognized"] for r in rows)
with_det = sum(1 for r in rows if r["detections"] > 0)
print(f"\n{len(rows)} images, {total_det} detections, {total_rec} recognized "
f"({100*total_rec//max(1,total_det)}%), images with any detection: {with_det}")
json.dump(rows, open(os.path.join(ROOT, "eval.json"), "w"), ensure_ascii=False, indent=1)
print(f"summary: {os.path.join(ROOT, 'eval.json')}")

788
recognization/code/eval_identity.py
View file

@ -1,788 +0,0 @@
"""Score identity lanes on labelled shelf facings. Architecture-neutral.
The original matcher is a candidate final architecture, not just a baseline.
Every lane is scored the same way, on the same crops, at the same resolutions:
local RootSIFT pyramid + OCR text lane (the shipped default)
matcher full-catalog SIFT/ORB/BRISK + RANSAC (the original demo)
gallery DINOv2 embedding, raw or with the trained head
hybrid-verify DINO shortlist, then the matcher re-ranks inside it
Reported per lane, as four separate ceilings rather than one conflated number:
catalog ceiling in-catalog facings with an enrolled view / all in-catalog
identity ceiling ungated top-1 rank correct on enrolled / enrolled
decision ceiling accepted-and-correct on enrolled / enrolled
end-to-end coverage accepted-and-correct / all in-catalog facings
plus precision over every accept — including accepts of unenrolled in-catalog
facings, which are false accepts and the most dangerous kind — the prediction
histogram, and the largest single-SKU share of accepts (the L09-apocalypse
detector). The hybrid lane also reports shortlist recall, because a true SKU
outside the shortlist is a hard ceiling no verifier can lift.
python code/eval_identity.py \
--labels eval/labels/image6.json \
--image add/image6.jpg --stock-dir stock-lopia-clean --lanes matcher --live
Labels are {"image", "facings": [{"box", "label"}]} with "unknown" for facings
not in the catalog and "exclude" for rows two label passes disagreed on. A
facing whose crop is a catalog template is refused as leakage, not scored:
those crops are how the store was enrolled.
"""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
import os
import subprocess
import sys
import time
from collections import Counter
from typing import Any, Collection, Dict, List, Optional, Sequence, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
import numpy as np # noqa: E402
from rebuild_catalog import sku_of # noqa: E402
from identity import LOCAL_ACCEPT # noqa: E402
UNKNOWN = "unknown"
# Two independent label passes disagreed here, so the row is not ground truth.
# Scoring it would be scoring a coin flip.
EXCLUDE = "exclude"
LANES = ("local", "matcher", "van", "gallery", "hybrid-verify")
# A single SKU above this share of accepts is the failure the last audit found:
# every dark package collapsing onto one enrolled product.
HUB_SHARE = 0.30
# Live frames arrive at this long side; the original demo classified a 2048 px
# still. Scoring both is how resolution stops being an invisible variable.
LIVE_LONG_SIDE = 1280
# Pixel equality is not a leak test: a template re-cut from the same photo with
# a different box passes a same-shape comparison. A 64x64 grayscale signature
# catches most of those, but it is a heuristic, not a proof — measured on
# the corrected labels against stock-lopia, re-cuts in the realistic 0.94..1.06 box band
# land 7..37 MAD from their facing's signature while the closest pair of
# genuinely distinct same-SKU facings is 23.1, so the distributions overlap and
# no threshold is clean. 20.0 catches 81% of that band with zero false
# positives on 5,886 genuine facing pairs and 7 clean packshots. The real fix is
# the enrollment protocol's session manifest (DECISION §8); this is the cheap
# backstop that runs without one.
LEAK_SIGNATURE_MAD = 20.0
# A signature hit is confirmed by correlation before it counts as a leak. A
# genuine re-cut, re-encode or rescale of an enrolled view correlates at 0.99+;
# two different photographs of the same product sit near 0.3. Measured on the
# l09 pair that the signature alone misjudged (0.28) and on re-encoded and
# rescaled copies of the enrolled view (0.999, 0.997).
LEAK_CORRELATION = 0.90
def load_labels(path: str) -> Dict[str, Any]:
"""Load a label file, refusing one whose rows carry no provenance.
F-01: the v1 labels were wrong on 23% of facings and nothing in the file
said where any of them came from, so a bad label was indistinguishable from
a good one. Every row must now name its `source`; a row that cannot be
traced is a row that cannot be argued with.
"""
with open(path) as handle:
payload = json.load(handle)
if not isinstance(payload, dict) or "facings" not in payload:
raise SystemExit(f"{path}: expected {{'image', 'facings': [...]}}")
missing = [
row.get("index", i)
for i, row in enumerate(payload["facings"])
if not row.get("source")
]
if missing:
raise SystemExit(
f"{path}: {len(missing)} facings carry no 'source' "
f"(first: index {missing[0]}). A label without provenance cannot be "
"audited; see the enrollment protocol in DECISION.md."
)
return payload
def file_hash(path: str) -> Optional[str]:
"""SHA-256 of a file's bytes, or None when it cannot be read."""
try:
with open(path, "rb") as handle:
return hashlib.sha256(handle.read()).hexdigest()
except OSError:
return None
def catalog_hash(stock_dir: str) -> Optional[str]:
"""SHA-256 over the catalog's CSV bytes and each view's bytes.
The view bytes are hashed, not just their names. Names alone were not
enough: replacing `sku_l09.jpg` with a different product's packshot left
the hash unchanged, so a report could claim the same catalog while every
l09 decision in it had moved. Re-encoding the same image does move the
hash, which is the price of not being able to lie about what was scored.
"""
digest = hashlib.sha256()
csv_path = os.path.join(stock_dir, "stock.csv")
try:
with open(csv_path, "rb") as handle:
digest.update(handle.read())
except OSError:
digest.update(b"missing-csv")
try:
names = sorted(
name for name in os.listdir(stock_dir)
if os.path.splitext(name)[1].lower() in (".jpg", ".jpeg", ".png")
)
except OSError:
names = []
for name in names:
digest.update(b"\n")
digest.update(name.encode("utf-8"))
try:
with open(os.path.join(stock_dir, name), "rb") as handle:
digest.update(handle.read())
except OSError:
digest.update(b"unreadable")
return digest.hexdigest()
def code_revision() -> Optional[str]:
"""The checked-out revision, or None outside a git worktree."""
try:
result = subprocess.run(
["git", "rev-parse", "HEAD"],
cwd=os.path.dirname(os.path.abspath(__file__)),
capture_output=True, text=True, timeout=10,
)
except (OSError, subprocess.SubprocessError):
return None
return result.stdout.strip() or None if result.returncode == 0 else None
def crop_of(image: Any, box: Sequence[int]) -> Any:
x1, y1, x2, y2 = [int(v) for v in box]
return image[max(0, y1):y2, max(0, x1):x2]
def at_live_resolution(image: Any, long_side: int = LIVE_LONG_SIDE) -> Any:
height, width = image.shape[:2]
if max(height, width) <= long_side:
return image
scale = long_side / float(max(height, width))
return cv2.resize(image, (int(width * scale), int(height * scale)), interpolation=cv2.INTER_AREA)
def _signature(image: Any) -> Any:
"""64x64 grayscale signature. Cheap, and tolerant of a re-cut."""
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) if image.ndim == 3 else image
return cv2.resize(gray, (64, 64), interpolation=cv2.INTER_AREA).astype(np.float32)
def _signatures(image: Any) -> List[Any]:
"""The crop's signature and a centre 90% crop's.
A re-cut box shifts the content inside the frame. Comparing a second,
tighter crop against the template absorbs that shift: measured on the
realistic re-cut band it catches 81% of re-cuts against 68% for the single
signature, at no cost in false positives.
"""
height, width = image.shape[:2]
dy, dx = int(height * 0.05), int(width * 0.05)
centre = image[dy:height - dy, dx:width - dx]
return [_signature(image), _signature(centre if centre.size else image)]
_TEMPLATES: Dict[str, List[Tuple[str, str, Tuple[int, int], List[Any]]]] = {}
def _catalog_templates(stock_dir: str) -> List[Tuple[str, str, Tuple[int, int], List[Any]]]:
"""(stem, name, shape, signatures) per catalog image, read once per stock dir.
Cached because the signature is what catches a re-cut crop, and re-reading
every template for every facing is what made the old check expensive.
"""
cached = _TEMPLATES.get(stock_dir)
if cached is None:
cached = []
for name in sorted(os.listdir(stock_dir)):
stem, ext = os.path.splitext(name)
if ext.lower() not in (".jpg", ".jpeg", ".png"):
continue
template = cv2.imread(os.path.join(stock_dir, name))
if template is None:
continue
cached.append((stem, name, template.shape[:2], _signatures(template)))
_TEMPLATES[stock_dir] = cached
return cached
def _correlation(left: Any, right: Any) -> float:
"""Mean-subtracted correlation of two images, resizing `right` to `left`.
Scale- and re-encode-tolerant, unlike a pixel comparison: the enrolled view
and a re-saved or rescaled copy of it correlate at 0.99+, while two
different photographs of the same product sit near 0.3.
"""
if right.shape[:2] != left.shape[:2]:
right = cv2.resize(right, (left.shape[1], left.shape[0]),
interpolation=cv2.INTER_AREA)
a = cv2.cvtColor(left, cv2.COLOR_BGR2GRAY).astype(np.float32).ravel()
b = cv2.cvtColor(right, cv2.COLOR_BGR2GRAY).astype(np.float32).ravel()
a -= a.mean()
b -= b.mean()
denom = float(np.linalg.norm(a) * np.linalg.norm(b))
return float((a * b).sum() / denom) if denom else 0.0
def leaked(image: Any, box: Sequence[int], stock_dir: str) -> Optional[str]:
"""Name of the catalog template this crop is, or None when it is held out.
Byte-identical crops and near-identical pastes both count: enrolment crops
scored as test data are training-set accuracy, not recognition.
Pixel equality is not a leak test. A template re-cut from the same photo
with a slightly different box is the same leak and passes a same-shape
comparison, so grayscale signatures are compared as well. That check is a
heuristic with overlapping distributions (see LEAK_SIGNATURE_MAD): it
catches most re-cuts and no genuine facing in the measured set, but a
determined re-cut can still pass. The enrollment protocol's session manifest
is the real refusal; this is the backstop for catalogs that lack one.
"""
crop = crop_of(image, box)
if crop.size == 0:
return None
crop_signatures = _signatures(crop)
for stem, name, shape, signatures in _catalog_templates(stock_dir):
close = min(float(np.abs(signature - crop_signature).mean())
for signature in signatures for crop_signature in crop_signatures)
if close < LEAK_SIGNATURE_MAD:
# The signature is a 64x64 grayscale thumbnail, so it is coarse
# enough to fire on two different photos of the same product: the
# l09 rear label from a retailer page sits 19.5 from the enrolled
# front packshot, under the 20.0 line, while being a different
# photograph (resized MAD 26.3). Confirm before calling it a leak.
# A genuine re-cut or re-encode correlates at 0.99+; that pair
# correlates at 0.28, so the two signals together are decisive
# where either alone is not.
template = cv2.imread(os.path.join(stock_dir, name))
if template is None or _correlation(crop, template) >= LEAK_CORRELATION:
return stem
continue
if shape != crop.shape[:2]:
continue
template = cv2.imread(os.path.join(stock_dir, name))
if template is None:
continue
if np.array_equal(template, crop):
return stem
left = crop.astype(np.float32).ravel()
right = template.astype(np.float32).ravel()
denom = float(np.linalg.norm(left) * np.linalg.norm(right))
if denom and float((left * right).sum() / denom) >= 0.98:
return stem
return None
def catalog_sku_ids(stock_dir: str) -> List[str]:
"""SKU ids from the store's catalog CSV, falling back to enrolled images.
Candidate sets must come from the catalog, never from the labels: a lane
told which SKUs the shelf holds is being scored against the answer key, and
a retrieval failure can never show up.
"""
path = os.path.join(stock_dir, "stock.csv")
if os.path.isfile(path):
with open(path, newline="") as handle:
ids = [row["SKU_ID"].strip() for row in csv.DictReader(handle)
if row.get("SKU_ID", "").strip()]
if ids:
return sorted(set(ids))
return sorted({sku_of(name) for name in os.listdir(stock_dir)
if os.path.splitext(name)[1].lower() in (".jpg", ".jpeg", ".png")})
def group_by_sku(scores: Dict[str, float]) -> Dict[str, float]:
"""Collapse per-view template names onto their SKU.
The matcher scores `sku_l01_1` and `sku_l01_2` separately; a SKU's score is
its best view, which is also how the gallery ranks several views. `sku_of`
is the canonical collapse, so a view outside the `sku_l` prefix groups with
its SKU instead of ranking as a product of its own.
"""
grouped: Dict[str, float] = {}
for name, value in (scores or {}).items():
sku = sku_of(name)
grouped[sku] = max(grouped.get(sku, 0.0), float(value))
return grouped
class Lane:
"""One identity architecture, scored identically to every other."""
name = "lane"
def rank(self, crop: Any) -> List[Tuple[str, float]]:
"""Catalog candidates, best first. Empty means nothing to answer."""
raise NotImplementedError
def shortlist(self, crop: Any, size: int) -> List[str]:
"""Candidate SKUs the verifier is allowed to consider."""
return [sku for sku, _ in self.rank(crop)[:size]]
def decide(self, crop: Any, ranked: List[Tuple[str, float]], *, accept: float,
margin: float, scale: str) -> Dict[str, Any]:
"""The lane's own decision. Default: the shared gate over its ranking.
A lane whose decision is not `decide_identity` over one ranking - the
fused local+text lane is one - overrides this, so every lane is still
scored through the same four ceilings.
"""
from identity import decide_identity
return decide_identity(dict(ranked), method=self.name, scale=scale,
min_score=accept, min_margin=margin)
class MatcherLane(Lane):
"""The original demo: local features and RANSAC over the whole catalog."""
name = "matcher"
def __init__(self, stock_dir: str, method: str = "hybrid") -> None:
from feature_matcher import ProductFeatureMatcher
self.matcher = ProductFeatureMatcher(stock_dir=stock_dir)
self.method = method
def rank(self, crop: Any) -> List[Tuple[str, float]]:
scores, _ = self.matcher.score_crop(crop, method=self.method)
return sorted(group_by_sku(scores).items(), key=lambda kv: kv[1], reverse=True)
class VanLane(MatcherLane):
"""Van's contract: the matcher's top SKU, always answered.
The `matcher` lane applies `decide_identity`, which is not what the original
demo did - it answered every crop with its best-scoring SKU and had no
accept line at all. Comparing the new system against the gated matcher
understates the baseline's coverage and overstates its precision, so the
completion gate's "Van" row is this lane: same scores, no gate.
"""
name = "van"
def decide(self, crop: Any, ranked: List[Tuple[str, float]], *, accept: float,
margin: float, scale: str) -> Dict[str, Any]:
from identity import IDENTITY_ACCEPTED
if not ranked:
return {
"status": IDENTITY_ACCEPTED, "method": self.name, "best_sku": None,
"score": 0.0, "confidence": 0.0, "identity_level": IDENTITY_ACCEPTED,
"ranked_candidates": [], "details": {}, "message": "no candidate",
"source": None,
}
best, score = ranked[0]
return {
"status": IDENTITY_ACCEPTED, "method": self.name, "best_sku": best,
"score": round(float(score), 2), "confidence": round(float(score), 4),
"identity_level": IDENTITY_ACCEPTED,
"ranked_candidates": [{"sku_id": sku, "score": round(float(v), 2)}
for sku, v in ranked[:5]],
"details": {}, "message": "Always-argmax baseline", "source": "van",
}
class LocalLaneAdapter(Lane):
"""The shipped default: RootSIFT pyramid plus the OCR text lane.
Wraps `identity_lanes` so the production decision is what gets scored,
rather than a re-implementation of it. `rank` returns the lane's own order
(its rank key, not its gate value), and `decide` runs the same
`DecisionPolicy` the server runs.
"""
name = "local"
def __init__(self, stock_dir: str, ocr: Any = None, use_text: bool = True) -> None:
from identity_lanes import LocalLane, TextLane
self.lane = LocalLane(stock_dir)
# E-3 asks what the text lane is worth, which is only answerable by
# running the same crops without it. `--no-text` therefore drops the
# lane whole: nothing in the decision reads it, and an ablation that
# still had to build an OCR engine could not be run on a machine
# without one.
self.use_text = use_text
self.text_lane = (
TextLane.from_catalog(
_catalog_rows(stock_dir), stock_dir, ocr if ocr is not None else _ocr_engine()
)
if use_text else None
)
def rank(self, crop: Any) -> List[Tuple[str, float]]:
return self.lane.rank(crop)
def decide(self, crop: Any, ranked: List[Tuple[str, float]], *, accept: float,
margin: float, scale: str) -> Dict[str, Any]:
from identity_lanes import DecisionPolicy
if not self.use_text:
# E-3's other arm: the same policy, over the same ranking, with no
# text evidence. `dict(ranked)` would not do - it drops the lane's
# order, and the value is not the order (see `LocalLane.rank`).
return DecisionPolicy(local_accept=accept).decide(ranked)
read = self.text_lane.read(crop)
text_rank = sorted(
self.text_lane.scores_from_text(read).items(),
key=lambda kv: kv[1], reverse=True,
)
return DecisionPolicy(
local_accept=accept, signatures=self.text_lane.signatures
).decide(ranked, text_rank, text_read=read)
def _catalog_rows(stock_dir: str) -> List[Dict[str, Any]]:
"""The catalog's rows, in the shape TextLane.from_catalog expects."""
from inventory import InventoryManager
return InventoryManager(stock_dir).products.values()
def _ocr_engine() -> Any:
from ocr import build_ocr_engine
return build_ocr_engine()
class GalleryLane(Lane):
"""DINOv2 embedding, with the store's trained head when it has one."""
name = "gallery"
def __init__(self, stock_dir: str, sku_ids: Sequence[str], image_size: int = 518) -> None:
from gallery import ProductGallery
self.gallery = ProductGallery(stock_dir, list(sku_ids), image_size=image_size)
if not self.gallery.load():
raise SystemExit(f"gallery unavailable for {stock_dir}")
def rank(self, crop: Any) -> List[Tuple[str, float]]:
# Ungated: the lane's own accept floor is applied once, by score_lane.
# Gating here too would reject before the floor is even consulted.
return self.gallery.rank(crop)
class HybridLane(Lane):
"""DINO shortlist, then the original matcher re-ranks inside it."""
name = "hybrid-verify"
def __init__(self, shortlist_lane: Lane, verifier: MatcherLane, size: int = 5) -> None:
self.shortlist_lane = shortlist_lane
self.verifier = verifier
self.size = int(size)
self.recall_hits = 0
self.recall_total = 0
def rank(self, crop: Any) -> List[Tuple[str, float]]:
allowed = self.shortlist_lane.shortlist(crop, self.size)
if not allowed:
return []
scores, _ = self.verifier.matcher.score_crop(crop, method=self.verifier.method)
grouped = group_by_sku(scores)
ranked = [(sku, float(grouped.get(sku, 0.0))) for sku in allowed]
return sorted(ranked, key=lambda kv: kv[1], reverse=True)
def score_lane(
lane: Lane,
facings: Sequence[Dict[str, Any]],
image: Any,
*,
accept: float,
margin: float,
scale: str,
enrolled_skus: Collection[str],
) -> Dict[str, Any]:
"""One lane over one shelf. Accepts only unique, above-floor answers.
`top1` and `coverage` were both accepted-correct over enrolled facings, so
ranking and deciding were never separated. They are now four quantities:
catalog_ceiling in-catalog facings with an enrolled view / all in-catalog
identity_ceiling ungated top-1 rank correct on enrolled / enrolled
decision_ceiling accepted-and-correct on enrolled / enrolled
end_to_end_coverage accepted-and-correct / all in-catalog facings
Precision divides by every accept, unenrolled in-catalog facings included:
the SKU is on the shelf but has no reference, so an accept there is wrong
and is the failure the enrollment protocol exists to prevent.
"""
in_catalog = [f for f in facings if f["label"] != UNKNOWN]
out_catalog = [f for f in facings if f["label"] == UNKNOWN]
enrolled_count = sum(1 for f in in_catalog if f["label"] in enrolled_skus)
unenrolled_count = len(in_catalog) - enrolled_count
correct = wrong = rejected = top1_correct = 0
# The decision ceiling is over enrolled facings only. The text lane can
# answer an unenrolled sibling correctly, and counting that in a ratio
# whose denominator is the enrolled count reports a ceiling above 100%.
enrolled_correct = 0
unknown_false = unenrolled_false = 0
accepts: Counter = Counter()
per_sku: Dict[str, List[int]] = {}
started = time.perf_counter()
for facing in facings:
crop = crop_of(image, facing["box"])
if crop.size == 0:
continue
ranked = lane.rank(crop)
decision = lane.decide(crop, ranked, accept=accept, margin=margin, scale=scale)
# decide_identity reports an accept as status "success"; identity_level
# is the stable field across both the inlier and cosine scales.
answered = decision.get("best_sku") if decision.get("identity_level") == "accepted" else None
truth = facing["label"]
if truth != UNKNOWN:
bucket = per_sku.setdefault(truth, [0, 0])
bucket[1] += 1
if truth in enrolled_skus and ranked and ranked[0][0] == truth:
top1_correct += 1
if answered == truth:
correct += 1
bucket[0] += 1
if truth in enrolled_skus:
enrolled_correct += 1
elif answered:
wrong += 1
if truth not in enrolled_skus:
unenrolled_false += 1
elif answered:
wrong += 1
unknown_false += 1
if answered:
accepts[answered] += 1
else:
rejected += 1
elapsed = time.perf_counter() - started
total_accepts = sum(accepts.values())
hub = max(accepts.values()) / total_accepts if total_accepts else 0.0
return {
"lane": lane.name,
"accept": accept,
"margin": margin,
"in_catalog": len(in_catalog),
"out_catalog": len(out_catalog),
"enrolled": enrolled_count,
"correct": correct,
"wrong": wrong,
"rejected": rejected,
"catalog_ceiling": round(enrolled_count / len(in_catalog), 4) if in_catalog else None,
"identity_ceiling": round(top1_correct / enrolled_count, 4) if enrolled_count else None,
"decision_ceiling": round(enrolled_correct / enrolled_count, 4) if enrolled_count else None,
"end_to_end_coverage": round(correct / len(in_catalog), 4) if in_catalog else None,
"false_accepts": wrong,
"unknown_false_accepts": unknown_false,
"unenrolled_false_accepts": unenrolled_false,
"false_accept_rate": round(unknown_false / len(out_catalog), 4) if out_catalog else None,
"near_unknown_false_accept_rate": round(unenrolled_false / unenrolled_count, 4) if unenrolled_count else None,
"precision": round(correct / total_accepts, 4) if total_accepts else None,
"max_sku_share": round(hub, 4),
"hub_sku": accepts.most_common(1)[0][0] if accepts else None,
"hub_alarm": hub > HUB_SHARE,
"histogram": dict(accepts.most_common()),
"per_sku": {sku: {"correct": hit, "total": seen} for sku, (hit, seen) in sorted(per_sku.items())},
"ms_per_crop": round(elapsed * 1000 / max(1, len(facings)), 1),
}
def shortlist_recall(
lane: HybridLane,
facings: Sequence[Dict[str, Any]],
image: Any,
enrolled_skus: Collection[str],
) -> Dict[str, Any]:
"""Share of enrolled facings whose true SKU survives the shortlist.
This is the hybrid's hard ceiling: a verifier cannot pick a SKU the
shortlist already dropped. The denominator is enrolled facings only — a
facing whose SKU has no reference cannot be retrieved by any shortlist, so
counting it here reports a retrieval failure that is really a catalog gap.
Measured on the corrected labels, that distinction is 67% vs 98% at top-5.
"""
hits = total = 0
misses: List[Dict[str, Any]] = []
for facing in facings:
if facing["label"] == UNKNOWN or facing["label"] not in enrolled_skus:
continue
crop = crop_of(image, facing["box"])
if crop.size == 0:
continue
total += 1
allowed = lane.shortlist_lane.shortlist(crop, lane.size)
if facing["label"] in allowed:
hits += 1
else:
misses.append({"index": facing.get("index"), "truth": facing["label"], "shortlist": allowed})
return {
"size": lane.size,
"hits": hits,
"total": total,
"recall": round(hits / total, 4) if total else None,
"misses": misses,
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--labels", required=True, help="labelled facings JSON")
parser.add_argument("--image", help="shelf photo; defaults to the labels' own image")
parser.add_argument("--stock-dir", default="stock-lopia")
parser.add_argument("--lanes", default="local,matcher,gallery,hybrid-verify")
parser.add_argument("--shortlist", type=int, default=5)
parser.add_argument("--accept", type=float, default=None, help="override the lane's own floor")
parser.add_argument("--no-text", action="store_true",
help="run the local lane without the text lane (E-3)")
parser.add_argument("--margin", type=float, default=None)
parser.add_argument("--live", action="store_true", help="also score at live resolution")
parser.add_argument("--json", help="write the report here")
args = parser.parse_args()
payload = load_labels(args.labels)
image_path = args.image or payload["image"]
image = cv2.imread(image_path)
if image is None:
raise SystemExit(f"cannot read {image_path}")
facings = payload["facings"]
held: List[Dict[str, Any]] = []
refused: List[Dict[str, Any]] = []
for facing in facings:
if facing["label"] == EXCLUDE:
continue
template = leaked(image, facing["box"], args.stock_dir)
if template:
refused.append({**facing, "template": template})
else:
held.append(facing)
wanted = [name.strip() for name in args.lanes.split(",") if name.strip()]
# Candidate sets come from the catalog, never from the labels: a lane told
# which SKUs the shelf holds is being scored against the answer key.
catalog_skus = catalog_sku_ids(args.stock_dir)
# A SKU with no enrolled view cannot be answered by any lane. Counting those
# facings as lane failures hides the real ceiling: catalog coverage.
enrolled_skus = {sku_of(name) for name in os.listdir(args.stock_dir)
if os.path.splitext(name)[1].lower() in (".jpg", ".jpeg", ".png")}
unenrolled = Counter(f["label"] for f in held if f["label"] != UNKNOWN and f["label"] not in enrolled_skus)
lanes: Dict[str, Lane] = {}
if "matcher" in wanted or "hybrid-verify" in wanted or "van" in wanted:
lanes["matcher"] = MatcherLane(args.stock_dir)
if "van" in wanted:
lanes["van"] = VanLane(args.stock_dir)
if "local" in wanted:
lanes["local"] = LocalLaneAdapter(args.stock_dir, use_text=not args.no_text)
if "gallery" in wanted or "hybrid-verify" in wanted:
lanes["gallery"] = GalleryLane(args.stock_dir, catalog_skus)
if "hybrid-verify" in wanted:
lanes["hybrid-verify"] = HybridLane(lanes["gallery"], lanes["matcher"], size=args.shortlist)
# Boxes are in the original image's coordinates. Scoring a downscaled copy
# needs the boxes scaled with it, or every crop is taken from the wrong place.
resolutions = [("full", image, 1.0)]
if args.live:
live = at_live_resolution(image)
resolutions.append(("live", live, live.shape[1] / float(image.shape[1])))
report: Dict[str, Any] = {
"image": image_path,
"stock_dir": args.stock_dir,
"labels": args.labels,
# The enrollment protocol's versioning rule: every report records the
# catalog hash, the label-file hash and the code revision, so a number
# can be traced to the exact inputs that produced it.
"provenance": {
"catalog_hash": catalog_hash(args.stock_dir),
"labels_hash": file_hash(args.labels),
"image_hash": file_hash(image_path),
"code_revision": code_revision(),
},
"facings": len(facings),
"leakage": {"excluded": len(refused), "labels": refused},
"hub_share_limit": HUB_SHARE,
"enrollment": {
"catalog_skus": catalog_skus,
"enrolled_skus": sorted(enrolled_skus),
"unenrolled_skus": dict(unenrolled.most_common()),
"facings_without_enrollment": sum(unenrolled.values()),
"scored_facings": len(held),
},
"results": {},
}
for resolution, picture, factor in resolutions:
scaled = held if factor == 1.0 else [
{**f, "box": [int(round(v * factor)) for v in f["box"]]} for f in held
]
rows = []
for name in wanted:
lane = lanes[name]
scale = {"gallery": "cosine", "local": "local"}.get(name, "inlier")
default_accept = {"cosine": 0.60, "local": LOCAL_ACCEPT}.get(scale, 1.0)
default_margin = 0.04 if scale == "cosine" else 0.15
accept = args.accept if args.accept is not None else default_accept
margin = args.margin if args.margin is not None else default_margin
row = score_lane(lane, scaled, picture, accept=accept, margin=margin, scale=scale,
enrolled_skus=enrolled_skus)
if isinstance(lane, HybridLane):
row["shortlist_recall"] = shortlist_recall(lane, scaled, picture, enrolled_skus)
rows.append(row)
report["results"][resolution] = rows
if args.json:
with open(args.json, "w") as handle:
json.dump(report, handle, indent=1)
enrollment = report["enrollment"]
if enrollment["unenrolled_skus"]:
print(f"\nenrollment ceiling: {enrollment['facings_without_enrollment']} facing(s) belong to "
f"{len(enrollment['unenrolled_skus'])} SKU(s) with no enrolled view — "
f"unanswerable by any lane: {enrollment['unenrolled_skus']}")
for resolution, rows in report["results"].items():
print(f"\n{resolution} resolution — {enrollment['scored_facings']} scored facings "
f"({sum(1 for f in held if f['label'] != UNKNOWN)} in-catalog, "
f"{sum(1 for f in held if f['label'] == UNKNOWN)} unknown)")
print(f" {'lane':<14} {'catalog':>8} {'identity':>9} {'decision':>9} {'end-to-end':>11} "
f"{'precision':>10} {'false acc':>10} {'max SKU':>8} {'ms/crop':>8}")
for row in rows:
cells = [f"{row[key]:.0%}" if row[key] is not None else "-" for key in
("catalog_ceiling", "identity_ceiling", "decision_ceiling",
"end_to_end_coverage", "precision")]
flag = " HUB!" if row["hub_alarm"] else ""
print(f" {row['lane']:<14} {cells[0]:>8} {cells[1]:>9} {cells[2]:>9} {cells[3]:>11} "
f"{cells[4]:>10} {row['false_accepts']:>10} {row['max_sku_share']:>7.0%} "
f"{row['ms_per_crop']:>8.1f}{flag}")
if "shortlist_recall" in row:
recall = row["shortlist_recall"]
share = f"{recall['recall']:.0%}" if recall["recall"] is not None else "-"
print(f" shortlist top-{recall['size']} recall: {share} "
f"({recall['hits']}/{recall['total']}) — hard ceiling for this lane")
if refused:
print(f"\nleakage: {len(refused)} facing(s) refused as catalog templates")
return 0
if __name__ == "__main__":
raise SystemExit(main())

861
recognization/code/eval_live.py
View file

@ -1,861 +0,0 @@
"""Replay a recorded walk through LiveSession and count what it actually decided.
The recording format is FrameRecorder's: DIR/<session>/index.jsonl plus the JPEG
bytes the client sent. Optional labels live in labels.json next to that index
(or at --labels). A label is {"frame", "box", "sku_id"}.
Template leakage is refused, not scored. A label whose crop matches a catalog
template (byte-identical, or IoU >= 0.98 against a same-sized template pasted
into the frame) is excluded and counted under leakage. Those crops are how the
store was enrolled; scoring them is training-set accuracy.
python code/eval_live.py RECORDING_DIR [--stock-dir DIR] [--labels PATH]
[--json PATH] [--iou 0.5]
[--replay] [--identity matcher|local|gallery|van] [--device cpu|mps|cuda]
Prints a human count table and writes the same report as JSON (default
<recording>/eval_live.json). Missing input exits 2 with {"error": ...} on
stdout. A recording that names identity_backend is scored as that backend unless
--identity repeats it. A recording that does not name one requires --identity;
omitting it does not score the matcher. --identity gallery uses the same
LiveSession path as the server and exits 2 when that gallery cannot load; it
does not score the matcher instead.
The report names the backend, whether the head loaded, and the weight pins.
The same pinned recording, catalog, weights, and labels produce the
same counts; timings are omitted because they are not.
No held-out labels means denominators and fingerprints only. That is not a
walk accuracy number.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import sys
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
LEVELS = ("accepted", "ambiguous", "rejected", "unknown", "unsettled")
TAG_STATES = ("readable", "unreadable", "absent", "detected")
class EvalInputError(ValueError):
"""The recording, labels, or catalog cannot be scored as given."""
def file_sha256(path: str) -> Optional[str]:
digest = hashlib.sha256()
try:
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(1 << 20), b""):
digest.update(chunk)
except OSError:
return None
return digest.hexdigest()
def content_sha256(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
# Every backend the server can be started with, so a recording made by the
# shipped `run-demo-server.sh` (--identity local) can be replayed. `local` is
# the default; `matcher` is Van's scorer kept as the E-1 baseline lane.
IDENTITY_BACKENDS = ("matcher", "local", "gallery", "van")
def _iou(a: Sequence[float], b: Sequence[float]) -> float:
if len(a) != 4 or len(b) != 4:
return 0.0
ax1, ay1, ax2, ay2 = (float(v) for v in a)
bx1, by1, bx2, by2 = (float(v) for v in b)
ix1, iy1 = max(ax1, bx1), max(ay1, by1)
ix2, iy2 = min(ax2, bx2), min(ay2, by2)
inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
if inter <= 0.0:
return 0.0
area_a = max(0.0, ax2 - ax1) * max(0.0, ay2 - ay1)
area_b = max(0.0, bx2 - bx1) * max(0.0, by2 - by1)
union = area_a + area_b - inter
return inter / union if union > 0.0 else 0.0
def load_index(recording_dir: str) -> List[Dict[str, Any]]:
path = os.path.join(recording_dir, "index.jsonl")
if not os.path.isfile(path):
raise EvalInputError(f"missing recording index: {path}")
rows: List[Dict[str, Any]] = []
with open(path, "r", encoding="utf-8") as handle:
for number, line in enumerate(handle, start=1):
line = line.strip()
if not line:
continue
try:
entry = json.loads(line)
except json.JSONDecodeError as exc:
raise EvalInputError(f"{path}:{number}: {exc.msg}") from exc
if not isinstance(entry, dict) or "frame" not in entry or "path" not in entry:
raise EvalInputError(f"{path}:{number}: line needs frame and path")
rows.append(entry)
if not rows:
raise EvalInputError(f"empty recording index: {path}")
return rows
def recorded_identity(rows: Sequence[Dict[str, Any]]) -> Optional[str]:
"""Backend named by the recording, when every present value agrees.
FrameRecorder writes identity_backend on each line when the server constructed
it with the active backend. Older indexes may omit it, or name it once under
identity.backend. Silence is not a backend. A mix is not a default.
"""
found: List[str] = []
for row in rows:
raw = row.get("identity_backend")
if raw is None and isinstance(row.get("identity"), dict):
raw = row["identity"].get("backend")
if raw is None:
continue
name = str(raw).strip().lower()
if name not in IDENTITY_BACKENDS:
raise EvalInputError(
f"recorded identity backend is not one of {', '.join(IDENTITY_BACKENDS)}: {raw}"
)
found.append(name)
unique = set(found)
if len(unique) > 1:
raise EvalInputError("recording names more than one identity backend")
return found[0] if found else None
def resolve_identity(requested: Optional[str], recorded: Optional[str]) -> str:
"""CLI and recording must name the same backend. Silence is not matcher."""
chosen = (requested or recorded or "").strip().lower()
if not chosen:
raise EvalInputError(
"recording does not name an identity backend; pass --identity "
+ " or --identity ".join(IDENTITY_BACKENDS)
)
if chosen not in IDENTITY_BACKENDS:
raise EvalInputError(
f"identity must be one of {', '.join(IDENTITY_BACKENDS)}, not {requested or recorded}"
)
if recorded and requested and recorded != chosen:
raise EvalInputError(
f"recording identity is {recorded}; pass --identity {recorded}"
)
return chosen
def recording_fingerprint(recording_dir: str, rows: Sequence[Dict[str, Any]]) -> Dict[str, Any]:
"""Hash the JPEGs the client sent, in index order. Missing files are an error."""
digest = hashlib.sha256()
missing: List[str] = []
for row in rows:
name = str(row["path"])
path = os.path.join(recording_dir, name)
payload_hash = file_sha256(path)
if payload_hash is None:
missing.append(name)
continue
digest.update(f"{row['frame']}:{name}:{payload_hash}\n".encode("utf-8"))
if missing:
raise EvalInputError("missing recorded frames: " + ", ".join(missing[:8]))
return {"sha256": digest.hexdigest(), "frames": len(rows)}
def load_labels(path: Optional[str]) -> List[Dict[str, Any]]:
if not path:
return []
if not os.path.isfile(path):
raise EvalInputError(f"missing labels: {path}")
try:
with open(path, "r", encoding="utf-8") as handle:
payload = json.loads(handle.read())
except json.JSONDecodeError as exc:
raise EvalInputError(f"labels are not JSON: {exc.msg}") from exc
rows = payload.get("labels") if isinstance(payload, dict) else payload
if not isinstance(rows, list):
raise EvalInputError("labels must be a list or {\"labels\": [...]}")
cleaned: List[Dict[str, Any]] = []
for index, row in enumerate(rows):
if not isinstance(row, dict):
raise EvalInputError(f"labels[{index}] is not an object")
if "frame" not in row or "sku_id" not in row or "box" not in row:
raise EvalInputError(f"labels[{index}] needs frame, box, sku_id")
box = row["box"]
if not isinstance(box, (list, tuple)) or len(box) != 4:
raise EvalInputError(f"labels[{index}].box must be [x1, y1, x2, y2]")
cleaned.append(
{
"frame": int(row["frame"]),
"box": [int(v) for v in box],
"sku_id": str(row["sku_id"]),
"source": row.get("source"),
}
)
return cleaned
def catalog_fingerprint(stock_dir: str, stock_csv: str = "stock.csv") -> Dict[str, Any]:
"""Same inputs as RecognitionEngine.catalog_version, plus the full digest and head."""
digest = hashlib.sha256()
csv_path = os.path.join(stock_dir, stock_csv)
csv_hash = file_sha256(csv_path)
if csv_hash is None:
digest.update(b"missing-csv")
else:
digest.update(bytes.fromhex(csv_hash))
try:
entries = sorted(
(name, os.path.getsize(os.path.join(stock_dir, name)))
for name in os.listdir(stock_dir)
if name.lower().endswith((".jpeg", ".jpg", ".png"))
)
except OSError as exc:
raise EvalInputError(f"cannot read catalog {stock_dir}: {exc.strerror}") from exc
for name, size in entries:
digest.update(f"{name}:{size}".encode("utf-8"))
full = digest.hexdigest()
head_path = os.path.join(stock_dir, "identity_head.pt")
return {
"stock_dir": os.path.basename(os.path.abspath(stock_dir)),
"version": full[:12],
"sha256": full,
"templates": len(entries),
"csv_sha256": csv_hash,
"identity_head_sha256": file_sha256(head_path),
}
def model_fingerprint(
weights_path: str,
*,
identity: str,
gallery: Any = None,
) -> Dict[str, Any]:
"""YOLO pin plus, for gallery, the backbone and head that actually loaded."""
try:
import ultralytics
library = f"ultralytics-{ultralytics.__version__}"
except Exception:
library = "ultralytics-unknown"
pin: Dict[str, Any] = {
"version": f"{os.path.basename(weights_path)}@{library}" if weights_path else f"none@{library}",
"weights_sha256": file_sha256(weights_path) if weights_path else None,
"identity": identity,
}
if identity != "gallery" or gallery is None:
if identity == "gallery":
pin["gallery_loaded"] = False
pin["gallery_head_loaded"] = False
return pin
from gallery import DINOV2_SHA256, dinov2_weights_path
backbone = dinov2_weights_path()
digest = file_sha256(backbone)
pin.update(
{
"gallery_loaded": True,
"gallery_head": getattr(gallery, "head_reason", None),
"gallery_head_loaded": getattr(gallery, "_head", None) is not None,
"gallery_views": gallery.view_count() if hasattr(gallery, "view_count") else None,
"dinov2_path": backbone,
"dinov2_sha256": digest,
"dinov2_sha256_expected": DINOV2_SHA256,
"dinov2_pinned": digest == DINOV2_SHA256,
}
)
return pin
def _decode_image(payload: bytes) -> Any:
import cv2
import numpy as np
image = cv2.imdecode(np.frombuffer(payload, dtype=np.uint8), cv2.IMREAD_COLOR)
if image is None:
raise EvalInputError("recorded frame is not a decodable image")
return image
def template_crops(stock_dir: str) -> List[Tuple[str, Any]]:
"""Catalog photos keyed by SKU stem (sku_l05, sku_l05_1, ...)."""
import cv2
crops: List[Tuple[str, Any]] = []
try:
names = sorted(os.listdir(stock_dir))
except OSError:
return crops
for name in names:
stem, extension = os.path.splitext(name)
if extension.lower() not in (".jpeg", ".jpg", ".png"):
continue
image = cv2.imread(os.path.join(stock_dir, name))
if image is None or image.size == 0:
continue
crops.append((stem, image))
return crops
def _same_picture(crop: Any, template: Any) -> bool:
"""True when two same-sized images are the same photo after JPEG."""
import numpy as np
left = crop.astype("float32").reshape(-1)
right = template.astype("float32").reshape(-1)
if left.size < 16 or left.size != right.size:
return False
left = left - float(left.mean())
right = right - float(right.mean())
denom = float(np.sqrt((left * left).sum() * (right * right).sum()))
if denom <= 1e-6:
return bool(np.max(np.abs(crop.astype("int16") - template.astype("int16"))) <= 2)
return float((left * right).sum() / denom) >= 0.98
def leaked_labels(
recording_dir: str,
rows: Sequence[Dict[str, Any]],
labels: Sequence[Dict[str, Any]],
stock_dir: str,
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
"""Drop labels whose crop is a catalog template.
Byte identity catches a frame that is the template file. Geometric identity
catches a template pasted into a larger frame at the labelled box.
"""
if not labels:
return [], []
templates = template_crops(stock_dir)
template_hashes = {}
try:
names = os.listdir(stock_dir)
except OSError:
names = []
for name in names:
digest = file_sha256(os.path.join(stock_dir, name))
if digest:
template_hashes.setdefault(digest, os.path.splitext(name)[0])
by_frame: Dict[int, List[Dict[str, Any]]] = {}
for label in labels:
by_frame.setdefault(int(label["frame"]), []).append(label)
held: List[Dict[str, Any]] = []
leaked: List[Dict[str, Any]] = []
frames = {int(row["frame"]): row for row in rows}
for frame_id, frame_labels in by_frame.items():
row = frames.get(frame_id)
image = None
payload_hash = None
if row is not None:
path = os.path.join(recording_dir, str(row["path"]))
try:
with open(path, "rb") as handle:
payload = handle.read()
except OSError:
payload = b""
payload_hash = content_sha256(payload) if payload else None
if payload:
image = _decode_image(payload)
for label in frame_labels:
reason = None
matched = None
if payload_hash in template_hashes:
reason = "frame_is_template"
matched = template_hashes[payload_hash]
elif image is not None:
x1, y1, x2, y2 = label["box"]
crop = image[max(0, y1) : max(0, y2), max(0, x1) : max(0, x2)]
if crop.size:
import cv2
ch, cw = crop.shape[:2]
for stem, template in templates:
th, tw = template.shape[:2]
if abs(th - ch) > 1 or abs(tw - cw) > 1:
continue
resized = template if (th, tw) == (ch, cw) else cv2.resize(template, (cw, ch))
# JPEG re-encoding keeps layout and drops exact pixels.
if _same_picture(crop, resized):
reason = "crop_is_template"
matched = stem
break
if reason:
leaked.append({**label, "leak": reason, "template": matched})
else:
held.append(label)
return held, leaked
def identity_level(track: Dict[str, Any]) -> str:
identity = track.get("identity") or {}
level = identity.get("identity_level")
if level in LEVELS:
return str(level)
state = identity.get("state")
if state == "recognized":
return "accepted"
if state == "needs_confirmation":
return "ambiguous"
if state == "unrecognized":
return "rejected"
return "unsettled"
def tag_state(track: Dict[str, Any]) -> str:
tag = track.get("price_tag") or track.get("tag") or {}
state = tag.get("state")
if state in TAG_STATES:
return str(state)
if tag:
return "detected"
return "absent"
def score_tracks(tracks: Iterable[Dict[str, Any]]) -> Dict[str, Any]:
"""Denominators over settled decisions. Unsettled tracks are not in the rate."""
counts = {level: 0 for level in LEVELS}
tags = {state: 0 for state in TAG_STATES}
for track in tracks:
counts[identity_level(track)] += 1
tags[tag_state(track)] += 1
settled = counts["accepted"] + counts["ambiguous"] + counts["rejected"] + counts["unknown"]
return {
"tracks": sum(counts.values()),
"settled": settled,
"accepted": counts["accepted"],
"ambiguous": counts["ambiguous"],
"rejected": counts["rejected"],
"unknown": counts["unknown"],
"unsettled": counts["unsettled"],
"accepted_rate": _rate(counts["accepted"], settled),
"ambiguous_rate": _rate(counts["ambiguous"], settled),
"rejected_rate": _rate(counts["rejected"], settled),
"tags_detected": tags["detected"] + tags["readable"] + tags["unreadable"],
"tags_readable": tags["readable"],
"tags_unreadable": tags["unreadable"],
"tags_absent": tags["absent"],
}
def _rate(numerator: int, denominator: int) -> Optional[float]:
if denominator <= 0:
return None
return round(numerator / denominator, 4)
def track_churn(frames: Sequence[Dict[str, Any]]) -> Dict[str, Any]:
"""How often one track id changes the SKU it is willing to name.
A change counts only after the id has already named a SKU. Restarting or
dropping the label (relabels/restarts) is reported beside that, from the
counters LiveSession already publishes.
"""
previous: Dict[int, Optional[str]] = {}
changes: Dict[int, int] = {}
seen: Dict[int, int] = {}
relabels: Dict[int, int] = {}
restarts: Dict[int, int] = {}
for frame in frames:
for track in frame.get("tracks") or []:
track_id = track.get("track_id")
if track_id is None:
continue
track_id = int(track_id)
seen[track_id] = seen.get(track_id, 0) + 1
relabels[track_id] = max(relabels.get(track_id, 0), int(track.get("relabels") or 0))
restarts[track_id] = max(restarts.get(track_id, 0), int(track.get("restarts") or 0))
named = None
if identity_level(track) == "accepted":
named = (track.get("identity") or {}).get("sku_id")
prior = previous.get(track_id)
if prior and named and named != prior:
changes[track_id] = changes.get(track_id, 0) + 1
if named:
previous[track_id] = named
churned = sum(1 for count in changes.values() if count)
return {
"tracks": len(seen),
"tracks_with_sku_change": churned,
"sku_changes": sum(changes.values()),
"relabels": sum(relabels.values()),
"restarts": sum(restarts.values()),
"per_track": [
{
"track_id": track_id,
"observations": seen[track_id],
"sku_changes": changes.get(track_id, 0),
"relabels": relabels.get(track_id, 0),
"restarts": restarts.get(track_id, 0),
}
for track_id in sorted(seen)
if changes.get(track_id, 0) or relabels.get(track_id, 0) or restarts.get(track_id, 0)
],
}
def labelled_top1(
frames: Sequence[Dict[str, Any]],
labels: Sequence[Dict[str, Any]],
*,
iou_min: float,
) -> Dict[str, Any]:
"""Top-1 over held-out labels. Accepted wrong SKU is a miss, not a reject.
Denominator is labels that overlap a track. A label with no overlapping
track is unmatched and stays out of the top-1 rate.
"""
by_frame: Dict[int, List[Dict[str, Any]]] = {}
for frame in frames:
by_frame.setdefault(int(frame.get("frame", frame.get("frame_id"))), []).append(frame)
correct = wrong = rejected = ambiguous = missed = 0
matched = 0
for label in labels:
observations = by_frame.get(int(label["frame"]), [])
best_iou = 0.0
best: Optional[Dict[str, Any]] = None
for observation in observations:
for track in observation.get("tracks") or []:
score = _iou(label["box"], track.get("box") or [])
if score > best_iou:
best_iou = score
best = track
if best is None or best_iou < iou_min:
missed += 1
continue
matched += 1
level = identity_level(best)
guess = (best.get("identity") or {}).get("sku_id")
if level == "accepted" and guess == label["sku_id"]:
correct += 1
elif level == "accepted":
wrong += 1
elif level == "ambiguous":
ambiguous += 1
else:
rejected += 1
return {
"labels": len(labels),
"matched": matched,
"unmatched": missed,
"correct": correct,
"wrong": wrong,
"rejected": rejected,
"ambiguous": ambiguous,
"top1": _rate(correct, matched),
"iou_min": iou_min,
}
def score_recording(
recording_dir: str,
*,
stock_dir: str,
weights_path: str = "",
labels_path: Optional[str] = None,
iou_min: float = 0.5,
live_frames: Optional[Sequence[Dict[str, Any]]] = None,
identity: str,
gallery: Any = None,
) -> Dict[str, Any]:
"""Score one session directory. live_frames, when given, are the replayed results.
Without live_frames the recorded index is what the server already decided.
That still produces denominators; it is not a fresh model run. identity is
the backend those frames were named with, not a claim that this process
loaded it. gallery, when passed, is the loaded ProductGallery.
"""
recording_dir = os.path.abspath(recording_dir)
rows = load_index(recording_dir)
sample = recording_fingerprint(recording_dir, rows)
labels_file = labels_path or os.path.join(recording_dir, "labels.json")
labels = load_labels(labels_file if os.path.isfile(labels_file) or labels_path else None)
held, leaked = leaked_labels(recording_dir, rows, labels, stock_dir)
source = list(live_frames) if live_frames is not None else rows
tracks = [track for frame in source for track in (frame.get("tracks") or [])]
decisions = score_tracks(tracks)
backend = identity.strip().lower()
if backend not in IDENTITY_BACKENDS:
raise EvalInputError(
f"identity must be one of {', '.join(IDENTITY_BACKENDS)}, not {identity}"
)
report = {
"recording": recording_dir,
"source": "live_replay" if live_frames is not None else "recorded_index",
"identity": {
"backend": backend,
"gallery_loaded": gallery is not None if backend == "gallery" else None,
"gallery_head": getattr(gallery, "head_reason", None) if gallery is not None else None,
"gallery_head_loaded": (
bool(getattr(gallery, "_head", None) is not None) if gallery is not None else None
),
},
"fingerprints": {
"sample": sample,
"model": model_fingerprint(weights_path, identity=backend, gallery=gallery),
"catalog": catalog_fingerprint(stock_dir),
},
"decisions": decisions,
"tags": {
"detected": decisions["tags_detected"],
"readable": decisions["tags_readable"],
"unreadable": decisions["tags_unreadable"],
"absent": decisions["tags_absent"],
},
"churn": track_churn(source),
"leakage": {"excluded": len(leaked), "labels": leaked},
"labelled_top1": labelled_top1(source, held, iou_min=iou_min),
"evidence": _evidence(labels, held, leaked, live_frames is not None),
}
return report
def _evidence(
labels: Sequence[Dict[str, Any]],
held: Sequence[Dict[str, Any]],
leaked: Sequence[Dict[str, Any]],
replayed: bool,
) -> List[str]:
gaps: List[str] = []
if not replayed:
gaps.append("counts are from the recorded index, not a fresh LiveSession replay")
if not labels:
gaps.append("no held-out labels; labelled top-1 is not a walk accuracy")
elif not held:
gaps.append("every label matched a catalog template and was excluded")
if leaked:
gaps.append(f"{len(leaked)} template-derived label(s) excluded")
return gaps
def replay_live(
recording_dir: str,
engine: Any,
*,
settings: Optional[Any] = None,
) -> List[Dict[str, Any]]:
"""Feed recorded JPEGs through one LiveSession, in index order.
Classification is asynchronous. Each frame is scored only after its
in-flight crops finish, then processed again so the label is on the
payload. Without that wait the same recording does not replay twice.
"""
from live import LiveSession
rows = load_index(recording_dir)
session = LiveSession("eval", engine, settings=settings)
session.refresh_catalog()
if hasattr(session, "enable_tags"):
session.enable_tags()
frames: List[Dict[str, Any]] = []
for row in rows:
path = os.path.join(recording_dir, str(row["path"]))
with open(path, "rb") as handle:
payload = handle.read()
frame_id = int(row["frame"])
result = session.process_frame(frame_id, payload)
# min_hits is 2, so the first pass only detects. Keep passing the same
# bytes until a pass schedules nothing and everything in flight lands.
for _ in range(4):
_wait_until_settled(session)
result = session.process_frame(frame_id, payload)
if int((session.snapshot().get("stats") or {}).get("classify_pending") or 0) == 0:
break
result["frame"] = frame_id
frames.append(result)
engine_shutdown = getattr(engine, "shutdown", None)
if callable(engine_shutdown):
engine_shutdown()
return frames
def _wait_until_settled(session: Any, timeout: float = 30.0) -> None:
import time
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
pending = int((session.snapshot().get("stats") or {}).get("classify_pending") or 0)
if pending == 0:
return
time.sleep(0.02)
raise EvalInputError("classification did not finish before the replay timeout")
def format_report(report: Dict[str, Any]) -> str:
decisions = report["decisions"]
tags = report["tags"]
top1 = report["labelled_top1"]
churn = report["churn"]
fingerprints = report["fingerprints"]
identity = report.get("identity") or {}
model = fingerprints["model"]
head = identity.get("gallery_head")
head_note = f" head {head}" if head else ""
lines = [
f"source: {report['source']}",
f"identity: {identity.get('backend')}{head_note}",
(
f"decisions: accepted {decisions['accepted']} ambiguous {decisions['ambiguous']} "
f"rejected {decisions['rejected']} unknown {decisions['unknown']} "
f"unsettled {decisions['unsettled']} settled {decisions['settled']}"
),
(
f"rates: accepted {decisions['accepted_rate']} "
f"ambiguous {decisions['ambiguous_rate']} rejected {decisions['rejected_rate']} "
f"(denominator settled)"
),
(
f"tags: detected {tags['detected']} readable {tags['readable']} "
f"unreadable {tags['unreadable']} absent {tags['absent']}"
),
(
f"churn: tracks {churn['tracks']} sku_changes {churn['sku_changes']} "
f"relabels {churn['relabels']} restarts {churn['restarts']}"
),
(
f"labelled top-1: {top1['correct']}/{top1['matched']} = {top1['top1']} "
f"wrong {top1['wrong']} unmatched {top1['unmatched']} "
f"leakage excluded {report['leakage']['excluded']}"
),
(
f"fingerprints: sample {fingerprints['sample']['sha256'][:12]} "
f"model {model['version']} "
f"weights {str(model.get('weights_sha256') or 'none')[:12]} "
f"catalog {fingerprints['catalog']['version']}"
),
]
if model.get("identity") == "gallery":
lines.append(
f"gallery: loaded {model.get('gallery_loaded')} "
f"head {model.get('gallery_head')} "
f"dinov2 {str(model.get('dinov2_sha256') or 'missing')[:12]} "
f"pinned {model.get('dinov2_pinned')}"
)
for gap in report["evidence"]:
lines.append(f"gap: {gap}")
return "\n".join(lines)
def replay_engine(stock_dir: str, weights: str, identity: str, device: Optional[str]) -> Tuple[Any, str, Any]:
"""Build the engine the server would, and refuse a gallery that did not load.
The backbone file is checked before YOLO loads. A missing or unpinned file
is the same refusal the server makes, without scoring the matcher.
"""
from engine import EngineSettings, build_default_engine
if identity == "gallery":
from gallery import DINOV2_SHA256, dinov2_weights_path
backbone = dinov2_weights_path()
digest = file_sha256(backbone)
if digest is None:
raise EvalInputError(
f"gallery identity backend requires local DINOv2 weights at {backbone}; "
"refusing to score the matcher"
)
if digest != DINOV2_SHA256:
raise EvalInputError(
f"DINOv2 weights at {backbone} are sha256 {digest}, expected {DINOV2_SHA256}"
)
settings = EngineSettings(stock_dir=stock_dir, identity_backend=identity)
if weights:
settings.weights_path = weights
if device:
settings.device = device
engine = build_default_engine(settings)
chosen = (engine.settings.identity_backend or "matcher").lower()
if chosen != identity:
engine.shutdown()
raise EvalInputError(f"engine identity is {chosen}, requested {identity}")
gallery = None
if identity == "gallery":
gallery = engine.identity_gallery()
if gallery is None:
engine.shutdown()
raise EvalInputError(
"gallery identity backend did not load; refusing to score the matcher"
)
return engine, engine.settings.weights_path, gallery
def main(argv: Optional[Sequence[str]] = None) -> int:
parser = argparse.ArgumentParser(description="Score a recorded live walk.")
parser.add_argument("recording", nargs="?", help="session directory with index.jsonl")
parser.add_argument("--stock-dir", default="stock")
parser.add_argument("--weights", default="")
parser.add_argument("--labels", default=None)
parser.add_argument("--json", dest="json_path", default=None)
parser.add_argument("--iou", type=float, default=0.5)
parser.add_argument(
"--replay",
action="store_true",
help="re-run frames through LiveSession (needs the recognition stack)",
)
parser.add_argument("--method", default="hybrid")
parser.add_argument(
"--identity",
default=None,
choices=list(IDENTITY_BACKENDS),
help="which mechanism names products (required when the recording omits identity_backend)",
)
parser.add_argument("--device", default=None, help="cpu | mps | cuda | cuda:0")
args = parser.parse_args(argv)
def fail(message: str) -> int:
print(json.dumps({"error": message}, ensure_ascii=False))
print(f"error: {message}", file=sys.stderr)
return 2
if not args.recording:
return fail("recording directory is required")
recording = os.path.abspath(args.recording)
if not os.path.isdir(recording):
return fail(f"recording directory not found: {recording}")
try:
rows = load_index(recording)
identity = resolve_identity(args.identity, recorded_identity(rows))
live_frames = None
gallery = None
weights = args.weights
if args.replay:
from live import LiveSettings
engine, weights, gallery = replay_engine(
args.stock_dir, args.weights, identity, args.device
)
live_frames = replay_live(
recording, engine, settings=LiveSettings(method=args.method, tags=True)
)
report = score_recording(
recording,
stock_dir=args.stock_dir,
weights_path=weights,
labels_path=args.labels,
iou_min=args.iou,
live_frames=live_frames,
identity=identity,
gallery=gallery,
)
except EvalInputError as exc:
return fail(str(exc))
text = json.dumps(report, ensure_ascii=False, indent=1, sort_keys=True)
destination = args.json_path or os.path.join(recording, "eval_live.json")
try:
with open(destination, "w", encoding="utf-8") as handle:
handle.write(text + "\n")
except OSError as exc:
return fail(f"cannot write {destination}: {exc.strerror}")
print(format_report(report))
print(f"json: {destination}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

6
recognization/code/fast_test.py Normal file
View file

@ -0,0 +1,6 @@
from ultralytics import YOLO
model = YOLO("/Users/nguyenhongvan/recongnize/runs/detect/train/weights/best.pt")
image_path = "/Users/nguyenhongvan/recongnize/data/add/image7.jpg"
results = model(image_path)
results[0].show()

148
recognization/code/feature_matcher.py
View file

@ -7,9 +7,7 @@ the methodology of Electronics 2023, 12(17), 3640.
import os
import cv2
import numpy as np
import threading
from typing import Dict, List, Tuple, Optional, Any
from identity import decide_identity
class LocalFeatureExtractor:
@ -124,18 +122,6 @@ class ProductFeatureMatcher:
def __init__(self, stock_dir: str = "stock"):
self.db = TemplateFeatureDatabase(stock_dir)
self.extractors = self.db.extractors
self._thread_local = threading.local()
def thread_extractors(self) -> Dict[str, "LocalFeatureExtractor"]:
"""One extractor set per thread: OpenCV Feature2D objects are not safe to
call concurrently from several threads."""
extractors = getattr(self._thread_local, "extractors", None)
if extractors is None:
extractors = {
name: LocalFeatureExtractor(name) for name in ("surf", "orb", "brisk")
}
self._thread_local.extractors = extractors
return extractors
def match_descriptors(
self,
@ -187,29 +173,26 @@ class ProductFeatureMatcher:
except Exception:
return len(good_matches), good_matches, None
def score_crop(
def classify_crop(
self,
crop_bgr: np.ndarray,
method: str = "hybrid"
) -> Tuple[Dict[str, float], Dict[str, Dict[str, Any]]]:
) -> Dict[str, Any]:
"""
Scores a product crop against every SKU template.
Returns (scores, details); raises ValueError for an unsupported method and
returns empty results for an empty crop.
Classifies a product crop against all SKU templates.
Methods supported: 'surf', 'orb', 'brisk', 'hybrid'.
"""
if crop_bgr is None or crop_bgr.size == 0:
return {}, {}
return {"status": "error", "message": "Empty crop image"}
crop_gray = cv2.cvtColor(crop_bgr, cv2.COLOR_BGR2GRAY)
method = method.lower()
extractors = self.thread_extractors()
scores: Dict[str, float] = {}
details: Dict[str, Dict[str, Any]] = {}
if method in ["surf", "orb", "brisk"]:
extractor = extractors.get(method)
extractor = self.extractors.get(method)
if extractor is None:
extractor = LocalFeatureExtractor(method)
kp_query, des_query = extractor.detect_and_compute(crop_gray)
@ -248,7 +231,7 @@ class ProductFeatureMatcher:
# Pre-compute features for query crop across all 3 extractors
query_features = {}
for sm in sub_methods:
kp_q, des_q = extractors[sm].detect_and_compute(crop_gray)
kp_q, des_q = self.extractors[sm].detect_and_compute(crop_gray)
query_features[sm] = (kp_q, des_q)
for sku_id, tmpl in self.db.templates.items():
@ -259,7 +242,7 @@ class ProductFeatureMatcher:
kp_q, des_q = query_features[sm]
kp_t = tmpl["features"][sm]["keypoints"]
des_t = tmpl["features"][sm]["descriptors"]
norm_t = extractors[sm].norm_type
norm_t = self.extractors[sm].norm_type
matches = self.match_descriptors(des_q, des_t, norm_t, ratio_thresh=0.75)
inliers, inlier_list, _ = self.compute_geometric_inliers(kp_q, kp_t, matches)
@ -283,103 +266,30 @@ class ProductFeatureMatcher:
"kp_query": query_features["surf"][0],
"kp_tmpl": tmpl["features"]["surf"]["keypoints"]
}
else:
raise ValueError(f"Unknown method: {method}")
return scores, details
def score_crop_fast(
self,
crop_bgr: np.ndarray,
shortlist: int = 5
) -> Tuple[Dict[str, float], Dict[str, Dict[str, Any]]]:
"""
ORB-shortlist then hybrid-score the top candidates only.
Same scoring formula as 'hybrid', but the expensive SURF/ORB/BRISK
descriptor matching runs on `shortlist` templates instead of all of them.
Every other template scores 0.0, so decide_identity() still ranks the
whole catalog. This is a live-mode speed path; validate it against
'hybrid' with bench_engine.py --compare before trusting it.
"""
if crop_bgr is None or crop_bgr.size == 0:
return {}, {}
crop_gray = cv2.cvtColor(crop_bgr, cv2.COLOR_BGR2GRAY)
extractors = self.thread_extractors()
orb = extractors["orb"]
kp_orb, des_orb = orb.detect_and_compute(crop_gray)
orb_scores: Dict[str, float] = {}
for sku_id, tmpl in self.db.templates.items():
kp_t = tmpl["features"]["orb"]["keypoints"]
des_t = tmpl["features"]["orb"]["descriptors"]
matches = self.match_descriptors(des_orb, des_t, orb.norm_type, ratio_thresh=0.75)
inliers, _, _ = self.compute_geometric_inliers(kp_orb, kp_t, matches)
denom = max(1, min(len(kp_orb), len(kp_t)))
orb_scores[sku_id] = inliers * 1.5 + (inliers / denom) * 80.0
candidates = sorted(orb_scores.items(), key=lambda kv: kv[1], reverse=True)[:shortlist]
scores: Dict[str, float] = {sku_id: 0.0 for sku_id in self.db.templates}
details: Dict[str, Dict[str, Any]] = {}
sub_methods = ["surf", "orb", "brisk"]
weights = {"surf": 0.45, "orb": 0.25, "brisk": 0.30}
query_features = {}
for sm in sub_methods:
kp_q, des_q = extractors[sm].detect_and_compute(crop_gray)
query_features[sm] = (kp_q, des_q)
for sku_id, _ in candidates:
tmpl = self.db.templates[sku_id]
composite = 0.0
sku_sub_details = {}
for sm in sub_methods:
kp_q, des_q = query_features[sm]
kp_t = tmpl["features"][sm]["keypoints"]
des_t = tmpl["features"][sm]["descriptors"]
matches = self.match_descriptors(des_q, des_t, extractors[sm].norm_type, ratio_thresh=0.75)
inliers, _, _ = self.compute_geometric_inliers(kp_q, kp_t, matches)
denom = max(1, min(len(kp_q), len(kp_t)))
sub_score = inliers * 1.5 + (inliers / denom) * 80.0
composite += weights[sm] * sub_score
sku_sub_details[sm] = {
"inliers": inliers,
"matches": len(matches),
"kp_q_count": len(kp_q),
"kp_t_count": len(kp_t),
"sub_score": round(sub_score, 2),
}
scores[sku_id] = composite
details[sku_id] = {
"score": composite,
"sub_details": sku_sub_details,
"kp_query": query_features["surf"][0],
"kp_tmpl": tmpl["features"]["surf"]["keypoints"],
}
return scores, details
def classify_crop(
self,
crop_bgr: np.ndarray,
method: str = "hybrid"
) -> Dict[str, Any]:
"""
Classifies a product crop against all SKU templates.
Methods supported: 'surf', 'orb', 'brisk', 'hybrid', 'hybrid_fast'.
"""
if crop_bgr is None or crop_bgr.size == 0:
return {"status": "error", "message": "Empty crop image"}
method = method.lower()
if method == "hybrid_fast":
scores, details = self.score_crop_fast(crop_bgr)
elif method in ["surf", "orb", "brisk", "hybrid"]:
scores, details = self.score_crop(crop_bgr, method=method)
else:
return {"status": "error", "message": f"Unknown method: {method}"}
ranked = decide_identity(scores, details=details, method=method)
return ranked
# Rank templates by score
if not scores:
return {"status": "error", "message": "No templates available"}
sorted_candidates = sorted(scores.items(), key=lambda x: x[1], reverse=True)
top_sku, top_score = sorted_candidates[0]
runner_up_sku, runner_up_score = sorted_candidates[1] if len(sorted_candidates) > 1 else (None, 0.0)
confidence = round(min(1.0, top_score / max(1.0, (top_score + runner_up_score * 0.5))), 4)
return {
"status": "success",
"method": method,
"best_sku": top_sku,
"score": round(top_score, 2),
"confidence": confidence,
"ranked_candidates": [
{"sku_id": s, "score": round(sc, 2)} for s, sc in sorted_candidates[:5]
],
"details": details[top_sku]
}
def generate_match_visualization(
self,
@ -396,7 +306,7 @@ class ProductFeatureMatcher:
crop_gray = cv2.cvtColor(crop_bgr, cv2.COLOR_BGR2GRAY)
eval_method = method if method in ["surf", "orb", "brisk"] else "surf"
extractor = self.thread_extractors()[eval_method]
extractor = self.extractors[eval_method]
kp_q, des_q = extractor.detect_and_compute(crop_gray)
kp_t = tmpl["features"][eval_method]["keypoints"]
des_t = tmpl["features"][eval_method]["descriptors"]

413
recognization/code/finetune_identity.py
View file

@ -1,413 +0,0 @@
"""
Fine-tune the identity embedding on the labels we can actually produce.
Measured: off-the-shelf DINOv2 tops out at ~25 % on a real milk shelf where the
products differ only by small text and a colour band, while it reaches 72-76 % on the
demo shelf, whose facings are near-identical crops of one photograph. That gap is the
whole problem, and the literature says the same thing (RP2K and RPC fine-tune;
GroceryVision reports a 17.5-point drop from Recall@5 to Recall@1).
This trains a small projection head on frozen DINOv2 features and compares, on the
same held-out facings, three ways of naming a product:
raw cosine the gallery as shipped
fine-tuned cosine the same matching, on projected features
fine-tuned + margin the same, trained with an angular margin
**What this can and cannot show.** The labels come from one shelf, and the held-out
facings are other facings of the same photograph, so the number is an upper bound: it
measures whether a trained head separates these products at all, not whether it
survives a second visit. Augmentation (crop, scale, brightness, blur) stands in for
the visit we do not have; a cross-visit number needs cross-visit data.
python code/finetune_identity.py [--epochs 60] [--views 2]
"""
from __future__ import annotations
import argparse
import os
import sys
import cv2
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
# Labelled by eye from a contact sheet of the detections (docs/EXTERNAL-EVAL.md).
MILK_SHELF_LABELS = {
"meiji_oishii": [13, 14, 15, 16, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36],
"tokunou45": [9, 10, 11, 12],
"megmilk": [20, 21, 22, 23, 24, 25, 26, 37, 38, 46, 47, 54, 55, 56, 57, 58, 59,
64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 90, 94, 95, 96, 97, 98, 99],
"hokkaido": [49, 50, 51, 60, 61, 62, 63, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,
100, 101, 102, 103, 104],
"teishibou": [2, 3, 4],
"meiji_love": [5, 6, 7, 8],
"oishii_tounyu": [42, 43, 44, 45],
"rakunou36": [17, 18],
}
def enrollment_report(stock: str, csv_name: str = "stock.csv") -> dict:
"""Catalog versus enrolled views. Does not load a model or write files.
A SKU with no readable view cannot be recognized. One view is reported, not
treated as enough: the measured knee is two views. Duplicate CSV rows and a
head whose class list disagrees with the catalog are blockers.
"""
import csv
import re
from gallery import ProductGallery
names = []
try:
names = os.listdir(stock)
except OSError:
names = []
images = [name for name in names if os.path.splitext(name)[1].lower() in (".jpeg", ".jpg", ".png")]
rows = []
csv_path = os.path.join(stock, csv_name)
if os.path.exists(csv_path):
with open(csv_path, encoding="utf-8") as handle:
rows = list(csv.DictReader(handle))
sku_ids = []
seen = set()
duplicate_rows = []
for row in rows:
sku_id = (row.get("SKU_ID") or "").strip()
if not sku_id:
continue
if sku_id in seen:
duplicate_rows.append(sku_id)
seen.add(sku_id)
sku_ids.append(sku_id)
gallery = ProductGallery(stock, sku_ids or [])
views = {sku_id: gallery._views_for(sku_id) for sku_id in dict.fromkeys(sku_ids)}
missing = [sku_id for sku_id, paths in views.items() if not paths]
single = [sku_id for sku_id, paths in views.items() if len(paths) == 1]
claimed = {os.path.basename(path) for paths in views.values() for path in paths}
orphans = sorted(name for name in images if name not in claimed)
# ponytail: prefix collision check. A view named sku_001 must not also match sku_00.
collisions = []
for sku_id, paths in views.items():
bare = [os.path.basename(path) for path in paths if os.path.splitext(os.path.basename(path))[0] != sku_id
and not re.fullmatch(re.escape(sku_id) + r"[_-]\d+", os.path.splitext(os.path.basename(path))[0])]
if bare:
collisions.append(sku_id)
blockers = []
if duplicate_rows:
blockers.append("duplicate_csv_sku")
if missing:
blockers.append("sku_without_view")
if collisions:
blockers.append("view_prefix_collision")
head_path = os.path.join(stock, "identity_head.pt")
head_note = None
if os.path.exists(head_path):
head_note = "present_not_opened"
# Do not torch.load here: the head can be large, and this report must stay cheap.
# Class-list drift is reported when the caller passes class_ids.
return {
"stock": stock,
"sku_ids": list(dict.fromkeys(sku_ids)),
"images": len(images),
"missing": missing,
"single_view": single,
"orphans": orphans,
"duplicate_rows": duplicate_rows,
"collisions": collisions,
"blockers": blockers,
"head": head_note,
}
def enrollment_report_text(report: dict) -> str:
return (
f"{report['stock']}: {len(report['sku_ids'])} SKUs, {report['images']} images, "
f"missing={len(report['missing'])}, single_view={len(report['single_view'])}, "
f"orphans={len(report['orphans'])}, blockers={report['blockers'] or 'none'}"
)
def build_model(image_size: int = 518):
import torch
from gallery import load_dinov2
device = "mps" if torch.backends.mps.is_available() else "cpu"
# Same hash-pinned local file as inference. pretrained=True would fetch a
# different copy even with YOLO_OFFLINE=1.
model = load_dinov2(image_size)
return model.eval().to(device), device
def prepare(image, size: int = 518):
mean = np.array([0.485, 0.456, 0.406], np.float32)
std = np.array([0.229, 0.224, 0.225], np.float32)
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
height, width = rgb.shape[:2]
side = max(height, width)
canvas = np.zeros((side, side, 3), np.uint8)
canvas[(side - height) // 2:(side - height) // 2 + height,
(side - width) // 2:(side - width) // 2 + width] = rgb
resized = cv2.resize(canvas, (size, size), interpolation=cv2.INTER_AREA).astype(np.float32) / 255.0
return ((resized - mean) / std).transpose(2, 0, 1)
def embed(model, device, images, size: int = 518, batch: int = 16):
import torch
out = []
for start in range(0, len(images), batch):
tensor = torch.from_numpy(np.stack([prepare(image, size) for image in images[start:start + batch]])).to(device)
with torch.no_grad():
features = model(tensor)
out.append(torch.nn.functional.normalize(features, dim=1).cpu().numpy())
return np.concatenate(out, axis=0)
def augment(image, rng):
"""Stand in for a second visit: crop, scale, brightness, contrast, blur."""
height, width = image.shape[:2]
# crop 82-100 % of the box, keeping the product centred
keep = rng.uniform(0.82, 1.0)
ch, cw = int(height * keep), int(width * keep)
y = rng.randint(0, max(1, height - ch + 1))
x = rng.randint(0, max(1, width - cw + 1))
out = image[y:y + ch, x:x + cw]
# photometric
out = cv2.convertScaleAbs(out, alpha=float(rng.uniform(0.8, 1.2)), beta=float(rng.uniform(-18, 18)))
if rng.random() < 0.4:
out = cv2.GaussianBlur(out, (3, 3), 0)
# small rotation
if rng.random() < 0.6:
angle = float(rng.uniform(-5, 5))
matrix = cv2.getRotationMatrix2D((out.shape[1] / 2, out.shape[0] / 2), angle, 1.0)
out = cv2.warpAffine(out, matrix, (out.shape[1], out.shape[0]), borderMode=cv2.BORDER_REPLICATE)
return out
def train_head(features, labels, class_ids, epochs, device, margin: float = 0.0, seed: int = 7):
"""A cosine classifier on frozen features: small, and hard to overfit."""
import torch
torch.manual_seed(seed)
dimension = features.shape[1]
projection = torch.nn.Linear(dimension, 128).to(device)
prototypes = torch.nn.Parameter(torch.randn(len(class_ids), 128, device=device))
torch.nn.init.xavier_uniform_(prototypes)
optimiser = torch.optim.AdamW(list(projection.parameters()) + [prototypes], lr=3e-3, weight_decay=1e-2)
x = torch.from_numpy(features).float().to(device)
y = torch.tensor([class_ids.index(label) for label in labels], device=device)
for epoch in range(epochs):
projected = torch.nn.functional.normalize(projection(x), dim=1)
unit_prototypes = torch.nn.functional.normalize(prototypes, dim=1)
logits = projected @ unit_prototypes.T
if margin > 0:
# angular margin: pull each sample away from the other prototypes
one_hot = torch.nn.functional.one_hot(y, len(class_ids)).float()
logits = logits - margin * one_hot
loss = torch.nn.functional.cross_entropy(logits * 16.0, y, label_smoothing=0.05)
optimiser.zero_grad()
loss.backward()
optimiser.step()
return projection, torch.nn.functional.normalize(prototypes.detach(), dim=1)
def train_from_stock(args) -> None:
"""Train the head on a store's own enrolled views, for the gallery to load.
The head belongs to the store: the views in its stock directory are its products,
and a head trained on them is what turns the gallery from a lookup that cannot
tell two milk cartons apart into one that can. One to four views per SKU is
enough, with augmentation standing in for the rest.
"""
import glob
import json
import torch
from gallery import DINOV2_MODEL, ProductGallery, catalog_digest, head_class_set
stock = args.stock
report = enrollment_report(stock)
print(enrollment_report_text(report))
if report["blockers"]:
print(f"{stock}: not training; enrollment blockers: {', '.join(report['blockers'])}")
return
gallery = ProductGallery(stock, report["sku_ids"])
views = {sku: gallery._views_for(sku) for sku in report["sku_ids"]}
views = {sku: paths for sku, paths in views.items() if paths}
if len(views) < 2:
print(f"{stock}: needs at least two products with views, found {len(views)}")
return
crops, labels = [], []
for sku, paths in sorted(views.items()):
for path in paths:
image = cv2.imread(path)
if image is not None and image.size:
crops.append(image)
labels.append(sku)
class_ids = sorted(views)
print(f"{stock}: {len(crops)} views over {len(class_ids)} products")
model, device = build_model()
rng = np.random.RandomState(11)
augmented, augmented_labels = [], []
for crop, label in zip(crops, labels):
augmented.append(crop)
augmented_labels.append(label)
for _ in range(args.augment):
augmented.append(augment(crop, rng))
augmented_labels.append(label)
print(f"embedding {len(augmented)} crops on {device}...")
features = embed(model, device, augmented)
projection, prototypes = train_head(features, augmented_labels, class_ids,
args.epochs, device, margin=0.25)
digest = catalog_digest(stock, class_ids)
if head_class_set(class_ids) is None or digest is None:
print(f"{stock}: refusing to write a head; class ids are empty or duplicated")
return
payload = {
"class_ids": class_ids,
"catalog_digest": digest,
"image_size": 518,
"base_model": DINOV2_MODEL,
"projection": {key: value.cpu() for key, value in projection.state_dict().items()},
"prototypes": prototypes.cpu(),
"views": {sku: len(paths) for sku, paths in sorted(views.items())},
}
path = os.path.join(stock, "identity_head.pt")
torch.save(payload, path)
print(f"wrote {path} ({len(class_ids)} products, {len(crops)} views)")
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--epochs", type=int, default=60)
parser.add_argument("--views", type=int, default=2, help="enrolment views per class")
parser.add_argument("--augment", type=int, default=12, help="augmented copies per enrolment view")
parser.add_argument("--image", default=os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"data", "external-shelves", "wikimedia-commons"))
parser.add_argument("--stock", default=None,
help="train on a store's enrolled views instead of the milk shelf, "
"and write <stock>/identity_head.pt for the gallery to load")
args = parser.parse_args()
if args.stock:
train_from_stock(args)
return
import json
from engine import build_default_engine, EngineSettings
manifest = json.load(open(os.path.join(args.image, "manifest.json")))
entry = next(e for e in manifest["images"] if "2011年4月の牛乳" in e["file"])
image = cv2.imread(os.path.join(args.image, entry["file"]))
height, width = image.shape[:2]
scale = 1600.0 / max(height, width)
image = cv2.resize(image, (int(width * scale), int(height * scale)), interpolation=cv2.INTER_AREA)
engine = build_default_engine(EngineSettings(stock_dir="stock"))
boxes = engine.detect(image, conf=0.1)
boxes.sort(key=lambda b: (b["box"][1] // 120, b["box"][0]))
crops = [engine.crop(image, box["box"]) for box in boxes]
train_crops, train_labels, test_crops, test_labels = [], [], [], []
for label, indices in MILK_SHELF_LABELS.items():
for position, index in enumerate(indices):
if position < args.views:
train_crops.append(crops[index])
train_labels.append(label)
else:
test_crops.append(crops[index])
test_labels.append(label)
class_ids = sorted(MILK_SHELF_LABELS)
print(f"{len(train_crops)} enrolment views, {len(test_crops)} held-out facings, {len(class_ids)} products")
model, device = build_model()
print(f"embedding {len(train_crops) * (args.augment + 1) + len(test_crops)} crops on {device}...")
rng = np.random.RandomState(11)
augmented, augmented_labels = [], []
for crop, label in zip(train_crops, train_labels):
augmented.append(crop)
augmented_labels.append(label)
for _ in range(args.augment):
augmented.append(augment(crop, rng))
augmented_labels.append(label)
train_features = embed(model, device, augmented)
test_features = embed(model, device, test_crops)
def score(similarity, label, threshold=None):
"""Top-1 over every held-out facing, with optional rejection.
Balanced accuracy is reported alongside the raw rate because the shelf is
unbalanced (MEGMILK has 34 facings, 酪農3.6 has two), and a system that only
ever answers "MEGMILK" would look respectable on the raw number.
"""
per_class = {class_id: [0, 0] for class_id in class_ids}
correct = rejected = 0
for index, truth in enumerate(test_labels):
best = int(similarity[index].argmax())
if threshold is not None and similarity[index][best] < threshold:
rejected += 1
per_class[truth][1] += 1
continue
guess = class_ids[best]
per_class[truth][1] += 1
if guess == truth:
correct += 1
per_class[truth][0] += 1
recalls = [found / total for found, total in per_class.values() if total]
balanced = sum(recalls) / max(1, len(recalls))
answered = len(test_labels) - rejected
answered_accuracy = correct / max(1, answered)
print(f" {label:<26} top-1 {correct:>3}/{len(test_labels)} = {100*correct//len(test_labels):>3}% "
f"balanced {100*balanced:>3.0f}% answered {answered:>3} "
f"({100*answered_accuracy:>3.0f}% of those right) rejected {rejected}")
return per_class
import torch
def class_prototypes(features):
prototypes = []
for class_id in class_ids:
members = [i for i, label in enumerate(train_labels) if label == class_id]
mean = features[members].mean(axis=0)
prototypes.append(mean / (np.linalg.norm(mean) + 1e-8))
return np.stack(prototypes)
# 1. raw features, matched the way the shipped gallery matches: best enrolment view
raw_prototypes = class_prototypes(train_features)
score(test_features @ raw_prototypes.T, "raw DINOv2 cosine")
score(test_features @ raw_prototypes.T, "raw, rejecting below 0.55", threshold=0.55)
# 2. the trained classifier's prototypes, with an angular margin
projection, prototypes = train_head(train_features, augmented_labels, class_ids,
args.epochs, device, margin=0.25)
with torch.no_grad():
projected_test = torch.nn.functional.normalize(
projection(torch.from_numpy(test_features).float().to(device)), dim=1).cpu().numpy()
similarity = projected_test @ prototypes.cpu().numpy().T
score(similarity, "fine-tuned + margin")
per_class = score(similarity, "fine-tuned, rejecting < 0.4", threshold=0.4)
print(" per class, answered only:")
for class_id in class_ids:
found, total = per_class[class_id]
if total:
print(f" {class_id:<16} {found:>2}/{total:<3} {100*found//total:>4}%")
print("\n note: enrolment and held-out facings come from one photograph, so these are")
print(" upper bounds. A cross-visit number needs a second visit, not a second split.")
if __name__ == "__main__":
main()

712
recognization/code/gallery.py
View file

@ -1,712 +0,0 @@
"""
A product gallery: identity by retrieval, with several views per SKU.
The matcher this replaces scores a crop against one template per SKU. Measured on
the demo shelf, that is the wrong shape for a store:
* one enrolled view identifies 33 % of facings (matcher) / 57 % (embedding);
* two views per SKU lift the embedding to 72 %, three to 75 %;
* the matcher gets *worse* with more views (33 % -> 5 %), because its raw inlier
score is not calibrated to be aggregated across views - and it cannot even
express two views of one SKU, since it keys templates by filename.
So the gallery is embedding-based. DINOv2 ViT-S/14 over aspect-preserved crops,
cosine similarity, several views per SKU, and a rejection threshold so an unknown
product comes back as unknown instead of as the nearest stranger.
Cost, measured on an Apple M5 (24 crops, batched and one at a time):
224 px CPU 25 ms/crop MPS 2.4 ms/crop
518 px CPU 159 ms/crop MPS 3.6 ms/crop
A live frame only needs its *new* tracks embedded (2-4 of them), so ~10 ms a frame
on the GPU. That is why this can sit in the live loop at all.
"""
from __future__ import annotations
import hashlib
import os
import re
import threading
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Sequence, Tuple
# Below this cosine similarity the nearest view is not the same product. Chosen
# from the measured distribution: same-product pairs sit above 0.6, different
# products below 0.5, and the gap is where a rejection belongs.
DEFAULT_THRESHOLD = 0.55
# The same line for a fine-tuned head, whose similarity scale is its own.
DEFAULT_HEAD_THRESHOLD = 0.40
# DINOv2 ViT-S/14, LVD-142M. timm's default source is the Hugging Face copy of
# these weights (Apache-2.0); pretrained=True fetches it even when YOLO_OFFLINE=1.
# The pin is the sha256 of that model.safetensors (timm hub snapshot 4610ca14).
# Not in Git (~84 MB). Place it at weights/dinov2_vits14_lvd142m.safetensors, or
# point DORAYAKI_DINOV2_WEIGHTS at a copy. See README "DINOv2 weights".
DINOV2_MODEL = "vit_small_patch14_dinov2.lvd142m"
DINOV2_SHA256 = "04d27f3400d059fc0cfd7d17dd1909a75bf3ea8fb3eeb48b97cb99e57ee20081"
DINOV2_FILENAME = "dinov2_vits14_lvd142m.safetensors"
# What load_dinov2 returns: the same 384 the fine-tuned head is Linear(384, 128)
# on, and the width a cached matrix has to have to belong to this backbone.
DINOV2_EMBED_DIM = 384
# Bumped when _prepare() or the cache layout changes. A matrix embedded under an
# older preprocessing pass is stale, not merely old, so it must be re-embedded.
CACHE_VERSION = 1
# One thread at a time runs the gallery backbone. Metal's shader library is
# process-global and is not thread-safe while kernels are still being compiled
# into it, which is what the first forward on a fresh process does; two threads
# reaching it together segfault the whole server. Module-level because the state
# being protected is the process's, not one gallery's.
# ponytail: serializes every embed. Audit C5's batching embedding worker is the
# upgrade path, and a second MPS consumer (--device mps puts YOLO on Metal too)
# would need to take this same lock rather than its own.
_MPS_INFER_LOCK = threading.Lock()
def dinov2_weights_path() -> str:
override = os.environ.get("DORAYAKI_DINOV2_WEIGHTS")
if override:
return override
here = os.path.dirname(os.path.abspath(__file__))
return os.path.join(os.path.dirname(here), "weights", DINOV2_FILENAME)
def _sha256(path: str) -> str:
import hashlib
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_dinov2(image_size: int) -> Any:
"""Build the gallery backbone from the pinned local file. Never downloads.
A missing or altered file raises FileNotFoundError naming the path and the
expected sha256. The head in finetune_identity.py is a Linear(384, 128) on
this model's features; a different backbone will not load that head.
"""
import timm
from safetensors.torch import load_file
from timm.models.vision_transformer import checkpoint_filter_fn
path = dinov2_weights_path()
if not os.path.isfile(path):
raise FileNotFoundError(
f"DINOv2 weights not found at {path}. "
f"Copy model.safetensors (sha256 {DINOV2_SHA256}) from the local "
f"Hugging Face cache of timm/{DINOV2_MODEL}, or set DORAYAKI_DINOV2_WEIGHTS. "
f"The gallery does not download weights."
)
digest = _sha256(path)
if digest != DINOV2_SHA256:
raise FileNotFoundError(
f"DINOv2 weights at {path} are sha256 {digest}, expected {DINOV2_SHA256}."
)
model = timm.create_model(
DINOV2_MODEL,
pretrained=False,
num_classes=0,
img_size=int(image_size),
dynamic_img_size=True,
)
# The file is the 518px checkpoint. timm's own filter resizes pos_embed when
# the caller asks for another size; pretrained=True would do this and also
# download, so it stays off.
state = checkpoint_filter_fn(load_file(path), model)
missing, unexpected = model.load_state_dict(state, strict=False)
unexpected = [key for key in unexpected if not key.startswith("head.")]
if missing or unexpected:
raise RuntimeError(
f"DINOv2 weights at {path} do not match {DINOV2_MODEL} "
f"(missing={missing}, unexpected={unexpected})."
)
return model
@dataclass
class GalleryEntry:
sku_id: str
view: str
embedding: Any = None
@dataclass
class GalleryMatch:
sku_id: Optional[str]
similarity: float
runner_up: Optional[str] = None
runner_up_similarity: float = 0.0
rejected: bool = False
@property
def margin(self) -> float:
return self.similarity - self.runner_up_similarity
def head_class_set(sku_ids: Sequence[str]) -> Optional[List[str]]:
"""Catalog order, or None when an id is empty or repeated."""
cleaned = [str(sku_id).strip() for sku_id in sku_ids]
if not cleaned or any(not sku_id for sku_id in cleaned):
return None
if len(set(cleaned)) != len(cleaned):
return None
return cleaned
def catalog_digest(stock_dir: str, sku_ids: Sequence[str], csv_name: str = "stock.csv") -> Optional[str]:
"""Pin for identity_head.pt: stock.csv bytes plus the catalog id list.
Image sizes are not included, so another view of the same SKU does not
invalidate a head. A changed row or id does. None if the id list is unsafe.
"""
cleaned = head_class_set(sku_ids)
if cleaned is None:
return None
digest = hashlib.sha256()
csv_path = os.path.join(stock_dir, csv_name)
try:
with open(csv_path, "rb") as handle:
digest.update(handle.read())
except OSError:
digest.update(b"missing-csv")
digest.update(b"\n")
digest.update("\n".join(sorted(cleaned)).encode("utf-8"))
return digest.hexdigest()
class ProductGallery:
"""Several views per SKU, embedded once, queried by cosine similarity."""
def __init__(
self,
stock_dir: str,
sku_ids: Sequence[str],
device: Optional[str] = None,
image_size: int = 518,
threshold: float = DEFAULT_THRESHOLD,
head_threshold: float = DEFAULT_HEAD_THRESHOLD,
) -> None:
self.stock_dir = stock_dir
self.sku_ids = list(sku_ids)
self.image_size = int(image_size)
self.threshold = float(threshold)
# A trained head produces a different similarity scale than raw features, so
# it needs its own line. Measured on a real milk shelf: the head answers 18 of
# 72 facings above 0.4 and is right on 94 % of them, while the raw threshold
# (0.55) rejects almost everything the head says.
self.head_threshold = float(head_threshold)
self.device = device
self._model = None
self._lock = threading.Lock()
self._entries: List[GalleryEntry] = []
self._matrix = None
self._available = False
# The source fingerprint this matrix was built from, and the per-SKU
# segment bounds derived from the entries it describes.
self._loaded_key = ""
self._rank = None
# A fine-tuned projection head, if the store has one (see
# finetune_identity.py). Measured on a real milk shelf: raw features put 1 %
# balanced accuracy on fine-grained products, a trained head 73 %.
self._head = None
self.head_reason = "not_loaded"
self._head_path = os.path.join(stock_dir, "identity_head.pt")
# ------------------------------------------------------------------ loading
def load_head(self) -> bool:
"""Load the store's fine-tuned head when its labels are this catalog's.
weights_only: the file is tensors and strings, not executable code.
A missing, malformed, or foreign head leaves the raw gallery in place
and records head_reason. Class ids must be unique, nonempty, and a
subset of this gallery: a head may cover only the SKUs it was trained
on. A catalog_digest, when present, must match stock.csv plus that
subset. A head written before the pin loads with head_reason
legacy_unpinned; it still cannot name a SKU outside this catalog.
"""
import torch
self._head = None
if not os.path.exists(self._head_path):
self.head_reason = "missing"
return False
try:
payload = torch.load(self._head_path, map_location="cpu", weights_only=True)
reason = self._accept_head(payload, torch)
except Exception:
self._head = None
self.head_reason = "malformed"
return False
if not reason.startswith("ok"):
self._head = None
self.head_reason = reason
return False
self.head_reason = "legacy_unpinned" if reason == "ok:legacy_unpinned" else "ok"
return True
def _accept_head(self, payload: Any, torch: Any) -> str:
if not isinstance(payload, dict):
return "malformed"
projection_state = payload.get("projection")
prototypes = payload.get("prototypes")
class_ids = payload.get("class_ids")
if not isinstance(projection_state, dict) or not torch.is_tensor(prototypes):
return "malformed"
if not isinstance(class_ids, (list, tuple)) or prototypes.ndim != 2:
return "malformed"
if prototypes.shape[0] != len(class_ids) or prototypes.shape[1] != 128:
return "malformed"
labels = []
for sku_id in class_ids:
if not isinstance(sku_id, str) or not sku_id.strip():
return "malformed: identity head class_ids must be nonempty strings"
labels.append(sku_id.strip())
if len(set(labels)) != len(labels):
return "malformed: identity head class_ids are not unique"
expected = head_class_set(self.sku_ids)
if expected is None or not set(labels) <= set(expected):
return "catalog_mismatch: head class_ids are not a subset of this catalog; retrain finetune_identity.py --stock"
recorded = payload.get("catalog_digest", None)
if recorded is None:
reason = "ok:legacy_unpinned"
else:
pinned = catalog_digest(self.stock_dir, labels)
if pinned is None or recorded != pinned:
return "catalog_mismatch: head catalog_digest does not match stock.csv and its class_ids; retrain finetune_identity.py --stock"
reason = "ok"
if payload.get("base_model") != DINOV2_MODEL:
return f"backbone_mismatch: head base_model is not {DINOV2_MODEL}"
if type(payload.get("image_size")) is not int or payload.get("image_size") != self.image_size:
return f"image_size_mismatch: head image_size is not {self.image_size}"
projection = torch.nn.Linear(384, 128)
projection.load_state_dict(projection_state)
projection.eval()
self._head = {
"projection": projection,
"prototypes": torch.nn.functional.normalize(prototypes.float(), dim=1),
"class_ids": labels,
}
return reason
def _load_model(self) -> Any:
if self._model is not None:
return self._model
import torch
device = self.device
if device is None:
device = "mps" if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available() else "cpu"
model = load_dinov2(self.image_size)
model = model.eval().to(device)
self.device = device
self._model = model
return model
def available(self) -> bool:
return self._available
def _views_for(self, sku_id: str) -> List[str]:
"""Images whose stem is the SKU or SKU plus _N / -N.
A bare prefix would enroll sku_001 as a view of sku_00.
"""
found: List[str] = []
extra = re.compile(rf"{re.escape(sku_id)}[_-]\d+$")
try:
names = sorted(os.listdir(self.stock_dir))
except OSError:
return found
for name in names:
stem, extension = os.path.splitext(name)
if extension.lower() not in (".jpeg", ".jpg", ".png"):
continue
if stem == sku_id or extra.fullmatch(stem):
found.append(os.path.join(self.stock_dir, name))
return found
def _enrolled(self) -> List[Tuple[str, str]]:
"""Every (sku, view) that would be embedded, in embed order."""
return [
(sku_id, path)
for sku_id in self.sku_ids
for path in self._views_for(sku_id)
]
def _source_key(self, enrolled: Sequence[Tuple[str, str]]) -> str:
"""A fingerprint of exactly the inputs an embedding depends on.
The view bytes, not the name and length: a re-encoded JPEG of the same
size moves the embedding, and a wrong row is worse than a slow start.
Empty when a view cannot be read, which no cache may answer for.
"""
digest = hashlib.sha256()
digest.update(f"v{CACHE_VERSION}|{self.image_size}|{DINOV2_SHA256}\n".encode("utf-8"))
try:
with open(os.path.join(self.stock_dir, "stock.csv"), "rb") as handle:
digest.update(handle.read())
except OSError:
digest.update(b"missing-csv")
for sku_id, path in enrolled:
digest.update(f"\n{sku_id}\x00{os.path.basename(path)}\x00".encode("utf-8"))
try:
with open(path, "rb") as handle:
digest.update(hashlib.sha256(handle.read()).digest())
except OSError:
return ""
return digest.hexdigest()
def _cache_path(self) -> str:
return os.path.join(self.stock_dir, f".gallery_cache_{self.image_size}.npz")
def _weights_pinned(self) -> bool:
"""The backbone on disk is still the one this matrix was embedded with.
A cache hit never opens load_dinov2, so without this a deleted or altered
checkpoint would read as a loaded gallery and only fail on the first
match - after the server had already reported itself ready.
"""
path = dinov2_weights_path()
return os.path.isfile(path) and _sha256(path) == DINOV2_SHA256
def _cache_read(self, key: str, enrolled: Sequence[Tuple[str, str]]) -> Optional[Tuple[List[Tuple[str, str]], Any]]:
"""The cached rows when they are provably this store's, else None."""
import numpy as np
try:
with np.load(self._cache_path(), allow_pickle=False) as payload:
if str(payload["key"].item()) != key:
return None
skus = [str(value) for value in payload["skus"].tolist()]
views = [str(value) for value in payload["views"].tolist()]
matrix = payload["matrix"]
if matrix.ndim != 2 or matrix.shape[1] != DINOV2_EMBED_DIM:
return None
if not matrix.shape[0] or matrix.shape[0] != len(skus) or len(skus) != len(views):
return None
if not np.issubdtype(matrix.dtype, np.floating) or not np.isfinite(matrix).all():
return None
# Rows walk the enrolled list forward and never backwards, so a
# moved, repeated or foreign row has nowhere left to match. Views
# that could not be decoded are simply absent: everyone after one
# keeps its order, which is what stops a valid key from renaming a
# SKU.
expected = [(sku_id, os.path.basename(view)) for sku_id, view in enrolled]
rows: List[Tuple[str, str]] = []
position = 0
for sku_id, view in zip(skus, views):
while position < len(expected) and expected[position] != (sku_id, view):
position += 1
if position == len(expected):
return None
rows.append((sku_id, os.path.join(self.stock_dir, view)))
position += 1
if not self._weights_pinned():
return None
return rows, matrix.astype(np.float32, copy=False)
except (OSError, ValueError, TypeError, KeyError, IndexError):
return None # absent, malformed, or raced with a checkpoint removal
def _cache_write(self, key: str, rows: Sequence[Tuple[str, str]], matrix: Any) -> None:
"""Replace the cache file in one step. A failed write is not a failed load."""
import numpy as np
path = self._cache_path()
tmp = f"{path}.{os.getpid()}.tmp"
try:
with open(tmp, "wb") as handle:
np.savez(
handle,
key=np.array(key),
skus=np.array([sku_id for sku_id, _ in rows]),
views=np.array([os.path.basename(view) for _, view in rows]),
matrix=matrix,
)
os.replace(tmp, path)
except OSError: # read-only store, no space: embed again next time
try:
os.unlink(tmp)
except OSError:
pass
def _embed_views(self, enrolled: Sequence[Tuple[str, str]]) -> Optional[Tuple[List[Tuple[str, str]], Any]]:
"""Embed every readable view once. None when the backbone is unusable."""
import cv2
import numpy as np
import torch
rows: List[Tuple[str, str]] = []
kept: List[Any] = []
for sku_id, path in enrolled:
image = cv2.imread(path)
if image is None or image.size == 0:
continue
rows.append((sku_id, path))
kept.append(_prepare(image, self.image_size))
if not kept:
return None
try:
with _MPS_INFER_LOCK:
model = self._load_model()
tensor = torch.from_numpy(np.stack(kept)).to(self.device)
with torch.no_grad():
features = model(tensor)
features = torch.nn.functional.normalize(features, dim=1).cpu().numpy()
except Exception:
return None
return rows, features.astype(np.float32, copy=False)
def load(self) -> bool:
"""Embed every enrolled view. Returns False when there is nothing to load.
One lock covers the whole load: two callers cannot each embed the catalog.
A loaded gallery is kept only while the store still fingerprints the
same, so a replaced or reordered enrollment re-embeds instead of
answering from a matrix that no longer describes it.
"""
with self._lock:
if self._available and self._matrix is not None:
enrolled = self._enrolled()
if enrolled and self._source_key(enrolled) == self._loaded_key:
return True
return self._load_locked()
def _unload(self) -> None:
self._available = False
self._entries = []
self._matrix = None
self._rank = None
self._loaded_key = ""
def _load_locked(self) -> bool:
enrolled = self._enrolled()
if not enrolled:
self._unload()
return False
key = self._source_key(enrolled)
cached = self._cache_read(key, enrolled) if key else None
if cached is None:
embedded = self._embed_views(enrolled)
if embedded is None:
self._unload()
return False
rows, features = embedded
if key: # an unreadable view has no key; never write one we cannot read
self._cache_write(key, rows, features)
else:
rows, features = cached
# A cache hit skips the embedding forward, so without this the model
# is built and Metal's kernels are compiled by whichever classify
# worker asks first - up to eight racing into one process-wide cache.
# The readiness path loads the gallery before the socket accepts, so
# the warmup is free and no worker meets a cold model.
#
# Best-effort on purpose: a cache hit is served without a usable
# backbone by contract, and _MPS_INFER_LOCK already makes the lazy
# path safe. This only spares the first worker the compile.
self._warm_backbone()
self._entries = [
GalleryEntry(sku_id=sku_id, view=path, embedding=features[index])
for index, (sku_id, path) in enumerate(rows)
]
self._matrix = features
self._available = True
self._rank = None
self._loaded_key = key
self.load_head()
return True
# ------------------------------------------------------------------ querying
def embed(self, crop: Any) -> Any:
import numpy as np
import torch
with _MPS_INFER_LOCK:
model = self._load_model()
tensor = torch.from_numpy(np.stack([_prepare(crop, self.image_size)])).to(self.device)
with torch.no_grad():
features = model(tensor)
features = torch.nn.functional.normalize(features, dim=1).cpu().numpy()
return features[0]
def _warm_backbone(self) -> None:
"""Build the backbone and compile its kernels, single-threaded.
Called under self._lock from the load path, so the lock order is
self._lock then _MPS_INFER_LOCK - the same order _embed_views takes, and
no path takes them reversed. The forward is a dummy canvas: what is
being warmed is the model and the shader library, not any one crop.
Metal only. A CPU gallery has no shader library to compile and a cache
hit is served without touching the backbone by contract, so warming one
would spend a model load to protect against nothing.
"""
import numpy as np
if self._resolved_device() != "mps":
return
try:
self.embed(np.zeros((self.image_size, self.image_size, 3), dtype=np.uint8))
except Exception:
pass
def _resolved_device(self) -> str:
"""The device the backbone will run on, without building it."""
if self.device is not None:
return self.device
import torch
if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
return "mps"
return "cpu"
def _rank_index(self) -> Tuple[List[str], Any, Any]:
"""Segment bounds for the per-SKU maximum, in first-appearance order.
Entries may arrive in any order, so the grouping is derived from them, not
assumed. Memoised against the list it was built from: replacing the list
rebuilds it, and a SKU list is never longer than the views behind it.
"""
import numpy as np
if self._rank is not None and self._rank[0] is self._entries:
return self._rank[1], self._rank[2], self._rank[3]
order: List[str] = []
first_seen: Dict[str, int] = {}
rows: List[int] = []
for entry in self._entries:
index = first_seen.get(entry.sku_id)
if index is None:
index = len(order)
first_seen[entry.sku_id] = index
order.append(entry.sku_id)
rows.append(index)
column = np.asarray(rows, dtype=np.intp)
# A stable sort groups the rows by SKU; reduceat then takes each segment's
# max, which is a SKU's best view whatever order its views were enrolled in.
grouped = np.argsort(column, kind="stable")
starts = np.searchsorted(column[grouped], np.arange(len(order)))
self._rank = (self._entries, order, grouped, starts)
return order, grouped, starts
def match(self, crop: Any, threshold: Optional[float] = None) -> GalleryMatch:
"""Nearest enrolled product, with a rejection when nothing is close enough."""
if not self._available or self._matrix is None:
return GalleryMatch(sku_id=None, similarity=0.0, rejected=True)
import numpy as np
limit = threshold
if limit is None:
limit = self.head_threshold if self._head is not None else self.threshold
limit = float(limit)
embedding = self.embed(crop)
if self._head is not None:
import torch
# Two stages, because the two mechanisms fail in opposite directions.
# Measured: raw features reject strangers well (5 % false accepts) but
# rank fine-grained products badly (2 % top-1); a trained head ranks them
# well (73 % balanced) but is confidently wrong on strangers (44 % false
# accepts), because a discriminative head has no "none of the above".
# So the raw features decide *whether* to answer, and the head decides
# *what* to answer.
raw_similarity = self._matrix @ embedding
best_view = float(raw_similarity.max()) if len(raw_similarity) else 0.0
if best_view < self.threshold:
return GalleryMatch(sku_id=None, similarity=best_view, rejected=True)
with torch.no_grad():
projected = torch.nn.functional.normalize(
self._head["projection"](torch.from_numpy(embedding).float()), dim=0
)
similarity = (self._head["prototypes"] @ projected).numpy()
ranked = sorted(zip(self._head["class_ids"], similarity), key=lambda kv: kv[1], reverse=True)
top_sku, top_score = ranked[0]
runner_sku, runner_score = ranked[1] if len(ranked) > 1 else (None, 0.0)
if float(top_score) < limit:
return GalleryMatch(sku_id=None, similarity=float(top_score), runner_up=top_sku,
runner_up_similarity=float(top_score), rejected=True)
return GalleryMatch(sku_id=top_sku, similarity=float(top_score), runner_up=runner_sku,
runner_up_similarity=float(runner_score))
similarity = self._matrix @ embedding
# Several views per SKU: a SKU's score is its best view.
order, grouped, starts = self._rank_index()
if not order:
return GalleryMatch(sku_id=None, similarity=0.0, rejected=True)
best_by_sku = np.maximum.reduceat(similarity[grouped], starts)
ranked = sorted(zip(order, best_by_sku.tolist()), key=lambda item: item[1], reverse=True)
top_sku, top_score = ranked[0]
runner_sku, runner_score = ranked[1] if len(ranked) > 1 else (None, 0.0)
if top_score < limit:
return GalleryMatch(sku_id=None, similarity=top_score, runner_up=top_sku,
runner_up_similarity=top_score, rejected=True)
return GalleryMatch(sku_id=top_sku, similarity=top_score, runner_up=runner_sku,
runner_up_similarity=runner_score)
def rank(self, crop: Any, limit: Optional[int] = None) -> List[Tuple[str, float]]:
"""Every enrolled SKU by similarity, best first, with no rejection.
`match()` answers "is anything close enough"; a shortlist needs "what are
the closest candidates", including when none of them is close. Gating a
shortlist at the raw threshold empties it for exactly the crops the
verifier exists to rescue.
"""
if not self._available or self._matrix is None:
return []
import numpy as np
embedding = self.embed(crop)
if self._head is not None:
import torch
with torch.no_grad():
projected = torch.nn.functional.normalize(
self._head["projection"](torch.from_numpy(embedding).float()), dim=0
)
similarity = (self._head["prototypes"] @ projected).numpy()
ranked = sorted(zip(self._head["class_ids"], similarity.tolist()),
key=lambda kv: kv[1], reverse=True)
else:
similarity = self._matrix @ embedding
order, grouped, starts = self._rank_index()
if not order:
return []
best_by_sku = np.maximum.reduceat(similarity[grouped], starts)
ranked = sorted(zip(order, best_by_sku.tolist()), key=lambda item: item[1], reverse=True)
return [(str(sku), float(score)) for sku, score in ranked[:limit]] if limit else \
[(str(sku), float(score)) for sku, score in ranked]
def head_loaded(self) -> bool:
return self._head is not None
def view_count(self) -> int:
return len(self._entries)
def sku_view_counts(self) -> Dict[str, int]:
counts: Dict[str, int] = {}
for entry in self._entries:
counts[entry.sku_id] = counts.get(entry.sku_id, 0) + 1
return counts
def _prepare(image: Any, size: int) -> Any:
"""Aspect-preserving square canvas: the crop's shape is part of the product."""
import cv2
import numpy as np
mean = np.array([0.485, 0.456, 0.406], np.float32)
std = np.array([0.229, 0.224, 0.225], np.float32)
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
height, width = rgb.shape[:2]
side = max(height, width)
canvas = np.zeros((side, side, 3), np.uint8)
canvas[(side - height) // 2:(side - height) // 2 + height,
(side - width) // 2:(side - width) // 2 + width] = rgb
resized = cv2.resize(canvas, (size, size), interpolation=cv2.INTER_AREA).astype(np.float32) / 255.0
return ((resized - mean) / std).transpose(2, 0, 1)

92
recognization/code/gui_app.py Normal file
View file

@ -0,0 +1,92 @@
"""
Native Desktop GUI for Supermarket Shelf Product Recognition
Opens a desktop OpenCV window displaying image.jpg with bounding boxes only.
Users can click on any product directly on screen to inspect price, stock,
and feature matching inliers in real-time.
"""
import os
import sys
import cv2
import numpy as np
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from pipeline import ShelfRecognitionPipeline
def main():
print("=" * 60)
print("STARTING NATIVE DESKTOP INTERFACE")
print("Image: image.jpg | Mode: Hybrid (Paper Method)")
print("Instructions: Click on ANY product to identify & view info.")
print("Press 'q' or 'ESC' on the window to exit.")
print("=" * 60)
pipeline = ShelfRecognitionPipeline()
init_res = pipeline.set_active_shelf_image("image.jpg")
if init_res.get("status") != "success":
print("[Error] Failed to load image.jpg")
return
boxes = init_res["boxes"]
base_image = pipeline.current_image.copy()
# Draw ONLY bounding boxes (no class names, no confidence labels)
display_image = pipeline.detector.draw_detections(base_image, boxes, draw_labels=False)
window_name = "Smart Shelf - Click Any Product (Press Q to quit)"
cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)
cv2.resizeWindow(window_name, 1280, 850)
def on_mouse(event, x, y, flags, param):
if event == cv2.EVENT_LBUTTONDOWN:
print(f"\n[Click detected at coordinates: ({x}, {y})]")
res = pipeline.process_click(x, y, method="hybrid")
if res.get("status") == "success":
prod = res["inventory"]
cls_info = res["classification"]
box_info = res["target_box"]
print("-" * 55)
print(f"📦 Product Identified : {prod['product_name']}")
print(f"🏷️ SKU ID : {prod['sku_id']}")
print(f"💰 Price : ¥{prod['price']}")
print(f"📊 In Stock (On-Shelf): {prod['number_on']} units")
print(f"🚚 Received / Sold : {prod['number_in']} in / {prod['number_out']} out")
print(f"📅 Expiration Date : {prod['expiration_date']}")
print(f"🎯 Match Confidence : {cls_info['confidence'] * 100:.1f}% (Score: {cls_info['score']})")
print("-" * 55)
# Update main window with selected box highlight (in red)
selected_display = pipeline.detector.draw_detections(
base_image, boxes, selected_box_id=box_info["id"], draw_labels=False
)
cv2.imshow(window_name, selected_display)
# Show visual match comparison window if available
if res.get("match_visualization") is not None:
match_win = "Feature Matcher (Shelf Crop vs Stock Template)"
cv2.namedWindow(match_win, cv2.WINDOW_NORMAL)
cv2.resizeWindow(match_win, 800, 400)
cv2.imshow(match_win, res["match_visualization"])
else:
print(f"[Info] {res.get('message', 'No product found at coordinates.')}")
cv2.setMouseCallback(window_name, on_mouse)
while True:
cv2.imshow(window_name, display_image)
key = cv2.waitKey(20) & 0xFF
if key in [ord('q'), ord('Q'), 27]:
break
cv2.destroyAllWindows()
if __name__ == "__main__":
main()

318
recognization/code/identity.py
View file

@ -1,318 +0,0 @@
"""Identity ranking for catalog scores.
Separates raw matcher scores from accept/unknown/ambiguous/reject so a
zero-score crop cannot become a SKU. Three score scales are recognized:
- `inlier`: Van's composite, kept as the selectable baseline lane (E-1).
- `local`: `identity_lanes.LocalLane`'s margin-over-runner-up, in inlier units.
- `cosine`: the embedding gallery, fenced off and not in the live path.
The `local` line is measured on the corrected image6 labels: 2.0 keeps 95% of
shown labels correct at 59% coverage with the text lane, 1.0 keeps 87% at 66%.
It is tuned on one photo, so it is a direction, not a calibration; E-4 replaces
it once Tier-2 held-out labels exist.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional, Sequence, Tuple
IDENTITY_ACCEPTED = "accepted"
IDENTITY_AMBIGUOUS = "ambiguous"
IDENTITY_UNKNOWN = "unknown"
IDENTITY_REJECTED = "rejected"
# ponytail: closed-set defaults for the baseline inlier scale. Recalibrate when
# held-out scores exist.
DEFAULT_MIN_SCORE = 1.0
DEFAULT_MIN_MARGIN = 0.15
DEFAULT_MIN_CONFIDENCE = 0.55
# `local` scale (identity_lanes.LocalLane). The value is already a margin over
# the runner-up, so the relative-margin test does not apply to it: a single
# absolute line is the whole gate. Kept equal to identity_lanes.LOCAL_ACCEPT,
# which is what the engine's DecisionPolicy uses; this copy exists for the
# evaluator's `--no-text` path, and two different lines would score the same
# crops two ways.
LOCAL_ACCEPT = 4.0
def relative_confidence(top_score: float, runner_up_score: float) -> float:
"""Existing matcher formula: top / (top + 0.5 * runner)."""
return round(min(1.0, top_score / max(1.0, (top_score + runner_up_score * 0.5))), 4)
# Cosine scale (gallery). Same-product pairs were measured above 0.6 and
# different products below 0.5. These are not a new calibration.
COSINE_ACCEPT = 0.60
COSINE_AMBIGUOUS = 0.50
COSINE_MARGIN = 0.04
VOTE_CAP = 3
def _finite(value: Any) -> Optional[float]:
try:
number = float(value)
except (TypeError, ValueError):
return None
if number != number or number in (float("inf"), float("-inf")):
return None
return number
def aggregate_votes(
observations: Sequence[Dict[str, float]],
*,
cap: int = VOTE_CAP,
) -> Dict[str, float]:
"""Mean of the newest observations, at most `cap`.
Older frames are dropped, not averaged in. An empty observation still
counts, so a later miss can outvote an earlier weak hit. Inlier and
cosine scores must not be mixed in one list.
"""
window = list(observations)[-max(1, int(cap)):]
if not window:
return {}
totals: Dict[str, float] = {}
for scores in window:
for sku_id, score in (scores or {}).items():
number = _finite(score)
if number is None:
continue
totals[str(sku_id)] = totals.get(str(sku_id), 0.0) + number
count = len(window)
return {sku_id: total / count for sku_id, total in totals.items()}
def _cosine_decision(
sorted_candidates: List[Tuple[str, float]],
*,
method: str,
details: Optional[Dict[str, Any]],
accept: float,
ambiguous: float,
margin_min: float,
) -> Dict[str, Any]:
top_sku, top_score = sorted_candidates[0]
runner_score = sorted_candidates[1][1] if len(sorted_candidates) > 1 else 0.0
margin = top_score - runner_score
if top_score >= accept and margin >= margin_min:
level = IDENTITY_ACCEPTED
elif top_score >= ambiguous:
level = IDENTITY_AMBIGUOUS
else:
level = IDENTITY_REJECTED
return {
"status": level,
"method": method,
"best_sku": top_sku if level == IDENTITY_ACCEPTED else None,
"provisional_sku": top_sku,
"score": float(top_score),
"confidence": float(top_score),
"identity_level": level,
"ranked_candidates": [
{"sku_id": sku, "score": float(value)} for sku, value in sorted_candidates[:3]
],
"details": details or {},
"message": None,
}
def decision_for_match(
match: Any,
*,
method: str = "gallery",
margin: float = COSINE_MARGIN,
) -> Dict[str, Any]:
"""Rank one gallery match. Rejection is the gallery's, not a second threshold.
`match()` already applied the raw 0.55 line or the head's 0.40 line. Re-ranking
those scores against COSINE_ACCEPT (0.60) would refuse a head match the gallery
accepted. The only extra gate here is the published cosine margin.
"""
similarity = _finite(getattr(match, "similarity", 0.0)) or 0.0
if match is None or getattr(match, "rejected", False) or not getattr(match, "sku_id", None):
return {
"status": IDENTITY_REJECTED,
"method": method,
"best_sku": None,
"score": similarity,
"confidence": similarity,
"identity_level": IDENTITY_REJECTED,
"ranked_candidates": [],
"details": {},
"message": "no enrolled product is close enough",
}
runner = getattr(match, "runner_up", None)
runner_score = _finite(getattr(match, "runner_up_similarity", 0.0)) or 0.0
gap = similarity - runner_score
close = bool(runner) and gap < margin
level = IDENTITY_AMBIGUOUS if close else IDENTITY_ACCEPTED
ranked = [{"sku_id": str(match.sku_id), "score": similarity}]
if runner:
ranked.append({"sku_id": str(runner), "score": runner_score})
return {
"status": level,
"method": method,
"best_sku": None if close else str(match.sku_id),
"score": similarity,
"confidence": similarity,
"identity_level": level,
"ranked_candidates": ranked,
"details": {},
"message": None,
}
def decide_identity(
scores: Dict[str, float],
*,
min_score: float = DEFAULT_MIN_SCORE,
min_margin: float = DEFAULT_MIN_MARGIN,
min_confidence: float = DEFAULT_MIN_CONFIDENCE,
details: Optional[Dict[str, Any]] = None,
method: str = "hybrid",
scale: str = "inlier",
) -> Dict[str, Any]:
"""Rank catalog scores into accepted / ambiguous / unknown / rejected.
Zero or missing evidence is rejected. A weak but unique top score is
unknown. Two close scores are ambiguous. Only a unique, above-threshold
score is accepted.
"""
if scale == "cosine":
if min_score == DEFAULT_MIN_SCORE:
min_score = COSINE_ACCEPT
if min_margin == DEFAULT_MIN_MARGIN:
min_margin = COSINE_MARGIN
if min_confidence == DEFAULT_MIN_CONFIDENCE:
min_confidence = COSINE_AMBIGUOUS
elif scale == "local" and min_score == DEFAULT_MIN_SCORE:
min_score = LOCAL_ACCEPT
cleaned: Dict[str, float] = {}
for sku_id, score in (scores or {}).items():
number = _finite(score)
if number is None:
continue
cleaned[str(sku_id)] = number
if not cleaned:
return {
"status": IDENTITY_REJECTED,
"method": method,
"best_sku": None,
"provisional_sku": None,
"score": 0.0,
"confidence": 0.0,
"identity_level": IDENTITY_REJECTED,
"ranked_candidates": [],
"details": {},
"message": "No templates available",
}
sorted_candidates: List[Tuple[str, float]] = sorted(
cleaned.items(), key=lambda item: item[1], reverse=True
)
if scale == "cosine":
return _cosine_decision(
sorted_candidates,
method=method,
details=details,
accept=min_score,
ambiguous=min_confidence,
margin_min=min_margin,
)
top_sku, top_score = sorted_candidates[0]
runner_up_sku, runner_up_score = (
sorted_candidates[1] if len(sorted_candidates) > 1 else (None, 0.0)
)
confidence = relative_confidence(top_score, runner_up_score)
ranked = [{"sku_id": s, "score": round(sc, 2)} for s, sc in sorted_candidates[:5]]
top_details = (details or {}).get(top_sku, {})
if scale == "local":
# The score is already a margin over the runner-up, so the relative
# margin test below would apply the same correction twice. One absolute
# line is the whole gate; a candidate under it is unknown, not ambiguous,
# because the lane has already said which one it prefers.
if top_score <= 0.0:
level, message, best = IDENTITY_REJECTED, "No matching evidence", None
elif top_score < min_score:
level, message, best = IDENTITY_UNKNOWN, "Top score below accept threshold", None
else:
level, message, best = IDENTITY_ACCEPTED, "Accepted unique catalog match", top_sku
return {
"status": "success" if level == IDENTITY_ACCEPTED else level,
"method": method,
"best_sku": best,
"provisional_sku": top_sku if top_score > 0.0 else None,
"score": round(top_score, 2),
"confidence": round(top_score, 4),
"identity_level": level,
"ranked_candidates": ranked,
"details": top_details,
"message": message,
}
if scale == "van":
# Van's contract: the top SKU, always answered. No gate, no abstention -
# this is the baseline the decision is measured against, and it exists
# so the stream comparison is reproducible rather than asserted.
if top_score <= 0.0:
level, message, best = IDENTITY_REJECTED, "No matching evidence", None
else:
level, message, best = IDENTITY_ACCEPTED, "Always-argmax baseline", top_sku
return {
"status": "success" if level == IDENTITY_ACCEPTED else level,
"method": method,
"best_sku": best,
"provisional_sku": top_sku if top_score > 0.0 else None,
"score": round(top_score, 2),
"confidence": confidence,
"identity_level": level,
"ranked_candidates": ranked,
"details": top_details,
"message": message,
}
if top_score <= 0.0:
level = IDENTITY_REJECTED
message = "No matching evidence"
best = None
elif (
runner_up_sku is not None
and top_score > 0.0
and (top_score - runner_up_score) < min_margin * max(top_score, 1.0)
):
level = IDENTITY_AMBIGUOUS
message = f"Close scores: {top_sku} vs {runner_up_sku}"
best = None
elif top_score < min_score or confidence < min_confidence:
level = IDENTITY_UNKNOWN
message = "Top score below accept threshold"
best = None
else:
level = IDENTITY_ACCEPTED
message = "Accepted unique catalog match"
best = top_sku
return {
"status": "success" if level == IDENTITY_ACCEPTED else level,
"method": method,
"best_sku": best,
"provisional_sku": top_sku if top_score > 0.0 else None,
"score": round(top_score, 2),
"confidence": confidence,
"identity_level": level,
"ranked_candidates": ranked,
"details": top_details,
"message": message,
}
def identity_from_ranked(
ranked_candidates: Sequence[Dict[str, Any]],
**kwargs: Any,
) -> Dict[str, Any]:
scores = {str(row["sku_id"]): float(row["score"]) for row in ranked_candidates}
return decide_identity(scores, **kwargs)

712
recognization/code/identity_lanes.py
View file

@ -1,712 +0,0 @@
"""Identity lanes and the decision that fuses them.
Two lanes answer "which catalog row is this crop", and one policy turns their
evidence into accepted / ambiguous / unknown / rejected:
- `LocalLane` reads printed geometry (RootSIFT over a reference pyramid,
similarity-constrained RANSAC, a hue check). It is the only primitive that
ranks fine-grained siblings correctly: 40/42 enrolled facings first at full
resolution, against 32/42 for Van's composite.
- `TextLane` reads the package text and compares it to each catalog row's
printed name. It is the only measured mechanism that rejects a near-unknown
sibling, and it can name catalog rows that have no enrolled image at all.
Why a policy and not a threshold: the matcher's raw score is not comparable
across references of different resolution, and a single unverified match used
to clear the old floor. Measured on the corrected image6 labels, the old gate
accepts 22-28 of 48 unknowns; the local gate alone reaches 79% precision at 43%
coverage, and adding the text lane reaches 95% precision at 59% coverage.
`ponytail:` the policy is a hand-set pair of lines tuned on one photo, not a
calibrated model. E-4 replaces it once Tier-2 held-out labels exist.
"""
from __future__ import annotations
import math
import os
import re
import sys
from dataclasses import dataclass
from typing import Any, Dict, List, Optional, Sequence, Set, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
import numpy as np # noqa: E402
from rebuild_catalog import bigrams, normalize_text, signature_bigrams, sku_of # noqa: E402
# The local lane's accept line, in the units of `confidence()` below. Measured
# on the corrected image6 labels: 2.0 keeps 95% of shown labels correct at 59%
# coverage, 1.0 keeps 87% at 66%. The stricter line is the default because a
# wrong label on a shelf is worse than a missing one.
#
# 4.0 since 2026-09-29: the line was tuned on image6 alone, where every crop is
# a facing of a product the shelf actually holds, so that set could not price
# it. Re-measured with `code/sweep_accept_lines.py` on 4580 crops from 64
# unrelated Wikimedia shelf photos - 1280-long-side frames at
# `LiveSettings.conf`, the configuration the phone runs - against
# `TEXT_MIN_SHARED = 4`: 2.0 accepts 60 of them (1.31%) and 4.0 accepts 3
# (0.07%), against G3's 1%. On image6 at full resolution 4.0 shows 44 labels of
# which 42 are right - 95.5% precision, and 65.6% of the 64 facings that hold a
# catalog row - and it is the highest-coverage cell in the sweep that passes
# both G2 and G3.
LOCAL_ACCEPT = 4.0
# The text lane's own accept line: a bigram overlap this high, with this much
# margin over the runner-up, names the row on its own. Measured: the text lane
# picks the right SKU on 41-42 of the 55 facings where it reads anything.
TEXT_ACCEPT = 0.5
TEXT_MARGIN = 0.15
# How many of the read's bigrams the winning row must actually contain before
# that read may name a product. A floor on the amount of shared evidence, not
# on the ratio: a two-pair read the row happens to contain scores 1.0, and
# "サソース" named sku_l08 that way.
#
# It gates the text lane's *answer* and nothing else. Zeroing the row's score
# would be a different policy under the same name: it reorders the ranking, so
# a zeroed top row promotes a runner-up and the margin gate ends up comparing a
# different pair than the one it was measured on. It would also gate the veto,
# which is backwards - the veto only ever refuses a local accept, so a floor
# under it would let a two-bigram read rescue a match the local lane earned.
# `code/sweep_accept_lines.py` measures the line by calling the policy below
# with this value, so the sweep and the shipped lane cannot disagree about what
# it means.
#
# The ratio stays the ranking's own gate, so a long read of shared brand copy
# still scores low and is still refused by the margin.
TEXT_MIN_SHARED = 4
# Reference pyramid heights. A ~1000 px packshot against a 110-250 px crop is a
# 4-8x ratio, and ORB's 8x1.2 pyramid tops out at 3.6x, which is why ORB and
# BRISK find zero inliers at live resolution. Scaling each reference to the
# crop's own size removes the ratio instead of widening the pyramid.
REFERENCE_HEIGHTS = (200, 400)
# Lowe's ratio test.
RATIO_THRESH = 0.75
# RANSAC reprojection tolerance, as a fraction of the reference's long side.
REPROJ_FRACTION = 0.02
# Hue histogram: chromatic pixels only, so a white background cannot vote.
HUE_BINS = 18
HUE_SATURATION_MIN = 60
HUE_VALUE_MIN = 50
# Quantity tokens: a number with its unit, in either script. Cut whole, because
# the unit's own letters ("g", "ml", "%") are what survived a digits-only rule.
_QUANTITY = re.compile(r"\d+(?:\.\d+)?\s*(?:KG|G|ML|L|%|個|本|枚|入|袋|缶|パック)")
# A package's own size, in mass units. Only mass can contradict a package size:
# a percentage ("カロリー 50% カット") is a claim about the contents, not the
# package, and counting it made every low-calorie crop look like a 50 g jar.
_MASS = re.compile(r"(\d+(?:\.\d+)?)\s*(KG|G)", re.IGNORECASE)
# The range a retail package of this class falls in. Below it are fragments of a
# longer number ("20" read out of "200g"); above it are two numbers run together
# ("5210" out of "210g"). Both are OCR noise, not a different package, and
# either would veto a correct answer if taken at face value.
PACKAGE_MASS_RANGE = (100.0, 2000.0)
# How many surviving bigrams a read needs before it may name a product.
IDENTITY_BIGRAM_FLOOR = 3
@dataclass
class Evidence:
"""What one reference view says about one crop."""
inliers: int = 0
cover: float = 0.0 # inlier hull area / reference area
qcover: float = 0.0 # inlier hull area / crop area
color: float = 0.0 # hue histogram intersection
def _hue_histogram(image: Any) -> np.ndarray:
"""Hue distribution over the chromatic pixels, L1-normalized."""
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
chromatic = ((hsv[..., 1] > HUE_SATURATION_MIN) & (hsv[..., 2] > HUE_VALUE_MIN))
hist = cv2.calcHist([hsv], [0], chromatic.astype(np.uint8), [HUE_BINS], [0, 180]).ravel()
return hist / (hist.sum() + 1e-6)
def _root_sift(descriptors: Optional[np.ndarray]) -> Optional[np.ndarray]:
"""RootSIFT: L1-normalize then square-root. Cheap, and it beats raw SIFT."""
if descriptors is None:
return None
normalized = descriptors / (np.abs(descriptors).sum(1, keepdims=True) + 1e-7)
return np.sqrt(normalized).astype(np.float32)
class LocalLane:
"""RootSIFT over a reference pyramid, similarity-RANSAC, hue re-rank.
Scores every enrolled reference view and reports the best view per SKU, so
a multi-view catalog cannot make one product compete with itself.
"""
name = "local"
def __init__(
self,
stock_dir: str,
*,
heights: Sequence[int] = REFERENCE_HEIGHTS,
nfeatures: int = 2000,
) -> None:
self.stock_dir = stock_dir
self.sift = cv2.SIFT_create(nfeatures=nfeatures)
self.matcher = cv2.BFMatcher(cv2.NORM_L2)
self.views: List[Dict[str, Any]] = []
self.sku_ids: List[str] = []
self._load(heights)
def _load(self, heights: Sequence[int]) -> None:
if not os.path.exists(self.stock_dir):
return
names = sorted(
name for name in os.listdir(self.stock_dir)
if name.lower().endswith((".jpg", ".jpeg", ".png"))
)
for name in names:
image = cv2.imread(os.path.join(self.stock_dir, name))
if image is None:
continue
sku = sku_of(name)
if sku not in self.sku_ids:
self.sku_ids.append(sku)
for height in heights:
if height > image.shape[0]:
# Never upscale a reference. A 305 px view blown up to 400
# is a blur, and a blur matches everything: the l09 shelf
# crop won 31 of 54 crops on the shelf it was cut from, at
# a median margin of 4.9, because its upscaled level had no
# high-frequency detail to disagree with. A view is used at
# the levels it actually has.
continue
scaled = cv2.resize(
image,
(max(1, int(image.shape[1] * height / image.shape[0])), int(height)),
interpolation=cv2.INTER_AREA,
)
gray = cv2.cvtColor(scaled, cv2.COLOR_BGR2GRAY)
keypoints, descriptors = self.sift.detectAndCompute(gray, None)
self.views.append(
{
"sku": sku,
"keypoints": keypoints,
"descriptors": _root_sift(descriptors),
"shape": gray.shape,
"hist": _hue_histogram(scaled),
}
)
def view_count(self) -> int:
return len(self.views)
def evidence(self, crop: Any) -> Dict[str, Evidence]:
"""Best evidence per canonical SKU. Empty when the crop has no features."""
if crop is None or getattr(crop, "size", 0) == 0:
return {}
gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
keypoints, descriptors = self.sift.detectAndCompute(gray, None)
descriptors = _root_sift(descriptors)
if descriptors is None or len(descriptors) < 2:
return {}
query_hist = _hue_histogram(crop)
crop_area = float(gray.shape[0] * gray.shape[1])
best: Dict[str, Evidence] = {}
for view in self.views:
reference = view["descriptors"]
if reference is None or len(reference) < 2:
continue
pairs = self.matcher.knnMatch(descriptors, reference, k=2)
good = [
first for first, second in (pair for pair in pairs if len(pair) == 2)
if first.distance < RATIO_THRESH * second.distance
]
if len(good) < 4:
continue
source = np.float32([keypoints[m.queryIdx].pt for m in good])
target = np.float32([view["keypoints"][m.trainIdx].pt for m in good])
_, mask = cv2.estimateAffinePartial2D(
source, target, method=cv2.RANSAC,
ransacReprojThreshold=REPROJ_FRACTION * max(view["shape"]),
)
if mask is None:
continue
inliers = mask.ravel().astype(bool)
count = int(inliers.sum())
if count < 3:
continue
reference_area = float(view["shape"][0] * view["shape"][1])
evidence = Evidence(
inliers=count,
cover=float(cv2.contourArea(cv2.convexHull(target[inliers])) / reference_area),
qcover=float(cv2.contourArea(cv2.convexHull(source[inliers])) / crop_area),
color=float(np.minimum(query_hist, view["hist"]).sum()),
)
current = best.get(view["sku"])
if current is None or evidence.inliers > current.inliers:
best[view["sku"]] = evidence
return best
@staticmethod
def rank_key(evidence: Evidence) -> float:
"""Order candidates. The hue term is what separates colour-coded variants
(pink ゼロ from red ハーフ), which grayscale SIFT cannot see at all."""
return evidence.inliers * (0.25 + evidence.color)
@staticmethod
def confidence(evidence: Evidence, runner_up: Evidence) -> float:
"""How much this crop's best candidate stands out, in inlier units.
The runner-up is subtracted in inlier space, so a sibling that matches
the same artwork cannot be separated by a high score alone; the hue
agreement then scales the result, because a match that disagrees on
colour is not the same package.
"""
margin = 1.0 - runner_up.inliers / max(evidence.inliers, 1)
return evidence.inliers * margin * evidence.color
def rank(self, crop: Any) -> List[Tuple[str, float]]:
"""Candidates best-first, each with the confidence the gate reads.
Order and value are different quantities on purpose: the order comes
from `rank_key` (inliers weighted by hue), the value from `confidence`
(the margin over the runner-up). Sorting by the value would reorder the
candidates, which is how a lane that ranks 40/42 correctly ends up
gating on the wrong one.
"""
evidence = self.evidence(crop)
if not evidence:
return []
ranked = sorted(evidence.items(), key=lambda kv: self.rank_key(kv[1]), reverse=True)
return [
(
sku,
self.confidence(
item,
ranked[index + 1][1] if index + 1 < len(ranked) else Evidence(),
),
)
for index, (sku, item) in enumerate(ranked)
]
def scores(self, crop: Any) -> Dict[str, float]:
"""`rank()` as a dict, in rank order. Convenience for callers that only
need the values; the policy uses `rank()` so the order survives."""
return {sku: value for sku, value in self.rank(crop)}
# Text normalization and bigrams come from `rebuild_catalog`, which uses them
# to match a catalog name against a packshot's OCR. Two copies of the same
# folding rules drift: the lane's copy folded ユ to itself while the catalog's
# did not, so the same read scored differently depending on which one ran.
def identity_bigrams(read: str) -> set:
"""Bigrams carrying a non-digit character, after quantity tokens are cut.
A crop reading "400g" matches every 400g product's size line, so size and
number tokens must not name a product on their own. Cutting the whole
quantity token - not just the digits - is what makes that true: "400g 50%"
leaves nothing behind, where counting non-digit bigrams alone left "0G" and
"0%" and let the read through. They still count for the veto, where the
local lane has already supplied the evidence and the text only has to
contradict it.
"""
return {gram for gram in bigrams(_QUANTITY.sub(" ", normalize_text(read))) if not gram.isdigit()}
def carries_identity(read: str) -> bool:
"""Whether a read holds enough name text to answer for a product.
Two bigrams is not enough: "カロリ" is two, and it is a descriptor that any
low-calorie package prints, not a name. Three is the floor - four
characters of surviving text - which keeps "コクウマ" (from "コクうま 65%")
and refuses "カロリ", "ハーフ" and every pure quantity string. The veto is
unaffected: there the local lane has already supplied the evidence and the
"""
return len(identity_bigrams(read)) >= IDENTITY_BIGRAM_FLOOR
def text_similarity(
read: str,
signature: str,
weights: Optional[Dict[str, float]] = None,
vocabulary: Optional[Set[str]] = None,
) -> float:
"""Share of the read text's bigrams that the signature explains.
Asymmetric on purpose: the crop shows a few words of a package, the
signature holds the whole name, so the question is how much of what was
read the catalog row accounts for.
`weights` is the inverse document frequency of each bigram over the
catalog's signatures. Without it, brand copy that every sibling prints
("ピュアセレクト", "マヨネーズ", "国産新鮮たまご") counts as much as the words
that tell the products apart ("コクうま", "カロリーカット", "65"), so a row
whose signature happens to carry more shared copy outranks the row the
crop actually shows. Measured on image6: every l10 facing reads
"ピュア セレクト マヨネーズ" with no "コクうま" and no "65", yet l09 tied or
won on shared brand text alone.
`vocabulary` is every bigram any signature can produce. A read's pairs
that no catalog name contains are OCR noise - a misread character, a
date, or a pair that spans two words the package prints separately
("セレクト マヨネーズ" yields "トマ"). Counting them in the denominator
charges the row for text it could never have explained, which is how a
correct read of l10 fell to 0.46 while a garbage read of l11 stayed at
0.24: the noise is proportional to how much was read, so the rows with
the most real text to match were punished hardest. Bounded this way the
same reads score 0.86 and 1.0 against their true rows, and the garbage
read ties across three rows and is refused by the margin.
`vocabulary` defaults to the weight table's own keys, which are by
construction the bigrams some signature produces. A second copy of that set
on the lane is a second thing to keep in step, and it is the drift the
module already paid for once with the folding rules.
"""
read_grams = bigrams(read)
if vocabulary is None and weights is not None:
vocabulary = set(weights)
if vocabulary is not None:
read_grams &= vocabulary
if not read_grams:
return 0.0
shared = read_grams & signature_bigrams(signature)
if not shared:
return 0.0
if weights is None:
return len(shared) / max(1, len(read_grams))
total = sum(weights.get(gram, 1.0) for gram in read_grams)
if total <= 0:
return 0.0
return sum(weights.get(gram, 1.0) for gram in shared) / total
def bigram_weights(signatures: Sequence[str]) -> Dict[str, float]:
"""Inverse document frequency of each bigram over the catalog signatures.
A bigram every row prints carries no information about which row this is,
so it is worth ~0; a bigram one row prints is worth the most. Smoothed and
floored at 0.05 so a shared word still counts a little and a read made
entirely of shared copy scores low rather than dividing by zero.
"""
counts: Dict[str, int] = {}
for signature in signatures:
for gram in set(signature_bigrams(signature)):
counts[gram] = counts.get(gram, 0) + 1
total = max(1, len(signatures))
return {
gram: max(0.05, math.log((total + 1) / (count + 1)))
for gram, count in counts.items()
}
class TextLane:
"""Package text against each catalog row's printed name.
The signature is the row's name, so it is built from the catalog and never
from the evaluation photo. Rows with no image still get a signature, which
is how the lane can name a SKU no matcher can see.
"""
name = "text"
def __init__(
self,
signatures: Dict[str, str],
ocr_engine: Any = None,
lock: Any = None,
) -> None:
self.signatures = {str(sku): text for sku, text in signatures.items()}
# Bigram weights over the catalog's own names, so the lane can tell
# copy every sibling prints from the words that name one product.
self.weights = bigram_weights(list(self.signatures.values()))
self.ocr_engine = ocr_engine
# The engine's OCR lock. Vision tolerates concurrency but the CPU does
# not, and this lane runs one OCR per crop on the classify pool, so it
# shares the same lock as tag reading instead of adding a second one.
self.lock = lock
@classmethod
def from_catalog(
cls,
catalog_rows: Sequence[Dict[str, Any]],
stock_dir: str,
ocr_engine: Any = None,
lock: Any = None,
) -> "TextLane":
signatures: Dict[str, str] = {}
for row in catalog_rows:
sku = str(row.get("sku_id") or "")
if not sku:
continue
signatures[sku] = str(row.get("product_name") or "")
return cls(signatures, ocr_engine=ocr_engine, lock=lock)
def read(self, crop: Any) -> str:
"""The text on one crop, upsampled when the crop is small.
Below ~200 px tall the recognizer misses the small variant words that
are the whole point of this lane, so the crop is doubled first.
"""
if self.ocr_engine is None or crop is None or getattr(crop, "size", 0) == 0:
return ""
if not getattr(self.ocr_engine, "available", lambda: False)():
return ""
image = crop
if image.shape[0] < 200:
image = cv2.resize(image, None, fx=2.0, fy=2.0, interpolation=cv2.INTER_CUBIC)
try:
if self.lock is not None:
with self.lock:
result = self.ocr_engine.read(image)
else:
result = self.ocr_engine.read(image)
except Exception:
return ""
return " ".join(line.text for line in result.lines)
def scores(self, crop: Any) -> Dict[str, float]:
"""Bigram similarity per catalog row. Empty when nothing was read."""
return self.scores_from_text(self.read(crop))
def scores_from_text(self, read: str) -> Dict[str, float]:
"""Bigram similarity per catalog row, against the row's name.
The name only. Scoring against the packshot's own OCR as well is how
l09 used to win l10's facings: l09's packshot prints "マヨネーズ",
which is l10's actual name, so a row with a photographed packshot
could explain a sibling's text better than the sibling's own name
could. Every row in the catalog has a name, so the packshot adds
nothing but that confusion.
"""
if len(normalize_text(read)) < 4:
return {}
return {
sku: text_similarity(read, signature, self.weights)
for sku, signature in self.signatures.items()
}
def _ranked(scores: Any) -> List[Tuple[str, float]]:
"""Candidates best-first.
A lane hands over an already-ordered sequence and it is used as given: the
order is the lane's ranking, and re-sorting by the confidence value would
silently replace it. A plain dict is sorted by value, which is the only
order a dict has.
"""
if scores is None:
return []
if isinstance(scores, dict):
items = scores.items()
return sorted(
((str(sku), float(value)) for sku, value in items),
key=lambda kv: kv[1],
reverse=True,
)
return [(str(sku), float(value)) for sku, value in scores]
class DecisionPolicy:
"""Turn two lanes' evidence into one identity decision.
The local lane answers first, because it is the only lane that ranks
siblings correctly. The text lane then does one of two jobs: it vetoes a
local answer whose package text names a different catalog row, and it
answers on its own when the local lane has nothing. Both jobs are measured:
the veto takes precision from 62% to 82% for one lost correct label, and
the lane adds ten correct labels at live resolution, including SKUs with no
enrolled image.
"""
def __init__(
self,
local_accept: float = LOCAL_ACCEPT,
text_accept: float = TEXT_ACCEPT,
text_margin: float = TEXT_MARGIN,
text_min_shared: int = TEXT_MIN_SHARED,
signatures: Optional[Dict[str, str]] = None,
method: str = "rootsift",
) -> None:
self.local_accept = float(local_accept)
self.text_accept = float(text_accept)
self.text_margin = float(text_margin)
self.text_min_shared = int(text_min_shared)
# The catalog names, so the evidence floor can ask how many of the
# read's pairs the winning row actually contains. Without them the
# floor cannot be applied and the ratio stands alone, which is the
# behaviour a caller that has no catalog is asking for.
self.signatures = {str(sku): str(text) for sku, text in (signatures or {}).items()}
self.method = method
def _shared_evidence(self, read: str, sku: str) -> int:
"""How many of the read's bigrams the row's name contains."""
signature = self.signatures.get(sku)
if signature is None:
return self.text_min_shared
return len(bigrams(read) & signature_bigrams(signature))
def _contradicts_size(self, read: str, sku: str) -> bool:
"""Does the read print a package size the row's name does not have?
The near-unknown case the text ranking cannot catch. A crop of a
Kewpie 450 g bag reads "マヨネーズ 450g", and every mayonnaise row in
the catalog explains "マヨネーズ" equally well, so the ranking ties at
1.0 and the strict-greater veto above never fires - the local lane's
answer stands. The size is the part that differs, and it is printed
plainly, so it is the part worth checking.
Only a mass in `PACKAGE_MASS_RANGE` counts. Outside it the read is
noise, not a package: "20" is a fragment of "200g" and "5210" is two
numbers run together, and vetoing on either would cost correct
answers. Measured on image6, this vetoes 3 false accepts at full
resolution and 1 at live, and no correct answer at either.
"""
signature = self.signatures.get(sku)
if signature is None:
return False
read_masses = {
float(number) for number, _unit in _MASS.findall(normalize_text(read))
}
plausible = {
mass for mass in read_masses
if PACKAGE_MASS_RANGE[0] <= mass <= PACKAGE_MASS_RANGE[1]
}
if not plausible:
return False
name_masses = {
float(number) for number, _unit in _MASS.findall(normalize_text(signature))
}
return not (read_masses & name_masses)
def decide(
self,
local_rank: Any,
text_rank: Any = None,
*,
details: Optional[Dict[str, Any]] = None,
text_read: Optional[str] = None,
) -> Dict[str, Any]:
"""One decision, in the shape `decide_identity` already returns.
`text_read` is the raw OCR text behind `text_rank`. It gates the
fill-in job only: a read that is nothing but a size or a number cannot
name a product, but it can still veto one. Omitting it means no
fill-in - a caller that has a ranking and no read has not shown the
text can name anything, so the safe reading is the default one.
"""
from identity import (
IDENTITY_ACCEPTED,
IDENTITY_AMBIGUOUS,
IDENTITY_REJECTED,
IDENTITY_UNKNOWN,
)
local = _ranked(local_rank)
text = _ranked(text_rank)
text_top = text[0] if text else None
text_runner = text[1][1] if len(text) > 1 else 0.0
text_names = (
text_top[0]
if text_top
and text_top[1] >= self.text_accept
and (text_top[1] - text_runner) >= self.text_margin
and carries_identity(text_read or "")
and self._shared_evidence(text_read or "", text_top[0]) >= self.text_min_shared
# A row the read contradicts on size cannot be named by that read
# either. Without this the veto above only moves the answer: the
# local lane's row is blocked and the text lane then names the same
# row, because "マヨネーズ" is what both of them matched.
and not self._contradicts_size(text_read or "", text_top[0])
else None
)
answer: Optional[str] = None
source: Optional[str] = None
if local and local[0][1] >= self.local_accept:
candidate = local[0][0]
# A local match whose package text names a different row is the
# near-unknown case no threshold can reject: the sibling shares the
# artwork, so the geometry is real and only the words differ.
if text_top and text_top[0] != candidate and text_top[1] > local_text_floor(text, candidate):
answer = None
elif self._contradicts_size(text_read or "", candidate):
answer = None
else:
answer, source = candidate, "local"
if answer is None and text_names:
answer, source = text_names, "text"
ranked = [
{"sku_id": sku, "score": round(value, 2)} for sku, value in local[:5]
]
if not ranked and text:
ranked = [{"sku_id": sku, "score": round(value, 3)} for sku, value in text[:5]]
if answer is not None:
level = IDENTITY_ACCEPTED
message = "Accepted unique catalog match"
elif local and local[0][1] > 0.0:
level = IDENTITY_AMBIGUOUS if len(local) > 1 else IDENTITY_UNKNOWN
message = "No candidate cleared the accept line"
elif text:
level = IDENTITY_UNKNOWN
message = "Text read, but it does not name one catalog row"
else:
level = IDENTITY_REJECTED
message = "No matching evidence"
# Coverage-first: the row the evidence supports well enough to *show*,
# even when it does not clear the accept line. `None` means the crop
# carried no usable evidence at all, which is the only case that still
# ends blank - a size-only read has no identity text and stays None.
if answer is not None:
provisional = answer
elif local and local[0][1] > 0.0:
# A vetoed local answer is still the best geometric guess, unless
# the text named a different row - then that row is the better one.
vetoed = bool(
text_top
and text_top[0] != local[0][0]
and text_top[1] > local_text_floor(text, local[0][0])
)
provisional = text_top[0] if vetoed else local[0][0]
elif (
text
and carries_identity(text_read or "")
and self._shared_evidence(text_read or "", text[0][0]) >= self.text_min_shared
):
provisional = text[0][0]
else:
provisional = None
top_score = local[0][1] if local else (text_top[1] if text_top else 0.0)
return {
"status": "success" if level == IDENTITY_ACCEPTED else level,
"method": self.method,
"best_sku": answer,
"provisional_sku": provisional,
"score": round(float(top_score), 2),
"confidence": round(float(top_score), 4),
"identity_level": level,
"ranked_candidates": ranked,
"details": (details or {}).get(answer, {}) if answer else {},
"message": message,
"source": source,
}
def local_text_floor(text: List[Tuple[str, float]], candidate: str) -> float:
"""The text score the local candidate itself explains.
A veto needs the text to *prefer* another row, not merely to mention it: a
crop that reads "マヨネーズ" scores every mayonnaise row, and the local
candidate is usually one of them. Only a row the text explains better than
the local answer's own row is a contradiction.
"""
for sku, value in text:
if sku == candidate:
return value
return 0.0

1755
recognization/code/live.py
View file

File diff suppressed because it is too large Load diff

235
recognization/code/ocr.py
View file

@ -1,235 +0,0 @@
"""
OCR engines for the recognition server.
One interface, several backends, chosen by availability and configuration - the
same shape as the app kit's platform bridges. The engine only answers "what text
is in this image and where"; everything tag-specific (band geometry, clustering,
field parsing, association) lives in tags.py.
Backends:
- VisionOcr macOS, via pyobjc/Vision. Measured 182-246 ms per rail band.
- FakeOcr deterministic lines for tests and offline demos.
- (RapidOCR) deliberately not shipped: the owner chose the macOS backend, and the
interface keeps a cross-platform engine a drop-in later.
The engine never guesses: a line that is not recognized is simply absent, and the
caller decides what absence means (see the compliance rules, which fail closed on
unreadable evidence rather than reporting a violation).
"""
from __future__ import annotations
import os
import time
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Sequence, Tuple
DEFAULT_LANGUAGES = ("ja-JP", "en-US")
@dataclass
class OcrLine:
"""One recognized text line, in the pixel space of the image it was read from."""
text: str
confidence: float
box: List[int] # [x1, y1, x2, y2], origin top-left
def as_dict(self) -> Dict[str, Any]:
return {"text": self.text, "confidence": round(float(self.confidence), 3), "box": list(self.box)}
@dataclass
class OcrResult:
lines: List[OcrLine] = field(default_factory=list)
engine: str = "none"
elapsed_ms: float = 0.0
def as_dict(self) -> Dict[str, Any]:
return {
"engine": self.engine,
"elapsed_ms": round(self.elapsed_ms, 1),
"lines": [line.as_dict() for line in self.lines],
}
class OcrUnavailable(RuntimeError):
"""Raised when a backend is asked to read but cannot run here."""
class OcrEngine:
"""Base class: subclasses implement read()."""
name = "none"
def available(self) -> bool: # pragma: no cover - overridden
return False
def read(
self,
image: Any,
*,
languages: Sequence[str] = DEFAULT_LANGUAGES,
fast: bool = False,
) -> OcrResult: # pragma: no cover - overridden
raise OcrUnavailable(f"engine '{self.name}' is not available")
class VisionOcr(OcrEngine):
"""macOS Vision text recognition through pyobjc."""
name = "vision"
def __init__(self, languages: Sequence[str] = DEFAULT_LANGUAGES):
self.languages = list(languages)
self._vision = None
self._quartz = None
self._load_error: Optional[str] = None
self._load()
def _load(self) -> None:
if os.uname().sysname != "Darwin": # pragma: no cover - platform guard
self._load_error = "not macOS"
return
try:
import Quartz # noqa: F401
import Vision # noqa: F401
self._quartz = Quartz
self._vision = Vision
except Exception as exc: # pragma: no cover - import guard
self._load_error = str(exc)
def available(self) -> bool:
return self._vision is not None and self._quartz is not None
def read(
self,
image: Any,
*,
languages: Optional[Sequence[str]] = None,
fast: bool = False,
) -> OcrResult:
if not self.available():
raise OcrUnavailable(f"Vision is unavailable here ({self._load_error})")
import cv2
import numpy as np
from Foundation import NSData # type: ignore
started = time.perf_counter()
height, width = image.shape[:2]
ok, encoded = cv2.imencode(".png", image)
if not ok:
return OcrResult(engine=self.name, elapsed_ms=(time.perf_counter() - started) * 1000.0)
raw = encoded.tobytes()
data = NSData.dataWithBytes_length_(raw, len(raw))
source = self._quartz.CGImageSourceCreateWithData(data, None)
cg_image = self._quartz.CGImageSourceCreateImageAtIndex(source, 0, None)
request = self._vision.VNRecognizeTextRequest.alloc().init()
request.setRecognitionLevel_(
self._vision.VNRequestTextRecognitionLevelFast
if fast
else self._vision.VNRequestTextRecognitionLevelAccurate
)
request.setRecognitionLanguages_(list(languages or self.languages))
request.setUsesLanguageCorrection_(False)
handler = self._vision.VNImageRequestHandler.alloc().initWithCGImage_options_(cg_image, None)
performed, error = handler.performRequests_error_([request], None)
elapsed = (time.perf_counter() - started) * 1000.0
if not performed:
raise OcrUnavailable(f"Vision request failed: {error}")
lines: List[OcrLine] = []
for observation in request.results() or []:
candidates = observation.topCandidates_(1)
if not candidates:
continue
top = candidates[0]
box = observation.boundingBox()
x1 = box.origin.x * width
x2 = (box.origin.x + box.size.width) * width
# Vision uses a bottom-left origin; the rest of the system is top-left.
y1 = (1.0 - box.origin.y - box.size.height) * height
y2 = (1.0 - box.origin.y) * height
lines.append(
OcrLine(
text=str(top.string()),
confidence=float(top.confidence()),
box=[int(round(x1)), int(round(y1)), int(round(x2)), int(round(y2))],
)
)
return OcrResult(lines=lines, engine=self.name, elapsed_ms=elapsed)
class FakeOcr(OcrEngine):
"""Deterministic engine for tests: returns whatever it was told to.
Accepts either a list of OcrLine (returned as-is, in the caller's pixel space)
or a callable(image) -> list[OcrLine].
"""
name = "fake"
def __init__(self, lines: Optional[Sequence[OcrLine]] = None, reader=None):
self._lines = list(lines or [])
self._reader = reader
self.calls = 0
self.last_image_shape: Optional[Tuple[int, int]] = None
def available(self) -> bool:
return True
def read(self, image: Any, *, languages: Sequence[str] = DEFAULT_LANGUAGES, fast: bool = False) -> OcrResult:
self.calls += 1
self.last_image_shape = tuple(image.shape[:2])
lines = self._reader(image) if self._reader is not None else [OcrLine(**vars(line)) for line in self._lines]
return OcrResult(lines=list(lines), engine=self.name, elapsed_ms=0.0)
class NullOcr(OcrEngine):
"""Explicitly no OCR: used when the feature is switched off."""
name = "none"
def available(self) -> bool:
return False
def read(self, image: Any, *, languages: Sequence[str] = DEFAULT_LANGUAGES, fast: bool = False) -> OcrResult:
return OcrResult(lines=[], engine=self.name, elapsed_ms=0.0)
def build_ocr_engine(kind: Optional[str] = None) -> OcrEngine:
"""Pick a backend. DORAYAKI_OCR = vision | fake | none (default: vision on macOS)."""
kind = (kind or os.environ.get("DORAYAKI_OCR") or "auto").lower()
if kind == "none":
return NullOcr()
if kind == "fake":
return FakeOcr()
if kind in ("auto", "vision"):
engine = VisionOcr()
if engine.available():
return engine
if kind == "vision":
raise OcrUnavailable(f"Vision requested but unavailable: {engine._load_error}")
return NullOcr()
if __name__ == "__main__": # smoke: python code/ocr.py <image> [<image> ...]
import sys
import cv2
engine = build_ocr_engine()
print(f"engine: {engine.name} (available: {engine.available()})")
for path in sys.argv[1:]:
image = cv2.imread(path)
if image is None:
print(f"{path}: cannot read")
continue
result = engine.read(image)
print(f"--- {path} {result.elapsed_ms:.0f} ms, {len(result.lines)} lines ---")
for line in result.lines[:12]:
print(f" {line.confidence:.2f} {line.text} {line.box}")

0
recognization/code/orb.py Normal file
View file

181
recognization/code/pipeline.py Normal file
View file

@ -0,0 +1,181 @@
"""
End-to-End Supermarket Shelf Recognition Pipeline
Integrates:
1. YOLO object detector for shelf product localization
2. SURF / ORB / BRISK / Hybrid local feature matcher for classification
3. Stock CSV inventory lookup for real-time pricing and stock quantities
"""
import os
import sys
import cv2
import numpy as np
from typing import Dict, Optional, Any
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from detector import ShelfProductDetector
from feature_matcher import ProductFeatureMatcher
from inventory import InventoryManager
class ShelfRecognitionPipeline:
"""Master pipeline managing interactive supermarket shelf recognition."""
def __init__(
self,
weights_path: str = "runs/detect/train/weights/best.pt",
stock_dir: str = "stock",
stock_csv: str = "stock.csv"
):
print("[Pipeline] Initializing Inventory Manager...")
self.inventory = InventoryManager(stock_dir=stock_dir, csv_filename=stock_csv)
print("[Pipeline] Initializing Feature Matcher...")
self.matcher = ProductFeatureMatcher(stock_dir=stock_dir)
print("[Pipeline] Initializing Shelf Detector...")
self.detector = ShelfProductDetector(weights_path=weights_path)
self.current_image_path: Optional[str] = None
self.current_image: Optional[np.ndarray] = None
self.current_detections: list = []
self.current_empty_slots: list = []
def set_active_shelf_image(self, image_input: Any) -> Dict[str, Any]:
"""Loads and runs detection on a new shelf image."""
if isinstance(image_input, str):
self.current_image_path = image_input
self.current_image = cv2.imread(image_input)
else:
self.current_image_path = "memory_buffer"
self.current_image = image_input
if self.current_image is None:
return {"status": "error", "message": "Invalid or unreadable image"}
det_result = self.detector.detect(self.current_image)
self.current_detections = det_result.get("boxes", [])
# Detect empty shelf spaces (out-of-stock gaps)
self.current_empty_slots = self.detector.detect_empty_slots(
self.current_image.shape, self.current_detections
)
return {
"status": "success",
"image_shape": self.current_image.shape,
"total_detected": len(self.current_detections),
"boxes": self.current_detections,
"empty_slots": self.current_empty_slots
}
def process_click(
self,
x: int,
y: int,
method: str = "hybrid",
tolerance_px: int = 40
) -> Dict[str, Any]:
"""
Handles user click at coordinate (x, y) on the shelf image:
1. Identifies the bounding box
2. Crops product ROI
3. Classifies via SURF/ORB/BRISK/Hybrid matching
4. Queries inventory for price, quantity, and metadata
"""
if self.current_image is None:
return {"status": "error", "message": "No active shelf image loaded."}
target_box = self.detector.find_box_at_point(
x, y, self.current_detections, tolerance_px=tolerance_px
)
if target_box is None:
# Check if user clicked an empty shelf space
for slot in getattr(self, "current_empty_slots", []):
sx1, sy1, sx2, sy2 = slot["box"]
if (sx1 - 10 <= x <= sx2 + 10) and (sy1 - 10 <= y <= sy2 + 10):
return {
"status": "empty_slot",
"clicked_point": (x, y),
"empty_slot": slot,
"message": f"Chỗ trống trên kệ (Khu vực hàng hết / Row {slot['row_index']}). Ước tính thiếu ~{slot['missing_items_est']} sản phẩm."
}
return {
"status": "not_found",
"clicked_point": (x, y),
"message": f"No product detected near coordinate ({x}, {y}). Please click inside a product packaging."
}
# Crop ROI
crop = self.detector.crop_roi(self.current_image, target_box["box"])
# Classify with selected method
cls_result = self.matcher.classify_crop(crop, method=method)
if cls_result.get("status") != "success":
return {
"status": "classification_failed",
"target_box": target_box,
"message": cls_result.get("message", "Feature matching failed.")
}
best_sku = cls_result["best_sku"]
product_info = self.inventory.get_product(best_sku)
# Generate match visualization
vis_match = self.matcher.generate_match_visualization(crop, best_sku, method=method)
# Generate annotated shelf image highlighting the selected item
annotated_shelf = self.detector.draw_detections(
self.current_image, self.current_detections, selected_box_id=target_box["id"]
)
return {
"status": "success",
"clicked_point": (x, y),
"target_box": target_box,
"classification": {
"method": method,
"matched_sku": best_sku,
"confidence": cls_result["confidence"],
"score": cls_result["score"],
"top_candidates": cls_result["ranked_candidates"],
},
"inventory": product_info,
"crop_image": crop,
"annotated_shelf": annotated_shelf,
"match_visualization": vis_match
}
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Test Shelf Recognition Pipeline")
parser.add_argument("--image", type=str, default="runs/detect/train/train_batch0.jpg")
parser.add_argument("--method", type=str, default="hybrid", choices=["surf", "orb", "brisk", "hybrid"])
args = parser.parse_args()
pipeline = ShelfRecognitionPipeline()
print(f"Loading image: {args.image}")
init_res = pipeline.set_active_shelf_image(args.image)
print(f"Total detected items: {init_res['total_detected']}")
if init_res['boxes']:
# Simulate click on the center of the first detected product
first_box = init_res['boxes'][0]
cx, cy = first_box['center']
print(f"Simulating click on Product #{first_box['id']} at ({cx}, {cy})...")
res = pipeline.process_click(cx, cy, method=args.method)
print("\n--- Recognition & Inventory Result ---")
print(f"Status: {res['status']}")
if res['status'] == 'success':
prod = res['inventory']
print(f"Matched SKU: {res['classification']['matched_sku']} (Score: {res['classification']['score']})")
print(f"Product Name: {prod['product_name']}")
print(f"Price: {prod['price']} JPY")
print(f"In Stock: {prod['number_on']} units (Received: {prod['number_in']}, Sold: {prod['number_out']})")
print(f"Expiration Date: {prod['expiration_date']}")

629
recognization/code/rebuild_catalog.py
View file

@ -1,629 +0,0 @@
"""Rebuild a store catalog from packshots, refusing unsafe enrollment.
The LOPIA catalog shipped with views cut from the test shelf itself: 71-119 px
wide, one 18 px sliver, duplicate crops, and five SKUs with no view at all. That
is why every dark package collapsed onto one SKU and why the head was trained on
its own exam.
This rebuilds a catalog directory from named packshots and refuses, per view:
* a view shorter than MIN_HEIGHT, narrower than MIN_WIDTH, or shorter than
PYRAMID_FLOOR (a source under it cannot be stored as the protocol's
200/400/800 px pyramid, so its scores would not be comparable with the rest),
* a view whose capture session appears in --eval-manifest,
* a packshot whose catalog name is not supported by its OCR text,
* a view that is a near-duplicate of another view of the same SKU,
* a SKU left with fewer than --min-views distinct views.
Every enrolled image must have a CSV row; a SKU with an image and no row refuses
the build. CSV rows with no image are kept and reported as text-only coverage
gaps, because the text lane names and vetoes those products from the CSV alone.
Capture session ids come from the source manifest (``manifest.json`` beside the
images by default), whose image entries use ``file`` plus
``capture_session_id`` or ``session_id``. Pixel equality is not a leak test:
re-cut crops pass it, so leakage is refused by capture session instead.
Two checks report rather than refuse, because both are properties of the
catalog as a whole: cross-SKU near-duplicates are flagged as
text-discriminated siblings (the near-unknown case only package text
separates), and ``--shelf-tags`` turns the completeness rule into a live-ready
verdict. Without that list the verdict is ``unknown``, not a guess.
python code/rebuild_catalog.py --source /tmp/lopia-clean \
--output stock-lopia-clean --csv stock-lopia/stock.csv
python code/rebuild_catalog.py --source /tmp/lopia-clean \
--output stock-lopia-clean --csv stock-lopia/stock.csv --replace
Views are named <sku>.jpg, <sku>_2.jpg, ... The CSV is copied byte-for-byte, so
text-only products stay in the catalog and its content hash does not move.
"""
from __future__ import annotations
import argparse
import csv
import json
import os
import shutil
import sys
import tempfile
import unicodedata
from typing import Any, Dict, List, Optional, Sequence, Set, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
import numpy as np # noqa: E402
from ocr import OcrUnavailable, build_ocr_engine # noqa: E402
# A view narrower than this cannot carry the logo and text geometry that tells
# two fine-grained products apart. The shipped LOPIA views were 71-119 px.
MIN_WIDTH = 120
# The enrollment protocol's floor: a reference view under 160 px tall carries no
# text geometry (facings under 60 px wide failed every measured lane).
MIN_HEIGHT = 160
# The name/OCR gate, re-measured on the shipped packshots with the strict name
# side in place, each image read four ways and scored against all 12 catalog
# names: correct packshots 0.9167-1.0000 (seven read their whole name; l11's
# read omits "マヨネーズ" and is the floor), the wrong-product packshot in
# references/rejected/ 0.0476-0.5833. 0.75 is the midpoint: 0.167 of headroom
# above every correct packshot and 0.167 below the wrong-product ceiling, so
# OCR noise on either side does not move a verdict. Siblings are not this
# gate's job - the コクうま packshot scores 0.85 against the マヨネーズ name,
# which is what text_discriminated_siblings flags, not a gap a threshold closes.
MIN_NAME_OCR = 0.75
# Vision drops small print at native scale: the l11 packshot reads ハーフ only
# when upscaled (0.57 -> 0.79). Reading at 2x is what makes the check about the
# package rather than about Vision's recall.
OCR_SCALE = 2.0
# Two views of one SKU this similar are the same photograph counted twice.
DUPLICATE_COSINE = 0.98
# The enrollment protocol's pyramid target: a source at least this tall can be
# stored as a 200/400/800 px pyramid, so a view under it cannot be scored
# against the rest. Refused, not reported: the l09 defect was a 305 px view
# whose inlier scores (85-129) dwarfed every other reference (10-40) and it won
# facings it should not, and a report is exactly the kind of thing that gets
# read once and then ignored. The fix is re-sourcing, which is a data task
# this tool should not let through the door in the first place.
PYRAMID_FLOOR = 800
# Official pack-shot pages publish a single front view per JAN, so a catalog
# built from them runs at 1 and says so; the floor exists to catch accidental
# single-view enrollment from shelf crops, where several views are available.
MIN_VIEWS = 2
IMAGE_EXTENSIONS = (".jpg", ".jpeg", ".png")
SESSION_KEYS = ("capture_session_id", "session_id")
SESSION_LIST_KEYS = ("sessions", "session_ids", "evaluation_sessions")
class RebuildError(Exception):
"""Refused input. The destination, if it existed, is unchanged."""
def sku_of(filename: str) -> str:
"""`sku_l01_2.jpg` -> `sku_l01`; `sku_l01.jpg` -> `sku_l01`."""
stem = os.path.splitext(filename)[0]
parts = stem.split("_")
if len(parts) >= 2 and parts[0] == "sku":
return "_".join(parts[:2])
return stem
def view_rank(filename: str) -> int:
"""Order views so the primary view sorts first: `sku_l01.jpg` before `_2`."""
stem = os.path.splitext(filename)[0]
parts = stem.split("_")
return int(parts[2]) if len(parts) > 2 and parts[2].isdigit() else 1
def cosine(left: np.ndarray, right: np.ndarray) -> float:
a = left.astype(np.float32).ravel()
b = right.astype(np.float32).ravel()
denom = float(np.linalg.norm(a) * np.linalg.norm(b))
return float((a * b).sum() / denom) if denom else 0.0
def upscale(image: np.ndarray) -> np.ndarray:
"""Enlarge for OCR only; the stored view keeps its own pixels."""
return cv2.resize(image, None, fx=OCR_SCALE, fy=OCR_SCALE, interpolation=cv2.INTER_CUBIC)
def normalize_text(text: str) -> str:
"""NFKC, uppercase, small kana folded, alnum and `%` kept."""
value = unicodedata.normalize("NFKC", text).upper()
# Vision varies between small and full-size kana, and Kewpie prints
# キユーピー with a small ユ; folding both keeps the comparison about the
# product name rather than the typesetter.
value = value.translate(str.maketrans("ッャュョァィゥェォヮヵヶ", "ツヤユヨアイウエオワカケ"))
value = value.replace("キユーピー", "キユピ").replace("キューピー", "キユピ")
return "".join(char for char in value if char.isalnum() or char == "%")
def signature_bigrams(text: str) -> Set[str]:
"""Character pairs of a catalog name, never across a word boundary.
A pair that spans two words is not a substring of either one, so it can
only ever be noise - and expensive noise, because it is rare across the
catalog and IDF therefore weights it as highly distinctive. "キユーピー
アマニ油" used to contribute "ピア", which let the garbage read
"キユピアク5C" be accepted as that row.
"""
pairs: Set[str] = set()
for token in text.split():
value = normalize_text(token)
pairs.update(value[index:index + 2] for index in range(len(value) - 1))
return pairs
def bigrams(text: str) -> Set[str]:
"""Character pairs of a read, across whitespace.
Deliberately not `signature_bigrams`: OCR puts spaces where the package
has none, so "ピュア セレクト" has to keep matching the catalog's
"ピュアセレクト". The catalog name is authoritative about where its words
end; a read is noisy evidence about the same characters, and dropping its
spaces is the only way the two can meet. The read is the side that gets
the benefit of the doubt, so the fake-pair problem is fixed where it
originates - in the signature - rather than by refusing to match.
"""
value = normalize_text(text)
return {value[index:index + 2] for index in range(len(value) - 1)}
def name_ocr_similarity(name: str, ocr_text: str) -> float:
"""Share of the catalog name's bigrams that the OCR text explains."""
name_bigrams = signature_bigrams(name)
return len(name_bigrams & bigrams(ocr_text)) / max(1, len(name_bigrams))
def best_orientation_text(ocr_engine: Any, image: np.ndarray, name: str = "") -> str:
"""OCR text from whichever of the four right-angle rotations matches best.
A packshot's identifying text is not always horizontal: the やまや
めんたいマヨネーズ packshot reads 明太手屋さんが only at 270 degrees, and
the upright pass returns three stray characters. Checking one orientation
refused a correct image, which is the worst failure this check has - it
rejects good data while claiming to validate it.
The winner is the rotation that explains the most of the catalog name, not
the one that reads the most characters: the upright pass on that packshot
returns stray kana that concatenate longer than the real read, so length
picks the wrong one. With no name to compare against, length is the only
signal left.
"""
rotations = (
image,
cv2.rotate(image, cv2.ROTATE_90_CLOCKWISE),
cv2.rotate(image, cv2.ROTATE_180),
cv2.rotate(image, cv2.ROTATE_90_COUNTERCLOCKWISE),
)
reads = [" ".join(line.text for line in ocr_engine.read(upscale(rotated)).lines)
for rotated in rotations]
if name:
return max(reads, key=lambda text: name_ocr_similarity(name, text))
return max(reads, key=lambda text: len(text.strip()))
def collect(source: str) -> Dict[str, List[str]]:
by_sku: Dict[str, List[str]] = {}
for name in sorted(os.listdir(source)):
if os.path.splitext(name)[1].lower() not in IMAGE_EXTENSIONS:
continue
by_sku.setdefault(sku_of(name), []).append(name)
for views in by_sku.values():
views.sort(key=view_rank)
return by_sku
def read_catalog(csv_path: str) -> Tuple[List[str], List[Dict[str, str]], Dict[str, str]]:
"""Columns, rows, and SKU -> product_name. Refuses a malformed catalog."""
with open(csv_path, newline="", encoding="utf-8-sig") as handle:
reader = csv.DictReader(handle)
fieldnames = list(reader.fieldnames or [])
rows = list(reader)
if "SKU_ID" not in fieldnames or "product_name" not in fieldnames:
raise RebuildError(f"{csv_path}: needs SKU_ID and product_name columns")
names: Dict[str, str] = {}
duplicates: Set[str] = set()
for row in rows:
sku = (row.get("SKU_ID") or "").strip()
if not sku:
raise RebuildError(f"{csv_path}: a row has an empty SKU_ID")
if sku in names:
duplicates.add(sku)
names[sku] = (row.get("product_name") or "").strip()
if duplicates:
raise RebuildError("duplicate CSV row(s): " + ", ".join(sorted(duplicates)))
return fieldnames, rows, names
def _session_values(payload: Any) -> Set[str]:
"""Every session id anywhere in a manifest, whatever its shape."""
found: Set[str] = set()
if isinstance(payload, dict):
for key, value in payload.items():
if key in SESSION_KEYS and isinstance(value, (str, int)):
found.add(str(value))
elif key in SESSION_LIST_KEYS and isinstance(value, list):
for item in value:
found.update([str(item)] if isinstance(item, (str, int)) else _session_values(item))
elif isinstance(value, (dict, list)):
found.update(_session_values(value))
elif isinstance(payload, list):
for item in payload:
found.update(_session_values(item))
return found
def load_eval_sessions(path: str) -> Set[str]:
"""Sessions an evaluation used. A manifest naming none is refused, not trusted."""
with open(path, encoding="utf-8") as handle:
sessions = _session_values(json.load(handle))
if not sessions:
raise RebuildError(f"{path}: evaluation manifest names no capture session")
return sessions
def load_capture_sessions(path: Optional[str]) -> Dict[str, str]:
"""Image basename -> capture session, from an enrollment manifest."""
if not path:
return {}
with open(path, encoding="utf-8") as handle:
payload = json.load(handle)
if isinstance(payload, dict):
entries = payload.get("images", [])
default = payload.get("capture_session_id") or payload.get("session_id")
else:
entries, default = payload, None
if not isinstance(entries, list):
raise RebuildError(f"{path}: 'images' must be a list")
sessions: Dict[str, str] = {}
for entry in entries:
if not isinstance(entry, dict):
continue
filename = entry.get("file") or entry.get("filename") or entry.get("image")
session = entry.get("capture_session_id") or entry.get("session_id") or default
if filename and session is not None:
sessions[os.path.basename(str(filename))] = str(session)
return sessions
def audit(
source: str,
min_views: int = MIN_VIEWS,
names: Optional[Dict[str, str]] = None,
ocr_engine: Any = None,
capture_sessions: Optional[Dict[str, str]] = None,
eval_sessions: Optional[Set[str]] = None,
) -> Tuple[Dict[str, List[str]], List[Dict[str, str]]]:
"""Keep the views that pass, and report every refusal with its reason."""
kept: Dict[str, List[str]] = {}
refused: List[Dict[str, str]] = []
capture_sessions = capture_sessions or {}
eval_sessions = eval_sessions or set()
for sku, views in sorted(collect(source).items()):
accepted: List[Tuple[str, np.ndarray]] = []
for name in views:
image = cv2.imread(os.path.join(source, name))
if image is None:
refused.append({"sku": sku, "view": name, "reason": "unreadable"})
continue
# Leakage first: a view from an evaluation session is refused even if
# it would fail another check, so the leak is never hidden by one.
session = capture_sessions.get(name)
if session and session in eval_sessions:
refused.append({"sku": sku, "view": name,
"reason": f"capture session {session} is in the evaluation manifest"})
continue
if image.shape[0] < MIN_HEIGHT:
refused.append({"sku": sku, "view": name,
"reason": f"{image.shape[0]}px tall, below the {MIN_HEIGHT}px floor"})
continue
if image.shape[1] < MIN_WIDTH:
refused.append({"sku": sku, "view": name,
"reason": f"{image.shape[1]}px wide, below the {MIN_WIDTH}px floor"})
continue
if image.shape[0] < PYRAMID_FLOOR:
refused.append({"sku": sku, "view": name,
"reason": f"{image.shape[0]}px tall, below the {PYRAMID_FLOOR}px "
"pyramid floor; its scores would not be comparable "
"with the rest - re-source this view"})
continue
if ocr_engine is not None and names is not None:
try:
text = best_orientation_text(ocr_engine, image, names.get(sku, ""))
except OcrUnavailable as error:
print(f"OCR skipped for {name}: {error}", file=sys.stderr)
else:
similarity = name_ocr_similarity(names.get(sku, ""), text)
if similarity < MIN_NAME_OCR:
refused.append({"sku": sku, "view": name,
"reason": f"name/OCR {similarity:.2f} below {MIN_NAME_OCR:.2f}: "
f"{text[:100] or '<no text read>'}"})
continue
duplicate = next((other for other, pixels in accepted
if image.shape == pixels.shape and cosine(image, pixels) >= DUPLICATE_COSINE), None)
if duplicate:
refused.append({"sku": sku, "view": name,
"reason": f"near-duplicate of {duplicate}"})
continue
accepted.append((name, image))
if len(accepted) < min_views:
for name, _ in accepted:
refused.append({"sku": sku, "view": name,
"reason": f"only {len(accepted)} usable view(s), {min_views} required"})
continue
kept[sku] = [name for name, _ in accepted]
return kept, refused
def unattributed(source: str) -> List[str]:
"""SKUs whose views have no entry in the source's ATTRIBUTION.json.
Reported, not refused. A view with no recorded source is still a usable
reference - the six that predate the attribution file are independent of
the evaluation photo and their OCR matches their catalog names - but the
provenance is gone, and provenance is what lets a later audit tell a
re-sourced packshot from a leaked shelf crop. Naming them keeps the gap
from growing quietly.
"""
path = os.path.join(source, "ATTRIBUTION.json")
try:
with open(path) as handle:
recorded = json.load(handle)
except (OSError, ValueError):
recorded = {}
return sorted(sku for sku in collect(source) if sku not in recorded)
def text_discriminated_siblings(
kept: Dict[str, List[str]], source: str, names: Optional[Dict[str, str]] = None
) -> List[Dict[str, Any]]:
"""Cross-SKU near-duplicates: the same artwork on two different products.
These are the near-unknown case no threshold rejects. The geometry is real
because the sibling shares the brand's artwork, so only the package text
separates them. They are kept - they are legitimately different products -
but flagged, because a catalog that enrolls one and not the other will
accept the missing one as the enrolled one.
"""
views: List[Tuple[str, str, np.ndarray]] = []
for sku, filenames in sorted(kept.items()):
for filename in filenames:
image = cv2.imread(os.path.join(source, filename))
if image is not None:
views.append((sku, filename, image))
flagged: List[Dict[str, Any]] = []
for index, (sku, filename, image) in enumerate(views):
for other_sku, other_name, other in views[index + 1:]:
if other_sku == sku or image.shape != other.shape:
continue
similarity = cosine(image, other)
if similarity < DUPLICATE_COSINE:
continue
flagged.append({
"sku": sku,
"view": filename,
"sibling_sku": other_sku,
"sibling_view": other_name,
"cosine": round(similarity, 4),
"name_similarity": round(
name_ocr_similarity(
(names or {}).get(sku, ""), (names or {}).get(other_sku, "")
), 3,
),
})
return flagged
def live_ready_verdict(
kept: Dict[str, List[str]], names: Optional[Dict[str, str]], shelf_skus: Optional[Set[str]]
) -> Dict[str, Any]:
"""The enrollment protocol's completeness rule, as a verdict.
A bay is live-ready when every SKU on its shelf tags is enrolled or
explicitly listed as text-only. Without a shelf-tag list the rule cannot be
evaluated, and the verdict says so rather than guessing.
"""
catalog = set(names or {})
enrolled = set(kept)
text_only = catalog - enrolled
if shelf_skus is None:
return {
"live_ready": None,
"reason": "no --shelf-tags given; the completeness rule needs the shelf's SKU list",
"enrolled": len(enrolled),
"text_only": sorted(text_only),
}
missing = sorted(shelf_skus - enrolled - text_only)
return {
"live_ready": not missing,
"reason": (
"every shelf SKU is enrolled or text-only"
if not missing
else "on the shelf but absent from the catalog: " + ", ".join(missing)
),
"enrolled": len(enrolled),
"text_only": sorted(text_only),
"shelf_skus": len(shelf_skus),
"missing_from_catalog": missing,
}
def load_shelf_skus(path: Optional[str]) -> Optional[Set[str]]:
"""SKUs on the shelf's tags: one per line, or a JSON list."""
if not path:
return None
with open(path, encoding="utf-8") as handle:
text = handle.read()
try:
payload = json.loads(text)
except json.JSONDecodeError:
return {line.strip() for line in text.splitlines() if line.strip()}
if isinstance(payload, list):
return {str(item).strip() for item in payload if str(item).strip()}
raise RebuildError(f"{path}: expected a JSON list or one SKU per line")
def write_catalog(kept: Dict[str, List[str]], source: str, output: str, csv_path: Optional[str]) -> None:
"""Copy the kept views and the whole CSV into a staging dir, then swap it in."""
staging = tempfile.mkdtemp(prefix=".catalog-", dir=os.path.dirname(os.path.abspath(output)) or ".")
try:
for views in kept.values():
for name in views:
shutil.copy2(os.path.join(source, name), os.path.join(staging, name))
if csv_path:
# Byte-for-byte, not re-serialized: gallery.catalog_digest hashes the
# raw file, and csv.DictWriter's default CRLF would move the digest
# and invalidate a trained identity head over a newline.
shutil.copy2(csv_path, os.path.join(staging, "stock.csv"))
if os.path.isdir(output):
# The documented rebuild runs with --source == --output, and the
# catalog directory also holds provenance this tool does not manage
# (README, references/rejected/, label files). Carry those over so a
# rebuild replaces the views and the CSV, not the record of where
# they came from.
for entry in os.listdir(output):
if entry == "stock.csv" or os.path.splitext(entry)[1].lower() in IMAGE_EXTENSIONS:
continue
origin, target = os.path.join(output, entry), os.path.join(staging, entry)
if os.path.isdir(origin):
shutil.copytree(origin, target)
else:
shutil.copy2(origin, target)
if os.path.exists(output):
shutil.rmtree(output)
os.replace(staging, output)
except BaseException:
shutil.rmtree(staging, ignore_errors=True)
raise
def main(argv: Optional[Sequence[str]] = None) -> int:
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--source", required=True, help="directory of pack shots")
parser.add_argument("--output", required=True, help="catalog directory to write")
parser.add_argument("--csv", help="stock.csv to validate and carry over whole")
parser.add_argument("--replace", action="store_true", help="overwrite an existing output")
parser.add_argument("--min-views", type=int, default=MIN_VIEWS,
help=f"views required per SKU (default {MIN_VIEWS}; "
"official pack-shot pages publish one front view per JAN)")
parser.add_argument("--ocr", choices=("auto", "vision", "none"), default=None,
help="packshot/name OCR backend (default: DORAYAKI_OCR, else vision "
"on macOS; an unavailable backend is skipped, not fatal)")
parser.add_argument("--source-manifest",
help="image capture metadata (default: SOURCE/manifest.json when present)")
parser.add_argument("--eval-manifest",
help="manifest of evaluation sessions; images captured in them are refused")
parser.add_argument("--shelf-tags",
help="the shelf's SKU list (JSON list or one per line) for the "
"completeness rule; without it the verdict is 'unknown'")
parser.add_argument("--json", help="write the audit here")
args = parser.parse_args(argv)
if os.path.exists(args.output) and not args.replace:
raise SystemExit(f"{args.output} exists; pass --replace to overwrite")
try:
if args.csv:
_, _, names = read_catalog(args.csv)
missing_rows = set(collect(args.source)) - set(names)
if missing_rows:
raise RebuildError("enrolled image SKU(s) with no CSV row: " + ", ".join(sorted(missing_rows)))
else:
names = {}
print("no --csv: image/CSV match and name/OCR checks skipped", file=sys.stderr)
source_manifest = args.source_manifest
default_manifest = os.path.join(args.source, "manifest.json")
if source_manifest is None and os.path.isfile(default_manifest):
source_manifest = default_manifest
capture_sessions = load_capture_sessions(source_manifest)
eval_sessions = load_eval_sessions(args.eval_manifest) if args.eval_manifest else set()
if eval_sessions:
unlabelled = sorted(name for views in collect(args.source).values() for name in views
if name not in capture_sessions)
if unlabelled:
print("eval-session check cannot cover image(s) with no capture_session_id: "
+ ", ".join(unlabelled), file=sys.stderr)
# Read the shelf list before anything is written: a bad path or a
# malformed list must fail here, with the tool's own message, rather
# than after write_catalog has already replaced the catalog.
shelf_skus = load_shelf_skus(args.shelf_tags)
engine = build_ocr_engine(args.ocr)
ocr_engine = engine if engine.available() else None
if ocr_engine is None:
print("name/OCR check skipped: no OCR backend available", file=sys.stderr)
kept, refused = audit(args.source, min_views=args.min_views, names=names,
ocr_engine=ocr_engine, capture_sessions=capture_sessions,
eval_sessions=eval_sessions)
if refused:
print(f"refused {len(refused)} view(s):", file=sys.stderr)
for row in refused:
print(f" {row['sku']:<10} {row['view']:<20} {row['reason']}", file=sys.stderr)
if not kept:
raise RebuildError("no SKU has usable views; nothing written")
write_catalog(kept, args.source, args.output, args.csv)
except (OSError, ValueError, json.JSONDecodeError, RebuildError, OcrUnavailable) as error:
raise SystemExit(str(error)) from error
# A row with no kept image is a text-only entry: the text lane still names
# and vetoes it from the CSV alone, so it is a coverage gap, not a refusal.
text_only = sorted(set(names) - set(kept))
siblings = text_discriminated_siblings(kept, args.source, names)
no_source = unattributed(args.source)
verdict = live_ready_verdict(kept, names, shelf_skus)
report = {
"source": args.source,
"output": args.output,
"min_width": MIN_WIDTH,
"min_height": MIN_HEIGHT,
"min_views": args.min_views,
"duplicate_cosine": DUPLICATE_COSINE,
"min_name_ocr": MIN_NAME_OCR,
"ocr": engine.name if ocr_engine is not None else "skipped",
"text_only_skus": text_only,
"text_discriminated_siblings": siblings,
"unattributed_skus": no_source,
"completeness": verdict,
"skus": {sku: views for sku, views in sorted(kept.items())},
"refused": refused,
}
if args.json:
with open(args.json, "w", encoding="utf-8") as handle:
json.dump(report, handle, indent=1, ensure_ascii=False)
print(f"{len(kept)} SKU(s), {sum(len(v) for v in kept.values())} view(s) -> {args.output}")
for sku, views in sorted(kept.items()):
print(f" {sku:<10} {len(views)} view(s): {', '.join(views)}")
if text_only:
print(f"\ntext-only CSV row(s), no enrolled image ({len(text_only)}): {', '.join(text_only)}")
if siblings:
print(f"\ntext-discriminated sibling(s) ({len(siblings)}):")
for row in siblings:
print(f" {row['sku']:<10} {row['view']:<20} ~ {row['sibling_sku']} "
f"({row['sibling_view']}) cosine {row['cosine']}")
if no_source:
print(f"\nno recorded source ({len(no_source)}): {', '.join(no_source)} - "
"usable, but provenance is missing; re-sourcing would close it")
if verdict["live_ready"] is None:
print(f"\ncompleteness: unknown - {verdict['reason']}")
elif verdict["live_ready"]:
print(f"\ncompleteness: live-ready ({verdict['reason']})")
else:
print(f"\ncompleteness: NOT live-ready - {verdict['reason']}")
if refused:
print(f"\nrefused {len(refused)} view(s):")
for row in refused:
print(f" {row['sku']:<10} {row['view']:<20} {row['reason']}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

920
recognization/code/reconcile.py
View file

@ -1,920 +0,0 @@
"""
Reconciliation (item vs price tag) and the first compliance rule family.
Two pure layers, no I/O:
- `reconcile()` compares one accepted identity with its tag and the catalog entry,
and publishes a state plus the individual signals. It never changes the product
identity: a disagreement is a second opinion, displayed next to the first.
- `evaluate_compliance()` turns settled evidence into findings. Every rule fails
closed on evidence quality: a row whose tags are merely unreadable produces a
capture hint, never a violation.
"""
from __future__ import annotations
import os
import re
import sys
import unicodedata
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from tags import TAG_STATE_ABSENT, TAG_STATE_READABLE, TAG_STATE_UNREADABLE # noqa: E402
RECONCILED_CONFIRMED = "confirmed"
RECONCILED_MISMATCH = "mismatch"
RECONCILED_UNVERIFIED = "unverified"
RECONCILED_NO_TAG = "no_tag"
RECONCILED_TAG_ONLY = "tag_only"
RULE_MISSING_TAG = "missing_price_tag"
RULE_MISSING_TAG_GROUP = "missing_tag_group"
# How far along a row a tag may name facings: enough to cover the product's own run
# of facings (measured: up to four facings, 70-120 px each), not enough to reach
# across the shelf. The cap exists because a product whose own tag was not detected
# would otherwise inherit whichever readable tag happens to be nearest.
TAG_REACH_FACINGS = 2.5
TAG_REACH_PX = 200
# The name similarity a tag needs before it may overrule the packaging channel.
TAG_OVERRIDE_SIMILARITY = 0.6
RULE_ORPHAN_TAG = "orphan_tag"
RULE_PRICE_MISMATCH = "price_mismatch"
RULE_DUPLICATE_TAG = "duplicate_tag"
RULE_UNREADABLE_TAGS = "unreadable_tags"
NAME_MATCH_THRESHOLD = 0.4
GROUP_MIN_FACINGS = 3
STABILITY_FRAMES = 3
STABILITY_WINDOW = 5
# ---------------------------------------------------------------------------
# text matching
# ---------------------------------------------------------------------------
def normalize_name(text: Optional[str]) -> str:
if not text:
return ""
text = unicodedata.normalize("NFKC", text)
text = re.sub(r"<[^>]*>", "", text)
text = re.sub(r"[\s・、。,.\-–—_/]+", "", text)
return text.lower()
def longest_common_substring_ratio(a: str, b: str) -> float:
"""Length of the longest common substring over the shorter string."""
if not a or not b:
return 0.0
shorter, longer = (a, b) if len(a) <= len(b) else (b, a)
best = 0
for start in range(len(shorter)):
for end in range(start + best + 1, len(shorter) + 1):
if shorter[start:end] in longer:
best = end - start
else:
break
return best / len(shorter)
# ---------------------------------------------------------------------------
# reconciliation
# ---------------------------------------------------------------------------
@dataclass
class Signal:
kind: str # "barcode" | "name" | "price"
agrees: bool
item: Any
tag: Any
note: Optional[str] = None
score: Optional[float] = None
def as_dict(self) -> Dict[str, Any]:
payload = {"kind": self.kind, "agrees": self.agrees, "item": self.item, "tag": self.tag}
if self.note:
payload["note"] = self.note
if self.score is not None:
payload["score"] = round(float(self.score), 3)
return payload
def reconcile(
identity: Optional[Dict[str, Any]],
tag: Optional[Dict[str, Any]],
product: Optional[Dict[str, Any]],
) -> Optional[Dict[str, Any]]:
"""Compare one facing with its tag.
Returns None when there is nothing to reconcile (no accepted identity and no
readable tag). A tag with no product above it is `tag_only`.
"""
identity_state = (identity or {}).get("state")
tag_state = (tag or {}).get("state", TAG_STATE_ABSENT)
accepted = identity_state == "recognized"
readable_tag = tag_state == TAG_STATE_READABLE
if not accepted and not readable_tag:
return None
if accepted and tag_state == TAG_STATE_ABSENT:
return {"state": RECONCILED_NO_TAG, "signals": [], "message": None}
if accepted and tag_state == TAG_STATE_UNREADABLE:
return {
"state": RECONCILED_UNVERIFIED,
"signals": [],
"message": "tag found but its text was not read",
}
if not accepted and readable_tag:
return {"state": RECONCILED_TAG_ONLY, "signals": [], "message": None}
signals: List[Signal] = []
# A barcode is the only exact signal; a checksummed code is not a guess. The
# number a tag prints large beside the sale price is NOT a barcode - it is the
# regular price (verified against the manufacturer: 4901577092381 for a tag
# printing 357), so it is compared as a price below.
item_barcode = str((product or {}).get("jan") or "").strip()
tag_barcode = str((tag or {}).get("barcode") or "").strip()
if item_barcode and tag_barcode:
signals.append(Signal("barcode", item_barcode == tag_barcode, item_barcode, tag_barcode))
item_name = normalize_name((product or {}).get("product_name"))
tag_name = normalize_name((tag or {}).get("name"))
if item_name and tag_name:
ratio = longest_common_substring_ratio(item_name, tag_name)
signals.append(
Signal("name", ratio >= NAME_MATCH_THRESHOLD, item_name[:32], tag_name[:32],
note=f"match {ratio:.2f}", score=ratio)
)
item_price = (product or {}).get("price")
tag_price = (tag or {}).get("price")
tag_regular = (tag or {}).get("regular_price")
if item_price is not None and (tag_price is not None or tag_regular is not None):
candidates = [value for value in (tag_price, tag_regular) if value is not None]
agrees = any(abs(float(item_price) - float(value)) < 0.5 for value in candidates)
signals.append(Signal("price", agrees, item_price, tag_price if tag_price is not None else tag_regular))
if not signals:
return {
"state": RECONCILED_UNVERIFIED,
"signals": [],
"message": "tag read, but nothing comparable in the catalog",
}
if any(signal.kind == "barcode" for signal in signals):
decided = [signal for signal in signals if signal.kind == "barcode"][0]
state = RECONCILED_CONFIRMED if decided.agrees else RECONCILED_MISMATCH
elif any(signal.agrees for signal in signals):
state = RECONCILED_CONFIRMED
elif all(not signal.agrees for signal in signals):
state = RECONCILED_MISMATCH
else:
state = RECONCILED_UNVERIFIED
message = None
if state == RECONCILED_MISMATCH:
price_signal = next((signal for signal in signals if signal.kind == "price"), None)
name_signal = next((signal for signal in signals if signal.kind == "name"), None)
if price_signal is not None:
message = f"tag says {price_signal.tag}, catalog says {price_signal.item}"
elif name_signal is not None:
message = "tag name does not match the catalog entry"
return {"state": state, "signals": [signal.as_dict() for signal in signals], "message": message}
# ---------------------------------------------------------------------------
# compliance
# ---------------------------------------------------------------------------
@dataclass
class ComplianceSettings:
group_min_facings: int = GROUP_MIN_FACINGS
stability_frames: int = STABILITY_FRAMES
stability_window: int = STABILITY_WINDOW
# A row may only be judged when enough of its facings carry tag evidence
# (readable or unreadable). Below this, the row is "not evaluated": claiming a
# tag is missing because our detector found none would be a false violation.
# 0.7 measured: at 0.5 a row where our own tag detection was patchy produced a
# false "group without price tag" on a shelf where every facing has a tag.
min_row_tag_coverage: float = 0.7
@dataclass
class Finding:
rule: str
severity: str # "violation" | "hint"
row_index: Optional[int]
box: List[int]
count: int = 1
skus: List[str] = field(default_factory=list)
detail: Optional[str] = None
def key(self) -> Tuple[Any, ...]:
return (self.rule, self.row_index, self.box[0] // 32, self.box[1] // 32, self.count)
def as_dict(self) -> Dict[str, Any]:
payload = {
"rule": self.rule,
"severity": self.severity,
"row_index": self.row_index,
"box": list(self.box),
"count": self.count,
"skus": list(self.skus),
}
if self.detail:
payload["detail"] = self.detail
return payload
class ComplianceTracker:
"""Keeps findings only while they persist, so the UI never flickers."""
def __init__(self, settings: Optional[ComplianceSettings] = None):
self.settings = settings or ComplianceSettings()
self._history: List[set] = []
def update(self, findings: Sequence[Finding]) -> List[Finding]:
current = {finding.key(): finding for finding in findings}
self._history.append(set(current))
if len(self._history) > self.settings.stability_window:
self._history.pop(0)
stable: List[Finding] = []
for key, finding in current.items():
seen = sum(1 for frame in self._history if key in frame)
if seen >= self.settings.stability_frames:
stable.append(finding)
return stable
def reset(self) -> None:
self._history.clear()
def _sku_of_track(track: Dict[str, Any]) -> Optional[str]:
identity = track.get("identity") or {}
if identity.get("state") == "recognized" and identity.get("sku_id"):
return str(identity["sku_id"])
return None
def group_facings(
row_tracks: Sequence[Dict[str, Any]],
max_gap_ratio: float = 0.35,
size_tolerance: float = 0.45,
) -> List[List[Dict[str, Any]]]:
"""Group a row's facings into products.
A store prints one price tag per *product*, not per facing: measured on the
demo shelf, a row carries 16-18 facings but only 4-5 tags, because the four
facings of one mayonnaise share a single tag. Counting facings therefore
invents missing tags on a shelf that is perfectly labelled, so the rules count
*groups*.
Identity wins over geometry when both facings were recognised: measured, a
run of ピュアセレクト facings was split by geometry alone because one facing
was detected narrower than its neighbours, and the product was then reported
as missing the tag that was in fact sitting further along its own run.
"""
ordered = sorted(
(track for track in row_tracks if track.get("box") or track.get("raw_box")),
key=lambda track: (track.get("box") or track.get("raw_box"))[0],
)
groups: List[List[Dict[str, Any]]] = []
for track in ordered:
box = track.get("box") or track.get("raw_box")
if groups:
previous = groups[-1][-1]
pbox = previous.get("box") or previous.get("raw_box")
gap = box[0] - pbox[2]
width = max(1, pbox[2] - pbox[0])
height = max(1, pbox[3] - pbox[1])
width2 = max(1, box[2] - box[0])
height2 = max(1, box[3] - box[1])
sku, previous_sku = _sku_of_track(track), _sku_of_track(previous)
close = gap <= max_gap_ratio * width
if sku and previous_sku:
same_product = sku == previous_sku and close
else:
same_product = close and (
abs(height2 - height) / height <= size_tolerance
and abs(width2 - width) / width <= size_tolerance
)
if same_product:
groups[-1].append(track)
continue
groups.append([track])
return groups
def cluster_into_rows(tracks: Sequence[Dict[str, Any]]) -> List[List[Dict[str, Any]]]:
"""Split facings into shelf rows by vertical overlap.
`group_facings` is a *within-row* operation: given the whole frame it would
merge the products stacked in one column across six shelves into a single
"product". Identity assignment needs rows first.
"""
rows: List[List[Dict[str, Any]]] = []
for track in sorted(
(track for track in tracks if track.get("box") or track.get("raw_box")),
key=lambda track: (track.get("box") or track.get("raw_box"))[1],
):
box = track.get("box") or track.get("raw_box")
placed = False
for row in rows:
row_box = _group_box(row)
overlap = min(row_box[3], box[3]) - max(row_box[1], box[1])
if overlap > 0.5 * min(row_box[3] - row_box[1], box[3] - box[1]):
row.append(track)
placed = True
break
if not placed:
rows.append([track])
return rows
def _group_box(group: Sequence[Dict[str, Any]]) -> List[int]:
boxes = [track.get("box") or track.get("raw_box") for track in group]
return [
min(box[0] for box in boxes),
min(box[1] for box in boxes),
max(box[2] for box in boxes),
max(box[3] for box in boxes),
]
def evaluate_compliance(
tracks: Sequence[Dict[str, Any]],
tag_by_track: Dict[int, Dict[str, Any]],
all_tags: Sequence[Dict[str, Any]],
bands: Sequence[Any],
reconciliations: Optional[Dict[int, Dict[str, Any]]] = None,
settings: Optional[ComplianceSettings] = None,
) -> List[Finding]:
"""Findings from settled evidence. Rules with unreadable evidence stay silent.
`tag_by_track` is the per-facing attachment; `all_tags` is every tag the pass
found, so a tag nobody claimed can be reported as an orphan.
"""
settings = settings or ComplianceSettings()
findings: List[Finding] = []
by_row: Dict[int, List[Dict[str, Any]]] = {}
for track in tracks:
row_index = _row_for(track.get("box") or track.get("raw_box"), bands)
by_row.setdefault(row_index, []).append(track)
for row_index, row_tracks in by_row.items():
row_tracks.sort(key=lambda track: (track.get("box") or [0])[0])
row_tags = [tag_by_track.get(int(track["track_id"]), {}) for track in row_tracks]
readable = [tag for tag in row_tags if tag.get("state") == TAG_STATE_READABLE]
evidenced = [
tag for tag in row_tags if tag.get("state") in (TAG_STATE_READABLE, TAG_STATE_UNREADABLE)
]
# Count products, not facings: one tag serves a run of identical facings.
products = group_facings(row_tracks)
product_tags = [
[
tag_by_track.get(int(track["track_id"]), {})
for track in group
]
for group in products
]
products_evidenced = [
group
for group, tags in zip(products, product_tags)
if any(tag.get("state") in (TAG_STATE_READABLE, TAG_STATE_UNREADABLE) for tag in tags)
]
coverage = len(products_evidenced) / max(1, len(products))
if not readable or coverage < settings.min_row_tag_coverage:
# Fail closed on evidence quality. Either nothing in this row could be
# read, or too few facings carry any tag evidence at all - in both cases
# the honest answer is "not evaluated", never "tag missing".
if evidenced:
detail = (
"move closer: the price tags in this row could not be read"
if not readable
else (
f"tags detected for {len(products_evidenced)} of {len(products)} products "
f"in this row"
)
)
findings.append(
Finding(
rule=RULE_UNREADABLE_TAGS,
severity="hint",
row_index=row_index,
box=_row_box(row_tracks),
count=len(row_tracks),
detail=detail,
)
)
continue
# missing_price_tag: one finding per row (the UI highlights the row), with
# the products that lack a tag listed rather than one finding per facing.
# A product's tag may sit anywhere along its run of facings, so a group
# only counts as missing when no tag exists for its SKU elsewhere in the
# row either. This is the domain rule, applied as a guard.
skus_with_tags = {
_sku_of_track(track)
for track, tag in zip(row_tracks, row_tags)
if tag.get("state") in (TAG_STATE_READABLE, TAG_STATE_UNREADABLE)
} - {None}
# A fragment of a product is not a product. Measured without identities: a
# facing detected at 54 px wide beside 150 px neighbours was split off and
# reported as missing the tag that belonged to its own run. Anything much
# narrower than the row's typical product is not judged.
widths = sorted(
max(1, (_group_box(group)[2] - _group_box(group)[0])) for group in products
)
typical_width = widths[len(widths) // 2] if widths else 0
missing_groups = [
group
for group, tags in zip(products, product_tags)
if all(tag.get("state") == TAG_STATE_ABSENT for tag in tags)
and not (_sku_of_track(group[0]) in skus_with_tags)
and (_group_box(group)[2] - _group_box(group)[0]) >= 0.6 * typical_width
]
if missing_groups:
findings.append(
Finding(
rule=RULE_MISSING_TAG,
severity="violation",
row_index=row_index,
box=_row_box([track for group in missing_groups for track in group]),
count=len(missing_groups),
skus=[_sku_of(group[0]) for group in missing_groups],
detail=(
f"{len(missing_groups)} of {len(products)} products in this row "
f"have no price tag"
),
)
)
# missing_tag_group: consecutive products without a tag
missing_ids = {id(group) for group in missing_groups}
run: List[List[Dict[str, Any]]] = []
for group, tags in zip(products, product_tags):
if id(group) in missing_ids:
run.append(group)
continue
if len(run) >= settings.group_min_facings:
findings.append(
_group_finding(row_index, [track for g in run for track in g], len(run))
)
run = []
if len(run) >= settings.group_min_facings:
findings.append(_group_finding(row_index, [track for g in run for track in g], len(run)))
# price_mismatch from reconciliation
for track in tracks:
track_id = int(track["track_id"])
reconciliation = (reconciliations or {}).get(track_id)
if not reconciliation or reconciliation.get("state") != RECONCILED_MISMATCH:
continue
signals = reconciliation.get("signals", [])
if not any(signal.get("kind") == "price" for signal in signals):
continue
# A mismatch needs one trustworthy anchor before it is a finding. Measured:
# a wrong identity plus a garbled tag read (name match 0.11, no JAN)
# produced a confident-looking "tag says 90, catalog says 188" that was
# really just noise. With an anchor - the tag's JAN, or a name that
# actually resembles the product - the disagreement is worth reporting.
tag_for_track = tag_by_track.get(track_id, {})
name_signal = next(
(signal for signal in signals if signal.get("kind") == "name"), None
)
# The anchor is the tag's own arithmetic agreeing, or a name that actually
# resembles the product. (It used to be `tag.jan`, which held a *price*.)
flags = set(tag_for_track.get("flags") or [])
corroborated = bool({"tax_consistent", "discount_consistent"} & flags)
name_score = float(name_signal.get("score") or 0.0) if name_signal else 0.0
has_anchor = corroborated or name_score >= 0.35
if not has_anchor:
findings.append(
Finding(
rule=RULE_UNREADABLE_TAGS,
severity="hint",
row_index=_row_for(track.get("box") or track.get("raw_box"), bands),
box=list(track.get("box") or track.get("raw_box") or [0, 0, 0, 0]),
count=1,
skus=[_sku_of(track)],
detail="the tag under this product could not be read well enough to compare",
)
)
continue
findings.append(
Finding(
rule=RULE_PRICE_MISMATCH,
severity="violation",
row_index=_row_for(track.get("box") or track.get("raw_box"), bands),
box=list(track.get("box") or track.get("raw_box") or [0, 0, 0, 0]),
count=1,
skus=[_sku_of(track)],
detail=reconciliation.get("message"),
)
)
# orphan_tag: a readable tag that no facing claims (a tag with no product above
# it is either an empty slot with a leftover tag or a mis-clustered read).
attached_ids = {
tag.get("id") for tag in tag_by_track.values() if tag.get("id")
}
for tag in all_tags:
if tag.get("state") != TAG_STATE_READABLE or not tag.get("id"):
continue
if tag["id"] in attached_ids:
continue
findings.append(
Finding(
rule=RULE_ORPHAN_TAG,
severity="violation",
row_index=None,
box=list(tag.get("box") or [0, 0, 0, 0]),
count=1,
detail="price tag with no product above it",
)
)
return findings
def _group_finding(row_index: int, run: Sequence[Dict[str, Any]], products: int = 0) -> Finding:
boxes = [track.get("box") or track.get("raw_box") or [0, 0, 0, 0] for track in run]
x1 = min(box[0] for box in boxes)
y1 = min(box[1] for box in boxes)
x2 = max(box[2] for box in boxes)
y2 = max(box[3] for box in boxes)
return Finding(
rule=RULE_MISSING_TAG_GROUP,
severity="violation",
row_index=row_index,
box=[x1, y1, x2, y2],
count=len(run),
skus=[_sku_of(track) for track in run],
detail=(
f"{products or len(run)} products in a row without a price tag"
),
)
def _row_box(tracks: Sequence[Dict[str, Any]]) -> List[int]:
boxes = [track.get("box") or track.get("raw_box") or [0, 0, 0, 0] for track in tracks]
return [
min(box[0] for box in boxes),
min(box[1] for box in boxes),
max(box[2] for box in boxes),
max(box[3] for box in boxes),
]
def _row_for(box: Optional[Sequence[int]], bands: Sequence[Any]) -> int:
"""Which rail band (i.e. row) a facing sits above."""
if not box or not bands:
return 0
x_center = (box[0] + box[2]) / 2.0
y_bottom = box[3]
best_index = 0
best_distance = None
for band in bands:
bx1, by1, bx2, by2 = band.box
if not (bx1 - 40 <= x_center <= bx2 + 40):
continue
distance = abs(by1 - y_bottom)
if best_distance is None or distance < best_distance:
best_distance = distance
best_index = band.row_index
return best_index
def _sku_of(track: Dict[str, Any]) -> str:
identity = track.get("identity") or {}
return str(identity.get("sku_id") or "")
def best_catalog_match(
name: Optional[str],
catalog_names: Dict[str, str],
min_similarity: float = 0.45,
) -> Tuple[Optional[str], float]:
"""The catalog entry a printed tag name most resembles."""
if not name:
return None, 0.0
normalized = normalize_name(name)
best_sku, best_score = None, 0.0
for sku_id, catalog_name in catalog_names.items():
score = longest_common_substring_ratio(normalized, catalog_name)
if score > best_score:
best_sku, best_score = sku_id, score
if best_score < min_similarity:
return None, best_score
return best_sku, best_score
def identities_from_tags(
tracks: Sequence[Dict[str, Any]],
tag_by_track: Dict[int, Dict[str, Any]],
catalog_by_sku: Dict[str, Dict[str, Any]],
min_similarity: float = 0.40,
all_tags: Optional[Sequence[Dict[str, Any]]] = None,
) -> Dict[int, Dict[str, Any]]:
"""Identify a product from the tag printed under it.
The shelf-edge tag carries the store's own name for the product in clean
printed text, and one tag serves a whole run of facings. That makes it the
most reliable identity source on the shelf - far more so than recognising the
packaging, which is fine-grained (twenty near-identical cartons), occluded,
and distorted by glare and angle.
Measured with an off-the-shelf DINOv2 embedding: identical products do cluster
(two orange juices, two apple juices, two almond drinks) but fine-grained
cases do not - 32 cartons collapsed into one cluster and nine different
bottled drinks grouped by *container shape*.
Assignment is by *tag position*, not by geometric grouping: the tags on a rail
mark where each product sits, so a facing belongs to the nearest tag along the
row. Grouping facings by size and spacing was measured to merge different
products (a 945 px "product" spanning half a shelf), which then swallowed the
wrong tag.
"""
catalog_names = {
sku_id: normalize_name(product.get("product_name"))
for sku_id, product in catalog_by_sku.items()
if product.get("product_name")
}
if not catalog_names:
return {}
tags = [dict(tag) for tag in (all_tags or []) if tag.get("box")]
for tag in tags:
if tag.get("sku_from_name") is None:
sku_id, score = best_catalog_match(tag.get("name"), catalog_names, min_similarity)
tag["sku_from_name"] = sku_id
tag["sku_score"] = score
if not tags:
return {}
found: Dict[int, Dict[str, Any]] = {}
for row in cluster_into_rows(tracks):
row_box = _group_box(row)
row_height = max(1, row_box[3] - row_box[1])
# The rail's tags sit just under the row they describe.
row_tags = [
tag
for tag in tags
if row_box[3] - 0.5 * (tag["box"][3] - tag["box"][1])
<= tag["box"][1]
<= row_box[3] + 1.5 * (tag["box"][3] - tag["box"][1])
]
if not row_tags:
continue
for track in row:
box = track.get("box") or track.get("raw_box")
center = (box[0] + box[2]) / 2.0
facing_width = max(1, box[2] - box[0])
# A tag names the product at its own position on the rail. Without a
# distance cap a facing whose own tag was not detected inherits the
# nearest readable tag's product, which can belong to a different
# product several columns away.
limit = max(TAG_REACH_FACINGS * facing_width, TAG_REACH_PX)
candidates = [
tag for tag in row_tags
if abs(center - (tag["box"][0] + tag["box"][2]) / 2.0) <= limit
]
if not candidates:
continue
nearest = min(
candidates,
key=lambda tag: abs(center - (tag["box"][0] + tag["box"][2]) / 2.0),
)
sku_id = nearest.get("sku_from_name")
if not sku_id:
continue
found[int(track["track_id"])] = {
"sku_id": sku_id,
"score": round(float(nearest.get("sku_score") or 0.0), 3),
"source": "tag_name",
}
return found
def attach_tags_to_boxes(
tag_service: Any,
image: Any,
boxes: Sequence[Dict[str, Any]],
catalog_by_sku: Dict[str, Dict[str, Any]],
identities: Optional[Dict[int, Dict[str, Any]]] = None,
) -> Dict[str, Any]:
"""One-shot tag pass for a still image (photo mode / `?tags=true`).
Runs the passes synchronously - a still is not a stream, so there is no frame
loop to protect - then attaches tags, reconciles, and evaluates compliance.
"""
image_shape = image.shape
# A still has no later frame to pick up the remaining bands, and `plan` only
# hands out a few per call (a stream must stay responsive). Drain them, with a
# cap so a huge shelf cannot turn one request into minutes of OCR. Draining
# also feeds the vote: a band read in two rounds is two observations.
bands: List[Any] = []
rounds = 0
while rounds < 4:
planned = tag_service.plan(boxes, image_shape)
if not planned:
break
for band in planned:
# Two independent reads (different upscale) so the vote has evidence
# to work with; a single read has nothing to vote against.
tag_service.run_pass(image, band, scale_bias=1.0)
tag_service.run_pass(image, band, scale_bias=1.7)
bands.append(band)
rounds += 1
tracks = [
{
"track_id": int(box["id"]),
"box": list(box["box"]),
"identity": (identities or {}).get(int(box["id"])) or {},
"tag_identity": {},
}
for box in boxes
]
tag_by_track, all_tags, counts = tag_service.attach(tracks)
# Tag-first identity, applied before reconciliation so the comparison below
# sees the identity the tag itself provides.
tag_identities = identities_from_tags(
tracks, tag_by_track, catalog_by_sku, all_tags=all_tags
)
for track in tracks:
derived = tag_identities.get(int(track["track_id"]))
if not derived:
continue
# Always keep the tag's claim, even when packaging recognition already
# named the product: two independent identity channels that can be
# compared are worth more than one that silently wins. A disagreement is
# itself a finding (a misplaced tag, or a product in the wrong place).
track["tag_identity"] = dict(derived)
current = track["identity"] or {}
if current.get("state") == "recognized":
if current.get("sku_id") != derived["sku_id"]:
# Only a corroborated tag, or a name that closely resembles the
# product, may overrule the packaging channel. A weak 0.40 name
# match used to be enough, which is not evidence.
tag_flags = set((tag_by_track.get(int(track["track_id"])) or {}).get("flags") or [])
strong = bool({"tax_consistent", "discount_consistent"} & tag_flags) or \
float(derived.get("score") or 0.0) >= TAG_OVERRIDE_SIMILARITY
if not strong:
track["identity_conflict"] = {
"packaging": current.get("sku_id"),
"tag": derived["sku_id"],
"packaging_score": current.get("score"),
"note": "tag name match too weak to overrule packaging",
}
continue
# Two channels disagree. Measured on the demo shelf: where both
# named the same facing they agreed only 69% of the time, and the
# packaging side of every disagreement carried a low score - so
# the store's own label wins, and the disagreement is recorded
# rather than hidden (a misplaced tag is a real finding).
track["identity_conflict"] = {
"packaging": current.get("sku_id"),
"tag": derived["sku_id"],
"packaging_score": current.get("score"),
}
track["identity"] = {
"state": "recognized",
"sku_id": derived["sku_id"],
"identity_level": "tag_over_packaging",
"confidence": derived["score"],
"score": derived["score"],
"source": "tag_name",
"candidates": [],
"attempts": 1,
"method": "tag_name",
}
continue
track["identity"] = {
"state": "recognized",
"sku_id": derived["sku_id"],
"identity_level": "tag_derived",
"confidence": derived["score"],
"score": derived["score"],
"source": "tag_name",
"candidates": [],
"attempts": 1,
"method": "tag_name",
}
reconciliations: Dict[int, Dict[str, Any]] = {}
for track in tracks:
track_id = track["track_id"]
tag = tag_by_track.get(track_id)
product = catalog_by_sku.get((track["identity"] or {}).get("sku_id") or "")
decision = reconcile(track["identity"], tag, product)
if decision is not None:
reconciliations[track_id] = decision
findings = evaluate_compliance(tracks, tag_by_track, all_tags, bands, reconciliations)
summary = summarize_findings(findings)
return {
"by_box": tag_by_track,
"tags": all_tags,
"counts": counts,
"reconciliations": reconciliations,
"tag_identities": tag_identities,
"bands": [{"row_index": band.row_index, "box": band.box} for band in bands],
"bands_objects": list(bands),
"compliance": {
"violations": summary["violations"],
"by_rule": summary["by_rule"],
"rows": [
{
"row_index": finding.row_index,
"rule": finding.rule,
"box": finding.box,
"missing": finding.count,
"skus": finding.skus,
}
for finding in findings
if finding.severity == "violation"
],
"hints": summary["hints"],
},
}
def reconcile_bundle(
bundle: Dict[str, Any],
detections: Sequence[Dict[str, Any]],
catalog_by_sku: Dict[str, Dict[str, Any]],
) -> Dict[str, Any]:
"""Re-run reconciliation and compliance against a cached tag bundle.
Used when identities arrive after the tag pass (photo mode: recognize paints
boxes, classify names them). Without this, every tag would look like `tag_only`
because it was compared against an identity that did not exist yet.
"""
tracks = [
{
"track_id": int(detection["id"]),
"box": list(detection["box"]),
"identity": detection.get("identity") or {},
}
for detection in detections
]
tag_by_track = bundle.get("by_box") or {}
reconciliations: Dict[int, Dict[str, Any]] = {}
for track in tracks:
track_id = track["track_id"]
tag = tag_by_track.get(track_id)
product = catalog_by_sku.get((track["identity"] or {}).get("sku_id") or "")
decision = reconcile(track["identity"], tag, product)
if decision is not None:
reconciliations[track_id] = decision
findings = evaluate_compliance(
tracks, tag_by_track, bundle.get("tags") or [], bundle.get("bands_objects") or [], reconciliations
)
summary = summarize_findings(findings)
return {
"reconciliations": reconciliations,
"compliance": {
"violations": summary["violations"],
"by_rule": summary["by_rule"],
"rows": [
{
"row_index": finding.row_index,
"rule": finding.rule,
"box": finding.box,
"missing": finding.count,
"skus": finding.skus,
}
for finding in findings
if finding.severity == "violation"
],
"hints": summary["hints"],
},
}
def summarize_findings(findings: Sequence[Finding]) -> Dict[str, Any]:
by_rule: Dict[str, int] = {}
for finding in findings:
if finding.severity != "violation":
continue
by_rule[finding.rule] = by_rule.get(finding.rule, 0) + 1
return {
"violations": sum(by_rule.values()),
"by_rule": by_rule,
"hints": [finding.as_dict() for finding in findings if finding.severity != "violation"],
}

147
recognization/code/record.py
View file

@ -1,147 +0,0 @@
"""
Record a live session so it can become data.
Everything measured so far comes from one or two photographs, which is the largest
weakness in the project. The fix is a store walk recorded from the device that will
do the scanning, with enough context to label it afterwards.
`--record-frames DIR` writes each processed frame plus a JSON line describing what
the server saw in it. The lines are the labelling input: a facing that the tag
channel named is a candidate label, and the confirmation screen is where a human
accepts or corrects it. Nothing here is a training set on its own - it is the raw
material for one.
Layout:
DIR/<session>/000123.jpg the frame, exactly as the client sent it
DIR/<session>/index.jsonl one line per frame:
{"frame": 123, "path": "...", "at": 1699..., "identity_backend": "gallery",
"summary": {...},
"tracks": [{"box": [...], "identity": {...}, "tag": {...}}]}
Frames are written at most every `every` processed frames (default 3) so a long
walk does not fill a disk, and the index keeps the frame's own numbering so the
sampling is visible rather than silent.
"""
from __future__ import annotations
import json
import os
import threading
import time
from typing import Any, Dict, List, Optional
class FrameRecorder:
"""Write frames and their results to disk, for labelling later."""
def __init__(
self,
root: str,
every: int = 3,
max_frames_per_session: int = 4000,
identity_backend: Optional[str] = None,
) -> None:
self.root = root
self.every = max(1, int(every))
self.max_frames_per_session = int(max_frames_per_session)
self.identity_backend = (identity_backend or "").strip().lower() or None
self._lock = threading.Lock()
self._counts: Dict[str, int] = {}
self._written: Dict[str, int] = {}
os.makedirs(self.root, exist_ok=True)
def _session_dir(self, session_id: str) -> str:
path = os.path.join(self.root, session_id)
os.makedirs(path, exist_ok=True)
return path
def write(self, session_id: str, frame_id: int, payload: bytes, result: Dict[str, Any]) -> None:
"""Record one processed frame and what was seen in it."""
with self._lock:
seen = self._counts.get(session_id, 0) + 1
self._counts[session_id] = seen
if seen % self.every:
return
written = self._written.get(session_id, 0)
if written >= self.max_frames_per_session:
return
self._written[session_id] = written + 1
directory = self._session_dir(session_id)
name = f"{written:06d}.jpg"
try:
with open(os.path.join(directory, name), "wb") as handle:
handle.write(payload)
except OSError:
return
tracks: List[Dict[str, Any]] = []
for track in result.get("tracks") or []:
tracks.append(
{
"box": track.get("box"),
"identity": track.get("identity"),
"tag_identity": track.get("tag_identity"),
"tag": track.get("price_tag"),
}
)
line = {
"frame": frame_id,
"path": name,
"at": round(time.time(), 3),
"summary": result.get("summary"),
"tracks": tracks,
}
if self.identity_backend:
line["identity_backend"] = self.identity_backend
try:
with open(os.path.join(directory, "index.jsonl"), "a", encoding="utf-8") as handle:
handle.write(json.dumps(line, ensure_ascii=False) + "\n")
except OSError:
return
def stats(self) -> Dict[str, Any]:
with self._lock:
return {
"root": self.root,
"every": self.every,
"sessions": len(self._counts),
"frames_written": sum(self._written.values()),
}
def labelling_candidates(index_path: str) -> List[Dict[str, Any]]:
"""Turn a recorded index into candidate labels for a human to confirm.
A facing that the tag channel named carries a product name and a corroborated
price; those are the seeds. The rest are candidates grouped by position, to be
labelled in the confirmation screen rather than by hand from scratch.
"""
seeds: List[Dict[str, Any]] = []
with open(index_path, "r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line:
continue
entry = json.loads(line)
for track in entry.get("tracks") or []:
tag_identity = track.get("tag_identity") or {}
tag = track.get("tag") or {}
if not tag_identity.get("sku_id"):
continue
seeds.append(
{
"frame": entry["frame"],
"box": track.get("box"),
"sku_id": tag_identity["sku_id"],
"name_similarity": tag_identity.get("score"),
"tag_price": tag.get("price"),
"corroborated": bool(
{"tax_consistent", "tax_pair_consistent", "discount_consistent"}
& set(tag.get("flags") or [])
),
}
)
return seeds

328
recognization/code/score_stream.py
View file

@ -1,328 +0,0 @@
"""Score a streamed clip's labels against ground truth: the G1 number.
The stream path was reported as "53 of 131 tracks", which is not comparable to
Van's 57% of labelled facings - the two count different things. But the clip is
built from `add/image6.jpg`, which has labels, so matching each confirmed
track's `label_box` to the labelled facings gives stream-path coverage directly.
python code/score_stream.py TRACKS.json --labels eval/labels/image6.json
`TRACKS.json` is what `stream_client.py --tracks` writes: one row per track that
settled on a label, with the box that label was measured on, already mapped back
to source-image coordinates.
Reported per run: correct / in-catalog facings, precision, unknown accepts, and
tracks that matched no labelled facing. A track whose box overlaps a facing the
catalog does not enrol is an unenrolled false accept - the dangerous kind, and
the one the evaluator's own precision denominator used to hide.
G1 has two halves - new covers at least as much as Van, and the lower bound of
the paired-bootstrap difference clears -2 pp - and the second half needs the
per-facing pairing, so every row carries the `facing` index it matched. Two
reports plus the labels then answer the gate on the same facings:
python code/score_stream.py NEW.json --van VAN.json
"""
from __future__ import annotations
import argparse
import json
import os
import random
import sys
from typing import Any, Dict, List, Optional, Sequence, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
# A facing clipped by the window edge is not something a run could have named,
# so the denominator keeps only facings at least this far inside the window.
VISIBLE_MIN = 0.5
# G1's second half: the lower bound of the paired difference must clear -2 pp.
G1_BOUND_MIN = -0.02
# Resampling draws for the paired bootstrap, and the seed that makes the bound
# a number this repo can print twice.
BOOTSTRAP_DRAWS = 10000
BOOTSTRAP_SEED = 0
def iou(a: Sequence[int], b: Sequence[int]) -> float:
ax1, ay1, ax2, ay2 = a
bx1, by1, bx2, by2 = b
inter_w = max(0, min(ax2, bx2) - max(ax1, bx1))
inter_h = max(0, min(ay2, by2) - max(ay1, by1))
inter = inter_w * inter_h
area_a = max(0, ax2 - ax1) * max(0, ay2 - ay1)
area_b = max(0, bx2 - bx1) * max(0, by2 - by1)
union = area_a + area_b - inter
return inter / union if union > 0 else 0.0
def load_facings(labels_path: str) -> List[Dict[str, Any]]:
with open(labels_path) as handle:
payload = json.load(handle)
return payload["facings"]
def enrolled_skus(stock_dir: str) -> set:
"""Canonical SKUs with at least one enrolled view."""
from rebuild_catalog import sku_of
return {
sku_of(name) for name in os.listdir(stock_dir)
if name.lower().endswith((".jpg", ".jpeg", ".png"))
}
def visible_fraction(box: Sequence[int], window: Sequence[int]) -> float:
"""Share of a facing's area inside the window the client actually showed."""
wx0, wy0, wx1, wy1 = window
bx0, by0, bx1, by1 = box
inter_w = max(0, min(bx1, wx1) - max(bx0, wx0))
inter_h = max(0, min(by1, wy1) - max(by0, wy0))
area = max(0, bx1 - bx0) * max(0, by1 - by0)
return (inter_w * inter_h) / area if area > 0 else 0.0
def reachable(facings: Sequence[Dict[str, Any]],
windows: Optional[Sequence[Sequence[int]]]
) -> List[Dict[str, Any]]:
"""Facings a run's windows showed well enough for the run to have named."""
if not windows:
return list(facings)
return [f for f in facings
if max(visible_fraction(f["box"], w) for w in windows) >= VISIBLE_MIN]
def correct_facings(rows: Sequence[Dict[str, Any]]) -> set:
"""Facing indices at least one track got right - one hit each, not one per track."""
return {row["facing"] for row in rows
if row["outcome"] == "correct" and row["facing"] is not None}
def score(tracks: Sequence[Dict[str, Any]], facings: Sequence[Dict[str, Any]],
stock_dir: str, iou_min: float,
windows: Optional[Sequence[Sequence[int]]] = None
) -> Dict[str, Any]:
"""Score the labels a stream settled on.
`windows` are the source-image rects the client showed, as
[x0, y0, x1, y1] each. The denominator is the facings at least half inside
one of them, in x and y both: a 1280x720 window over a 1500x2000 photo sees
24 of the 63 in-catalog facings, and a facing clipped by the window edge is
not something any run could have named. A panned run shows a different rect
each frame, so the reachable set is the union. Without it the denominator is
every in-catalog facing.
A correct answer is counted before the unenrolled and unknown branches. The
text lane names SKUs the image lane has no view for, which is the point of
it, and scoring that as a false accept punishes the lane for working.
"""
facings = reachable(facings, windows)
in_catalog = [f for f in facings if f["label"] != "unknown"]
enrolled = enrolled_skus(stock_dir)
correct = 0
wrong = 0
unenrolled_false = 0
unknown_false = 0
unmatched = 0
rows: List[Dict[str, Any]] = []
# Coverage is per facing, precision is per label. A track that restarts on
# the same product, or two tracks on one facing, is two claims but one
# facing: counting tracks as coverage reports 138% of a 24-facing window.
named: Dict[int, str] = {}
for track in tracks:
box = track.get("label_box") or []
if len(box) != 4:
unmatched += 1
continue
best, best_iou = None, 0.0
for facing in facings:
value = iou(box, facing["box"])
if value > best_iou:
best, best_iou = facing, value
if best is None or best_iou < iou_min:
unmatched += 1
rows.append({"track_id": track["track_id"], "predicted": track["sku_id"],
"facing": None, "truth": None, "iou": round(best_iou, 3),
"outcome": "unmatched"})
continue
truth = best["label"]
predicted = track["sku_id"]
if predicted == truth:
correct += 1
outcome = "correct"
named[best["index"]] = predicted
elif truth == "unknown":
unknown_false += 1
outcome = "unknown_false_accept"
elif truth not in enrolled:
unenrolled_false += 1
outcome = "unenrolled_false_accept"
else:
wrong += 1
outcome = "wrong"
rows.append({"track_id": track["track_id"], "predicted": predicted,
"facing": best["index"], "truth": truth,
"iou": round(best_iou, 3), "outcome": outcome})
shown = correct + wrong + unenrolled_false + unknown_false
return {
"in_catalog_facings": len(in_catalog),
"facings_named": len(named),
"tracks_scored": len(tracks),
"correct": correct,
"wrong": wrong,
"unenrolled_false_accepts": unenrolled_false,
"unknown_false_accepts": unknown_false,
"unmatched_tracks": unmatched,
"shown": shown,
"coverage": round(len(named) / len(in_catalog), 4) if in_catalog else 0.0,
"precision": round(correct / shown, 4) if shown else 0.0,
# The window set travels with the report: a paired comparison has to
# know what each run was shown, not just what it named.
"windows": [list(w) for w in windows] if windows else None,
"rows": rows,
}
def paired_bootstrap(van: Dict[str, Any], new: Dict[str, Any],
facings: Sequence[Dict[str, Any]],
draws: int = BOOTSTRAP_DRAWS,
seed: int = BOOTSTRAP_SEED) -> Dict[str, Any]:
"""G1's second half: new minus Van, per facing, with the 95 % lower bound.
Both reports are graded on one facing set - the in-catalog facings either
run's windows showed - so a facing one run panned past cannot flatter the
other. Resampling draws facings with replacement rather than comparing two
independent rates, because the two runs saw the same photo: pairing removes
the facing-to-facing variance that otherwise swamps a 27-facing sample.
Percentile method on the sorted draw differences; the seed is fixed so the
number in a report is one this repo reproduces.
"""
windows = list(van.get("windows") or []) + list(new.get("windows") or [])
shared = sorted(f["index"] for f in reachable(facings, windows)
if f["label"] != "unknown")
if not shared:
return {"facings": 0, "difference": 0.0, "lower_95": 0.0, "upper_95": 0.0,
"van": 0, "new": 0, "draws": draws, "seed": seed}
van_hits = correct_facings(van["rows"])
new_hits = correct_facings(new["rows"])
van_vector = [1 if i in van_hits else 0 for i in shared]
new_vector = [1 if i in new_hits else 0 for i in shared]
n = len(shared)
rng = random.Random(seed)
spread = []
for _ in range(draws):
van_sum = new_sum = 0
for _ in range(n):
# The same draw index into both vectors: that is the pairing.
drawn = rng.randrange(n)
van_sum += van_vector[drawn]
new_sum += new_vector[drawn]
spread.append((new_sum - van_sum) / n)
spread.sort()
tail = max(1, int(0.025 * draws))
return {
"facings": n,
"van": sum(van_vector),
"new": sum(new_vector),
"van_rate": round(sum(van_vector) / n, 4),
"new_rate": round(sum(new_vector) / n, 4),
"difference": round((sum(new_vector) - sum(van_vector)) / n, 4),
"lower_95": round(spread[tail - 1], 4),
"upper_95": round(spread[min(draws - 1, int(0.975 * draws))], 4),
"draws": draws,
"seed": seed,
}
def load_run(path: str, window: Optional[str] = None
) -> Tuple[Dict[str, Any], Optional[List[Tuple[int, ...]]]]:
"""A stream_client --tracks dump and the source-image windows it showed."""
with open(path) as handle:
payload = json.load(handle)
if window:
windows = [tuple(int(part) for part in window.split(","))]
elif payload.get("windows"):
windows = [tuple(w) for w in payload["windows"]]
else:
windows = None
return payload, windows
def main(argv: Optional[Sequence[str]] = None) -> int:
parser = argparse.ArgumentParser(description="Score streamed labels against ground truth")
parser.add_argument("tracks", help="JSON written by stream_client.py --tracks")
parser.add_argument("--labels", default="eval/labels/image6.json")
parser.add_argument("--stock-dir", default="stock-lopia-clean")
parser.add_argument("--iou", type=float, default=0.5)
parser.add_argument("--json", dest="json_path", default=None)
parser.add_argument("--window", default=None,
help="x0,y0,x1,y1 the client showed; the denominator is "
"restricted to facings at least half inside it")
parser.add_argument("--van", default=None,
help="Van baseline track dump on the same clip; scores G1's "
"paired-bootstrap half against it")
parser.add_argument("--draws", type=int, default=BOOTSTRAP_DRAWS)
parser.add_argument("--seed", type=int, default=BOOTSTRAP_SEED)
parser.add_argument("--verbose", action="store_true", help="print every scored track")
args = parser.parse_args(argv)
facings = load_facings(args.labels)
payload, windows = load_run(args.tracks, args.window)
report = score(payload["tracks"], facings, args.stock_dir, args.iou, windows)
report["source"] = payload.get("source")
report["pace"] = payload.get("pace")
report["seconds"] = payload.get("seconds")
print(f"{os.path.basename(args.tracks)} pace={report['pace']} "
f"{report['seconds']}s")
print(f" facings named {report['facings_named']}/{report['in_catalog_facings']} "
f"in-catalog ({report['coverage']:.0%}) labels correct {report['correct']} "
f"precision {report['precision']:.0%} shown {report['shown']}")
print(f" wrong {report['wrong']} unenrolled false accepts "
f"{report['unenrolled_false_accepts']} unknown false accepts "
f"{report['unknown_false_accepts']} unmatched tracks {report['unmatched_tracks']}")
if args.verbose:
for row in report["rows"]:
print(f" t{row['track_id']:4d} {row['outcome']:24s} "
f"facing={row['facing']} pred={row['predicted']} "
f"truth={row['truth']} iou={row['iou']}")
if args.van:
van_payload, van_windows = load_run(args.van, args.window)
van_report = score(van_payload["tracks"], facings, args.stock_dir,
args.iou, van_windows)
paired = paired_bootstrap(van_report, report, facings, args.draws, args.seed)
report["van"] = {k: van_report[k] for k in
("in_catalog_facings", "facings_named", "coverage",
"correct", "wrong", "precision", "shown")}
report["g1_paired"] = paired
print(f" G1 vs {os.path.basename(args.van)}: new {paired['new']}/"
f"{paired['facings']} ({paired['new_rate']:.0%}) - van {paired['van']}/"
f"{paired['facings']} ({paired['van_rate']:.0%}) = "
f"{paired['difference'] * 100:+.1f} pp")
print(f" paired bootstrap {paired['draws']} draws seed {paired['seed']}: "
f"95% bounds {paired['lower_95'] * 100:+.1f} pp to "
f"{paired['upper_95'] * 100:+.1f} pp; lower bound clears "
f"{G1_BOUND_MIN * 100:+.0f} pp: "
f"{'yes' if paired['lower_95'] >= G1_BOUND_MIN else 'NO'}")
if args.json_path:
with open(args.json_path, "w") as handle:
json.dump(report, handle, indent=2)
return 0
if __name__ == "__main__":
sys.exit(main())

348
recognization/code/shelf_map.py
View file

@ -1,348 +0,0 @@
"""
The shelf as a plane, not a stream of boxes.
Every measurement so far treats a frame as the world: a facing is "a track id",
and a track id dies the moment the camera pans past and comes back. Label
inheritance, drift checks and restart counters were each correct fixes for that,
and all of them are symptoms. The shelf does not move; the camera does.
This module keeps a per-session **shelf map**. Each frame is registered to the map
by its content shift, every detection is placed in shelf coordinates, and a
detection that lands where a facing already is *merges* into it. Identity votes and
tag readings accumulate on the physical facing rather than on a transient track.
That is what the current literature does for retail: RetailGlue (CVPR 2026W) builds
shelf panoramas by matching products across frames and reports 88.4 % stitching F1,
and the 7-Eleven Taiwan system (Sci. Rep. 2025, 7,000+ stores) stitches "virtual
shelves" before checking planograms.
**What is measured, and what is not.** The synthetic walk registers to within
0.01 px of its ground truth - but it is a pure translation of a single photograph,
so that validates the harness, not handheld video. The number that matters (real
registration error, and the duplicate rate that follows from it) needs real
footage, which `record.py` exists to collect.
"""
from __future__ import annotations
import threading
import uuid
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Sequence, Tuple
# A frame whose content shift cannot be estimated this confidently is treated as
# discontinuous: the map starts a new section rather than guessing a position.
MIN_REGISTRATION_RESPONSE = 0.20
# Two detections are the same physical facing when their shelf boxes overlap this
# much. Measured: adjacent facings of one product overlap ~0 by construction.
MERGE_IOU = 0.45
def _iou(a: Sequence[int], b: Sequence[int]) -> float:
x1, y1 = max(a[0], b[0]), max(a[1], b[1])
x2, y2 = min(a[2], b[2]), min(a[3], b[3])
intersection = max(0, x2 - x1) * max(0, y2 - y1)
area_a = max(0, a[2] - a[0]) * max(0, a[3] - a[1])
area_b = max(0, b[2] - b[0]) * max(0, b[3] - b[1])
return intersection / max(1.0, float(area_a + area_b - intersection))
@dataclass
class Facing:
"""One physical position on the shelf, with everything observed there."""
facing_id: str
shelf_box: List[int]
row: int = 0
observations: int = 0
first_seq: int = 0
last_seq: int = 0
identity_votes: Dict[str, float] = field(default_factory=dict)
tag_votes: Dict[str, float] = field(default_factory=dict)
tag_readings: List[Dict[str, Any]] = field(default_factory=list)
seen_track_ids: set = field(default_factory=set)
def vote_identity(self, sku_id: Optional[str], weight: float = 1.0) -> None:
if sku_id:
self.identity_votes[sku_id] = self.identity_votes.get(sku_id, 0.0) + weight
def vote_tag(self, sku_id: Optional[str], reading: Optional[Dict[str, Any]] = None) -> None:
if sku_id:
self.tag_votes[sku_id] = self.tag_votes.get(sku_id, 0.0) + 1.0
if reading:
self.tag_readings.append(reading)
def best_identity(self) -> Tuple[Optional[str], float]:
"""The identity the facing's pooled evidence supports, and its agreement."""
if not self.identity_votes:
return None, 0.0
sku_id, weight = max(self.identity_votes.items(), key=lambda item: item[1])
total = sum(self.identity_votes.values())
return sku_id, weight / max(1e-6, total)
def best_tag_identity(self) -> Tuple[Optional[str], float]:
if not self.tag_votes:
return None, 0.0
sku_id, weight = max(self.tag_votes.items(), key=lambda item: item[1])
total = sum(self.tag_votes.values())
return sku_id, weight / max(1e-6, total)
def as_payload(self) -> Dict[str, Any]:
identity, agreement = self.best_identity()
tag_identity, tag_agreement = self.best_tag_identity()
return {
"facing_id": self.facing_id,
"box": list(self.shelf_box),
"row": self.row,
"observations": self.observations,
"track_ids": len(self.seen_track_ids),
"sku_id": identity,
"identity_agreement": round(agreement, 3),
"tag_sku_id": tag_identity,
"tag_agreement": round(tag_agreement, 3),
"tag_readings": self.tag_readings[-3:],
}
class ShelfMap:
"""Registers frames and accumulates facings in shelf coordinates."""
def __init__(self, settings: Optional[Any] = None) -> None:
self._lock = threading.Lock()
self._prev_gray: Optional[Any] = None
self._offset: List[float] = [0.0, 0.0] # shelf position of the frame origin
self._facings: List[Facing] = []
self._orb = None
self.registrations = 0
self.registration_failures = 0
self.anchors = 0
self.continuations = 0
self.detections = 0
self.merges = 0
self.responses: List[float] = []
# ------------------------------------------------------------- registration
def _gray(self, frame: Any, scale: float = 0.5) -> Any:
import cv2
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
if scale != 1.0:
gray = cv2.resize(gray, None, fx=scale, fy=scale, interpolation=cv2.INTER_AREA)
return gray
def register(self, frame: Any) -> Optional[Tuple[float, float]]:
"""Content shift from the previous frame, in full-resolution pixels.
Phase correlation first: it is the right tool for a translating scene and
costs a couple of milliseconds. ORB + RANSAC only when the correlation is
weak (blur, a large jump, a scene with no texture), because that is the case
where a fast estimator quietly returns nonsense.
"""
import cv2
import numpy as np
gray = self._gray(frame)
with self._lock:
previous = self._prev_gray
self._prev_gray = gray
if previous is None or previous.shape != gray.shape:
return (0.0, 0.0)
window = cv2.createHanningWindow((gray.shape[1], gray.shape[0]), cv2.CV_32F)
(dx, dy), response = cv2.phaseCorrelate(previous.astype(np.float32), gray.astype(np.float32), window)
self.responses.append(float(response))
if response >= MIN_REGISTRATION_RESPONSE:
self.registrations += 1
return (dx / 0.5, dy / 0.5) # back to full resolution
shift = self._orb_shift(previous, gray)
if shift is None:
self.registration_failures += 1
return None
self.registrations += 1
return (shift[0] / 0.5, shift[1] / 0.5)
def _orb_shift(self, previous: Any, current: Any) -> Optional[Tuple[float, float]]:
import cv2
import numpy as np
if self._orb is None:
self._orb = cv2.ORB_create(nfeatures=1500, fastThreshold=10)
key_a, desc_a = self._orb.detectAndCompute(previous, None)
key_b, desc_b = self._orb.detectAndCompute(current, None)
if desc_a is None or desc_b is None or len(key_a) < 20 or len(key_b) < 20:
return None
matches = cv2.BFMatcher(cv2.NORM_HAMMING).knnMatch(desc_a, desc_b, k=2)
good = [m for m, n in matches if m.distance < 0.75 * n.distance]
if len(good) < 12:
return None
src = np.float32([key_a[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
dst = np.float32([key_b[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)
matrix, _ = cv2.estimateAffinePartial2D(src, dst, method=cv2.RANSAC, ransacReprojThreshold=3.0)
if matrix is None:
return None
return (float(matrix[0, 2]), float(matrix[1, 2]))
# ------------------------------------------------------------------ facings
def _anchor_to_map(
self, detections: Sequence[Sequence[int]], offset_x: float, offset_y: float
) -> Tuple[float, float]:
"""Correct the chained offset by matching detections against the map.
Chaining pairwise shifts drifts: measured over a 79-step walk with *perfect*
pairwise registration, the accumulated x offset reached 2047 px on a 1500 px
shelf, the same facing landed in two places, and a shelf of ~60 facings
became 334 with 113 of them seen once. Drift is inherent to chaining - the
fix is to anchor the frame to the structure already on the map, which is
what the retail-stitching literature does with product correspondences.
One ICP-style step is enough: for every detection that has a facing nearby,
the correction it implies is a vote, and the median vote wins.
"""
if not self._facings or len(detections) < 4:
return offset_x, offset_y
corrections = []
search = 40.0
for box in detections:
centre_x = (box[0] + box[2]) / 2.0 + offset_x
centre_y = (box[1] + box[3]) / 2.0 + offset_y
best, best_distance = None, search
for facing in self._facings:
facing_x = (facing.shelf_box[0] + facing.shelf_box[2]) / 2.0
facing_y = (facing.shelf_box[1] + facing.shelf_box[3]) / 2.0
distance = max(abs(facing_x - centre_x), abs(facing_y - centre_y))
if distance < best_distance:
best, best_distance = facing, distance
if best is not None:
corrections.append(
(
(best.shelf_box[0] + best.shelf_box[2]) / 2.0 - ((box[0] + box[2]) / 2.0 + offset_x),
(best.shelf_box[1] + best.shelf_box[3]) / 2.0 - ((box[1] + box[3]) / 2.0 + offset_y),
)
)
if len(corrections) < 4:
return offset_x, offset_y
dx = sorted(c[0] for c in corrections)[len(corrections) // 2]
dy = sorted(c[1] for c in corrections)[len(corrections) // 2]
self.anchors += 1
return offset_x + dx, offset_y + dy
def update(self, seq: int, frame: Any, tracks: Sequence[Dict[str, Any]]) -> None:
"""Register the frame, then place or merge every detection on the shelf."""
shift = self.register(frame)
boxes = [
[int(v) for v in (track.get("box") or track.get("raw_box"))]
for track in tracks
if (track.get("box") or track.get("raw_box")) and len(track.get("box") or track.get("raw_box")) == 4
]
with self._lock:
if shift is None:
# A discontinuous frame: keep the last known offset rather than
# inventing one, so its detections land where the shelf is.
shift = (0.0, 0.0)
# `shift` is how far the CONTENT moved in the frame, so the frame's
# origin moves the opposite way along the shelf. Adding it instead
# (measured) walks the map backwards: the same facing lands 50 px away
# in the next frame and a shelf of ~60 facings becomes 857.
self._offset[0] -= shift[0]
self._offset[1] -= shift[1]
if self._facings:
self._offset[0], self._offset[1] = self._anchor_to_map(
boxes, self._offset[0], self._offset[1]
)
offset_x, offset_y = self._offset
for track in tracks:
box = track.get("box") or track.get("raw_box")
if not box or len(box) != 4:
continue
self.detections += 1
shelf_box = [
int(round(box[0] + offset_x)), int(round(box[1] + offset_y)),
int(round(box[2] + offset_x)), int(round(box[3] + offset_y)),
]
track_id = track.get("track_id")
facing = self._find(shelf_box, int(track_id) if track_id is not None else None)
if facing is None:
facing = Facing(
facing_id=f"face_{uuid.uuid4().hex[:8]}",
shelf_box=shelf_box,
first_seq=seq,
)
self._facings.append(facing)
else:
self.merges += 1
# Average the box: repeated observations of a static object
# should converge, and the average is what the eye expects.
for index in range(4):
facing.shelf_box[index] = int(round((facing.shelf_box[index] * facing.observations + shelf_box[index]) / (facing.observations + 1)))
facing.observations += 1
facing.last_seq = seq
identity = track.get("identity") or {}
if identity.get("state") == "recognized":
facing.vote_identity(identity.get("sku_id"), float(identity.get("score") or 1.0))
tag_identity = track.get("tag_identity") or {}
tag = track.get("price_tag") or {}
if tag_identity.get("sku_id") or tag.get("state") == "readable":
facing.vote_tag(tag_identity.get("sku_id"), tag if tag else None)
if track.get("track_id") is not None:
facing.seen_track_ids.add(int(track["track_id"]))
def _find(self, shelf_box: Sequence[int], track_id: Optional[int] = None) -> Optional[Facing]:
"""The facing a detection belongs to.
Continuity first: if this track id was already placed nearby, it is the same
facing, and no overlap test is needed - the tracker already decided that.
Geometry only when the id is new (a pan that outran the tracker, or the
tracker renumbering). Measured: with geometry alone, box jitter split the
same facing often enough that the map held 210 facings where the tracker
held 156 ids - it was adding duplicates, not removing them.
"""
width = max(1, shelf_box[2] - shelf_box[0])
height = max(1, shelf_box[3] - shelf_box[1])
if track_id is not None:
radius = 0.75 * max(width, height)
best, best_distance = None, radius
for facing in self._facings:
if track_id not in facing.seen_track_ids:
continue
distance = max(
abs((facing.shelf_box[0] + facing.shelf_box[2]) / 2.0 - (shelf_box[0] + shelf_box[2]) / 2.0),
abs((facing.shelf_box[1] + facing.shelf_box[3]) / 2.0 - (shelf_box[1] + shelf_box[3]) / 2.0),
)
if distance < best_distance:
best, best_distance = facing, distance
if best is not None:
self.continuations += 1
return best
best, best_overlap = None, MERGE_IOU
for facing in self._facings:
overlap = _iou(shelf_box, facing.shelf_box)
if overlap > best_overlap:
best, best_overlap = facing, overlap
return best
# ------------------------------------------------------------------ reporting
def facings(self) -> List[Facing]:
with self._lock:
return list(self._facings)
def stats(self) -> Dict[str, Any]:
with self._lock:
facings = list(self._facings)
responses = list(self.responses[-200:])
consistent = 0
for facing in facings:
sku_id, agreement = facing.best_identity()
if sku_id and agreement >= 0.8:
consistent += 1
return {
"facings": len(facings),
"detections": self.detections,
"merges": self.merges,
"continuations": self.continuations,
"registrations": self.registrations,
"anchored": self.anchors,
"registration_failures": self.registration_failures,
"median_response": round(sorted(responses)[len(responses) // 2], 3) if responses else None,
"identity_consistent": consistent,
"offset": [round(value, 1) for value in self._offset],
}

365
recognization/code/stream_client.py
View file

@ -1,365 +0,0 @@
"""
Reference live client for the Dorayaki recognition server.
Streams frames over WebSocket /v1/stream and prints what the server sees. This
is the documented example for capture sources that are not a browser: a cart
camera, a wearable camera, a rail camera, a robot.
Usage (from recognization/):
.venv/bin/python code/stream_client.py --source image.jpg --pan 12 --seconds 20
.venv/bin/python code/stream_client.py --source 0 # webcam index
.venv/bin/python code/stream_client.py --source clip.mp4 --seconds 30
.venv/bin/python code/stream_client.py --url ws://host:8080/v1/stream --token SECRET
Frame protocol: 4-byte big-endian frame id + JPEG bytes.
"""
from __future__ import annotations
import argparse
import json
import os
import statistics
import sys
import time
from typing import Any, Dict, List, Optional
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(current_dir)
if os.path.exists(os.path.join(project_root, "stock")):
os.chdir(project_root)
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
from websockets.sync.client import connect # noqa: E402
def percentile(values: List[float], pct: float) -> float:
if not values:
return 0.0
ordered = sorted(values)
index = min(len(ordered) - 1, max(0, int(round((pct / 100.0) * (len(ordered) - 1)))))
return ordered[index]
class FrameSource:
"""Yields BGR frames from a camera, a video file, or a still image pan."""
def __init__(self, source: str, *, width: int, pan_step: int, pan_width: int = 1280):
self.source = source
self.width = width
self.pan_step = pan_step
self.pan_width = pan_width
self.capture: Optional[cv2.VideoCapture] = None
self.image = None
self.pan_x = 0
self.pan_direction = 1
# The downscale `frame()` applied to the window. A box the server
# measured is in the sent frame's pixels, so mapping it back to the
# source divides by this.
self.scale = 1.0
# Where the last frame was cut from the source image. (0, 0) for a
# camera or a video, whose frames are the whole image.
self.offset = (0, 0)
# The source-image rects the window showed, so an evaluator can restrict
# its denominator to the facings this run could actually have seen.
# A list, because a panned run shows a different rect each frame and the
# reachable set is their union. Both axes matter: a 1280-wide window
# over a 1500-wide photo is x-limited as well as y-limited.
self.windows: List[Tuple[int, int, int, int]] = []
if source.isdigit():
self.capture = cv2.VideoCapture(int(source))
elif os.path.isfile(source) and source.lower().endswith((".mp4", ".mov", ".avi", ".mkv", ".webm")):
self.capture = cv2.VideoCapture(source)
else:
self.image = cv2.imread(source)
if self.image is None:
raise SystemExit(f"cannot read source {source}")
def frame(self, index: int):
"""One frame, plus where it was cut from the source image.
The offset is what lets an evaluator map a track's box back to the
source image's coordinates: a label measured on a pan window is only
comparable to ground truth once the window's origin is added back.
"""
if self.image is not None:
height, width = self.image.shape[:2]
# Full height, 9:16 wide: the same window the app's replay source
# sends. A window only `pan_width` tall shows one shelf row, which
# reads as a matcher failure when it is only the crop.
window_h = height
window_w = min(int(window_h * 9 / 16), width)
max_x = max(0, width - window_w)
x = int(max(0, min(self.pan_x, max_x)))
window = self.image[:, x:x + window_w]
# Scale to the send size here, so the frame this yields is exactly
# what the app sends and `encode` has nothing left to do. The pan
# step is in sent-frame pixels, so it is divided by this scale.
self.scale = min(1.0, self.width / float(window_w)) if self.width else 1.0
if self.scale < 1.0:
window = cv2.resize(
window,
(int(round(window.shape[1] * self.scale)), int(round(window.shape[0] * self.scale))),
interpolation=cv2.INTER_AREA,
)
self.offset = (x, 0)
self.windows.append((x, 0, x + window_w, window_h))
self.pan_x += self.pan_direction * self.pan_step / self.scale
# Clamp the accumulator, not just the slice origin: a negative
# `pan_x` would satisfy `<= 0` on every later frame and flip the
# direction each time, parking the window at the left edge.
if self.pan_x >= max_x:
self.pan_x = float(max_x)
self.pan_direction = -1
elif self.pan_x <= 0:
self.pan_x = 0.0
self.pan_direction = 1
return window
if self.capture is not None:
ok, frame = self.capture.read()
if not ok: # loop video files
self.capture.set(cv2.CAP_PROP_POS_FRAMES, 0)
ok, frame = self.capture.read()
if not ok:
return None
return frame
return None
def release(self) -> None:
if self.capture is not None:
self.capture.release()
def encode(frame, width: int, quality: int) -> bytes:
height, current_width = frame.shape[:2]
if width and current_width > width:
scale = width / float(current_width)
frame = cv2.resize(frame, (int(width), int(round(height * scale))))
ok, buffer = cv2.imencode(".jpg", frame, [cv2.IMWRITE_JPEG_QUALITY, quality])
if not ok:
raise RuntimeError("JPEG encode failed")
return buffer.tobytes()
def main() -> None:
parser = argparse.ArgumentParser(description="Dorayaki live stream client")
parser.add_argument("--url", default="ws://127.0.0.1:8080/v1/stream")
parser.add_argument("--token", default=os.environ.get("DORAYAKI_API_TOKEN"))
parser.add_argument("--source", default="image.jpg", help="camera index, video path, or image path")
parser.add_argument("--pan", type=float, default=12.0, help="pixels per frame when panning a still image")
parser.add_argument("--seconds", type=float, default=20.0)
parser.add_argument("--json", action="store_true", help="print a JSON summary instead of the table")
parser.add_argument("--tracks", default=None,
help="write per-track labels and their boxes (source-image "
"coordinates) as JSON, for scoring against ground truth")
parser.add_argument("--quiet", action="store_true", help="only print the final summary")
parser.add_argument("--width", type=int, default=1280, help="send width")
parser.add_argument("--quality", type=int, default=75)
parser.add_argument("--max-inflight", type=int, default=1)
parser.add_argument("--pace", choices=["send", "result"], default="send",
help="'send': keep the socket busy (frames drop server-side); "
"'result': send the next frame only after a result, so the "
"pan stays continuous (the mode to use for stability checks)")
parser.add_argument("--method", default=None, help="surf | orb | brisk | hybrid")
args = parser.parse_args()
url = args.url
headers = {"Authorization": f"Bearer {args.token}"} if args.token else None
source = FrameSource(args.source, width=args.width, pan_step=args.pan)
rtts: List[float] = []
detect_times: List[float] = []
frames_sent = 0
results = 0
drops = 0
first_label: Dict[int, float] = {}
label_latencies: List[float] = []
labels: Dict[int, str] = {}
pending_frame: Optional[int] = None
# Per track: the label it settled on, the box that label was measured on,
# and the pan offset of the frame that box lives in. Written out with
# --tracks so an evaluator can match labels to ground truth.
track_records: Dict[int, Dict[str, Any]] = {}
frame_offsets: Dict[int, Tuple[int, int]] = {}
# The scale `encode` applied to that frame. A box the server measured is in
# the sent frame's pixels, so mapping it back to the source needs both the
# pan offset and this factor; without it a `--width` below the source size
# shrinks every box and the evaluator matches the wrong facing.
frame_scales: Dict[int, float] = {}
label_changes = 0
unexplained_changes = 0
relabel_seen: Dict[int, int] = {}
restart_seen: Dict[int, int] = {}
inherited_seen: Dict[int, bool] = {}
seen_tracks: set = set()
last_summary: Dict[str, Any] = {}
started = time.perf_counter()
in_flight: Dict[int, float] = {}
pending_frame: Optional[int] = None
with connect(url, max_size=8 * 1024 * 1024, additional_headers=headers) as socket:
hello = json.loads(socket.recv(timeout=10))
if hello.get("type") != "hello":
raise SystemExit(f"unexpected first message: {hello}")
if not args.quiet:
print(f"hello: session {hello['session_id']} model {hello['model']['version']} "
f"device {hello['model']['device']} config {hello['config']}")
if args.method:
socket.send(json.dumps({"type": "config", "method": args.method}))
ack = json.loads(socket.recv(timeout=10))
if not args.quiet:
print(f"config ack: {ack.get('config')}")
def consume(message: Dict[str, Any]) -> None:
"""Fold one result into the running metrics."""
nonlocal results, last_summary, label_changes, unexplained_changes
if message.get("type") != "result":
if not args.quiet:
print(f"server message: {message}")
return
received_at = time.perf_counter()
sent_at = in_flight.pop(message["frame_id"], None)
if sent_at is not None:
rtts.append((received_at - sent_at) * 1000.0)
detect_times.append(message["timing_ms"]["detect"])
results += 1
last_summary = message["summary"]
now = time.perf_counter()
for track in message["tracks"]:
track_id = track["track_id"]
seen_tracks.add(track_id)
identity = track["identity"]
if identity["state"] != "recognized":
continue
sku = identity["sku_id"]
relabels = track.get("relabels", 0)
restarts = track.get("restarts", 0)
if restarts > restart_seen.get(track_id, 0):
# The id was restarted: earlier labels describe another object.
labels.pop(track_id, None)
first_label.pop(track_id, None)
inherited = bool(track.get("inherited"))
if track_id not in first_label:
first_label[track_id] = now
if not inherited:
label_latencies.append((now - started) * 1000.0)
# Only a measured-to-measured change on the same object is a
# label change; inherited labels describe the same product
# under a new track id.
if track_id in labels and labels[track_id] != sku:
if not inherited and not inherited_seen.get(track_id):
label_changes += 1
if relabels <= relabel_seen.get(track_id, 0):
unexplained_changes += 1
labels[track_id] = sku
# The box the label is about, in source-image coordinates. The
# offset comes from the frame the box was measured on, not the
# frame this result arrived with: the pan has moved on since.
offset = frame_offsets.get(track.get("label_frame_id"), (0, 0))
inv = 1.0 / frame_scales.get(track.get("label_frame_id"), 1.0)
label_box = track.get("label_box") or track.get("box") or []
if len(label_box) == 4:
track_records[track_id] = {
"track_id": track_id,
"sku_id": sku,
"label_box": [
label_box[0] * inv + offset[0], label_box[1] * inv + offset[1],
label_box[2] * inv + offset[0], label_box[3] * inv + offset[1],
],
"label_frame_id": track.get("label_frame_id"),
"result_frame_id": message["frame_id"],
"inherited": inherited,
"restarts": restarts,
}
inherited_seen[track_id] = inherited
relabel_seen[track_id] = relabels
restart_seen[track_id] = restarts
if not args.quiet and results % 10 == 0:
print(f"f{message['seq']:04d} tracks={len(message['tracks']):3d} "
f"recognized={last_summary['recognized']:3d} "
f"unconfirmed={last_summary.get('unconfirmed', 0):3d} "
f"detect={message['timing_ms']['detect']:5.1f}ms "
f"rtt={rtts[-1]:6.1f}ms fps={results / (now - started):5.2f}")
frame_id = 0
index = 0
while time.perf_counter() - started < args.seconds:
frame = source.frame(index)
if frame is None:
break
index += 1
frame_id = (frame_id + 1) & 0xFFFFFFFF
frame_offsets[frame_id] = source.offset
frame_scales[frame_id] = source.scale
payload = encode(frame, args.width, args.quality)
if len(in_flight) >= args.max_inflight:
if pending_frame is not None:
drops += 1
pending_frame = frame_id
socket.send(frame_id.to_bytes(4, "big") + payload, text=False)
frames_sent += 1
else:
socket.send(frame_id.to_bytes(4, "big") + payload, text=False)
frames_sent += 1
in_flight[frame_id] = time.perf_counter()
if args.pace == "result":
# One frame in flight: the next frame is captured only after this
# one was answered, so the pan stays continuous.
while in_flight:
try:
consume(json.loads(socket.recv(timeout=5)))
except TimeoutError:
print("timeout waiting for a result")
break
continue
try:
consume(json.loads(socket.recv(timeout=5)))
except TimeoutError:
print("timeout waiting for a result")
break
source.release()
elapsed = time.perf_counter() - started
summary = {
"seconds": round(elapsed, 1),
"frames_sent": frames_sent,
"results": results,
"results_per_s": round(results / elapsed, 2) if elapsed else 0,
"frames_dropped_by_client": drops,
"rtt_ms_p50": round(statistics.median(rtts), 1) if rtts else None,
"rtt_ms_p95": round(percentile(rtts, 95), 1) if rtts else None,
"server_detect_ms_p50": round(statistics.median(detect_times), 1) if detect_times else None,
"server_detect_ms_p95": round(percentile(detect_times, 95), 1) if detect_times else None,
"tracks_seen": len(seen_tracks),
"tracks_labelled": len(labels),
"label_changes": label_changes,
"unexplained_label_changes": unexplained_changes,
"first_label_ms_p50": round(statistics.median(label_latencies), 0) if label_latencies else None,
"final_summary": last_summary,
}
if args.tracks:
with open(args.tracks, "w") as handle:
json.dump({
"windows": [list(w) for w in source.windows] or None,
"source": args.source,
"pace": args.pace,
"seconds": round(elapsed, 1),
"tracks": sorted(track_records.values(), key=lambda row: row["track_id"]),
}, handle, indent=2)
if args.json:
print(json.dumps(summary, indent=2))
else:
print("\n--- stream summary ---")
for key, value in summary.items():
if key != "final_summary":
print(f"{key:24s} {value}")
print(f"{'final_summary':24s} {json.dumps(last_summary, ensure_ascii=False)}")
if results == 0:
sys.exit(1)
if __name__ == "__main__":
main()

84
recognization/code/suft.py Normal file
View file

@ -0,0 +1,84 @@
import cv2
import matplotlib.pyplot as plt
import numpy as np
def suft_function(image):
image1 = cv2.imread(image)
# Convert the training image to RGB
training_image = cv2.cvtColor(image1, cv2.COLOR_BGR2RGB)
# Convert the training image to gray scale
training_gray = cv2.cvtColor(training_image, cv2.COLOR_RGB2GRAY)
# Create test image by adding Scale Invariance and Rotational Invariance
test_image = cv2.pyrDown(training_image)
test_image = cv2.pyrDown(test_image)
num_rows, num_cols = test_image.shape[:2]
rotation_matrix = cv2.getRotationMatrix2D((num_cols/2, num_rows/2), 30, 1)
test_image = cv2.warpAffine(test_image, rotation_matrix, (num_cols, num_rows))
test_gray = cv2.cvtColor(test_image, cv2.COLOR_RGB2GRAY)
# Display traning image and testing image
fx, plots = plt.subplots(1, 2, figsize=(20,10))
plots[0].set_title("Training Image")
plots[0].imshow(training_image)
plots[1].set_title("Testing Image")
plots[1].imshow(test_image)
# ## Detect keypoints and Create Descriptor
surf = cv2.xfeatures2d.SURF_create(800)
train_keypoints, train_descriptor = surf.detectAndCompute(training_gray, None)
test_keypoints, test_descriptor = surf.detectAndCompute(test_gray, None)
keypoints_without_size = np.copy(training_image)
keypoints_with_size = np.copy(training_image)
cv2.drawKeypoints(training_image, train_keypoints, keypoints_without_size, color = (0, 255, 0))
cv2.drawKeypoints(training_image, train_keypoints, keypoints_with_size, flags = cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
# Display image with and without keypoints size
fx, plots = plt.subplots(1, 2, figsize=(20,10))
plots[0].set_title("Train keypoints With Size")
plots[0].imshow(keypoints_with_size, cmap='gray')
plots[1].set_title("Train keypoints Without Size")
plots[1].imshow(keypoints_without_size, cmap='gray')
# Print the number of keypoints detected in the training image
print("Number of Keypoints Detected In The Training Image: ", len(train_keypoints))
# Print the number of keypoints detected in the query image
print("Number of Keypoints Detected In The Query Image: ", len(test_keypoints))
# ## Matching Keypoints
# Create a Brute Force Matcher object.
bf = cv2.BFMatcher(cv2.NORM_L1, crossCheck = False)
# Perform the matching between the SURF descriptors of the training image and the test image
matches = bf.match(train_descriptor, test_descriptor)
# The matches with shorter distance are the ones we want.
matches = sorted(matches, key = lambda x : x.distance)
result = cv2.drawMatches(training_image, train_keypoints, test_gray, test_keypoints, matches, test_gray, flags = 2)
# Display the best matching points
plt.rcParams['figure.figsize'] = [14.0, 7.0]
plt.title('Best Matching Points')
plt.imshow(result)
plt.show()
# Print total number of matching points between the training and query images
print("\nNumber of Matching Keypoints Between The Training and Query Images: ", len(matches))
return keypoints_with_size

225
recognization/code/sweep_accept_lines.py
View file

@ -1,225 +0,0 @@
"""Joint sweep of the two open-set gates: LOCAL_ACCEPT x TEXT_MIN_SHARED.
Both lines were fitted on image6, where every crop is a facing of a product the
shelf actually holds. That set can only measure false accepts on the products it
happens to contain, so it cannot price a line on its own - the far-unknown rate
comes from `data/external-shelves/*`: Wikimedia Commons shelf photos that share
nothing with the catalog, where every accept is a false accept by construction.
Each cell is produced by calling the shipped `DecisionPolicy` with that cell's
values and the catalog signatures, never by re-implementing the gate in the
sweep: a sweep that scores its own copy of the policy reports on a machine the
server does not run. `TEXT_MIN_SHARED` is a floor on the evidence behind the
text lane's *answer*. It is deliberately not a floor on the score - zeroing a
row reorders the ranking, so the margin gate would compare a different pair -
and deliberately not a floor on the veto, which only ever refuses. The constant
in `identity_lanes.py` says the same thing next to the code that does it.
The distractor side is measured the way the phone measures it - a 1280-long-side
frame at `LiveSettings.conf`, the path `code/eval_distractors.py` runs. The
shelf is scored at both resolutions: the full-resolution column is the one the
published table quotes, the live column is what the phone sees.
RootSIFT and OCR dominate the runtime and neither depends on the gates, so the
per-crop evidence is cached. Without `--cache` the sweep rebuilds it every run
and prints the same numbers, only slower.
.venv/bin/python code/sweep_accept_lines.py \\
--cache data/evaluations/2026-09-29-session/scratch/accept_line_cache.pkl
"""
from __future__ import annotations
import argparse
import glob
import os
import pickle
import sys
from typing import Any, Dict, List, Optional, Sequence, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
from eval_identity import ( # noqa: E402
EXCLUDE,
LIVE_LONG_SIDE,
UNKNOWN,
LocalLaneAdapter,
at_live_resolution,
crop_of,
load_labels,
)
from identity_lanes import LOCAL_ACCEPT, TEXT_MIN_SHARED, DecisionPolicy # noqa: E402
SHELF_LABELS = "eval/labels/image6.json"
SHELF_IMAGE = "add/image6.jpg"
DISTRACTOR_DIRS = (
"data/external-shelves/wikimedia-commons",
"data/external-shelves/wikimedia-commons-2",
)
LOCAL_ACCEPTS = (2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0)
TEXT_MIN_SHAREDS = (2, 3, 4, 5, 6)
CACHE_VERSION = 3
# One crop's evidence: the local lane's ranking and the text lane's raw read.
Evidence = Tuple[Any, str]
# One facing: its ground-truth label plus that evidence.
Facing = Tuple[str, Any, str]
def _evidence(lane: LocalLaneAdapter, frame: Any, box: Sequence[int]) -> Optional[Evidence]:
"""Rank and read one crop, or None when the box is off the frame."""
crop = crop_of(frame, box)
if crop.size == 0:
return None
return lane.rank(crop), lane.text_lane.read(crop)
def build_cache(stock_dir: str) -> Dict[str, Any]:
"""Every crop's evidence, gathered once and swept many times."""
from engine import EngineSettings, build_default_engine
from live import LiveSettings
lane = LocalLaneAdapter(stock_dir)
image = cv2.imread(SHELF_IMAGE)
live = at_live_resolution(image)
factor = live.shape[1] / float(image.shape[1])
shelf_full: List[Facing] = []
shelf_live: List[Facing] = []
for facing in load_labels(SHELF_LABELS)["facings"]:
label = facing.get("label")
if label in (None, EXCLUDE):
continue
full = _evidence(lane, image, facing["box"])
if full is None:
continue
shelf_full.append((label, full[0], full[1]))
# A facing the live frame cuts to nothing is dropped rather than
# scored at full resolution: the two columns have to be the same crops
# at two sizes, or the resolution is not the only variable.
small = _evidence(lane, live, [int(round(v * factor)) for v in facing["box"]])
if small is not None:
shelf_live.append((label, small[0], small[1]))
engine = build_default_engine(EngineSettings(stock_dir=stock_dir))
conf = LiveSettings().conf
distractors: List[Evidence] = []
photos = 0
for directory in DISTRACTOR_DIRS:
paths = sorted(
path for path in glob.glob(os.path.join(directory, "*"))
if os.path.splitext(path)[1].lower() in (".jpg", ".jpeg", ".png")
)
for path in paths:
shelf_photo = cv2.imread(path)
if shelf_photo is None:
continue
photos += 1
frame = at_live_resolution(shelf_photo)
for detection in engine.detect(frame, conf=conf):
box = detection.get("box") or detection.get("bbox")
if not box:
continue
item = _evidence(lane, frame, [int(v) for v in box])
if item is not None:
distractors.append(item)
return {
"version": CACHE_VERSION,
"conf": conf,
"photos": photos,
"shelf_full": shelf_full,
"shelf_live": shelf_live,
"distractors": distractors,
}
def _score(lane: LocalLaneAdapter, policy: DecisionPolicy, facings: Sequence[Facing]) -> Tuple[int, int]:
"""Shown and correct labels for one shelf column under one policy."""
shown = correct = 0
for label, ranked, read in facings:
text = sorted(
lane.text_lane.scores_from_text(read).items(), key=lambda kv: kv[1], reverse=True
)
answer = policy.decide(ranked, text, text_read=read)["best_sku"]
if answer:
shown += 1
correct += answer == label
return shown, correct
def _false_accepts(lane: LocalLaneAdapter, policy: DecisionPolicy,
distractors: Sequence[Evidence]) -> int:
"""Every accept on an unrelated shelf is a false accept by construction."""
accepts = 0
for ranked, read in distractors:
text = sorted(
lane.text_lane.scores_from_text(read).items(), key=lambda kv: kv[1], reverse=True
)
if policy.decide(ranked, text, text_read=read)["best_sku"]:
accepts += 1
return accepts
def main(argv: Optional[List[str]] = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--stock", default="stock-lopia-clean")
parser.add_argument("--cache", default="", help="build once, reuse on later runs")
parser.add_argument("--rebuild", action="store_true", help="ignore an existing cache")
args = parser.parse_args(argv)
cache: Optional[Dict[str, Any]] = None
if args.cache and os.path.exists(args.cache) and not args.rebuild:
with open(args.cache, "rb") as handle:
cache = pickle.load(handle)
if cache.get("version") != CACHE_VERSION:
cache = None
if cache is None:
cache = build_cache(args.stock)
if args.cache:
os.makedirs(os.path.dirname(os.path.abspath(args.cache)), exist_ok=True)
with open(args.cache, "wb") as handle:
pickle.dump(cache, handle)
lane = LocalLaneAdapter(args.stock)
shelf_full: List[Facing] = cache["shelf_full"]
shelf_live: List[Facing] = cache["shelf_live"]
distractors: List[Evidence] = cache["distractors"]
known = sum(1 for label, _, _ in shelf_full if label != UNKNOWN)
print(f"catalog {args.stock}, {LIVE_LONG_SIDE}-long-side frames at detect_conf {cache['conf']}")
print(f"image6: {len(shelf_full)} facings, {known} holding a catalog row, "
f"{len(shelf_live)} of them cut from the live frame")
print(f"distractors: {len(distractors)} crops from {cache['photos']} unrelated shelf photos")
print()
print(f"{'accept':>6} {'floor':>5} | {'shown':>5} {'corr':>4} {'prec':>6} {'cover':>6} "
f"| {'shown':>5} {'corr':>4} {'prec':>6} | {'false':>5} {'rate':>6}")
print(f"{'':>6} {'':>5} | {'--- full resolution ---':^25} "
f"| {'---- live resolution ----':^25} | {'-- unknown shelves --':^14}")
for accept in LOCAL_ACCEPTS:
for floor in TEXT_MIN_SHAREDS:
policy = DecisionPolicy(
local_accept=accept, text_min_shared=floor,
signatures=lane.text_lane.signatures,
)
full_shown, full_correct = _score(lane, policy, shelf_full)
live_shown, live_correct = _score(lane, policy, shelf_live)
false = _false_accepts(lane, policy, distractors)
shown, correct = full_shown, full_correct
rate = false / len(distractors) * 100
shipped = " <- shipped" if (accept, floor) == (LOCAL_ACCEPT, TEXT_MIN_SHARED) else ""
print(
f"{accept:>6.1f} {floor:>5} | {shown:>5} {correct:>4} "
f"{correct / shown * 100 if shown else 0:>5.1f}% {correct / known * 100:>5.1f}% "
f"| {live_shown:>5} {live_correct:>4} "
f"{live_correct / live_shown * 100 if live_shown else 0:>5.1f}% "
f"| {false:>5} {rate:>5.2f}%{shipped}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

860
recognization/code/tag_cards.py
View file

@ -1,860 +0,0 @@
"""
Price tags as cards with fields, not rail bands.
The old tag pass OCR'd a whole shelf rail and then tried to split the text into
tags. Two things went wrong, both measured:
* the reported box was the *band*, so a tag box looked like a strip spanning four
or five tags and lined up with nothing;
* a band mixes several tags' text, so values were assigned to the wrong tag
(measured: a tag reading 649/754.9 came back as price 2630, name
"コプサラダトレッシング" - text from a different tag).
This module takes the opposite route: the OCR engine's own line boxes define the
cards. Lines are grouped into a card by horizontal proximity, the card's box is
the union of its lines (so it lines up with the printed tag by construction), and
the fields are read with the tag's layout as a prior.
The layout of a Japanese shelf-edge tag, top to bottom: a 毎日お買得-style header,
the product name, the JAN's last three digits beside a discount, the sale price in
large digits, and a small tax-included or unit price. Measured on a LOPIA tag:
269 x 1.08 = 290.52 379 x 1.08 = 409.32 199 x 1.08 = 214.92
so the small clean number and the big stylised one are linked by the consumption
tax. That relation is used as a *check*: a price is only trusted when the two
numbers agree, and when the big digits cannot be read at all the price is derived
from the small one and marked as derived rather than measured.
"""
from __future__ import annotations
import json
import math
import os
import re
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
# Defaults when a store has no store.json. Measured on the demo LOPIA tags:
# 8% consumption tax, brick-red rails, and shelf prices from ¥50. A store file
# replaces these for that store only.
DEFAULT_TAX_RATIOS = (1.08, 1.10)
DEFAULT_MIN_PRICE = 50
# OpenCV hue is 0-180. Brick red wraps past 0, so a pixel is rail-coloured when
# its hue is below hue_low or above hue_high.
DEFAULT_RAIL_HSV = {"hue_low": 14, "hue_high": 166, "sat_min": 70, "val_min": 70}
# Aliases the parser reads. load_store_policy() republishes them per store.
TAX_RATIOS = DEFAULT_TAX_RATIOS
MIN_PRICE = DEFAULT_MIN_PRICE
_RAIL: Optional["StorePolicy"] = None
def _remember_rail(policy: "StorePolicy") -> None:
global _RAIL
_RAIL = policy
class StorePolicyError(ValueError):
"""store.json exists but cannot be used. Never borrow another store's file."""
@dataclass(frozen=True)
class StorePolicy:
tax_ratios: Tuple[float, ...] = DEFAULT_TAX_RATIOS
min_price: int = DEFAULT_MIN_PRICE
rail_hue_low: int = 14
rail_hue_high: int = 166
rail_sat_min: int = 70
rail_val_min: int = 70
store_id: Optional[str] = None
def active(self) -> None:
global TAX_RATIOS, MIN_PRICE
TAX_RATIOS = self.tax_ratios
MIN_PRICE = self.min_price
_remember_rail(self)
def default_policy() -> StorePolicy:
policy = StorePolicy()
policy.active()
return policy
def _as_ratio(value: Any) -> float:
number = float(value)
if not 1.0 < number <= 1.5:
raise StorePolicyError(f"tax ratio {value!r} is outside 1-1.5")
return number
def _as_bound(value: Any, name: str, low: int, high: int) -> int:
number = int(value)
if not low <= number <= high:
raise StorePolicyError(f"{name} {value!r} is outside {low}-{high}")
return number
def load_store_policy(stock_dir: Optional[str]) -> StorePolicy:
"""Read `<stock_dir>/store.json`. A missing file uses the defaults.
A file that is present but invalid raises StorePolicyError naming that path.
"""
if not stock_dir:
return StorePolicy()
path = os.path.join(stock_dir, "store.json")
if not os.path.isfile(path):
return StorePolicy()
try:
with open(path, encoding="utf-8") as handle:
raw = json.load(handle)
except (OSError, json.JSONDecodeError) as exc:
raise StorePolicyError(f"{path}: {exc}") from exc
if not isinstance(raw, dict):
raise StorePolicyError(f"{path}: expected an object")
try:
ratios = raw.get("tax_ratios", list(DEFAULT_TAX_RATIOS))
if not isinstance(ratios, list) or not ratios:
raise StorePolicyError(f"{path}: tax_ratios must be a non-empty list")
rail = raw.get("rail_hsv", DEFAULT_RAIL_HSV)
if not isinstance(rail, dict):
raise StorePolicyError(f"{path}: rail_hsv must be an object")
policy = StorePolicy(
tax_ratios=tuple(_as_ratio(item) for item in ratios),
min_price=_as_bound(raw.get("min_price", DEFAULT_MIN_PRICE), "min_price", 1, 100000),
rail_hue_low=_as_bound(rail.get("hue_low", 14), "hue_low", 0, 180),
rail_hue_high=_as_bound(rail.get("hue_high", 166), "hue_high", 0, 180),
rail_sat_min=_as_bound(rail.get("sat_min", 70), "sat_min", 0, 255),
rail_val_min=_as_bound(rail.get("val_min", 70), "val_min", 0, 255),
store_id=str(raw["store_id"]) if raw.get("store_id") else None,
)
except (TypeError, ValueError) as exc:
if isinstance(exc, StorePolicyError):
raise
raise StorePolicyError(f"{path}: {exc}") from exc
policy.active()
return policy
HEADER_PATTERN = re.compile(r"お買得|お買い得|特価|価格|セール")
# The tag truncates its discount to a tenth of a 割, so agreement is one step.
DISCOUNT_TOLERANCE = 0.05
# A regular price above this multiple of the sale price is a misread, not a discount.
REGULAR_PRICE_CEILING = 1.8
DISCOUNT_PATTERN = re.compile(r"(\d+(?:\.\d)?)\s*割\s*引")
# A decimal whose point the OCR dropped or replaced with a space (measured: a
# tax-included 409.32 came back as "409 3").
LOOSE_DECIMAL_PATTERN = re.compile(r"(\d{2,4})[.,\s](\d{1,2})(?!\d)")
DECIMAL_PATTERN = re.compile(r"(\d{1,4})[.,](\d{2})(?!\d)")
INTEGER_PATTERN = re.compile(r"(?<!\d)(\d{2,4})(?!\d)")
JAN3_PATTERN = re.compile(r"(?<!\d)(\d{3})(?!\d)")
# Package sizes and other numbers that are not prices or JANs.
SIZE_PATTERN = re.compile(r"\d+\s*(?:g|kg|ml|mL|ℓ|L|個|枚|袋|本|入)", re.IGNORECASE)
NOISE_NUMBERS = {"108", "110"}
@dataclass
class OcrLine:
"""One OCR line: its text and its box in the image the OCR ran on."""
text: str
box: Tuple[int, int, int, int]
confidence: float = 0.0
@property
def width(self) -> int:
return self.box[2] - self.box[0]
@property
def height(self) -> int:
return self.box[3] - self.box[1]
@dataclass
class TagFields:
"""What a tag says, with the evidence for each value.
Field names match the rest of the tag pipeline (`tags.TagFields`) so a card's
reading can be folded into the session cache unchanged; `price_source` and the
corroboration flags are the additions this module brings.
`regular_price` is the number a LOPIA tag prints large beside the sale price
(357 beside 269). It is the price *before* the discount, not a product code:
checked against the manufacturer, this product's JAN is 4901577092381, whose
last three digits are 381. All three tags on the demo shelf agree with their
printed discount instead - (357-269)/357 = 24.6 % -> 2.4割引 - which makes the
discount a second arithmetic check alongside the tax-included price.
"""
price: Optional[int] = None
regular_price: Optional[int] = None
discount: Optional[float] = None
unit_price: Optional[float] = None
name: Optional[str] = None
# A full JAN read from the tag's barcode. Only decodable at close range
# (measured: 0.63 px per module at ~1 m needs ~3x more pixels), so it is
# normally None and the barcode stays a deliberate close-up step.
barcode: Optional[str] = None
# measured (the big digits agreed with the tax relation) | derived (taken from
# the tax relation because the big digits could not be read) | none
price_source: str = "none"
header: Optional[str] = None
confidence: float = 0.0
flags: List[str] = field(default_factory=list)
def corroborated(self) -> bool:
"""Did the tag's own arithmetic agree anywhere?
Measured on a live walk: single uncorroborated reads of the big stylised
digits produced prices of 20, 23, 72, 2002 and 4508. A price is only worth
publishing when the tag's arithmetic confirms it.
Three strengths, weakest to strongest: the tax-included price and the sale
price form an exact pair (`tax_pair_consistent`); the printed discount
agrees with the pair (`discount_consistent`); the stylised digits agree with
the pair as well (`tax_consistent`). The compliance rules demand one of the
two stronger ones before a tag may accuse anything.
"""
return bool(
{"tax_consistent", "tax_pair_consistent", "discount_consistent"} & set(self.flags)
)
def is_readable(self) -> bool:
"""Readable = an anchor, a price, and arithmetic that agrees."""
if self.price is None or self.price < MIN_PRICE:
return False
if not self.corroborated():
return False
return self.regular_price is not None or self.name is not None
def as_dict(self) -> Dict[str, Any]:
return {
"price": self.price,
"regular_price": self.regular_price,
"discount": self.discount,
"unit_price": self.unit_price,
"name": self.name,
"barcode": self.barcode,
"price_source": self.price_source,
"flags": list(self.flags),
}
def as_payload(self) -> Dict[str, Any]:
payload = self.as_dict()
payload["confidence"] = round(self.confidence, 3)
return payload
@dataclass
class TagCard:
"""A price tag: its box in the frame, and what it says."""
box: Tuple[int, int, int, int]
fields: TagFields
lines: List[OcrLine] = field(default_factory=list)
@property
def centre_x(self) -> float:
return (self.box[0] + self.box[2]) / 2.0
def as_payload(self) -> Dict[str, Any]:
payload = {"box": list(self.box), "state": "readable" if self.fields.is_readable() else "unreadable"}
payload.update(self.fields.as_payload())
return payload
# --------------------------------------------------------------------------- grouping
def cluster_rows(lines: Sequence[OcrLine], max_gap: int = 26) -> List[List[OcrLine]]:
"""Split OCR lines into horizontal rows.
A rail band's crop also catches the packaging text of the rows above and
below, which is what made an earlier grouping mix a product's text into a
tag. Rows are separated first, so a tag row can be chosen on its own.
The separator is a vertical *gap*, not strict overlap: a tag stacks small
lines (header, name, JAN, price) with space between them, so requiring
overlap split one tag across several rows. Shelf rows sit hundreds of pixels
apart, so a small gap threshold still separates them.
"""
if not lines:
return []
rows: List[List[OcrLine]] = []
for line in sorted(lines, key=lambda l: l.box[1]):
placed = False
for row in rows:
top = min(l.box[1] for l in row)
bottom = max(l.box[3] for l in row)
gap = max(0, max(top - line.box[3], line.box[1] - bottom))
if gap <= max_gap:
row.append(line)
placed = True
break
if not placed:
rows.append([line])
return rows
def tag_row_score(row: Sequence[OcrLine], header_pattern: Any = None) -> float:
"""How much a row looks like a line of price tags.
Tags carry a header, a discount, and a decimal (tax-included) price; product
packaging rarely carries all three on one line. The header pattern is bound
here: a missing module global must not fail the tag pass.
"""
def pattern(name: str, fallback: str) -> Any:
found = globals().get(name)
return found if found is not None else re.compile(fallback)
header = header_pattern if header_pattern is not None else pattern(
"HEADER_PATTERN", r"お買得|お買い得|特価|価格|セール"
)
discount = pattern("DISCOUNT_PATTERN", r"(\d+(?:\.\d)?)\s*割\s*引")
decimal = pattern("DECIMAL_PATTERN", r"(\d{1,4})[.,](\d{2})(?!\d)")
jan = pattern("JAN3_PATTERN", r"(?<!\d)(\d{3})(?!\d)")
score = 0.0
for line in row:
text = line.text
if header.search(text):
score += 2.0
if discount.search(text):
score += 1.5
if decimal.search(text):
score += 1.5
if jan.search(text) and not decimal.search(text):
score += 0.5
return score
def best_tag_rows(lines: Sequence[OcrLine], keep: int = 2) -> List[List[OcrLine]]:
rows = cluster_rows(lines)
scored = sorted(rows, key=tag_row_score, reverse=True)
return [row for row in scored[:keep] if tag_row_score(row) > 0]
def group_lines_into_cards(
lines: Sequence[OcrLine],
max_gap: int = 60,
max_card_width: int = 420,
) -> List[List[OcrLine]]:
"""Group OCR lines into cards by horizontal proximity.
Tags sit side by side with a visible gap between their text blocks, and the
decorative part of the card carries no text, so a horizontal gap is a
reliable separator. Lines are first bucketed into rough columns, then split
where the gap is too wide or the column is wider than a tag.
"""
if not lines:
return []
ordered = sorted(lines, key=lambda line: (line.box[0], line.box[1]))
groups: List[List[OcrLine]] = []
current: List[OcrLine] = []
current_span: Optional[Tuple[int, int]] = None
for line in ordered:
start, end = line.box[0], line.box[2]
if not current:
current, current_span = [line], (start, end)
continue
assert current_span is not None
overlaps = start <= current_span[1] + max_gap
wide_enough = max(current_span[1], end) - min(current_span[0], start) <= max_card_width
if overlaps and wide_enough:
current.append(line)
current_span = (min(current_span[0], start), max(current_span[1], end))
else:
groups.append(current)
current, current_span = [line], (start, end)
if current:
groups.append(current)
return groups
def card_box(lines: Iterable[OcrLine], pad_x: int = 6, pad_y: int = 4) -> Tuple[int, int, int, int]:
"""The card's box: the union of its lines, padded to the printed edge."""
xs1 = [line.box[0] for line in lines]
ys1 = [line.box[1] for line in lines]
xs2 = [line.box[2] for line in lines]
ys2 = [line.box[3] for line in lines]
return (max(0, min(xs1) - pad_x), max(0, min(ys1) - pad_y), max(xs2) + pad_x, max(ys2) + pad_y)
# --------------------------------------------------------------------------- parsing
def _to_float(whole: str, fraction: str) -> float:
return float(f"{whole}.{fraction}")
def _line_height(line: Any) -> int:
"""Line height, whichever line type the caller passes in.
`tag_cards` has its own line type, but the tag pipeline hands over the OCR
engine's lines, which carry only a box.
"""
height = getattr(line, "height", None)
if height is not None:
return int(height)
return int(line.box[3] - line.box[1])
def _vote(candidates: Sequence[Tuple[Any, float]]) -> Optional[Any]:
"""Pick a value by agreement across reads, not by one read's optimism.
Several passes over the same card produce several reads of the same field.
OCR confidence alone is a poor guide (measured: a wrong 12.4割引 came back
more confidently than the right 2.4割引), so agreement wins, and confidence
only breaks ties.
"""
if not candidates:
return None
scores: Dict[Any, List[float]] = {}
for value, confidence in candidates:
scores.setdefault(value, []).append(confidence)
return max(scores.items(), key=lambda item: (len(item[1]), sum(item[1]) / len(item[1])))[0]
# Every flag that describes the published field set. They are recomputed whenever
# the fields change (after voting), so a stale one - "no_unit_price" surviving a
# frame that did read the unit price - cannot linger.
CORROBORATION_FLAGS = {
"tax_consistent",
"tax_pair_consistent",
"digit_read_disagrees",
"price_from_unit",
"discount_consistent",
"discount_derived",
"discount_mismatch",
"regular_price_implausible",
"price_implausible",
"no_price",
"no_unit_price",
"no_regular_price",
}
def corroborate(fields: TagFields) -> List[str]:
"""Run the tag's arithmetic on a field set and return the flags it earns.
Used twice: while parsing one read, and again after the session has voted
field by field. Voting picks each field's value independently, so the winner
can be a price from one frame and a tax-included price from another - measured:
a record published price 149 beside unit price 214.92, a pair that never
appeared together (214.92 / 1.08 = 199). Re-running the arithmetic on the
values actually being published is what keeps the claim coherent.
"""
flags: List[str] = []
price = fields.price
unit = fields.unit_price
if unit:
derived = None
for ratio in TAX_RATIOS:
candidate = unit / ratio
rounded = round(candidate)
if abs(candidate - rounded) <= 0.02 and MIN_PRICE <= rounded <= 100000:
derived = float(rounded)
break
if derived is not None:
if price is None:
fields.price = int(round(derived))
fields.price_source = "derived"
flags.append("price_from_unit")
elif abs(price - derived) <= max(2.0, 0.01 * derived):
fields.price = int(round(derived))
flags.append("tax_consistent")
else:
# The pair is exact but the digits disagree; the pair wins.
fields.price = int(round(derived))
fields.price_source = "derived"
flags.extend(["tax_pair_consistent", "digit_read_disagrees"])
if price is None:
flags.append("no_price")
elif price < MIN_PRICE:
flags.append("price_implausible")
if fields.unit_price is None:
flags.append("no_unit_price")
if fields.regular_price and fields.price:
plausible = 1.02 * fields.price <= fields.regular_price <= REGULAR_PRICE_CEILING * fields.price
if not plausible:
flags.append("regular_price_implausible")
fields.regular_price = None
else:
ratio = (fields.regular_price - fields.price) / float(fields.regular_price)
expected = math.floor(ratio * 100.0) / 10.0
if fields.discount is None:
fields.discount = expected
flags.append("discount_derived")
elif abs(fields.discount - expected) <= DISCOUNT_TOLERANCE:
flags.append("discount_consistent")
else:
# The printed discount disagrees with the two prices. The prices
# corroborate each other, so publish the discount they imply and
# keep the disagreement on the record.
fields.discount = expected
flags.append("discount_mismatch")
if fields.regular_price is None:
flags.append("no_regular_price")
return flags
def parse_tag_fields(lines: Sequence[OcrLine]) -> TagFields:
"""Read a tag's fields from its OCR lines, using the tag layout as a prior."""
fields = TagFields()
if not lines:
return fields
ordered = sorted(lines, key=lambda line: (line.box[1], line.box[0]))
texts = [(line.text.strip(), line) for line in ordered if line.text.strip()]
if not texts:
return fields
header = next((text for text, _ in texts if HEADER_PATTERN.search(text)), None)
if header:
fields.header = header
# Discounts: vote across reads, and keep only a plausible rate (a 12.4割引
# came from a spurious leading 1 in one pass).
discount_candidates: List[Tuple[float, float]] = []
for text, line in texts:
match = DISCOUNT_PATTERN.search(text)
if match:
value = float(match.group(1))
if 0.1 <= value <= 9.9:
discount_candidates.append((value, line.confidence))
fields.discount = _vote(discount_candidates)
decimal_candidates: List[Tuple[float, float]] = [] # (value, line confidence)
for text, line in texts:
strict = DECIMAL_PATTERN.findall(text)
for whole, fraction in strict:
value = _to_float(whole, fraction)
if 10.0 <= value <= 100000.0:
decimal_candidates.append((value, line.confidence))
if not strict and not SIZE_PATTERN.search(text):
for whole, fraction in LOOSE_DECIMAL_PATTERN.findall(text):
value = _to_float(whole, fraction if len(fraction) == 2 else fraction + "0")
if 10.0 <= value <= 100000.0:
decimal_candidates.append((value, line.confidence * 0.8))
fields.unit_price = _vote(decimal_candidates)
# The name: the longest line that is not a number, a header, or a price.
def looks_like_name(text: str) -> bool:
if HEADER_PATTERN.search(text) or DISCOUNT_PATTERN.search(text):
return False
if DECIMAL_PATTERN.search(text):
return False
letters = sum(1 for char in text if not char.isdigit() and char not in " .,()（）<>＜＞%¥円")
return letters >= 4
name_candidates = [(text, line) for text, line in texts if looks_like_name(text)]
if name_candidates:
# The longest read is the most complete one; ties go to the more confident.
name_candidates.sort(key=lambda item: (-len(item[0]), -item[1].confidence))
fields.name = name_candidates[0][0]
# LARGEST text on the tag (its line is roughly twice as tall as the small
# print), and the JAN's last three digits sit on a small line of their own.
# Getting this order wrong made a 269 price look like a JAN and disappear.
integer_lines: List[Tuple[Any, List[str]]] = []
for text, line in texts:
if DECIMAL_PATTERN.search(text) or SIZE_PATTERN.search(text):
continue
numbers = [n for n in INTEGER_PATTERN.findall(text) if n not in NOISE_NUMBERS]
if numbers:
integer_lines.append((line, numbers))
price_line = None
measured = None
if integer_lines:
price_line, numbers = max(integer_lines, key=lambda item: _line_height(item[0]))
tall = max(_line_height(line) for line, _ in integer_lines)
# Only accept a price from a line that is actually the big print.
if _line_height(price_line) >= 0.75 * tall:
measured = float(max(numbers, key=int))
# The regular price: a small-line number that is *higher* than the sale price
# (it is the price before the discount), with package sizes excluded.
regular_candidates: List[Tuple[int, float]] = []
for line, numbers in integer_lines:
if line is price_line:
continue
for candidate in numbers:
value = int(candidate)
if value < MIN_PRICE:
continue
if measured is not None and value <= measured:
continue
regular_candidates.append((value, line.confidence))
if regular_candidates:
regular_candidates.sort(key=lambda item: -item[1])
fields.regular_price = regular_candidates[0][0]
# The price starts as what the big digits said; the arithmetic below may
# correct it from the tax-included price, which is the more reliable read.
if measured is not None:
fields.price = int(round(measured))
fields.price_source = "measured"
# The tag's arithmetic: the tax relation between the sale price and the
# tax-included price, and the discount between the regular price and the sale
# price. Both are checks the tag carries with it.
fields.flags = [flag for flag in fields.flags if flag not in CORROBORATION_FLAGS]
fields.flags.extend(corroborate(fields))
# Confidence: the anchors we actually trust, not the OCR's own optimism.
score = 0.0
if fields.price is not None:
score += 0.5 if fields.price_source == "measured" else 0.35
if fields.regular_price is not None:
score += 0.15
if fields.corroborated():
score += 0.2
if fields.name is not None:
score += 0.15
if fields.discount is not None:
score += 0.1
fields.confidence = min(1.0, score)
return fields
def card_from_lines(lines: Sequence[OcrLine]) -> TagCard:
return TagCard(box=card_box(lines), fields=parse_tag_fields(lines), lines=list(lines))
# --------------------------------------------------------------------------- finding unreadable tags
def find_rails(image: Any, min_fraction: float = 0.25) -> List[Tuple[int, int, int, int]]:
"""Find the shelf's tag rails.
In the stores in these images the price tags are printed *on* the rail: a
red brick-patterned strip that runs the width of the shelf, one tag per
product column. The rail is therefore the place to look for tags, and its
brick red is what makes it findable without reading anything.
"""
import cv2
import numpy as np
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
hue, sat, val = cv2.split(hsv)
rail = _RAIL or StorePolicy()
brick = (
((hue < rail.rail_hue_low) | (hue > rail.rail_hue_high))
& (sat > rail.rail_sat_min)
& (val > rail.rail_val_min)
).astype(np.uint8)
row_fraction = brick.mean(axis=1)
height, width = image.shape[:2]
rails: List[Tuple[int, int, int, int]] = []
start: Optional[int] = None
for y, fraction in enumerate(row_fraction):
if fraction >= min_fraction and start is None:
start = y
elif fraction < min_fraction and start is not None:
if y - start >= 20:
rails.append((0, start, width, y))
start = None
if start is not None and height - start >= 20:
rails.append((0, start, width, height))
return rails
def find_tag_blocks_on_rail(
image: Any,
rail: Tuple[int, int, int, int],
min_height_ratio: float = 0.35,
) -> List[Tuple[int, int, int, int]]:
"""Find the tags printed on a rail: bright text blocks against the brick.
Every facing has a tag, so this is what separates "this product has no tag"
from "this tag could not be read" - the distinction the compliance rules need
before they accuse a shelf of anything.
"""
import cv2
import numpy as np
x1, y1, x2, y2 = rail
strip = image[y1:y2, x1:x2]
if strip.size == 0:
return []
hsv = cv2.cvtColor(strip, cv2.COLOR_BGR2HSV)
hue, sat, val = cv2.split(hsv)
# The tag's text block is paper: brighter and much less saturated than the
# brick behind it. Tag text splits the paper into pieces, so the tag is found
# by projecting onto x (a tag is a run of columns that carry paper) rather
# than by connected components, which fragment at every printed line.
paper = ((val > 120) & (sat < 145)).astype(np.uint8)
column_fraction = paper.mean(axis=0)
rail_height = y2 - y1
blocks: List[Tuple[int, int, int, int]] = []
start: Optional[int] = None
for index, fraction in enumerate(column_fraction):
if fraction >= 0.18 and start is None:
start = index
elif fraction < 0.18 and start is not None:
blocks.append((start, index))
start = None
if start is not None:
blocks.append((start, len(column_fraction)))
tags: List[Tuple[int, int, int, int]] = []
for start_x, end_x in blocks:
if end_x - start_x < 40:
continue
window = paper[:, start_x:end_x]
rows = np.where(window.mean(axis=1) >= 0.15)[0]
if len(rows) < min_height_ratio * rail_height:
continue
tags.append((x1 + start_x, y1 + int(rows.min()), x1 + end_x, y1 + int(rows.max())))
return tags
def learn_header_template(image: Any, cards: Sequence[TagCard]) -> Optional[Any]:
"""Learn what this store's tag header looks like, from a tag we could read.
A tag whose text cannot be read still has to be *found*, or "no tag here" and
"I could not read this tag" become the same answer and the compliance rules
have nothing honest to say. The header is the one part of a tag that repeats
identically across the shelf, so a single readable tag teaches the detector
what to look for - no per-chain template to maintain.
"""
import cv2
for card in cards:
for line in card.lines:
if not HEADER_PATTERN.search(line.text or ""):
continue
x1, y1, x2, y2 = line.box
if x2 - x1 < 40 or y2 - y1 < 8:
continue
height, width = image.shape[:2]
# The header bar is wider than the text on it (measured: a 79 px text
# line sits on a ~196 px bar), so the template is padded out to the bar
# - a bare text template matches far fewer tags.
pad_x = max(6, int((x2 - x1) * 0.75))
pad_y = max(2, int((y2 - y1) * 0.35))
x1, x2 = max(0, x1 - pad_x), min(width, x2 + pad_x)
y1, y2 = max(0, y1 - pad_y), min(height, y2 + pad_y)
template = image[y1:y2, x1:x2]
if template.size:
return template
return None
def find_tags_by_template(
image: Any,
template: Any,
region: Optional[Tuple[int, int, int, int]] = None,
threshold: float = 0.55,
scales: Sequence[float] = (0.72, 0.82, 0.92, 1.0, 1.1, 1.2, 1.32),
) -> List[Tuple[int, int, int, int]]:
"""Find tag headers across an image (or one region) by matching the learned header.
Returns tag boxes: the header plus the text block beneath it, which is where
the name, the JAN and the price are printed.
"""
import cv2
import numpy as np
if template is None or not getattr(template, "size", 0):
return []
height, width = image.shape[:2]
x1, y1, x2, y2 = region or (0, 0, width, height)
x1, y1 = max(0, x1), max(0, y1)
x2, y2 = min(width, x2), min(height, y2)
crop = image[y1:y2, x1:x2]
if crop.size == 0:
return []
grey = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
hits: List[Tuple[float, int, int, int, int]] = []
for scale in scales:
scaled = template if scale == 1.0 else cv2.resize(
template, None, fx=scale, fy=scale, interpolation=cv2.INTER_AREA
)
if scaled.shape[0] >= grey.shape[0] or scaled.shape[1] >= grey.shape[1]:
continue
response = cv2.matchTemplate(grey, cv2.cvtColor(scaled, cv2.COLOR_BGR2GRAY), cv2.TM_CCOEFF_NORMED)
ys, xs = np.where(response >= threshold)
for x, y in zip(xs, ys):
hits.append((float(response[y, x]), int(x), int(y), scaled.shape[1], scaled.shape[0]))
hits.sort(reverse=True)
kept: List[Tuple[float, int, int, int, int]] = []
for hit in hits:
_, x, y, w, h = hit
if all(abs(x - kx) > w * 0.5 or abs(y - ky) > h for _, kx, ky, _, _ in kept):
kept.append(hit)
boxes = []
for _, x, y, w, h in kept:
# The tag continues below its header: the text block is about five header
# heights tall and a little wider than the header itself.
tag_x1 = x1 + x - 4
tag_y1 = y1 + y - 4
tag_x2 = min(width, tag_x1 + int(w * 1.30))
tag_y2 = min(height, tag_y1 + int(h * 5.0))
boxes.append((tag_x1, tag_y1, tag_x2, tag_y2))
return boxes
# --------------------------------------------------------------------------- reading
def read_card_lines(
engine: Any,
image: Any,
box: Tuple[int, int, int, int],
margin: int = 10,
scales: Sequence[float] = (1.0, 2.0, 3.0),
variants: Sequence[str] = ("gray", "red"),
) -> List[OcrLine]:
"""OCR one card several ways and pool the reads.
The fields on a tag are printed at very different sizes: the price is huge
and stylised, the JAN and the tax-included price are small and clean. A single
pass at one scale reads one of them well and the other badly, so the card is
read at three scales and in the red channel (where red-on-white text becomes
high contrast), and the results are pooled for voting.
"""
import cv2
height, width = image.shape[:2]
x1 = max(0, int(box[0]) - margin)
y1 = max(0, int(box[1]) - margin)
x2 = min(width, int(box[2]) + margin)
y2 = min(height, int(box[3]) + margin)
crop = image[y1:y2, x1:x2]
if crop.size == 0:
return []
pooled: List[OcrLine] = []
for variant in variants:
if variant == "red":
prepared = cv2.cvtColor(255 - crop[:, :, 2], cv2.COLOR_GRAY2BGR)
elif variant == "green":
prepared = cv2.cvtColor(crop[:, :, 1], cv2.COLOR_GRAY2BGR)
else:
prepared = crop
for scale in scales:
scaled = prepared if scale == 1.0 else cv2.resize(
prepared, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC
)
result = engine.read(scaled)
for line in result.lines:
text = (line.text or "").strip()
if not text:
continue
pooled.append(
OcrLine(
text=text,
box=(
x1 + int(line.box[0] / scale),
y1 + int(line.box[1] / scale),
x1 + int(line.box[2] / scale),
y1 + int(line.box[3] / scale),
),
confidence=float(line.confidence),
)
)
return pooled

1043
recognization/code/tags.py
View file

File diff suppressed because it is too large Load diff

25
recognization/code/test.py Normal file
View file

@ -0,0 +1,25 @@
from ultralytics import YOLO
import cv2
model = YOLO("/Users/nguyenhongvan/recongnize/runs/detect/train/weights/best.pt")
image_path = "/Users/nguyenhongvan/recongnize/data/add/image9.jpg"
results = model(image_path)
image = cv2.imread(image_path)
for box in results[0].boxes.xyxy.cpu().numpy():
x1, y1, x2, y2 = map(int, box)
cv2.rectangle(
image,
(x1, y1),
(x2, y2),
(0, 255, 0),
2
)
cv2.imshow("Bounding Boxes Only", image)
cv2.waitKey(0)
cv2.destroyAllWindows()

553
recognization/code/test_api.py
View file

@ -1,553 +0,0 @@
"""
Contract tests for the recognition server API.
A fake engine stands in for the model, so these run without weights. They defend
the wire contracts a client depends on: response shapes, the shared error shape,
auth, limits, and the WebSocket protocol (hello, frames, control messages,
recoverable errors).
"""
from __future__ import annotations
import base64
import json
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
import numpy as np # noqa: E402
from app import ServerSettings, create_app # noqa: E402
from engine import EngineSettings, RecognitionEngine # noqa: E402
from test_live import SmallBoxDetector # noqa: E402
from test_live import FakeDetector, FakeInventory, FakeMatcher # noqa: E402
from fastapi.testclient import TestClient # noqa: E402
def make_app(*, token=None, max_sessions=4, detector=None):
# The fake matcher is what these tests exercise, so the backend is pinned to
# `matcher`; the shipped default is the `local` lane, which reads real
# reference images and would ignore the fake.
engine = RecognitionEngine(
detector or FakeDetector(), FakeMatcher(), FakeInventory(),
settings=EngineSettings(classify_workers=2, identity_backend="matcher"),
)
settings = ServerSettings(token=token, max_sessions=max_sessions)
app = create_app(engine, settings, web_dir="/nonexistent")
return TestClient(app), engine
def jpeg_bytes(width=640, height=480):
ok, buffer = cv2.imencode(".jpg", np.zeros((height, width, 3), dtype="uint8"))
return buffer.tobytes()
def frame_payload(frame_id, payload=None):
payload = payload if payload is not None else jpeg_bytes()
return frame_id.to_bytes(4, "big") + payload
class RestContractTests(unittest.TestCase):
def setUp(self):
self.client, self.engine = make_app()
def tearDown(self):
self.engine.shutdown()
def test_health_needs_no_token_and_reports_model(self):
response = self.client.get("/v1/health")
self.assertEqual(response.status_code, 200)
body = response.json()
self.assertEqual(body["status"], "ok")
self.assertIn("version", body["model"])
self.assertEqual(body["catalog"]["sku_count"], 2)
def test_a_panned_sample_walks_and_stays_portrait(self):
# The debug replay source walks a shelf by asking for a frame index. A
# pan that returned the same window every time would freeze the stage
# while the boxes moved, which is the bug this pins.
first = self.client.get("/v1/samples/image6?pan=0")
self.assertEqual(first.status_code, 200)
self.assertEqual(first.headers["content-type"], "image/jpeg")
later = self.client.get("/v1/samples/image6?pan=40")
self.assertEqual(later.status_code, 200)
self.assertNotEqual(first.content, later.content, "a later pan must show a different window")
import cv2
import numpy as np
decoded = cv2.imdecode(np.frombuffer(later.content, dtype="uint8"), cv2.IMREAD_COLOR)
height, width = decoded.shape[:2]
self.assertEqual((width, height), (720, 1280), "the window must match the phone's portrait frame")
def test_a_sample_without_pan_is_the_whole_image(self):
response = self.client.get("/v1/samples/image6")
self.assertEqual(response.status_code, 200)
self.assertGreater(len(response.content), 1024)
def test_catalog_shape_has_no_local_paths(self):
body = self.client.get("/v1/catalog").json()
product = body["products"][0]
self.assertEqual(set(product), {
"sku_id", "product_name", "price", "number_in", "number_out", "number_on",
"receipt_date", "expiration_date", "image_url",
})
def test_catalog_image_404_for_unknown_and_traversal(self):
response = self.client.get("/v1/catalog/nope/image")
self.assertEqual(response.status_code, 404)
self.assertEqual(response.json()["error"]["code"], "NOT_FOUND")
# A traversal-looking id is rejected before it can reach the filesystem
# (either by routing or by the inventory lookup).
for sku in ("..%2F..%2Fstock.csv", "../../etc/passwd", "sku_a%2F..%2F..%2Fstock.csv"):
response = self.client.get(f"/v1/catalog/{sku}/image")
self.assertIn(response.status_code, (400, 404), sku)
def test_stock_update_validates_and_returns_product(self):
before = self.client.get("/v1/catalog").json()["products"]
before_on = next(p["number_on"] for p in before if p["sku_id"] == "sku_a")
response = self.client.post("/v1/catalog/sku_a/stock", json={"delta": -1})
self.assertEqual(response.status_code, 200)
self.assertEqual(response.json()["product"]["number_on"], before_on - 1)
self.assertEqual(self.client.post("/v1/catalog/sku_a/stock", json={"delta": 0}).status_code, 422)
self.assertEqual(self.client.post("/v1/catalog/sku_a/stock", json={"delta": 500}).status_code, 422)
self.assertEqual(self.client.post("/v1/catalog/ghost/stock", json={"delta": -1}).status_code, 404)
def test_recognize_accepts_raw_bytes(self):
response = self.client.post(
"/v1/recognize?classify=true", content=jpeg_bytes(),
headers={"Content-Type": "image/jpeg"},
)
self.assertEqual(response.status_code, 200)
body = response.json()
self.assertTrue(body["image_id"].startswith("img_"))
self.assertEqual(body["coordinate_convention"], "pixel-xyxy-origin-top-left")
self.assertEqual(len(body["detections"]), 1)
self.assertIn(body["detections"][0]["identity"]["state"], ("recognized", "unconfirmed"))
for key in ("width", "height", "catalog_version", "model_version", "summary", "timing_ms"):
self.assertIn(key, body)
def test_recognize_accepts_multipart(self):
response = self.client.post(
"/v1/recognize?classify=false",
files={"file": ("shelf.jpg", jpeg_bytes(), "image/jpeg")},
)
self.assertEqual(response.status_code, 200)
self.assertEqual(response.json()["detections"][0]["identity"]["state"], "detected")
def test_recognize_rejects_bad_input(self):
self.assertEqual(self.client.post("/v1/recognize", content=b"").status_code, 400)
response = self.client.post(
"/v1/recognize", content=b"not an image", headers={"Content-Type": "image/jpeg"}
)
self.assertEqual(response.status_code, 422)
self.assertEqual(response.json()["error"]["code"], "UNDECODABLE_IMAGE")
response = self.client.post(
"/v1/recognize", content=b"{}", headers={"Content-Type": "application/json"}
)
self.assertEqual(response.status_code, 415)
self.assertEqual(response.json()["error"]["code"], "UNSUPPORTED_MEDIA_TYPE")
def test_a_lying_content_length_is_rejected_while_streaming(self):
"""The ASGI receive stops once the body passes the cap, before decode."""
import anyio
from transport import BodyLimit
cap = 32
seen = {"bytes": 0}
async def app(scope, receive, send):
while True:
message = await receive()
if message["type"] != "http.request":
continue
seen["bytes"] += len(message.get("body") or b"")
if not message.get("more_body"):
break
await send({"type": "http.response.start", "status": 200, "headers": []})
await send({"type": "http.response.body", "body": b"ok"})
chunks = [b"x" * 20, b"y" * 20, b"z" * 20]
async def receive():
body = chunks.pop(0) if chunks else b""
return {"type": "http.request", "body": body, "more_body": bool(chunks)}
sent = []
async def send(message):
sent.append(message)
async def run():
await BodyLimit(app, cap)(
{"type": "http"}, receive, send
)
anyio.run(run)
self.assertLessEqual(seen["bytes"], cap + 20)
self.assertEqual(sent[0]["status"], 413)
self.assertIn(b"PAYLOAD_TOO_LARGE", sent[1]["body"])
def test_overflow_after_response_start_sends_no_second_start(self):
"""A body that overflows after headers began must not start again."""
import anyio
from transport import BodyLimit
async def app(scope, receive, send):
await send({"type": "http.response.start", "status": 200, "headers": []})
await receive()
await send({"type": "http.response.body", "body": b"ok"})
sent = []
async def receive():
return {"type": "http.request", "body": b"x" * 40, "more_body": False}
async def send(message):
sent.append(message)
anyio.run(BodyLimit(app, 8), {"type": "http"}, receive, send)
starts = [message for message in sent if message.get("type") == "http.response.start"]
self.assertEqual(len(starts), 1)
self.assertEqual(starts[0]["status"], 200)
self.assertFalse(any(message.get("status") == 413 for message in sent))
def test_classify_and_inspect_roundtrip(self):
recognized = self.client.post(
"/v1/recognize?classify=false", content=jpeg_bytes(),
headers={"Content-Type": "image/jpeg"},
).json()
image_id = recognized["image_id"]
box = recognized["detections"][0]["box"]
classified = self.client.post(
f"/v1/images/{image_id}/classify", json={"method": "hybrid"}
)
self.assertEqual(classified.status_code, 200)
self.assertEqual(len(classified.json()["identities"]), 1)
inspected = self.client.post(
f"/v1/images/{image_id}/inspect", json={"box": box, "method": "hybrid"}
)
self.assertEqual(inspected.status_code, 200)
body = inspected.json()
self.assertIn("identity", body)
self.assertTrue(body["crop_jpeg_b64"].startswith("data:image/jpeg;base64,"))
def test_unknown_method_is_a_400(self):
response = self.client.post(
"/v1/recognize?method=nope", content=jpeg_bytes(),
headers={"Content-Type": "image/jpeg"},
)
self.assertEqual(response.status_code, 400)
self.assertEqual(response.json()["error"]["code"], "BAD_REQUEST")
def test_expired_image_id(self):
response = self.client.post("/v1/images/img_missing/classify", json={"method": "hybrid"})
self.assertEqual(response.status_code, 404)
self.assertEqual(response.json()["error"]["code"], "IMAGE_EXPIRED")
def test_validation_errors_use_the_shared_shape(self):
response = self.client.post("/v1/images/img_x/inspect", json={"box": [1, 2, 3]})
self.assertEqual(response.status_code, 422)
self.assertIn("error", response.json())
class AuthTests(unittest.TestCase):
def test_token_is_required_when_configured(self):
client, engine = make_app(token="s3cret")
try:
self.assertEqual(client.get("/v1/health").status_code, 200)
self.assertEqual(client.get("/v1/catalog").status_code, 401)
self.assertEqual(client.get("/v1/catalog", headers={"Authorization": "Bearer wrong"}).status_code, 401)
self.assertEqual(client.get("/v1/catalog", headers={"Authorization": "Bearer s3cret"}).status_code, 200)
finally:
engine.shutdown()
class WebSocketProtocolTests(unittest.TestCase):
def setUp(self):
self.client, self.engine = make_app()
def tearDown(self):
self.engine.shutdown()
def test_hello_frame_result_and_control_messages(self):
with self.client.websocket_connect("/v1/stream") as socket:
hello = socket.receive_json()
self.assertEqual(hello["type"], "hello")
self.assertEqual(hello["protocol"], 1)
self.assertIn("session_id", hello)
self.assertIn("config", hello)
self.assertEqual(hello["limits"]["max_frame_bytes"], 4 * 1024 * 1024)
socket.send_json({"type": "ping", "t": 7})
self.assertEqual(socket.receive_json(), {"type": "pong", "t": 7})
socket.send_json({"type": "config", "method": "orb"})
ack = socket.receive_json()
self.assertEqual(ack["type"], "config_ack")
self.assertEqual(ack["config"]["method"], "orb")
socket.send_bytes(frame_payload(11))
result = socket.receive_json()
self.assertEqual(result["type"], "result")
self.assertEqual(result["frame_id"], 11)
self.assertEqual(result["seq"], 1)
self.assertIn("tracks", result)
self.assertIn("summary", result)
self.assertIn("quality", result)
socket.send_json({"type": "reset"})
self.assertEqual(socket.receive_json()["type"], "reset_ack")
socket.send_json({"type": "not-a-thing"})
error = socket.receive_json()
self.assertEqual(error["code"], "BAD_MESSAGE")
def test_short_frame_is_recoverable(self):
with self.client.websocket_connect("/v1/stream") as socket:
socket.receive_json() # hello
socket.send_bytes(b"\x00\x01")
error = socket.receive_json()
self.assertEqual(error["code"], "BAD_FRAME")
# the connection is still usable
socket.send_bytes(frame_payload(1))
self.assertEqual(socket.receive_json()["type"], "result")
def test_undecodable_frame_is_reported_and_keeps_the_socket(self):
with self.client.websocket_connect("/v1/stream") as socket:
socket.receive_json()
socket.send_bytes(frame_payload(5, b"not an image"))
error = socket.receive_json()
self.assertEqual(error["code"], "UNDECODABLE_IMAGE")
self.assertEqual(error["frame_id"], 5)
socket.send_bytes(frame_payload(6))
self.assertEqual(socket.receive_json()["frame_id"], 6)
def test_inspect_over_the_socket(self):
with self.client.websocket_connect("/v1/stream") as socket:
socket.receive_json()
socket.send_bytes(frame_payload(1))
result = socket.receive_json()
track_id = result["tracks"][0]["track_id"]
socket.send_json({"type": "inspect", "track_id": track_id, "request_id": "r1"})
reply = socket.receive_json()
self.assertEqual(reply["type"], "inspect_result")
self.assertEqual(reply["request_id"], "r1")
self.assertIn("identity", reply)
socket.send_json({"type": "inspect", "track_id": 999999, "request_id": "r2"})
missing = socket.receive_json()
self.assertEqual(missing["code"], "TRACK_NOT_IN_VIEW")
def crop_app(self):
"""A shelf whose only facing is too small for the frame crop to decide."""
client, engine = make_app(detector=SmallBoxDetector())
self.addCleanup(engine.shutdown)
return client
def test_capabilities_turn_on_the_crop_lane(self):
with self.client.websocket_connect("/v1/stream") as socket:
hello = socket.receive_json()
self.assertIn("max_crop_bytes", hello["limits"])
socket.send_json({"type": "capabilities", "crop": True, "max_crop_bytes": 65536})
ack = socket.receive_json()
self.assertEqual(ack["type"], "capabilities_ack")
self.assertEqual(ack["capabilities"], {"crop": True, "max_crop_bytes": 65536})
def test_a_capability_the_server_cannot_honour_is_refused(self):
"""A half-stated or malformed capability turns nothing on."""
for message in (
{"type": "capabilities", "max_crop_bytes": 65536},
{"type": "capabilities", "crop": False, "max_crop_bytes": 65536},
{"type": "capabilities", "crop": "true", "max_crop_bytes": 65536},
{"type": "capabilities", "crop": True},
{"type": "capabilities", "crop": True, "max_crop_bytes": 0},
{"type": "capabilities", "crop": True, "max_crop_bytes": -4096},
{"type": "capabilities", "crop": True, "max_crop_bytes": True},
{"type": "capabilities", "crop": True, "max_crop_bytes": "65536"},
{"type": "capabilities", "crop": True, "max_crop_bytes": 64 * 1024 * 1024},
):
with self.subTest(capability=message):
with self.crop_app().websocket_connect("/v1/stream") as socket:
socket.receive_json() # hello
socket.send_json(message)
self.assertEqual(socket.receive_json()["code"], "BAD_MESSAGE")
# A refused capability leaves this session on protocol v1: the
# facing it cannot report is never asked about.
for frame_id in (1, 2, 3):
socket.send_bytes(frame_payload(frame_id))
result = socket.receive_json()
self.assertEqual(result["stats"]["crops"]["requested"], 0)
for track in result["tracks"]:
self.assertNotIn("crop_request", track)
def test_a_v1_client_is_never_asked_for_a_crop(self):
with self.crop_app().websocket_connect("/v1/stream") as socket:
socket.receive_json() # hello
for frame_id in (1, 2, 3):
socket.send_bytes(frame_payload(frame_id))
result = socket.receive_json()
for track in result["tracks"]:
self.assertNotIn("crop_request", track)
self.assertEqual(result["stats"]["crops"]["requested"], 0)
def test_a_crop_round_trip_labels_the_track(self):
with self.crop_app().websocket_connect("/v1/stream") as socket:
socket.receive_json() # hello
socket.send_json({"type": "capabilities", "crop": True, "max_crop_bytes": 131072})
self.assertEqual(socket.receive_json()["type"], "capabilities_ack")
socket.send_bytes(frame_payload(1))
socket.receive_json()
socket.send_bytes(frame_payload(2))
result = socket.receive_json()
request = result["tracks"][0].get("crop_request")
self.assertIsNotNone(request, "a facing too small for the frame crop must be asked about")
self.assertEqual(request["frame_id"], 2)
socket.send_json({
"type": "crop",
"request_id": request["request_id"],
"track_id": request["track_id"],
"frame_id": request["frame_id"],
"jpeg": base64.b64encode(jpeg_bytes(80, 120)).decode("ascii"),
})
self.assertEqual(socket.receive_json()["type"], "crop_ack")
socket.send_bytes(frame_payload(3))
settled = socket.receive_json()
self.assertEqual(settled["stats"]["crops"]["accepted"], 1)
identities = [t["identity"] for t in settled["tracks"]]
self.assertTrue(any(i["source"] == "crop" for i in identities))
def test_a_crop_that_does_not_match_its_request_is_refused(self):
with self.crop_app().websocket_connect("/v1/stream") as socket:
socket.receive_json() # hello
socket.send_json({"type": "capabilities", "crop": True, "max_crop_bytes": 131072})
socket.receive_json()
socket.send_bytes(frame_payload(1))
socket.receive_json()
socket.send_bytes(frame_payload(2))
request = socket.receive_json()["tracks"][0]["crop_request"]
socket.send_json({
"type": "crop",
"request_id": request["request_id"],
"track_id": request["track_id"],
"frame_id": 4242,
"jpeg": base64.b64encode(jpeg_bytes(80, 120)).decode("ascii"),
})
refused = socket.receive_json()
self.assertEqual(refused["type"], "crop_error")
self.assertEqual(refused["code"], "CROP_MISMATCH")
socket.send_bytes(frame_payload(3))
self.assertEqual(socket.receive_json()["stats"]["crops"]["accepted"], 0)
def test_a_crop_that_is_not_an_image_is_refused(self):
with self.crop_app().websocket_connect("/v1/stream") as socket:
socket.receive_json() # hello
socket.send_json({"type": "capabilities", "crop": True, "max_crop_bytes": 131072})
socket.receive_json()
socket.send_bytes(frame_payload(1))
socket.receive_json()
socket.send_bytes(frame_payload(2))
request = socket.receive_json()["tracks"][0]["crop_request"]
socket.send_json({
"type": "crop",
"request_id": request["request_id"],
"track_id": request["track_id"],
"frame_id": request["frame_id"],
"jpeg": base64.b64encode(b"not an image").decode("ascii"),
})
self.assertEqual(socket.receive_json()["code"], "CROP_UNDECODABLE")
def test_a_small_jpeg_claiming_huge_pixels_is_refused_before_decode(self):
with self.crop_app().websocket_connect("/v1/stream") as socket:
socket.receive_json()
socket.send_json({"type": "capabilities", "crop": True, "max_crop_bytes": 131072})
socket.receive_json()
socket.send_bytes(frame_payload(1))
socket.receive_json()
socket.send_bytes(frame_payload(2))
request = socket.receive_json()["tracks"][0]["crop_request"]
jpeg = bytearray(jpeg_bytes(80, 120))
sof = jpeg.index(b"\xff\xc0")
jpeg[sof + 5:sof + 9] = (16384).to_bytes(2, "big") * 2
socket.send_json({
"type": "crop", "request_id": request["request_id"],
"track_id": request["track_id"], "frame_id": request["frame_id"],
"jpeg": base64.b64encode(jpeg).decode("ascii"),
})
refused = socket.receive_json()
self.assertEqual(refused["code"], "CROP_TOO_LARGE")
socket.send_json({"type": "ping", "t": 1})
self.assertEqual(socket.receive_json(), {"type": "pong", "t": 1})
def test_a_crop_without_an_image_is_refused(self):
with self.client.websocket_connect("/v1/stream") as socket:
socket.receive_json() # hello
socket.send_json({"type": "crop", "request_id": "c1", "track_id": 1, "frame_id": 1})
refused = socket.receive_json()
self.assertEqual(refused["code"], "CROP_BAD_MESSAGE")
socket.send_json({"type": "ping", "t": 1})
self.assertEqual(socket.receive_json(), {"type": "pong", "t": 1})
def test_token_is_enforced_on_the_socket(self):
"""Header or the first post-hello auth frame. A query token is rejected."""
client, engine = make_app(token="s3cret")
try:
with self.assertRaises(Exception):
with client.websocket_connect("/v1/stream") as socket:
self.assertEqual(socket.receive_json()["type"], "hello")
socket.send_json({"type": "ping", "t": 1})
socket.receive_json()
with self.assertRaises(Exception):
with client.websocket_connect("/v1/stream?token=s3cret") as socket:
socket.receive_json()
with client.websocket_connect(
"/v1/stream", headers={"Authorization": "Bearer s3cret"}
) as socket:
self.assertEqual(socket.receive_json()["type"], "hello")
with client.websocket_connect("/v1/stream") as socket:
self.assertEqual(socket.receive_json()["type"], "hello")
socket.send_json({"type": "auth", "token": "s3cret"})
socket.send_json({"type": "ping", "t": 1})
self.assertEqual(socket.receive_json(), {"type": "pong", "t": 1})
finally:
engine.shutdown()
def test_session_limit_closes_extra_clients(self):
client, engine = make_app(max_sessions=1)
try:
with client.websocket_connect("/v1/stream") as first:
first.receive_json()
with self.assertRaises(Exception):
with client.websocket_connect("/v1/stream") as second:
second.receive_json()
finally:
engine.shutdown()
def test_lifespan_shutdown_skips_a_reserved_slot(self):
"""A reservation is None until setup finishes. Shutdown must not call it."""
client, engine = make_app()
client.app.state.sessions["reserved"] = None
try:
with client:
pass
finally:
engine.shutdown()
if __name__ == "__main__":
unittest.main()

20
recognization/code/test_empty.py Normal file
View file

@ -0,0 +1,20 @@
import sys
import os
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from empty_shelf_detector import detect_and_draw_empty_slots
if __name__ == "__main__":
image_path = "image.jpg"
print(f"Processing empty shelf detection on {image_path}...")
annotated, empty_slots = detect_and_draw_empty_slots(
image_path,
output_path="detected_empty_slots.jpg"
)
print(f"Detected {len(empty_slots)} empty shelf slots:")
for s in empty_slots:
print(f" - Slot [{s['id']}]: Row {s['row_index']}, Box={s['box']}, Width={s['width']}px, Est. Missing Items={s['missing_items_est']}")
print(f"Visualization saved to: detected_empty_slots.jpg")

861
recognization/code/test_engine.py
View file

@ -1,861 +0,0 @@
"""
Contract tests for the recognition engine.
No YOLO weights and no network: a fake detector and a fake matcher stand in for
the model, so these tests defend the engine's own contracts - normalization,
locking, isolation between clients, empty-slot geometry, and the still-image
store.
"""
from __future__ import annotations
import os
import sys
import threading
import time
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from engine import ( # noqa: E402
EngineSettings,
ImageExpired,
ImageStore,
RecognitionEngine,
UndecodableImage,
UnknownMethod,
identity_payload,
)
class TextLaneSignatureTests(unittest.TestCase):
"""The lane must score a crop against the row's name, not its packshot.
l09's packshot prints "マヨネーズ", which is l10's actual name. Scoring the
packshot's OCR as part of l09's signature let l09 explain an l10 crop's
text better than l10's own name could, so l09 won l10's facings. Measured
on image6: l10 correct went 2 -> 5 of 11, wrong 3 -> 1, precision 95% ->
97%, and the l09 hub alarm cleared.
"""
ROWS = [
{"sku_id": "sku_l09", "product_name": "ピュアセレクト コクうま カロリーカット65% 360g"},
{"sku_id": "sku_l10", "product_name": "ピュアセレクト マヨネーズ 450g"},
{"sku_id": "sku_l03", "product_name": "キューピー からしマヨネーズ 200g"},
]
def lane(self):
from identity_lanes import TextLane
return TextLane.from_catalog(self.ROWS, "no-such-dir", ocr_engine=None)
def test_a_read_of_the_plain_mayonnaise_names_the_plain_mayonnaise(self):
scores = self.lane().scores_from_text(
"AJINOMOTO とれて3日以内の 国産新鮮たまご ピュア セレクト マヨネーズ"
)
self.assertEqual(max(scores, key=scores.get), "sku_l10")
def test_a_read_of_the_variant_names_the_variant(self):
scores = self.lane().scores_from_text("ピュアセレクト コクうま カロリーカット65%")
self.assertEqual(max(scores, key=scores.get), "sku_l09")
def test_shared_brand_copy_alone_does_not_name_a_row(self):
"""Every row prints マヨネーズ, so it must not decide between them.
The lane scores l10 and l03 equally on that word alone; the policy's
margin gate is what turns the tie into no answer.
"""
from identity_lanes import DecisionPolicy
scores = self.lane().scores_from_text("マヨネーズ")
self.assertEqual(scores["sku_l10"], scores["sku_l03"])
decision = DecisionPolicy().decide([], scores, text_read="マヨネーズ")
self.assertIsNone(decision["best_sku"])
def test_a_signature_never_pairs_across_two_words(self):
"""A catalog name must not claim text it does not contain.
"キューピー アマニ油" normalizes to "キユピ アマニ油"; with the space
dropped it was "キユピアマニ油", which contains "ピア" - a pair neither
word has. That pair is rare across the catalog, so IDF weighted it as
distinctive and the garbage read "キユピアク5C" was accepted as l02.
"""
from rebuild_catalog import signature_bigrams
name = "キューピー アマニ油マヨネーズ 200g"
self.assertNotIn("ピア", signature_bigrams(name))
self.assertIn("アマ", signature_bigrams(name))
def test_a_read_keeps_matching_across_the_spaces_ocr_inserts(self):
"""The read is the noisy side, so its spaces must not break a match.
Vision prints "ピュア セレクト" where the catalog name has
"ピュアセレクト". Refusing to pair across the read's spaces dropped
two correct l10 labels below the accept line, so the read pairs
across them and the signature is the side that stays strict.
"""
from rebuild_catalog import bigrams
self.assertIn("アセ", bigrams("ピュア セレクト マヨネーズ"))
def test_a_garbage_read_does_not_name_a_row(self):
"""The read that the fake pair used to accept must not name a row.
It shares only "キユ" and "ユピ" with l02's name, both of which every
Kewpie row prints. Bounded to the catalog's own vocabulary the read
keeps just those two pairs, so it scores the same against every
Kewpie row and the margin gate refuses it - the same mechanism that
already handles a read of shared brand copy alone.
"""
from identity_lanes import DecisionPolicy, TextLane
rows = self.ROWS + [{"sku_id": "sku_l02", "product_name": "キューピー アマニ油マヨネーズ 200g"}]
read = "キユピアク5C"
scores = TextLane.from_catalog(rows, "no-such-dir", ocr_engine=None).scores_from_text(read)
ranked = sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
self.assertEqual(ranked[0][1], ranked[1][1], "the read cannot tell the Kewpie rows apart")
decision = DecisionPolicy().decide([], ranked, text_read=read)
self.assertIsNone(decision["best_sku"])
def test_a_read_of_real_text_still_names_its_row(self):
"""Bounding the denominator must not cost a correct read its answer.
This is the facing the strict-read variant broke: l10's own name read
with OCR's spacing, which has to clear the accept line by margin.
"""
from identity_lanes import TEXT_ACCEPT, TEXT_MARGIN, TextLane
rows = self.ROWS + [{"sku_id": "sku_l02", "product_name": "キューピー アマニ油マヨネーズ 200g"}]
scores = TextLane.from_catalog(rows, "no-such-dir", ocr_engine=None).scores_from_text(
"213日以月A ピュア セレクト マヨネーズ"
)
ranked = sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
self.assertEqual(ranked[0][0], "sku_l10")
self.assertGreaterEqual(ranked[0][1], TEXT_ACCEPT)
self.assertGreaterEqual(ranked[0][1] - ranked[1][1], TEXT_MARGIN)
def test_a_thin_read_is_not_enough_to_name_a_row(self):
"""A ratio of 1.0 on a few pairs is not evidence.
"XXイス" contains exactly one pair the catalog knows, and the row that
contains it scores 1.0 with a full margin over every other row. On
image6 that shape named sku_l08 from the read "サソース". The floor asks
how much of the read the row actually explains, so one pair and two
pairs are both refused while a real read of the name is accepted.
Two pairs used to be enough. It is not: on 2297 crops from unrelated
Wikimedia shelves, 20 of 26 text-lane answers rested on exactly two
shared bigrams - "避くて おいしい" named sku_l12 through おい/いし - for
an 11.4% false accept rate. The floor is 4.
"""
from identity_lanes import DecisionPolicy, TextLane
rows = [
{"sku_id": "sku_l08", "product_name": "オイスターソース"},
{"sku_id": "sku_l01", "product_name": "キューピー マヨネーズ"},
]
lane = TextLane.from_catalog(rows, "no-such-dir", ocr_engine=None)
for thin_read in ("XXイス", "ZZソース"):
thin = sorted(lane.scores_from_text(thin_read).items(),
key=lambda kv: kv[1], reverse=True)
self.assertEqual(thin[0][1], 1.0, "the ratio alone cannot see the difference")
self.assertIsNone(
DecisionPolicy(signatures=lane.signatures).decide(
[], thin, text_read=thin_read
)["best_sku"],
f"{thin_read!r} shares too few pairs to name a row",
)
enough = sorted(lane.scores_from_text("オイスター").items(),
key=lambda kv: kv[1], reverse=True)
self.assertEqual(
DecisionPolicy(signatures=lane.signatures).decide(
[], enough, text_read="オイスター"
)["best_sku"],
"sku_l08",
)
def test_one_shared_pair_is_refused_by_the_floor_alone(self):
"""Everything but the evidence floor says answer, and it says no.
"アンース" holds one pair the catalog knows - "ース" - so the row scores
a perfect 1.0 with the whole margin over an empty runner-up, and the
read carries three pairs, so `carries_identity` passes as well. Only
`TEXT_MIN_SHARED` stands between a coincidental pair and a named
product, which is why the test asserts the score is 1.0: it pins the
floor as the only thing doing the refusing.
"""
from identity_lanes import DecisionPolicy, TextLane
lane = TextLane.from_catalog(
[{"sku_id": "sku_x", "product_name": "オイスターソース"}],
"no-such-dir",
ocr_engine=None,
)
read = "アンース"
self.assertEqual(lane.scores_from_text(read)["sku_x"], 1.0)
self.assertIsNone(
DecisionPolicy(signatures=lane.signatures).decide(
[], [("sku_x", 1.0)], text_read=read
)["best_sku"],
"one shared pair is not enough to name a row",
)
def test_a_thin_read_still_vetoes_even_though_it_cannot_name(self):
"""The floor is a floor on naming, not on contradicting.
"北海道塩" shares three bigrams with sku_b - one short of the floor -
so it cannot name that row. It is still the row the text prefers over
the local answer, and the local lane clears its own line at 5.0 against
a line of 4.0, so if the floor also sat under the veto this would come
back as sku_a: a three-bigram read would have rescued a match the
geometry earned.
"""
from identity_lanes import DecisionPolicy, TextLane
lane = TextLane.from_catalog(
[
{"sku_id": "sku_a", "product_name": "キューピー ハーフマヨネーズ 400g"},
{"sku_id": "sku_b", "product_name": "北海道塩"},
],
"no-such-dir",
ocr_engine=None,
)
read = "北海道塩"
ranked = sorted(lane.scores_from_text(read).items(),
key=lambda kv: kv[1], reverse=True)
self.assertEqual((ranked[0][0], ranked[0][1]), ("sku_b", 1.0))
self.assertIsNone(
DecisionPolicy(signatures=lane.signatures).decide(
[("sku_a", 5.0)], ranked, text_read=read
)["best_sku"],
"a read too thin to name a row still refuses the local answer",
)
class FakeDetector:
"""Records concurrency and returns fixed boxes."""
weights_path = "fake.pt"
def __init__(self, boxes=None):
self.boxes = boxes if boxes is not None else [
{"id": 0, "box": [10, 10, 60, 90], "confidence": 0.9, "class_id": 0,
"width": 50, "height": 80, "center": [35, 50]},
{"id": 1, "box": [70, 10, 120, 90], "confidence": 0.8, "class_id": 0,
"width": 50, "height": 80, "center": [95, 50]},
]
self.concurrent = 0
self.max_concurrent = 0
self.calls = 0
self.lock = threading.Lock()
def detect(self, image, conf_threshold=0.25, iou_threshold=0.45):
with self.lock:
self.calls += 1
self.concurrent += 1
self.max_concurrent = max(self.max_concurrent, self.concurrent)
time.sleep(0.01)
with self.lock:
self.concurrent -= 1
return {"status": "success", "boxes": [dict(b) for b in self.boxes]}
def detect_empty_slots(self, image_shape, boxes, min_gap_ratio=0.75):
# One gap between the two boxes, expressed in the frame the engine hands
# over. Scaling with that frame is what makes the normalization test
# meaningful: the engine must translate the answer back to the client's
# pixel space.
height, width = int(image_shape[0]), int(image_shape[1])
x1, x2 = int(width * 0.30), int(width * 0.35)
return [{
"id": "empty_1",
"box": [x1, int(height * 0.10), x2, int(height * 0.90)],
"width": x2 - x1,
"height": int(height * 0.80),
"missing_items_est": 1,
"center": [(x1 + x2) // 2, height // 2],
"row_index": 1,
}]
class FakeMatcher:
def __init__(self):
self.calls = 0
def score_crop(self, crop, method="hybrid"):
self.calls += 1
time.sleep(0.01)
return {"sku_a": 40.0, "sku_b": 2.0}, {"sku_a": {"score": 40.0}}
def score_crop_fast(self, crop, shortlist=5):
return self.score_crop(crop)
def classify_crop(self, crop, method="hybrid"):
from identity import decide_identity
scores, details = self.score_crop(crop, method=method)
return decide_identity(scores, details=details, method=method)
def generate_match_visualization(self, crop, sku, method="surf", max_matches=30):
return crop
class FakeInventory:
def __init__(self):
self.products = {
"sku_a": {"sku_id": "sku_a", "product_name": "Alpha", "price": 100.0,
"number_in": 10, "number_out": 2, "number_on": 8,
"receipt_date": "2026-01-01", "expiration_date": "2026-12-31",
"local_image_path": None},
"sku_b": {"sku_id": "sku_b", "product_name": "Beta", "price": 200.0,
"number_in": 5, "number_out": 1, "number_on": 4,
"receipt_date": "2026-01-02", "expiration_date": "2026-12-30",
"local_image_path": None},
}
self.lock = threading.Lock()
def list_all_skus(self):
return list(self.products)
def get_product(self, sku_id):
return self.products.get(sku_id)
def update_stock(self, sku_id, delta):
with self.lock:
product = self.products.get(sku_id)
if product is None:
return False
product["number_on"] = max(0, product["number_on"] + delta)
if delta < 0:
product["number_out"] += abs(delta)
else:
product["number_in"] += delta
return True
def make_engine(**overrides):
# The fake matcher is what these tests exercise, so the backend is pinned to
# `matcher`; the shipped default is the `local` lane, which reads real
# reference images and would ignore the fake.
overrides.setdefault("identity_backend", "matcher")
settings = EngineSettings(device="cpu", classify_workers=4, **overrides)
detector = FakeDetector()
matcher = FakeMatcher()
inventory = FakeInventory()
engine = RecognitionEngine(detector, matcher, inventory, settings=settings)
return engine, detector, matcher, inventory
def make_image(width=200, height=120):
import numpy as np
return np.zeros((height, width, 3), dtype="uint8")
class EngineBasicsTests(unittest.TestCase):
def test_detect_normalizes_and_reindexes(self):
engine, _, _, _ = make_engine()
boxes = engine.detect(make_image())
self.assertEqual(len(boxes), 2)
self.assertEqual([b["id"] for b in boxes], [0, 1])
self.assertEqual(boxes[0]["width"], 50)
self.assertEqual(boxes[0]["center"], [35, 50])
def test_detector_calls_never_overlap(self):
engine, detector, _, _ = make_engine()
threads = [threading.Thread(target=engine.detect, args=(make_image(),)) for _ in range(8)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
self.assertEqual(detector.calls, 8)
self.assertEqual(detector.max_concurrent, 1, "the model must be used by one caller at a time")
def test_unknown_method_is_rejected(self):
engine, _, _, _ = make_engine()
with self.assertRaises(UnknownMethod):
engine.score_crop(make_image(), method="not-a-method")
def test_undecodable_payload_raises(self):
engine, _, _, _ = make_engine()
with self.assertRaises(UndecodableImage):
engine.decode_image(b"this is not an image")
def test_crop_is_a_copy(self):
engine, _, _, _ = make_engine()
image = make_image()
crop = engine.crop(image, [0, 0, 10, 10])
crop[0, 0, 0] = 255
self.assertEqual(int(image[0, 0, 0]), 0)
class EngineClassificationTests(unittest.TestCase):
def test_classify_boxes_accepts_detection_dicts_without_size_fields(self):
"""The API hands classify_boxes stored detections, which carry no width/height."""
engine, _, _, _ = make_engine()
detections = [
{"id": 0, "box": [10, 10, 60, 90], "confidence": 0.9, "identity": identity_payload(None)},
{"id": 1, "box": [70, 10, 120, 90], "confidence": 0.8, "identity": identity_payload(None)},
]
rows = engine.classify_boxes(make_image(), detections, method="hybrid")
self.assertEqual(len(rows), 2)
self.assertEqual(rows[0]["identity"]["state"], "recognized")
self.assertEqual(rows[0]["identity"]["sku_id"], "sku_a")
def test_an_accepted_detection_carries_its_catalog_row(self):
"""The phone must not have to look up a name and price from a bare SKU.
`classify_boxes` returned only the identity, so every accepted
still-image answer reached the client as an id with no product row:
measured through the real server, 41 of 41 accepted detections had no
`product`. The live path attached it; the still path did not.
"""
engine, _, _, _ = make_engine()
detections = [
{"id": 0, "box": [10, 10, 60, 90], "confidence": 0.9,
"identity": identity_payload(None)},
]
rows = engine.classify_boxes(make_image(), detections, method="hybrid")
self.assertEqual(rows[0]["identity"]["state"], "recognized")
self.assertEqual(rows[0]["product"]["product_name"], "Alpha")
self.assertEqual(rows[0]["product"]["price"], 100.0)
result = engine.recognize(make_image(), method="hybrid")
accepted = [
row for row in result["detections"]
if row["identity"]["identity_level"] == "accepted"
]
self.assertTrue(accepted, "the fake detector and matcher accept a box")
for row in accepted:
self.assertIn("product", row, "an accepted answer must carry its row")
def test_a_failing_crop_reports_unrecognized_not_pending(self):
"""One bad crop must not fail the batch, and must not read as pending.
`identity_payload(None)` is the "not yet classified" shape; an
operational failure has to stay distinguishable from absence.
"""
engine, _, matcher, _ = make_engine()
def explode(crop, method=None):
raise RuntimeError("classifier blew up")
matcher.score_crop = explode
rows = engine.classify_boxes(make_image(), [
{"id": 0, "box": [10, 10, 60, 90], "confidence": 0.9},
], method="hybrid")
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["identity"]["state"], "unrecognized")
self.assertIsNone(rows[0]["identity"]["sku_id"])
def test_parallel_classification_matches_sequential(self):
engine, _, matcher, _ = make_engine()
boxes = [
{"id": index, "box": [10 + index, 10, 60 + index, 90], "confidence": 0.9,
"width": 50, "height": 80, "center": [35, 50]}
for index in range(12)
]
parallel = engine.classify_boxes(make_image(), boxes, method="hybrid")
sequential = [engine.score_crop(engine.crop(make_image(), box["box"]), "hybrid") for box in boxes]
self.assertEqual(len(parallel), 12)
self.assertEqual(len(sequential), 12)
for row in parallel:
self.assertEqual(row["identity"]["sku_id"], "sku_a")
def test_summary_counts_states(self):
engine, _, _, _ = make_engine()
detections = [
{"identity": {"state": "recognized", "sku_id": "sku_a"}},
{"identity": {"state": "recognized", "sku_id": "sku_a"}},
{"identity": {"state": "needs_confirmation", "sku_id": None}},
{"identity": {"state": "unrecognized", "sku_id": None}},
]
summary = engine.summarize(detections, [])
self.assertEqual(summary["recognized"], 2)
self.assertEqual(summary["needs_confirmation"], 1)
self.assertEqual(summary["unrecognized"], 1)
self.assertEqual(summary["by_sku"], [{"sku_id": "sku_a", "facings": 2}])
class EmptySlotNormalizationTests(unittest.TestCase):
def test_slots_scale_with_the_frame(self):
"""The detector's thresholds are absolute pixels, so a half-size frame must
produce the same slots (scaled) as the reference-size frame."""
engine, _, _, _ = make_engine()
boxes_small = [
{"box": [5, 5, 30, 45]},
{"box": [35, 5, 60, 45]},
]
slots_small = engine.empty_slots((60, 100), boxes_small)
boxes_big = [
{"box": [10, 10, 60, 90]},
{"box": [70, 10, 120, 90]},
]
slots_big = engine.empty_slots((120, 200), boxes_big)
self.assertEqual(len(slots_small), 1)
self.assertEqual(len(slots_big), 1)
# the small frame is scaled up to the reference width, so the returned
# slot comes back in the small frame's own pixels
self.assertLess(slots_small[0]["box"][2], slots_big[0]["box"][2])
class ImageStoreTests(unittest.TestCase):
def test_roundtrip_and_expiry(self):
store = ImageStore(max_items=2, ttl_s=0.05, max_bytes=10_000_000)
stored = store.put(make_image())
self.assertEqual(store.get(stored.image_id).width, 200)
time.sleep(0.08)
with self.assertRaises(ImageExpired):
store.get(stored.image_id)
def test_bounded_by_count(self):
store = ImageStore(max_items=2, ttl_s=60, max_bytes=10_000_000)
ids = [store.put(make_image()).image_id for _ in range(4)]
with self.assertRaises(ImageExpired):
store.get(ids[0])
self.assertEqual(store.get(ids[-1]).width, 200)
class CatalogTests(unittest.TestCase):
def test_catalog_and_stock_update(self):
engine, _, _, _ = make_engine()
catalog = engine.catalog()
self.assertEqual(len(catalog), 2)
self.assertEqual(catalog[0]["image_url"], "/v1/catalog/sku_a/image")
product = engine.update_stock("sku_a", -1)
self.assertEqual(product["number_on"], 7)
self.assertEqual(product["number_out"], 3)
self.assertIsNone(engine.update_stock("nope", -1))
class GalleryLoadTests(unittest.TestCase):
def test_a_failed_gallery_load_returns_none_and_is_not_retried(self):
engine, _, _, _ = make_engine(identity_backend="gallery", stock_dir="does-not-exist")
class BrokenGallery:
def __init__(self, *args, **kwargs):
pass
def load(self):
raise FileNotFoundError("DINOv2 weights not found")
import gallery as gallery_module
original = gallery_module.ProductGallery
gallery_module.ProductGallery = BrokenGallery
try:
self.assertIsNone(engine.identity_gallery())
self.assertIsNone(engine.identity_gallery())
finally:
gallery_module.ProductGallery = original
engine.shutdown()
self.assertIs(engine._gallery, False)
class LocalLaneTests(unittest.TestCase):
"""The shipped default backend, which the fake matcher cannot exercise.
A two-SKU catalog of real images is built in a temp dir, so the lane reads
genuine reference files and the decision is the one production makes.
"""
def make_catalog(self):
import tempfile
import cv2
import numpy as np
root = tempfile.mkdtemp(prefix="lane-catalog-")
rng = np.random.default_rng(7)
for sku, base in (("sku_a", (200, 60, 60)), ("sku_b", (60, 60, 200))):
image = np.zeros((400, 300, 3), dtype="uint8")
image[:, :] = base
# Structured, high-contrast marks: SIFT needs corners, not noise.
for index in range(12):
x = int(rng.integers(10, 280))
y = int(rng.integers(10, 380))
cv2.rectangle(image, (x, y), (x + 40, y + 40), (255, 255, 255), 3)
cv2.putText(image, sku[-1].upper(), (x + 5, y + 30),
cv2.FONT_HERSHEY_SIMPLEX, 1.0, (0, 0, 0), 2)
cv2.imwrite(os.path.join(root, f"{sku}.jpg"), image)
with open(os.path.join(root, "stock.csv"), "w", encoding="utf-8") as handle:
handle.write("SKU_ID,Product_Name,Price,Number_In,Number_Out,Number_On\n")
handle.write("sku_a,Alpha,100,1,0,1\n")
handle.write("sku_b,Beta,200,1,0,1\n")
return root
def test_decide_crop_names_the_matching_sku(self):
import cv2
root = self.make_catalog()
engine, _, _, _ = make_engine(identity_backend="local", stock_dir=root)
try:
self.assertEqual(engine.identity_scale, "local")
self.assertIsNotNone(engine.identity_lane())
# The reference itself is the easiest possible query for its own lane.
crop = cv2.imread(os.path.join(root, "sku_a.jpg"))
decision = engine.decide_crop(crop)
self.assertEqual(decision.get("best_sku"), "sku_a")
self.assertEqual(decision.get("identity_level"), "accepted")
finally:
engine.shutdown()
def test_score_crop_is_keyed_by_canonical_sku(self):
import cv2
root = self.make_catalog()
engine, _, _, _ = make_engine(identity_backend="local", stock_dir=root)
try:
scores = engine.score_crop(cv2.imread(os.path.join(root, "sku_b.jpg")))
self.assertTrue(scores)
# No view-suffixed ids: every key is a catalog SKU.
self.assertTrue(all("_" not in sku[4:] for sku in scores), scores)
finally:
engine.shutdown()
def test_a_featureless_crop_is_not_named(self):
import numpy as np
root = self.make_catalog()
engine, _, _, _ = make_engine(identity_backend="local", stock_dir=root)
try:
blank = np.zeros((200, 150, 3), dtype="uint8")
decision = engine.decide_crop(blank)
self.assertIsNone(decision.get("best_sku"))
finally:
engine.shutdown()
def test_warm_up_builds_both_lanes(self):
root = self.make_catalog()
engine, _, _, _ = make_engine(identity_backend="local", stock_dir=root)
try:
engine.warm_up()
# Built eagerly, so the first live frame never pays for them.
self.assertIsNotNone(engine._lane)
self.assertIsNotNone(engine._text_lane)
finally:
engine.shutdown()
class DecisionPolicyTests(unittest.TestCase):
"""The two jobs the text lane does, and the boundary between them."""
def policy(self):
from identity_lanes import DecisionPolicy
return DecisionPolicy()
def test_local_answer_stands_when_the_text_agrees(self):
decision = self.policy().decide([("sku_a", 5.0)], [("sku_a", 0.9)])
self.assertEqual(decision["best_sku"], "sku_a")
self.assertEqual(decision["source"], "local")
def test_text_vetoes_a_local_answer_naming_another_row(self):
"""The near-unknown case: real geometry, different words.
The veto blocks the local answer, and the text lane then names its own
row - the two jobs compose, so the result is the text's SKU, never the
vetoed local one.
"""
decision = self.policy().decide(
[("sku_a", 5.0)], [("sku_b", 0.9)], text_read="コクうま 65%"
)
self.assertEqual(decision["best_sku"], "sku_b")
self.assertEqual(decision["source"], "text")
def test_text_mentioning_the_local_candidate_is_not_a_veto(self):
"""A crop reading "マヨネーズ" scores every mayonnaise row.
The local candidate is usually one of them, so a veto needs the text to
*prefer* another row, not merely to mention it: here the text mentions
sku_b but explains sku_a better, so the local answer stands.
"""
decision = self.policy().decide([("sku_a", 5.0)], [("sku_b", 0.4), ("sku_a", 0.45)])
self.assertEqual(decision["best_sku"], "sku_a")
self.assertEqual(decision["source"], "local")
def test_the_lanes_own_order_wins_over_the_value_order(self):
"""Order and value are different quantities on the local lane.
The order comes from `rank_key` (inliers weighted by hue); the value is
the margin over the runner-up. Re-sorting by value names a different SKU
than the lane chose - here sku_b has the larger value but sku_a is the
lane's own first choice, and sku_a is what must be answered.
Both values clear LOCAL_ACCEPT, so the test is about the ordering and
not about the gate.
"""
decision = self.policy().decide([("sku_a", 4.5), ("sku_b", 5.0)], [])
self.assertEqual(decision["best_sku"], "sku_a")
def test_a_size_only_read_does_not_fill_in(self):
"""Size and number tokens are not identity text."""
decision = self.policy().decide([], [("sku_b", 0.9)], text_read="400g")
self.assertIsNone(decision["best_sku"])
def test_a_size_only_read_still_vetoes(self):
"""The veto keeps "65%" usable: the local lane supplied the evidence."""
decision = self.policy().decide(
[("sku_a", 5.0)], [("sku_b", 0.9)], text_read="65%"
)
self.assertIsNone(decision["best_sku"])
def test_a_read_printing_another_size_vetoes_the_local_answer(self):
"""The documented near-unknown: a Kewpie 450 g bag the catalog lacks.
The crop reads "マヨネーズ 450g", which every mayonnaise row explains
equally, so the text ranking ties at 1.0 and the strict-greater veto
cannot fire. The size is what differs, and the catalog's Kewpie rows
are 200 g, so the read contradicts the row the local lane chose.
"""
from identity_lanes import DecisionPolicy
policy = DecisionPolicy(signatures={"sku_a": "キューピー ハーフマヨネーズ 400g"})
decision = policy.decide(
[("sku_a", 5.0)], [("sku_a", 1.0)], text_read="ピーマヨネーズ 450g"
)
self.assertIsNone(decision["best_sku"])
def test_a_read_agreeing_on_size_does_not_veto(self):
"""The same read with the row's own size must still answer."""
from identity_lanes import DecisionPolicy
policy = DecisionPolicy(signatures={"sku_a": "キューピー ハーフマヨネーズ 400g"})
decision = policy.decide(
[("sku_a", 5.0)], [("sku_a", 1.0)], text_read="ピーマヨネーズ 400g"
)
self.assertEqual(decision["best_sku"], "sku_a")
def test_ocr_noise_around_a_size_does_not_veto(self):
"""A fragment or a run-together number is noise, not a package size.
Live reads of a correct 200 g row came back as "20" and "5210"; both
are outside the range a package of this class falls in, and vetoing on
either would cost a correct answer.
"""
from identity_lanes import DecisionPolicy
policy = DecisionPolicy(signatures={"sku_a": "キューピー ハーフマヨネーズ 200g"})
for read in ("アマニ油 20gs", "内臓脂訪を 滅与ず ッデュービーフィ5 210g", "n0gベ% 50"):
decision = policy.decide([("sku_a", 5.0)], [("sku_a", 1.0)], text_read=read)
self.assertEqual(decision["best_sku"], "sku_a", read)
def test_a_percentage_is_not_a_package_size(self):
"""A low-calorie claim must not read as a 50 g package."""
from identity_lanes import DecisionPolicy
policy = DecisionPolicy(signatures={"sku_a": "キューピー ハーフマヨネーズ 400g"})
decision = policy.decide(
[("sku_a", 5.0)], [("sku_a", 1.0)], text_read="カロリー 50% カット 400g"
)
self.assertEqual(decision["best_sku"], "sku_a")
def test_a_named_read_with_a_size_still_fills_in(self):
decision = self.policy().decide([], [("sku_b", 0.9)], text_read="コクうま 65%")
self.assertEqual(decision["best_sku"], "sku_b")
def test_text_fills_in_when_local_has_nothing(self):
decision = self.policy().decide([], [("sku_b", 0.9)], text_read="コクうま 65%")
self.assertEqual(decision["best_sku"], "sku_b")
self.assertEqual(decision["source"], "text")
def test_a_ranking_without_a_read_does_not_fill_in(self):
"""Omitting the read is the safe default, not an ungated one.
A caller that has a ranking and no read has not shown the text can name
anything, so the fill-in job stays closed.
"""
decision = self.policy().decide([], [("sku_b", 0.9)])
self.assertIsNone(decision["best_sku"])
def test_quantity_strings_are_not_names(self):
"""Every pure quantity string, including the ones a digits-only rule let through.
"400g 50%" and "450g カロリー50%" survived the first version of this
gate: it counted non-digit bigrams, and the unit's own letters ("0G",
"0%") are non-digits. Cutting the whole quantity token is what closes
it. "450g カロリー50%" still passes, and correctly so - "カロリー" is
four characters of real text, and it is the text lane's job to find
that no catalog row is named by it.
"""
from identity_lanes import carries_identity
for read in ("400g", "400g 50%", "500ml", "65%", "3日", "中辛"):
with self.subTest(read=read):
self.assertFalse(carries_identity(read), f"{read!r} is not a name")
for read in ("コクうま 65%", "450g カロリー50%"):
with self.subTest(read=read):
self.assertTrue(carries_identity(read), f"{read!r} carries name text")
def test_text_below_its_own_line_does_not_fill_in(self):
decision = self.policy().decide([], [("sku_b", 0.3)])
self.assertIsNone(decision["best_sku"])
def test_a_close_text_pair_does_not_fill_in(self):
"""The margin gate: two rows the text explains equally must not name one."""
decision = self.policy().decide([], [("sku_b", 0.9), ("sku_c", 0.85)])
self.assertIsNone(decision["best_sku"])
def test_no_evidence_is_rejected(self):
decision = self.policy().decide([], [])
self.assertIsNone(decision["best_sku"])
self.assertEqual(decision["identity_level"], "rejected")
class LocalLaneConcurrencyTests(unittest.TestCase):
"""The lane is shared by the classify pool, so `rank` must be thread-safe.
One `cv2.BFMatcher` is reused across calls. The classify pool runs up to
eight workers, so a lane that mutated matcher state per call would return
another crop's answer under load - the failure would look like a
recognition error, not a crash.
"""
def test_concurrent_ranking_matches_solo_ranking(self):
import threading
import cv2
from identity_lanes import LocalLane
stock_dir = os.path.join(os.path.dirname(current_dir), "stock-lopia-clean")
if not os.path.isdir(stock_dir):
self.skipTest("no catalog checked out")
image_path = os.path.join(os.path.dirname(current_dir), "add", "image6.jpg")
if not os.path.isfile(image_path):
self.skipTest("no evaluation photo checked out")
lane = LocalLane(stock_dir)
image = cv2.imread(image_path)
crops = [image[700:1000, 600 + 40 * i:800 + 40 * i] for i in range(8)]
solo = lane.rank(crops[0])
results = {}
errors = []
def work(index: int) -> None:
try:
for _ in range(6):
results[index] = lane.rank(crops[index])
except Exception as error: # noqa: BLE001 - reported below
errors.append(repr(error))
threads = [threading.Thread(target=work, args=(i,)) for i in range(8)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
self.assertEqual(errors, [])
self.assertEqual(results[0], solo, "a shared matcher must not cross crops")
class CatalogImageTests(unittest.TestCase):
def test_catalog_image_path_is_resolved_through_inventory(self):
engine, _, _, _ = make_engine()
self.assertIsNone(engine.catalog_image_path("../../etc/passwd"))
self.assertIsNone(engine.catalog_image_path("sku_a"))
if __name__ == "__main__":
unittest.main()

58
recognization/code/test_eval_labels.py
View file

@ -1,58 +0,0 @@
"""
The evaluator's label-file contract.
F-01: the v1 labels were wrong on 23% of facings and the file recorded nothing
about where any of them came from, so a bad label could not be told from a good
one. `load_labels` now refuses a file whose rows carry no provenance.
"""
from __future__ import annotations
import json
import os
import sys
import tempfile
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from eval_identity import load_labels # noqa: E402
def write_labels(payload) -> str:
handle = tempfile.NamedTemporaryFile("w", suffix=".json", delete=False)
json.dump(payload, handle)
handle.close()
return handle.name
class LoadLabelsTests(unittest.TestCase):
def test_a_traced_label_file_loads(self):
path = write_labels({
"image": "add/image6.jpg",
"facings": [{"index": 0, "label": "sku_l01", "source": "visual-audit"}],
})
self.addCleanup(os.unlink, path)
self.assertEqual(len(load_labels(path)["facings"]), 1)
def test_a_label_without_provenance_is_refused(self):
path = write_labels({
"image": "add/image6.jpg",
"facings": [{"index": 0, "label": "sku_l01"}],
})
self.addCleanup(os.unlink, path)
with self.assertRaises(SystemExit) as caught:
load_labels(path)
self.assertIn("source", str(caught.exception))
def test_a_file_without_facings_is_refused(self):
path = write_labels({"image": "add/image6.jpg"})
self.addCleanup(os.unlink, path)
with self.assertRaises(SystemExit):
load_labels(path)
if __name__ == "__main__":
unittest.main()

338
recognization/code/test_eval_live.py
View file

@ -1,338 +0,0 @@
"""Metric contracts for eval_live. No model, no real walk, no fabricated accuracy."""
from __future__ import annotations
import json
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
import numpy as np # noqa: E402
from eval_live import ( # noqa: E402
IDENTITY_BACKENDS,
EvalInputError,
identity_level,
labelled_top1,
leaked_labels,
load_index,
recorded_identity,
resolve_identity,
score_recording,
score_tracks,
track_churn,
)
def track(level, sku=None, track_id=1, box=None, relabels=0, restarts=0, tag=None):
identity = {"identity_level": level, "state": "recognized" if level == "accepted" else "unrecognized"}
if level == "accepted":
identity["sku_id"] = sku
row = {
"track_id": track_id,
"box": box or [0, 0, 10, 10],
"identity": identity,
"relabels": relabels,
"restarts": restarts,
}
if tag is not None:
row["price_tag"] = tag
return row
def write_jpeg(path, image):
ok, encoded = cv2.imencode(".jpg", image)
assert ok
with open(path, "wb") as handle:
handle.write(encoded.tobytes())
class DecisionDenominatorTests(unittest.TestCase):
def test_settled_excludes_unsettled_and_rates_use_that_denominator(self):
scored = score_tracks(
[
track("accepted", "sku_a"),
track("ambiguous"),
track("rejected"),
track("unknown"),
{"identity": {"state": "unconfirmed", "identity_level": None}},
]
)
self.assertEqual(scored["settled"], 4)
self.assertEqual(scored["unsettled"], 1)
self.assertEqual(scored["accepted_rate"], 0.25)
self.assertIsNone(score_tracks([{"identity": {"state": "unconfirmed"}}])["accepted_rate"])
def test_wire_state_fills_a_missing_level(self):
self.assertEqual(
identity_level({"identity": {"state": "needs_confirmation"}}), "ambiguous"
)
self.assertEqual(identity_level({"identity": {"state": "unconfirmed"}}), "unsettled")
def test_readable_tags_are_counted_apart_from_detected(self):
scored = score_tracks(
[
track("accepted", "sku_a", tag={"state": "readable"}),
track("rejected", tag={"state": "unreadable"}),
track("rejected", tag={"state": "absent"}),
track("rejected"),
]
)
self.assertEqual(scored["tags_readable"], 1)
self.assertEqual(scored["tags_detected"], 2)
self.assertEqual(scored["tags_absent"], 2)
class Top1AndChurnTests(unittest.TestCase):
def test_top1_counts_only_accepted_matches_over_overlapping_labels(self):
frames = [
{
"frame": 1,
"tracks": [
track("accepted", "sku_a", box=[0, 0, 10, 10]),
track("accepted", "sku_b", track_id=2, box=[100, 100, 110, 110]),
],
},
{"frame": 2, "tracks": [track("rejected", box=[0, 0, 10, 10])]},
{"frame": 3, "tracks": []},
]
labels = [
{"frame": 1, "box": [0, 0, 10, 10], "sku_id": "sku_a"},
{"frame": 1, "box": [100, 100, 110, 110], "sku_id": "sku_a"},
{"frame": 2, "box": [0, 0, 10, 10], "sku_id": "sku_a"},
{"frame": 3, "box": [0, 0, 10, 10], "sku_id": "sku_a"},
]
scored = labelled_top1(frames, labels, iou_min=0.5)
self.assertEqual((scored["correct"], scored["wrong"], scored["rejected"], scored["unmatched"]), (1, 1, 1, 1))
self.assertEqual(scored["top1"], 0.3333)
def test_sku_churn_counts_a_renamed_track_not_the_first_name(self):
frames = [
{"tracks": [track("accepted", "sku_a", track_id=7)]},
{"tracks": [track("accepted", "sku_a", track_id=7)]},
{"tracks": [track("accepted", "sku_b", track_id=7, relabels=1)]},
{"tracks": [track("rejected", track_id=8, restarts=2)]},
]
churn = track_churn(frames)
self.assertEqual(churn["sku_changes"], 1)
self.assertEqual(churn["tracks_with_sku_change"], 1)
self.assertEqual(churn["relabels"], 1)
self.assertEqual(churn["restarts"], 2)
self.assertEqual(churn["per_track"][0]["track_id"], 7)
class LeakageTests(unittest.TestCase):
def setUp(self):
self.root = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.root, ignore_errors=True)
def test_a_pasted_catalog_crop_is_excluded_and_a_real_facing_is_not(self):
stock = os.path.join(self.root, "stock")
os.makedirs(stock)
template = np.full((20, 16, 3), 40, dtype=np.uint8)
template[2:8, 2:8] = (255, 0, 0)
write_jpeg(os.path.join(stock, "sku_a.jpeg"), template)
frame = np.zeros((40, 40, 3), dtype=np.uint8)
frame[5:25, 10:26] = template
recording = os.path.join(self.root, "walk")
os.makedirs(recording)
write_jpeg(os.path.join(recording, "000000.jpg"), frame)
with open(os.path.join(recording, "index.jsonl"), "w", encoding="utf-8") as handle:
handle.write(json.dumps({"frame": 0, "path": "000000.jpg", "tracks": []}) + "\n")
labels = [
{"frame": 0, "box": [10, 5, 26, 25], "sku_id": "sku_a"},
{"frame": 0, "box": [0, 0, 8, 8], "sku_id": "sku_b"},
]
held, leaked = leaked_labels(recording, load_index(recording), labels, stock)
self.assertEqual(len(leaked), 1)
self.assertEqual(leaked[0]["leak"], "crop_is_template")
self.assertEqual([row["sku_id"] for row in held], ["sku_b"])
def test_score_recording_names_the_leak_and_refuses_it_as_top1(self):
stock = os.path.join(self.root, "stock")
os.makedirs(stock)
image = np.full((12, 12, 3), 9, dtype=np.uint8)
write_jpeg(os.path.join(stock, "sku_a.jpg"), image)
with open(os.path.join(stock, "stock.csv"), "w", encoding="utf-8") as handle:
handle.write("SKU_ID,product_name\nsku_a,A\n")
recording = os.path.join(self.root, "session")
os.makedirs(recording)
shutil.copy(os.path.join(stock, "sku_a.jpg"), os.path.join(recording, "000000.jpg"))
with open(os.path.join(recording, "index.jsonl"), "w", encoding="utf-8") as handle:
handle.write(
json.dumps(
{
"frame": 1,
"path": "000000.jpg",
"tracks": [track("accepted", "sku_a", box=[0, 0, 12, 12])],
}
)
+ "\n"
)
with open(os.path.join(recording, "labels.json"), "w", encoding="utf-8") as handle:
json.dump({"labels": [{"frame": 1, "box": [0, 0, 12, 12], "sku_id": "sku_a"}]}, handle)
report = score_recording(recording, stock_dir=stock, identity="matcher")
self.assertEqual(report["leakage"]["excluded"], 1)
self.assertEqual(report["labelled_top1"]["labels"], 0)
self.assertEqual(report["labelled_top1"]["top1"], None)
self.assertEqual(report["decisions"]["accepted"], 1)
self.assertTrue(report["fingerprints"]["sample"]["sha256"])
self.assertEqual(len(report["fingerprints"]["catalog"]["version"]), 12)
self.assertTrue(any("template-derived" in gap for gap in report["evidence"]))
class CliTests(unittest.TestCase):
def test_missing_recording_is_an_explicit_error(self):
completed = subprocess.run(
[sys.executable, os.path.join(current_dir, "eval_live.py"), "/no/such/recording"],
capture_output=True,
text=True,
check=False,
)
self.assertEqual(completed.returncode, 2)
self.assertIn("error", json.loads(completed.stdout))
def test_the_same_pinned_recording_scores_twice_the_same(self):
root = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, root, ignore_errors=True)
stock = os.path.join(root, "stock")
os.makedirs(stock)
with open(os.path.join(stock, "stock.csv"), "w", encoding="utf-8") as handle:
handle.write("SKU_ID\nsku_a\n")
recording = os.path.join(root, "session")
os.makedirs(recording)
payload = np.zeros((8, 8, 3), dtype=np.uint8)
write_jpeg(os.path.join(recording, "000000.jpg"), payload)
line = {
"frame": 4,
"path": "000000.jpg",
"tracks": [
track("accepted", "sku_a", tag={"state": "readable"}),
track("ambiguous", track_id=2),
],
}
with open(os.path.join(recording, "index.jsonl"), "w", encoding="utf-8") as handle:
handle.write(json.dumps(line) + "\n")
with open(os.path.join(recording, "labels.json"), "w", encoding="utf-8") as handle:
json.dump({"labels": [{"frame": 4, "box": [0, 0, 8, 8], "sku_id": "sku_a"}]}, handle)
first = score_recording(recording, stock_dir=stock, identity="matcher")
second = score_recording(recording, stock_dir=stock, identity="matcher")
self.assertEqual(first, second)
self.assertEqual(first["decisions"]["accepted"], 1)
self.assertEqual(first["decisions"]["ambiguous"], 1)
self.assertEqual(first["labelled_top1"]["correct"], 1)
self.assertIsNone(first["fingerprints"]["model"]["weights_sha256"])
def test_a_broken_index_line_is_an_input_error(self):
root = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, root, ignore_errors=True)
with open(os.path.join(root, "index.jsonl"), "w", encoding="utf-8") as handle:
handle.write("{not json}\n")
with self.assertRaises(EvalInputError):
load_index(root)
class IdentityContractTests(unittest.TestCase):
def test_missing_gallery_weights_exit_without_a_matcher_score(self):
root = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, root, ignore_errors=True)
recording = os.path.join(root, "session")
os.makedirs(recording)
write_jpeg(os.path.join(recording, "000000.jpg"), np.zeros((8, 8, 3), dtype=np.uint8))
with open(os.path.join(recording, "index.jsonl"), "w", encoding="utf-8") as handle:
handle.write(json.dumps({"frame": 1, "path": "000000.jpg", "tracks": []}) + "\n")
missing = os.path.join(root, "no-such-dinov2.safetensors")
report_path = os.path.join(recording, "eval_live.json")
env = os.environ.copy()
env["DORAYAKI_DINOV2_WEIGHTS"] = missing
completed = subprocess.run(
[
sys.executable,
os.path.join(current_dir, "eval_live.py"),
recording,
"--replay",
"--identity",
"gallery",
"--json",
report_path,
],
capture_output=True,
text=True,
check=False,
env=env,
)
self.assertEqual(completed.returncode, 2)
payload = json.loads(completed.stdout)
self.assertIn("DINOv2", payload["error"])
self.assertIn("matcher", payload["error"])
self.assertFalse(os.path.exists(report_path))
self.assertNotIn("accepted_rate", completed.stdout)
def test_a_silent_recording_does_not_become_the_matcher(self):
with self.assertRaises(EvalInputError):
resolve_identity(None, None)
self.assertEqual(resolve_identity("matcher", None), "matcher")
self.assertEqual(resolve_identity(None, "gallery"), "gallery")
with self.assertRaises(EvalInputError):
resolve_identity("matcher", "gallery")
def test_recorded_backend_is_read_and_a_mix_is_refused(self):
self.assertEqual(recorded_identity([{"identity_backend": "Gallery"}]), "gallery")
self.assertIsNone(recorded_identity([{"frame": 1}]))
self.assertEqual(
recorded_identity([{"identity": {"backend": "matcher", "sku_id": "sku_a"}}]),
"matcher",
)
with self.assertRaises(EvalInputError):
recorded_identity([
{"identity_backend": "gallery"},
{"identity_backend": "matcher"},
])
def test_van_is_accepted_by_the_cli_and_by_a_recording_that_names_it(self):
self.assertIn("van", IDENTITY_BACKENDS)
self.assertEqual(resolve_identity("van", None), "van")
self.assertEqual(recorded_identity([{"identity_backend": "van"}]), "van")
root = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, root, ignore_errors=True)
stock = os.path.join(root, "stock")
os.makedirs(stock)
recording = os.path.join(root, "session")
os.makedirs(recording)
write_jpeg(os.path.join(recording, "000000.jpg"), np.zeros((8, 8, 3), dtype=np.uint8))
line = {"frame": 1, "path": "000000.jpg", "tracks": [], "identity_backend": "van"}
with open(os.path.join(recording, "index.jsonl"), "w", encoding="utf-8") as handle:
handle.write(json.dumps(line) + "\n")
for extra in ([], ["--identity", "van"]):
report_path = os.path.join(root, "eval_live.json")
completed = subprocess.run(
[
sys.executable,
os.path.join(current_dir, "eval_live.py"),
recording,
"--stock-dir",
stock,
"--json",
report_path,
*extra,
],
capture_output=True,
text=True,
check=False,
)
self.assertEqual(completed.returncode, 0, completed.stdout + completed.stderr)
with open(report_path, "r", encoding="utf-8") as handle:
report = json.load(handle)
self.assertEqual(report["identity"]["backend"], "van")
self.assertEqual(report["fingerprints"]["model"]["identity"], "van")
if __name__ == "__main__":
unittest.main()

631
recognization/code/test_gallery.py
View file

@ -1,631 +0,0 @@
"""
Tests for the product gallery's matching logic.
The embedding model is injected, so these run without torch or a GPU: what is
being tested is the retrieval contract - best view wins, several views per SKU
count as one answer, and a stranger is rejected rather than named.
"""
from __future__ import annotations
import os
import sys
import unittest
import numpy as np
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(current_dir)
if os.path.exists(os.path.join(project_root, "stock")):
os.chdir(project_root)
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from gallery import ( # noqa: E402
DINOV2_MODEL,
DINOV2_SHA256,
GalleryEntry,
ProductGallery,
catalog_digest,
dinov2_weights_path,
head_class_set,
load_dinov2,
)
import gallery as gallery_module # noqa: E402
def a_gallery(views, threshold=0.55):
"""A gallery with hand-made embeddings: no model, no files."""
skus = sorted({sku for sku, _ in views})
gallery = ProductGallery("does-not-exist", skus, threshold=threshold)
embeddings = []
entries = []
for index, (sku, vector) in enumerate(views):
vector = np.asarray(vector, dtype=np.float32)
vector /= np.linalg.norm(vector)
embeddings.append(vector)
entries.append(GalleryEntry(sku_id=sku, view=f"{sku}_{index}", embedding=vector))
gallery._entries = entries
gallery._matrix = np.stack(embeddings)
gallery._available = True
return gallery
def unit(vector):
vector = np.asarray(vector, dtype=np.float32)
return (vector / np.linalg.norm(vector)).tolist()
class Backbone:
"""A deterministic stand-in for DINOv2: the prepared crop's channel means,
tiled to the width the real backbone returns."""
def __init__(self):
self.calls = 0
def __call__(self, tensor):
self.calls += 1
return tensor.mean(dim=(2, 3)).repeat(1, gallery_module.DINOV2_EMBED_DIM // 3)
def a_query():
"""A unit vector in the backbone's space, for match() without a model."""
query = np.zeros(gallery_module.DINOV2_EMBED_DIM, dtype=np.float32)
query[0] = 1.0
query[1] = 0.05
return query / np.linalg.norm(query)
class UnreachableBackbone:
def __call__(self, tensor):
raise AssertionError("the gallery loaded a backbone it should have cached")
def write_view(path, seed):
import cv2
image = np.random.RandomState(seed).randint(0, 256, (24, 32, 3), dtype=np.uint8)
cv2.imwrite(path, image)
def a_store(directory, names=("sku_a", "sku_a_1", "sku_b")):
"""A store dir: one JPEG per view plus the stock.csv the catalog is read from."""
with open(os.path.join(directory, "stock.csv"), "w", encoding="utf-8") as handle:
handle.write("SKU_ID,product_name\n")
for index, name in enumerate(names):
write_view(os.path.join(directory, f"{name}.jpeg"), index)
def a_stored_gallery(directory, names, backbone, image_size=32):
gallery = ProductGallery(directory, list(names), device="cpu", image_size=image_size)
gallery._load_model = lambda: backbone
return gallery
class MatchingTests(unittest.TestCase):
def test_the_nearest_view_names_the_product(self):
gallery = a_gallery([("sku_a", unit([1, 0, 0])), ("sku_b", unit([0, 1, 0]))])
gallery.embed = lambda crop: np.array([0.9, 0.1, 0.0], dtype=np.float32)
match = gallery.match(object())
self.assertFalse(match.rejected)
self.assertEqual(match.sku_id, "sku_a")
self.assertAlmostEqual(match.similarity, 0.9, places=3)
def test_several_views_of_one_sku_are_one_answer(self):
# The second view of sku_b is closer than any view of sku_a, but sku_a must
# still win on its best view: a SKU is not outvoted by having more photos.
gallery = a_gallery([
("sku_a", unit([1, 0, 0])),
("sku_b", unit([0, 1, 0])),
("sku_b", unit([0.2, 0.98, 0])),
])
gallery.embed = lambda crop: np.array([1.0, 0.05, 0.0], dtype=np.float32)
match = gallery.match(object())
self.assertEqual(match.sku_id, "sku_a")
self.assertEqual(match.runner_up, "sku_b")
def test_a_stranger_is_rejected_not_named(self):
gallery = a_gallery([("sku_a", unit([1, 0, 0])), ("sku_b", unit([0, 1, 0]))])
gallery.embed = lambda crop: np.array([0.0, 0.0, 1.0], dtype=np.float32)
match = gallery.match(object())
self.assertTrue(match.rejected)
self.assertIsNone(match.sku_id)
def test_the_threshold_moves_the_rejection_line(self):
gallery = a_gallery([("sku_a", unit([1, 0, 0]))], threshold=0.99)
gallery.embed = lambda crop: np.array([0.9, 0.4, 0.0], dtype=np.float32)
self.assertTrue(gallery.match(object()).rejected)
self.assertFalse(gallery.match(object(), threshold=0.5).rejected)
def test_margin_separates_a_confident_match_from_a_close_one(self):
gallery = a_gallery([("sku_a", unit([1, 0, 0])), ("sku_b", unit([0.9, 0.44, 0]))])
gallery.embed = lambda crop: np.array([1.0, 0.1, 0.0], dtype=np.float32)
confident = gallery.match(object())
gallery.embed = lambda crop: np.array([0.95, 0.3, 0.0], dtype=np.float32)
close = gallery.match(object())
self.assertGreater(confident.margin, close.margin)
def test_the_per_sku_max_does_not_depend_on_entry_order(self):
# Three SKUs, several views each, enrolled out of order: a SKU is scored
# by its best view, and the answer cannot move because a row moved.
gallery = a_gallery([
("sku_b", unit([0, 1, 0])),
("sku_a", unit([1, 0, 0])),
("sku_c", unit([0, 0, 1])),
("sku_b", unit([0.3, 0.95, 0])),
("sku_a", unit([0.98, 0.2, 0])),
])
gallery.embed = lambda crop: np.array([1.0, 0.05, 0.0], dtype=np.float32)
straight = gallery.match(object())
self.assertEqual(straight.sku_id, "sku_a")
self.assertEqual(straight.runner_up, "sku_b")
for order in ([4, 0, 3, 1, 2], [2, 3, 1, 4, 0]):
gallery._entries = [gallery._entries[index] for index in order]
gallery._matrix = gallery._matrix[order]
shuffled = gallery.match(object())
self.assertEqual(shuffled.sku_id, straight.sku_id)
self.assertEqual(shuffled.runner_up, straight.runner_up)
self.assertAlmostEqual(shuffled.similarity, straight.similarity, places=6)
self.assertAlmostEqual(shuffled.runner_up_similarity, straight.runner_up_similarity, places=6)
def test_a_tie_keeps_the_first_enrolled_sku(self):
# Exactly equidistant: first appearance decides, as it did before the
# per-SKU maximum became a segmented reduction.
gallery = a_gallery([("sku_a", unit([1, 0, 0])), ("sku_b", unit([0, 1, 0]))])
gallery.embed = lambda crop: np.array([1.0, 1.0, 0.0], dtype=np.float32)
match = gallery.match(object())
self.assertFalse(match.rejected)
self.assertEqual(match.sku_id, "sku_a")
self.assertEqual(match.runner_up, "sku_b")
self.assertAlmostEqual(match.similarity, match.runner_up_similarity, places=6)
self.assertAlmostEqual(match.margin, 0.0, places=6)
def test_an_unloaded_gallery_rejects_everything(self):
gallery = ProductGallery("does-not-exist", ["sku_a"])
match = gallery.match(object())
self.assertTrue(match.rejected)
class EnrollmentTests(unittest.TestCase):
def test_views_are_found_by_sku_prefix(self):
import tempfile
with tempfile.TemporaryDirectory() as directory:
for name in ("sku_a.jpeg", "sku_a_1.jpeg", "sku_a-2.jpeg", "sku_ab.jpeg", "notes.txt"):
open(os.path.join(directory, name), "wb").close()
gallery = ProductGallery(directory, ["sku_a", "sku_ab"])
self.assertEqual(len(gallery._views_for("sku_a")), 3,
"one view, two extra views, and not sku_ab")
self.assertEqual(len(gallery._views_for("sku_ab")), 1)
class FeatureCacheTests(unittest.TestCase):
"""The cached matrix must be this store's, or it must not be used."""
def setUp(self):
import hashlib
import tempfile
self.directory = tempfile.TemporaryDirectory()
self.addCleanup(self.directory.cleanup)
a_store(self.directory.name)
# A cache is only served when the pinned backbone is on disk, so point the
# pin at a stand-in file the module's expected digest is told about. The
# check itself - hash this file, compare - is what is under test.
self.weights = os.path.join(self.directory.name, "dinov2.safetensors")
self.write(self.weights, b"not really a checkpoint")
os.environ["DORAYAKI_DINOV2_WEIGHTS"] = self.weights
self.addCleanup(os.environ.pop, "DORAYAKI_DINOV2_WEIGHTS", None)
self.addCleanup(setattr, gallery_module, "DINOV2_SHA256", gallery_module.DINOV2_SHA256)
gallery_module.DINOV2_SHA256 = hashlib.sha256(b"not really a checkpoint").hexdigest()
def write(self, path, content):
with open(path, "wb") as handle:
handle.write(content)
return path
def real_gallery(self):
"""A gallery with the real _load_model: the server's own readiness path."""
return ProductGallery(self.directory.name, ["sku_a", "sku_b"], device="cpu", image_size=32)
def loaded(self, names=("sku_a", "sku_b"), image_size=32):
backbone = Backbone()
gallery = a_stored_gallery(self.directory.name, names, backbone, image_size=image_size)
self.assertTrue(gallery.load())
return gallery, backbone
def cache_payload(self, gallery):
"""The rows and matrix a loaded gallery left on disk."""
with np.load(gallery._cache_path(), allow_pickle=False) as payload:
return (
[str(value) for value in payload["skus"].tolist()],
[str(value) for value in payload["views"].tolist()],
payload["matrix"],
)
def rewrite_cache(self, gallery, skus, views, matrix):
"""The same valid key with different contents: only the rows can catch it."""
path = gallery._cache_path()
with np.load(path, allow_pickle=False) as payload:
key = str(payload["key"].item())
np.savez(path, key=key, skus=np.array(skus), views=np.array(views), matrix=matrix)
def re_embeds(self, first, why):
"""The one right answer to a rewritten cache: build it from the pixels."""
backbone = Backbone()
gallery = a_stored_gallery(self.directory.name, ("sku_a", "sku_b"), backbone)
self.assertTrue(gallery.load(), why)
self.assertEqual(backbone.calls, 1, f"{why} must not be served from cache")
np.testing.assert_array_equal(gallery._matrix, first._matrix)
self.assertEqual([entry.sku_id for entry in gallery._entries], ["sku_a", "sku_a", "sku_b"])
def test_a_second_start_answers_the_same_from_the_cache(self):
first, backbone = self.loaded()
self.assertEqual(backbone.calls, 1)
self.assertEqual(first.view_count(), 3)
self.assertTrue(os.path.isfile(first._cache_path()))
cached = a_stored_gallery(self.directory.name, ("sku_a", "sku_b"), UnreachableBackbone())
self.assertTrue(cached.load())
self.assertEqual(cached.view_count(), first.view_count())
np.testing.assert_array_equal(cached._matrix, first._matrix)
self.assertEqual([entry.view for entry in cached._entries],
[entry.view for entry in first._entries])
query = a_query()
first.embed = cached.embed = lambda crop: query
self.assertEqual(vars(cached.match(object())), vars(first.match(object())))
def test_changed_view_bytes_re_embed(self):
first, _ = self.loaded()
before = first._matrix.copy()
write_view(os.path.join(self.directory.name, "sku_b.jpeg"), seed=99)
second, backbone = self.loaded()
self.assertEqual(backbone.calls, 1, "a replaced view must not be served from cache")
self.assertFalse(np.array_equal(second._matrix, before))
def test_a_reordered_catalog_does_not_borrow_another_mapping(self):
self.loaded()
swapped, backbone = self.loaded(names=("sku_b", "sku_a"))
self.assertEqual(backbone.calls, 1)
self.assertEqual([entry.sku_id for entry in swapped._entries], ["sku_b", "sku_a", "sku_a"])
def test_another_image_size_is_never_served(self):
small, _ = self.loaded(image_size=32)
large, backbone = self.loaded(image_size=64)
self.assertEqual(backbone.calls, 1)
self.assertNotEqual(small._cache_path(), large._cache_path())
def test_a_corrupt_cache_re_embeds_the_same_matrix(self):
first, _ = self.loaded()
with open(first._cache_path(), "wb") as handle:
handle.write(b"not an npz at all")
second, backbone = self.loaded()
self.assertEqual(backbone.calls, 1)
np.testing.assert_array_equal(second._matrix, first._matrix)
def test_swapped_cache_rows_are_not_served(self):
# A current key says nothing about the rows inside the file. The two
# sku_a views are adjacent, so a swap has to cross SKUs to mean
# anything - and the matrix that moves with them is the dangerous part:
# a vector that would then be filed under the wrong product.
first, _ = self.loaded()
skus, views, matrix = self.cache_payload(first)
self.assertEqual(list(zip(skus, views)),
[("sku_a", "sku_a.jpeg"), ("sku_a", "sku_a_1.jpeg"), ("sku_b", "sku_b.jpeg")])
self.rewrite_cache(first, skus[::-1], views[::-1], matrix[::-1])
self.re_embeds(first, "rows swapped with their vectors")
def test_mislabelled_cache_rows_are_not_served(self):
# The views and the matrix stay where they are, so every row names a
# product it was not embedded for.
first, _ = self.loaded()
skus, views, matrix = self.cache_payload(first)
self.rewrite_cache(first, [skus[2], skus[1], skus[0]], views, matrix)
self.re_embeds(first, "labels swapped around the views")
def test_a_duplicated_cache_row_is_not_served(self):
# The first view written twice: one image would answer for two products.
first, _ = self.loaded()
skus, views, matrix = self.cache_payload(first)
self.rewrite_cache(first, skus[:1] * 2 + skus[2:], views[:1] * 2 + views[2:], matrix)
self.re_embeds(first, "a row repeated")
def test_a_foreign_sku_row_is_not_served(self):
# A row from another store's catalog: a SKU this gallery never enrolled.
first, _ = self.loaded()
skus, views, matrix = self.cache_payload(first)
self.rewrite_cache(first, skus[1:] + ["other_store"], views[1:] + ["other_store.jpeg"], matrix)
self.re_embeds(first, "another store's row")
def test_a_cache_may_omit_a_view_it_could_not_decode(self):
# The writer drops a view it could not read and keeps everyone else's
# order, so such a cache is still this store's - and still worth serving
# rather than re-embedding the whole catalog on every start.
first, _ = self.loaded()
skus, views, matrix = self.cache_payload(first)
self.rewrite_cache(first, skus[:1] + skus[2:], views[:1] + views[2:], matrix[[0, 2]])
backbone = Backbone()
gallery = a_stored_gallery(self.directory.name, ("sku_a", "sku_b"), backbone)
self.assertTrue(gallery.load())
self.assertEqual(backbone.calls, 0, "a readable shorter cache is still a hit")
self.assertEqual([entry.sku_id for entry in gallery._entries], ["sku_a", "sku_b"])
np.testing.assert_array_equal(gallery._matrix, first._matrix[[0, 2]])
def test_a_nonnumeric_cache_matrix_is_not_served(self):
# Only the float matrix _cache_write produces is a cache. A complex
# matrix is finite and casts without complaint, so it answers with a
# different matrix than the one on disk; integer, boolean and text
# matrices are not embeddings at all.
first, _ = self.loaded()
skus, views, matrix = self.cache_payload(first)
others = {
"complex": matrix.astype(np.complex128),
"timedelta": np.zeros_like(matrix, dtype="timedelta64[s]"),
"integer": (matrix * 8).astype(np.int32),
"boolean": np.ones_like(matrix, dtype=bool),
"text": np.zeros(matrix.shape, dtype="<U4"),
}
for name, other in others.items():
with self.subTest(matrix=name):
self.rewrite_cache(first, skus, views, other)
self.re_embeds(first, f"a {name} matrix")
def test_a_loaded_gallery_reloads_when_its_source_moves(self):
gallery, backbone = self.loaded()
self.assertTrue(gallery.load())
self.assertEqual(backbone.calls, 1, "an unchanged store must not re-embed")
write_view(os.path.join(self.directory.name, "sku_b.jpeg"), seed=99)
self.assertTrue(gallery.load())
self.assertEqual(backbone.calls, 2)
with open(os.path.join(self.directory.name, "stock.csv"), "a", encoding="utf-8") as handle:
handle.write("sku_c,new product\n")
self.assertTrue(gallery.load())
self.assertEqual(backbone.calls, 3, "stock.csv is part of what an embedding is pinned to")
self.assertEqual(gallery.view_count(), 3)
def test_a_valid_cache_needs_the_pinned_backbone_on_disk(self):
# The regression: a cache hit never opens load_dinov2, so a deleted or
# altered checkpoint used to read as a loaded gallery and then raise on
# the first match - after the server had reported itself ready.
self.loaded()
gallery = self.real_gallery()
self.assertTrue(gallery.load(), "the cache serves while the pin holds")
os.environ["DORAYAKI_DINOV2_WEIGHTS"] = os.path.join(self.directory.name, "gone.safetensors")
missing = self.real_gallery()
self.assertFalse(missing.load())
self.assertFalse(missing.available())
os.environ["DORAYAKI_DINOV2_WEIGHTS"] = self.write(
os.path.join(self.directory.name, "altered.safetensors"), b"a different checkpoint"
)
altered = self.real_gallery()
self.assertFalse(altered.load())
self.assertFalse(altered.available())
def test_an_altered_backbone_re_embeds_rather_than_serves(self):
# With a backbone that can still load, an altered file must re-embed.
self.loaded()
self.assertTrue(
a_stored_gallery(self.directory.name, ("sku_a", "sku_b"), UnreachableBackbone()).load(),
"the cache still serves on the pinned file",
)
self.write(self.weights, b"a different checkpoint")
backbone = Backbone()
rebuilt = a_stored_gallery(self.directory.name, ("sku_a", "sku_b"), backbone)
self.assertTrue(rebuilt.load())
self.assertEqual(backbone.calls, 1, "an unpinned cache must not be served")
class OfflineWeightsTests(unittest.TestCase):
def test_a_missing_weights_file_names_the_path_and_does_not_fetch(self):
previous = os.environ.get("DORAYAKI_DINOV2_WEIGHTS")
os.environ["DORAYAKI_DINOV2_WEIGHTS"] = os.path.join(
"does-not-exist", "dinov2_vits14_lvd142m.safetensors"
)
try:
with self.assertRaises(FileNotFoundError) as raised:
load_dinov2(224)
finally:
if previous is None:
os.environ.pop("DORAYAKI_DINOV2_WEIGHTS", None)
else:
os.environ["DORAYAKI_DINOV2_WEIGHTS"] = previous
message = str(raised.exception)
self.assertIn("does-not-exist", message)
self.assertIn(DINOV2_SHA256, message)
self.assertIn("does not download", message)
def test_the_default_path_stays_inside_the_repo(self):
previous = os.environ.pop("DORAYAKI_DINOV2_WEIGHTS", None)
try:
self.assertTrue(dinov2_weights_path().endswith(
os.path.join("weights", "dinov2_vits14_lvd142m.safetensors")
))
finally:
if previous is not None:
os.environ["DORAYAKI_DINOV2_WEIGHTS"] = previous
def _head_payload(class_ids, torch):
projection = torch.nn.Linear(384, 128)
prototypes = torch.nn.functional.normalize(torch.randn(len(class_ids), 128), dim=1)
return {
"class_ids": list(class_ids),
"image_size": 518,
"base_model": DINOV2_MODEL,
"projection": {key: value.detach().cpu() for key, value in projection.state_dict().items()},
"prototypes": prototypes.detach().cpu(),
}
class HeadPinTests(unittest.TestCase):
def test_a_stale_head_cannot_label_this_catalog(self):
import tempfile
import torch
with tempfile.TemporaryDirectory() as directory:
with open(os.path.join(directory, "stock.csv"), "w", encoding="utf-8") as handle:
handle.write("SKU_ID,product_name\nsku_a,A\nsku_b,B\nsku_c,C\n")
gallery = ProductGallery(directory, ["sku_a", "sku_b", "sku_c"])
gallery._available = True
path = gallery._head_path
subset = ["sku_c", "sku_a"]
digest = catalog_digest(directory, subset)
foreign = _head_payload(["other_store", "sku_a"], torch)
torch.save(foreign, path)
self.assertFalse(gallery.load_head())
self.assertIsNone(gallery._head)
self.assertIn("catalog_mismatch", gallery.head_reason)
self.assertTrue(gallery.available())
legacy = _head_payload(subset, torch)
torch.save(legacy, path)
self.assertTrue(gallery.load_head())
self.assertEqual(gallery.head_reason, "legacy_unpinned")
self.assertEqual(gallery._head["class_ids"], subset)
self.assertNotIn("other_store", gallery._head["class_ids"])
stale = _head_payload(subset, torch)
stale["catalog_digest"] = "0" * 64
torch.save(stale, path)
self.assertFalse(gallery.load_head())
self.assertIsNone(gallery._head)
self.assertIn("catalog_mismatch", gallery.head_reason)
self.assertTrue(gallery.available())
wrong_backbone = _head_payload(subset, torch)
wrong_backbone["catalog_digest"] = digest
wrong_backbone["base_model"] = "vit_base_patch14_dinov2.lvd142m"
torch.save(wrong_backbone, path)
self.assertFalse(gallery.load_head())
self.assertIn("backbone_mismatch", gallery.head_reason)
wrong_size = _head_payload(subset, torch)
wrong_size["catalog_digest"] = digest
wrong_size["image_size"] = 224
torch.save(wrong_size, path)
self.assertFalse(gallery.load_head())
self.assertIn("image_size_mismatch", gallery.head_reason)
torch.save(["not", "a", "head"], path)
self.assertFalse(gallery.load_head())
self.assertTrue(gallery.head_reason.startswith("malformed"))
self.assertTrue(gallery.available())
self.assertIsNone(gallery._head)
current = _head_payload(subset, torch)
current["catalog_digest"] = digest
torch.save(current, path)
self.assertTrue(gallery.load_head())
self.assertEqual(gallery.head_reason, "ok")
self.assertEqual(gallery._head["class_ids"], subset)
self.assertIsNone(head_class_set(["sku_a", "sku_a", ""]))
class ConcurrentInferenceTests(unittest.TestCase):
"""The backbone is never entered by two threads at once.
Metal's shader library is process-global and is not thread-safe while
kernels are still being compiled into it, so two threads reaching the first
forward together segfault the whole server. This is a contract test, not a
timing test: it records overlap rather than measuring speed.
"""
def test_two_threads_never_enter_the_backbone_together(self):
import time
import threading
import tempfile
with tempfile.TemporaryDirectory() as directory:
a_store(directory, names=("sku_a", "sku_b"))
gallery = ProductGallery(directory, ["sku_a", "sku_b"], device="cpu", image_size=32)
inside = 0
overlapped = []
gate = threading.Lock()
class SlowBackbone:
def __call__(self, tensor):
nonlocal inside
with gate:
inside += 1
if inside > 1:
overlapped.append(inside)
time.sleep(0.01)
with gate:
inside -= 1
return tensor.mean(dim=(2, 3)).repeat(1, gallery_module.DINOV2_EMBED_DIM // 3)
gallery._load_model = lambda: SlowBackbone()
crop = np.zeros((24, 32, 3), dtype=np.uint8)
errors = []
def work():
try:
for _ in range(6):
gallery.embed(crop)
except Exception as error: # pragma: no cover - reported below
errors.append(repr(error))
threads = [threading.Thread(target=work) for _ in range(4)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
self.assertEqual(errors, [])
self.assertEqual(overlapped, [], "two threads ran the backbone at the same time")
class TrainerPinTests(unittest.TestCase):
def test_head_training_loads_the_pinned_file_and_keeps_preprocess(self):
import finetune_identity
import gallery as gallery_module
seen = []
class FakeBackbone:
def eval(self):
return self
def to(self, device):
return self
sentinel = FakeBackbone()
def fake_load(image_size):
seen.append(image_size)
return sentinel
# build_model resolves the loader from gallery at call time, so that is
# the attribute to replace; finetune_identity has no module-level copy.
original = gallery_module.load_dinov2
gallery_module.load_dinov2 = fake_load
try:
model, device = finetune_identity.build_model(224)
finally:
gallery_module.load_dinov2 = original
self.assertIs(model, sentinel)
self.assertIn(device, ("cpu", "mps"))
self.assertEqual(seen, [224])
image = np.zeros((10, 20, 3), dtype=np.uint8)
self.assertEqual(finetune_identity.prepare(image, 32).shape, gallery_module._prepare(image, 32).shape)
self.assertTrue(np.allclose(
finetune_identity.prepare(image, 32),
gallery_module._prepare(image, 32),
))
if __name__ == "__main__":
unittest.main()

69
recognization/code/test_identity.py
View file

@ -1,69 +0,0 @@
"""Ranking contract for decide_identity: what a score set is allowed to name.
Moved out of test_observe.py when the legacy session model was deleted; the
ranking rules are the live path's, not the legacy one's.
"""
from __future__ import annotations
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from identity import ( # noqa: E402
IDENTITY_ACCEPTED,
IDENTITY_AMBIGUOUS,
IDENTITY_REJECTED,
IDENTITY_UNKNOWN,
decide_identity,
)
class IdentityRankingTests(unittest.TestCase):
def test_zero_scores_are_rejected_not_a_sku(self):
ranked = decide_identity({"sku_a": 0.0, "sku_b": 0.0})
self.assertEqual(ranked["identity_level"], IDENTITY_REJECTED)
self.assertIsNone(ranked["best_sku"])
self.assertEqual(ranked["status"], IDENTITY_REJECTED)
def test_empty_scores_are_rejected(self):
ranked = decide_identity({})
self.assertEqual(ranked["identity_level"], IDENTITY_REJECTED)
self.assertIsNone(ranked["best_sku"])
def test_weak_unique_score_is_unknown(self):
ranked = decide_identity({"sku_a": 0.4, "sku_b": 0.0})
self.assertEqual(ranked["identity_level"], IDENTITY_UNKNOWN)
self.assertIsNone(ranked["best_sku"])
def test_close_scores_are_ambiguous(self):
ranked = decide_identity({"sku_a": 20.0, "sku_b": 19.5})
self.assertEqual(ranked["identity_level"], IDENTITY_AMBIGUOUS)
self.assertIsNone(ranked["best_sku"])
def test_unique_strong_score_is_accepted(self):
ranked = decide_identity({"sku_a": 40.0, "sku_b": 5.0})
self.assertEqual(ranked["identity_level"], IDENTITY_ACCEPTED)
self.assertEqual(ranked["best_sku"], "sku_a")
self.assertEqual(ranked["status"], "success")
def test_van_answers_what_the_gate_would_refuse(self):
# Van's contract is the baseline the decision is measured against, so it
# must answer exactly where the shipped gate abstains. A weak score and
# a close pair are both refused above; here both are named.
weak = decide_identity({"sku_a": 0.4, "sku_b": 0.0}, scale="van")
self.assertEqual(weak["identity_level"], IDENTITY_ACCEPTED)
self.assertEqual(weak["best_sku"], "sku_a")
close = decide_identity({"sku_a": 20.0, "sku_b": 19.5}, scale="van")
self.assertEqual(close["best_sku"], "sku_a")
def test_van_still_rejects_when_there_is_no_evidence(self):
# Always-argmax is not always-answer: with nothing to rank there is no
# argmax, and naming a SKU from an empty score set would be a fabrication.
ranked = decide_identity({}, scale="van")
self.assertEqual(ranked["identity_level"], IDENTITY_REJECTED)
self.assertIsNone(ranked["best_sku"])

89
recognization/code/test_image.py Normal file
View file

@ -0,0 +1,89 @@
"""
Test Script for image.jpg
1. Runs YOLO detection on image.jpg.
2. Draws ONLY bounding box rectangles (no class names, no confidence labels).
3. Saves output to detected_image.jpg.
4. Matches the detected bounding boxes against stock templates.
"""
import os
import sys
import cv2
import numpy as np
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from detector import ShelfProductDetector
from feature_matcher import ProductFeatureMatcher
from inventory import InventoryManager
def main():
image_path = "image.jpg"
print(f"[1/4] Loading {image_path}...")
image = cv2.imread(image_path)
if image is None:
print(f"[Error] Failed to read {image_path}")
return
h, w = image.shape[:2]
print(f" Image resolution: {w}x{h}")
print("[2/4] Detecting products using YOLO...")
detector = ShelfProductDetector("runs/detect/train/weights/best.pt")
# You can adjust conf_threshold as needed (default 0.25)
det_res = detector.detect(image, conf_threshold=0.25)
boxes = det_res.get("boxes", [])
print(f" Total detected items: {len(boxes)}")
print("[3/4] Drawing bounding boxes ONLY (no class names, no confidence)...")
annotated = detector.draw_detections(image, boxes, draw_labels=False)
output_path = "detected_image.jpg"
cv2.imwrite(output_path, annotated)
print(f" Saved detection visualization to: {output_path}")
print("[4/4] Matching against stock catalogue (SURF, ORB, BRISK, Hybrid)...")
matcher = ProductFeatureMatcher("stock")
inv = InventoryManager("stock")
print("\n--- Stock Templates Present in image.jpg ---")
matched_skus = set()
sample_matches = []
# Test top detections to find corresponding stock SKUs
for b in boxes:
crop = detector.crop_roi(image, b["box"])
# Use hybrid method from the paper
res = matcher.classify_crop(crop, method="hybrid")
if res.get("status") == "success":
sku_id = res["best_sku"]
score = res["score"]
conf = res["confidence"]
prod = inv.get_product(sku_id)
if prod:
matched_skus.add(sku_id)
sample_matches.append({
"box_id": b["id"],
"box": b["box"],
"sku_id": sku_id,
"product_name": prod["product_name"],
"price": prod["price"],
"number_on": prod["number_on"],
"score": score,
"confidence": conf
})
# Sort matches by confidence
sample_matches.sort(key=lambda x: x["score"], reverse=True)
print(f"\nSuccessfully matched {len(matched_skus)} distinct SKUs across detected shelf boxes:")
for sm in sample_matches[:10]:
print(f"Box #{sm['box_id']:<3} -> [{sm['sku_id']}] {sm['product_name']:<30} | Price: ¥{sm['price']:<5} | On-Shelf: {sm['number_on']} | Score: {sm['score']}")
print("\n[Complete] Tested image.jpg successfully.")
if __name__ == "__main__":
main()

1346
recognization/code/test_live.py
View file

File diff suppressed because it is too large Load diff

475
recognization/code/test_ocr_tags.py
View file

@ -1,475 +0,0 @@
"""
Contract tests for OCR-backed price tags.
No OCR engine runs: a fake engine supplies text lines, so these tests defend the
tag layer's own logic - band geometry, clustering, Japanese price-tag parsing,
the multi-pass vote, and tag-to-product association.
"""
from __future__ import annotations
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import numpy as np # noqa: E402
from ocr import FakeOcr, NullOcr, OcrLine, OcrResult, OcrUnavailable, build_ocr_engine # noqa: E402
from tags import ( # noqa: E402
TAG_STATE_ABSENT,
TAG_STATE_DETECTED,
TAG_STATE_READABLE,
TAG_STATE_UNREADABLE,
TagService,
TagSettings,
cluster_tags,
lines_in_band,
parse_price_tag,
rail_bands,
)
from tag_cards import StorePolicy, StorePolicyError, load_store_policy # noqa: E402
import tag_cards # noqa: E402
def image(width=1000, height=600):
return np.zeros((height, width, 3), dtype="uint8")
def box(x1, y1, x2, y2, box_id=0):
return {"id": box_id, "box": [x1, y1, x2, y2], "confidence": 0.9,
"width": x2 - x1, "height": y2 - y1, "center": [(x1 + x2) // 2, (y1 + y2) // 2]}
# --------------------------------------------------------------------------- parsing
class PriceTagParsingTests(unittest.TestCase):
def test_reads_the_measured_tag(self):
"""Text as macOS Vision returned it from a real Japanese tag (image6)."""
fields = parse_price_tag([
"・ユービーゼロ ノンコレステロールく310g＞",
"357",
"2.4割引",
"269",
"290.52",
])
self.assertEqual(fields.discount, 0.24)
self.assertEqual(fields.unit_price, 290.52)
self.assertIn(fields.price, (269, 357))
self.assertIn("ゼロノンコレステロール", fields.name or "")
self.assertIn("sale", fields.flags)
self.assertIn("discounted", fields.flags)
def test_largest_printed_number_is_the_price(self):
lines = [
OcrLine("357", 1.0, [10, 10, 40, 30]),
OcrLine("269", 0.9, [60, 5, 150, 60]), # big sale price
OcrLine("2.4割引", 0.9, [10, 40, 80, 60]),
]
fields = parse_price_tag([line.text for line in lines], lines)
self.assertEqual(fields.price, 269)
self.assertEqual(fields.regular_price, 357)
def test_unit_price_is_not_mistaken_for_a_price(self):
fields = parse_price_tag(["290.52", "158"])
self.assertEqual(fields.price, 158)
self.assertEqual(fields.unit_price, 290.52)
def test_sale_banner_is_not_a_name(self):
fields = parse_price_tag(["毎日お買得", "198"])
self.assertIsNone(fields.name)
self.assertEqual(fields.price, 198)
def test_implausible_numbers_are_ignored(self):
fields = parse_price_tag(["123456", "0"])
self.assertIsNone(fields.price)
def test_empty_input_yields_nothing_not_zero(self):
fields = parse_price_tag([])
self.assertIsNone(fields.price)
self.assertIsNone(fields.name)
self.assertEqual(fields.flags, [])
def test_discount_out_of_range_is_ignored(self):
fields = parse_price_tag(["98割", "198"])
self.assertIsNone(fields.discount)
self.assertEqual(fields.price, 198)
# --------------------------------------------------------------------------- geometry
class BandGeometryTests(unittest.TestCase):
def test_band_sits_below_the_row(self):
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
bands = rail_bands(boxes, (600, 1000))
self.assertEqual(len(bands), 1)
x1, y1, x2, y2 = bands[0].box
self.assertLessEqual(x1, 100)
self.assertGreaterEqual(x2, 440)
self.assertGreaterEqual(y1, 280) # starts just inside the row
self.assertGreater(y2, 300) # and reaches below it
self.assertEqual(bands[0].row_index, 1)
def test_two_rows_give_two_bands(self):
boxes = [
box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2),
box(100, 400, 200, 600, 3), box(220, 400, 320, 600, 4), box(340, 400, 440, 600, 5),
]
bands = rail_bands(boxes, (800, 1000))
self.assertEqual(len(bands), 2)
self.assertLess(bands[0].box[3], bands[1].box[1])
def test_no_boxes_no_bands(self):
self.assertEqual(rail_bands([], (600, 1000)), [])
class ClusteringTests(unittest.TestCase):
def band(self):
return rail_bands([box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1),
box(340, 100, 440, 300, 2)], (600, 1000))[0]
def test_nearby_lines_join_one_tag(self):
band = self.band()
lines = [
OcrLine("キュービー", 0.9, [110, 318, 300, 338]),
OcrLine("269", 0.9, [110, 345, 240, 375]),
]
tags = cluster_tags(lines, band, (600, 1000))
self.assertEqual(len(tags), 1)
self.assertEqual(tags[0]["fields"].price, 269)
def test_distant_lines_become_separate_tags(self):
band = self.band()
lines = [
OcrLine("コーヒー", 0.9, [110, 318, 250, 338]),
OcrLine("200", 0.9, [110, 345, 230, 375]),
OcrLine("お茶", 0.9, [350, 318, 430, 338]),
OcrLine("300", 0.9, [350, 345, 430, 375]),
]
tags = cluster_tags(lines, band, (600, 1000))
self.assertEqual(len(tags), 2)
self.assertEqual({tag["fields"].price for tag in tags}, {200, 300})
def test_single_tiny_fragment_is_rejected(self):
band = self.band()
lines = [OcrLine("7", 0.4, [150, 350, 160, 360])]
self.assertEqual(cluster_tags(lines, band, (600, 1000)), [])
def test_lines_outside_the_band_are_ignored(self):
band = self.band()
lines = [OcrLine("999", 0.9, [120, 10, 180, 40])]
self.assertEqual(lines_in_band(lines, band), [])
# --------------------------------------------------------------------------- service
def vision_like_reader(image_shape=None):
"""A fake reader that returns tag text wherever it is called, in crop space."""
def reader(crop):
height, width = crop.shape[:2]
return [
OcrLine("キュービー ゼロ ノンコレステロール", 0.6, [int(width * 0.02), int(height * 0.05), int(width * 0.28), int(height * 0.3)]),
OcrLine("357", 0.9, [int(width * 0.02), int(height * 0.4), int(width * 0.12), int(height * 0.7)]),
OcrLine("269", 0.9, [int(width * 0.10), int(height * 0.4), int(width * 0.30), int(height * 0.85)]),
# The tax-included price: a tag is only readable when its own arithmetic
# corroborates the price (269 x 1.08 = 290.52).
OcrLine("290.52", 0.6, [int(width * 0.12), int(height * 0.55), int(width * 0.32), int(height * 0.78)]),
]
return reader
class TagServiceTests(unittest.TestCase):
def make(self, **overrides):
settings = TagSettings(**overrides)
engine = FakeOcr(reader=vision_like_reader())
service = TagService(engine, settings=settings)
return service, engine
def test_pass_produces_a_readable_tag_in_original_coordinates(self):
service, engine = self.make()
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
image_array = image()
bands = service.plan(boxes, image_array.shape)
self.assertTrue(bands)
service.run_pass(image_array, bands[0])
by_track, tags, counts = service.attach(
[{"track_id": 0, "box": [100, 100, 200, 300]}]
)
self.assertEqual(len(tags), 1)
self.assertEqual(tags[0]["state"], TAG_STATE_READABLE)
self.assertEqual(tags[0]["price"], 269)
self.assertEqual(counts[TAG_STATE_READABLE], 1)
# the tag box must be inside the image, i.e. mapped back from the crop
tag_box = tags[0]["box"]
self.assertGreaterEqual(tag_box[0], 0)
self.assertLessEqual(tag_box[2], image_array.shape[1])
def test_vote_overcomes_a_single_misread(self):
reads = {"count": 0}
def reader(crop):
reads["count"] += 1
height, width = crop.shape[:2]
# A card is read several times per pass (multi-scale + red channel), so
# one bad read among many is what the vote is for. (A misread that
# repeats on every read of the same channel would survive the vote -
# that is the documented consistent-misread limit.)
price = "26" if reads["count"] == 2 else "269"
return [
OcrLine("コーヒー", 0.6, [10, 10, 200, 30]),
OcrLine(price, 0.9, [20, 40, 190, 80]),
]
service = TagService(FakeOcr(reader=reader), settings=TagSettings(pass_interval_frames=0))
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
for _ in range(3):
bands = service.plan(boxes, array.shape)
for band in bands:
service.run_pass(array, band)
_, tags, _ = service.attach([{"track_id": 0, "box": [100, 100, 200, 300]}])
self.assertEqual(tags[0]["price"], 269, "the majority read must win")
def test_settled_bands_stop_being_planned(self):
service, _ = self.make(pass_interval_frames=0)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
for _ in range(5):
for band in service.plan(boxes, array.shape):
service.run_pass(array, band)
self.assertTrue(service.stats()["settled"] >= 1)
self.assertEqual(service.plan(boxes, array.shape), [], "a settled shelf costs no OCR")
def test_cadence_limits_passes(self):
service, _ = self.make(pass_interval_frames=15)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
self.assertTrue(service.plan(boxes, array.shape))
self.assertEqual(service.plan(boxes, array.shape), [], "not due yet")
def test_facing_without_a_tag_reports_absent(self):
service, _ = self.make()
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
by_track, _, counts = service.attach([{"track_id": 7, "box": [700, 100, 800, 300]}])
self.assertEqual(by_track[7]["state"], TAG_STATE_ABSENT)
self.assertEqual(counts[TAG_STATE_ABSENT], 1)
def test_unavailable_engine_plans_nothing(self):
service = TagService(NullOcr())
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
self.assertEqual(service.plan(boxes, image().shape), [])
by_track, tags, _ = service.attach([{"track_id": 0, "box": [100, 100, 200, 300]}])
self.assertEqual(tags, [])
self.assertEqual(by_track[0]["state"], TAG_STATE_ABSENT)
def test_reset_clears_the_cache(self):
service, _ = self.make()
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
for band in service.plan(boxes, array.shape):
service.run_pass(array, band)
self.assertTrue(service.stats()["tags"] >= 1)
service.reset()
self.assertEqual(service.stats()["tags"], 0)
def test_a_new_shelf_reusing_row_zero_is_read(self):
"""Screen row 0 on a newly panned shelf is not the row that settled."""
service, _ = self.make(pass_interval_frames=0, settle_observations=1, settle_passes=1)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
for _ in range(3):
for band in service.plan(boxes, array.shape, shelf_offset=(0.0, 0.0)):
service.run_pass(array, band)
self.assertEqual(
service.plan(boxes, array.shape, shelf_offset=(0.0, 0.0)),
[],
"the same shelf row stays settled",
)
again = service.plan(boxes, array.shape, shelf_offset=(0.0, 400.0))
self.assertTrue(again, "a shelf 400 px away reuses the screen row and must be read")
def test_a_seen_tag_with_no_text_is_detected_not_absent(self):
def reader(crop):
return []
service = TagService(FakeOcr(reader=reader), settings=TagSettings(pass_interval_frames=0))
array = image()
# A bright paper block on a brick rail, under the product row. OCR reads nothing.
array[300:420, :] = (40, 40, 180)
array[330:390, 120:280] = (235, 235, 235)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
bands = service.plan(boxes, array.shape)
self.assertTrue(bands)
service.run_pass(array, bands[0])
by_track, tags, counts = service.attach([{"track_id": 0, "box": [100, 100, 200, 300]}])
self.assertTrue(tags, "the paper block is a tag even though OCR returned nothing")
self.assertEqual(tags[0]["state"], TAG_STATE_DETECTED)
self.assertEqual(by_track[0]["state"], TAG_STATE_DETECTED)
self.assertEqual(counts[TAG_STATE_DETECTED], 1)
self.assertNotEqual(by_track[0]["state"], TAG_STATE_ABSENT)
def test_a_tag_pass_survives_a_missing_header_pattern(self):
"""Evaluation replay died with NameError: HEADER_PATTERN."""
import tag_cards
from tag_cards import OcrLine as CardLine
row = [CardLine(text="毎日お買得", box=(0, 0, 40, 16), confidence=0.9)]
saved = tag_cards.__dict__.pop("HEADER_PATTERN", None)
try:
score = tag_cards.tag_row_score(row)
finally:
if saved is not None:
tag_cards.HEADER_PATTERN = saved
self.assertGreater(score, 0, "a sale header still marks the row when the global is missing")
service, _ = self.make(pass_interval_frames=0)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
bands = service.plan(boxes, image().shape)
self.assertTrue(bands)
service.run_pass(image(), bands[0])
self.assertGreaterEqual(service.stats()["passes_run"], 1)
self.assertIsNone(service.stats()["last_pass_error"])
def test_a_queued_pass_keeps_the_shelf_it_was_planned_on(self):
"""OCR that finishes after a pan must clear its own rail, not the new one."""
service, _ = self.make(pass_interval_frames=0, settle_observations=1, settle_passes=1)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
planned = service.plan(boxes, array.shape, shelf_offset=(0.0, 0.0))
self.assertEqual(service.stats()["pending_bands"], 1)
moved = service.plan(boxes, array.shape, shelf_offset=(0.0, 400.0))
self.assertTrue(moved, "the new shelf is a different rail and must be queued too")
self.assertEqual(service.stats()["pending_bands"], 2)
service.run_pass(array, planned[0])
self.assertEqual(service.stats()["pending_bands"], 1, "the finished pass clears only its own key")
self.assertEqual(service._tags[next(iter(service._tags))].shelf_key[0], "shelf")
self.assertNotEqual(
service._tags[next(iter(service._tags))].shelf_key,
moved[0].key,
"the late read must not be filed under the shelf now in frame",
)
service.run_pass(array, moved[0])
self.assertEqual(service.stats()["pending_bands"], 0)
self.assertEqual(len({tag.shelf_key for tag in service._tags.values()}), 2)
def test_an_offscreen_tag_does_not_bind_the_new_facing(self):
"""A pan that reuses screen coordinates must not hand the old tag to the new facing."""
service, _ = self.make(pass_interval_frames=0, settle_observations=1, settle_passes=1)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
first = service.plan(boxes, array.shape, shelf_offset=(0.0, 0.0))
service.run_pass(array, first[0])
by_track, tags, _ = service.attach(
[{"track_id": 0, "box": [100, 100, 200, 300]}],
shelf_offset=(0.0, 0.0),
image_shape=array.shape,
)
self.assertEqual(by_track[0]["state"], TAG_STATE_READABLE)
self.assertEqual(by_track[0]["price"], 269)
self.assertGreaterEqual(tags[0]["box"][1], 0)
self.assertLess(tags[0]["box"][1], array.shape[0])
again = service.plan(boxes, array.shape, shelf_offset=(0.0, 400.0))
self.assertTrue(again)
by_track, tags, _ = service.attach(
[{"track_id": 9, "box": [100, 100, 200, 300]}],
shelf_offset=(0.0, 400.0),
image_shape=array.shape,
)
self.assertEqual(tags, [], "the old tag is above this frame")
self.assertEqual(by_track[9]["state"], TAG_STATE_ABSENT)
self.assertIsNone(by_track[9]["id"])
def test_attach_does_not_use_the_latest_plan_origin(self):
"""A later plan must not project this attach through its stored origin."""
service, _ = self.make(pass_interval_frames=0, settle_observations=1, settle_passes=1)
boxes = [box(100, 100, 200, 300, 0), box(220, 100, 320, 300, 1), box(340, 100, 440, 300, 2)]
array = image()
first = service.plan(boxes, array.shape, shelf_offset=(0.0, 0.0))
service.run_pass(array, first[0])
service.plan(boxes, array.shape, shelf_offset=(0.0, 400.0))
by_track, tags, _ = service.attach(
[{"track_id": 0, "box": [100, 100, 200, 300]}],
image_shape=array.shape,
)
self.assertTrue(tags, "omitting shelf_offset is screen coordinates, not the panned plan origin")
self.assertEqual(by_track[0]["state"], TAG_STATE_READABLE)
by_track, tags, _ = service.attach(
[{"track_id": 0, "box": [100, 100, 200, 300]}],
shelf_offset=(0.0, 0.0),
image_shape=array.shape,
)
self.assertEqual(by_track[0]["state"], TAG_STATE_READABLE)
class StorePolicyTests(unittest.TestCase):
def test_a_missing_file_uses_the_defaults(self):
policy = load_store_policy(None)
policy.active()
self.assertEqual(policy.tax_ratios, (1.08, 1.10))
self.assertEqual(policy.min_price, 50)
self.assertEqual(tag_cards.MIN_PRICE, 50)
def test_a_store_file_does_not_change_another_store(self):
import json
import tempfile
with tempfile.TemporaryDirectory() as root:
good = os.path.join(root, "lopia")
bad = os.path.join(root, "other")
os.makedirs(good)
os.makedirs(bad)
with open(os.path.join(good, "store.json"), "w", encoding="utf-8") as handle:
json.dump({"store_id": "lopia", "tax_ratios": [1.08], "min_price": 80}, handle)
with open(os.path.join(bad, "store.json"), "w", encoding="utf-8") as handle:
handle.write("{not json")
loaded = load_store_policy(good)
self.assertEqual(loaded.min_price, 80)
self.assertEqual(tag_cards.MIN_PRICE, 80)
with self.assertRaises(StorePolicyError) as raised:
load_store_policy(bad)
self.assertIn(bad, str(raised.exception))
self.assertEqual(tag_cards.MIN_PRICE, 80, "a bad file must not replace the store already loaded")
again = load_store_policy(good)
self.assertEqual(again.min_price, 80)
def test_a_ratio_outside_range_fails_clearly(self):
import json
import tempfile
load_store_policy(None)
with tempfile.TemporaryDirectory() as root:
with open(os.path.join(root, "store.json"), "w", encoding="utf-8") as handle:
json.dump({"tax_ratios": [2.5], "min_price": 999}, handle)
with self.assertRaises(StorePolicyError) as raised:
load_store_policy(root)
self.assertIn("tax ratio", str(raised.exception))
self.assertEqual(tag_cards.MIN_PRICE, 50)
class EngineSelectionTests(unittest.TestCase):
def test_explicit_none_and_fake(self):
self.assertFalse(build_ocr_engine("none").available())
self.assertTrue(build_ocr_engine("fake").available())
def test_vision_unavailable_here_is_honest(self):
"""On a machine without Vision the factory falls back rather than lying."""
engine = build_ocr_engine("auto")
if engine.available():
self.assertEqual(engine.name, "vision")
else:
self.assertIn(engine.name, ("none",))
def test_null_engine_returns_no_lines(self):
result = NullOcr().read(image())
self.assertEqual(result.lines, [])
self.assertEqual(result.engine, "none")
if __name__ == "__main__":
unittest.main()

122
recognization/code/test_real_model.py
View file

@ -1,122 +0,0 @@
"""
Opt-in integration test against the real model and the real catalog.
Skipped unless DORAYAKI_REAL_MODEL=1, because it loads the YOLO weights and
classifies a full shelf. It defends the end-to-end claim: the server detects the
whole shelf, accepts a meaningful share of the products, keeps a label stable
while the camera pans, and never names a SKU for a weak match.
cd recognization/code
DORAYAKI_REAL_MODEL=1 YOLO_OFFLINE=1 ../.venv/bin/python -m unittest test_real_model -v
"""
from __future__ import annotations
import math
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(current_dir)
if os.path.exists(os.path.join(project_root, "stock")):
os.chdir(project_root)
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
from engine import EngineSettings, build_default_engine # noqa: E402
from live import LiveSession, LiveSettings # noqa: E402
ENABLED = os.environ.get("DORAYAKI_REAL_MODEL") == "1"
# Measured baseline on image.jpg (2026-09-24): 170 detections, 105 recognized.
BASELINE_DETECTIONS = 170
BASELINE_RECOGNIZED = 105
@unittest.skipUnless(ENABLED, "set DORAYAKI_REAL_MODEL=1 to run against the real model")
class RealModelTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.engine = build_default_engine(EngineSettings(device="cpu"))
cls.engine.warm_up()
@classmethod
def tearDownClass(cls):
cls.engine.shutdown()
def test_recognize_whole_shelf(self):
with open("image.jpg", "rb") as handle:
payload = handle.read()
image = self.engine.decode_image(payload)
result = self.engine.recognize(image, classify=True)
summary = result["summary"]
self.assertGreaterEqual(
summary["in_view"], int(BASELINE_DETECTIONS * 0.95),
"detection must not regress below the measured baseline",
)
self.assertGreaterEqual(
summary["recognized"], int(BASELINE_RECOGNIZED * 0.9),
"recognition must not regress below the measured baseline",
)
# every named SKU must exist in the catalog
catalog = {product["sku_id"] for product in self.engine.catalog()}
for detection in result["detections"]:
identity = detection["identity"]
if identity["state"] == "recognized":
self.assertIn(identity["sku_id"], catalog)
else:
self.assertIsNone(identity["sku_id"], "only an accepted match may name a SKU")
def test_pan_keeps_labels_stable_and_advances_them(self):
image = cv2.imread("image.jpg")
height, width = image.shape[:2]
frame_w, frame_h = 1280, 720
session = LiveSession("real", self.engine, settings=LiveSettings(method="hybrid"))
session.refresh_catalog()
labels: dict = {}
relabels: dict = {}
restarts: dict = {}
unexplained = []
recognized_frames = 0
frames = 16
for index in range(frames):
phase = index / (frames - 1)
x = int((width - frame_w) * (0.5 - 0.5 * math.cos(2 * math.pi * phase)))
y = max(0, (height - frame_h) // 2)
window = image[y:y + frame_h, x:x + frame_w]
ok, buffer = cv2.imencode(".jpg", window, [cv2.IMWRITE_JPEG_QUALITY, 75])
session.submit_frame(index, buffer.tobytes())
pending = session.take_pending(timeout=5.0)
self.assertIsNotNone(pending)
result = session.process_frame(*pending)
import time
time.sleep(0.4) # let classification land, as a live client's next frame would
if result["summary"]["recognized"]:
recognized_frames += 1
for track in result["tracks"]:
identity = track["identity"]
if identity["state"] != "recognized":
continue
track_id = track["track_id"]
sku = identity["sku_id"]
if restarts.get(track_id, 0) < track["restarts"]:
labels.pop(track_id, None)
if track_id in labels and labels[track_id] != sku:
if track["relabels"] <= relabels.get(track_id, 0):
unexplained.append((track_id, labels[track_id], sku))
labels[track_id] = sku
relabels[track_id] = track["relabels"]
restarts[track_id] = track["restarts"]
self.assertGreater(recognized_frames, frames // 2, "labels should appear on most frames")
self.assertGreater(len(labels), 0, "the pan should label products")
self.assertEqual(unexplained, [], "a label must never change without re-verification")
if __name__ == "__main__":
unittest.main()

382
recognization/code/test_rebuild_catalog.py
View file

@ -1,382 +0,0 @@
"""
Contract tests for the enrollment tool's gates.
The refusal rules (size floors including the pyramid floor, session leakage,
name/OCR, duplicates) are what the tool exists for, and they are pinned here
against the measured distribution rather than against invented numbers. The
two checks that report rather than refuse are covered too, because both are
properties of the catalog as a whole and neither is visible from a single
view: cross-SKU near-duplicates, and the completeness rule that decides whether
a bay is live-ready.
"""
from __future__ import annotations
import os
import sys
import tempfile
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
import cv2 # noqa: E402
import numpy as np # noqa: E402
from rebuild_catalog import ( # noqa: E402
MIN_NAME_OCR,
PYRAMID_FLOOR,
RebuildError,
audit,
live_ready_verdict,
load_shelf_skus,
name_ocr_similarity,
text_discriminated_siblings,
)
def artwork(mark=None):
"""A package face; `mark` adds the small text that separates siblings."""
image = np.full((400, 300, 3), 200, dtype="uint8")
cv2.rectangle(image, (20, 20), (280, 380), (40, 40, 180), -1)
cv2.putText(image, "MAYO", (60, 200), cv2.FONT_HERSHEY_SIMPLEX, 2.0, (255, 255, 255), 4)
if mark:
cv2.putText(image, mark, (60, 300), cv2.FONT_HERSHEY_SIMPLEX, 1.2, (255, 255, 255), 3)
return image
class SiblingFlagTests(unittest.TestCase):
"""The near-unknown case: same artwork, different product."""
def make_source(self, images):
root = tempfile.mkdtemp(prefix="siblings-")
for name, image in images.items():
cv2.imwrite(os.path.join(root, name), image)
return root
def test_shared_artwork_across_skus_is_flagged(self):
root = self.make_source({"sku_x1.jpg": artwork(), "sku_x2.jpg": artwork("HALF")})
flagged = text_discriminated_siblings({"sku_x1": ["sku_x1.jpg"], "sku_x2": ["sku_x2.jpg"]}, root)
self.assertEqual(len(flagged), 1, flagged)
self.assertEqual(flagged[0]["sku"], "sku_x1")
self.assertEqual(flagged[0]["sibling_sku"], "sku_x2")
self.assertGreater(flagged[0]["cosine"], 0.98)
def test_distinct_artwork_is_not_flagged(self):
other = np.full((400, 300, 3), 30, dtype="uint8")
cv2.circle(other, (150, 200), 90, (200, 200, 40), -1)
root = self.make_source({"sku_x1.jpg": artwork(), "sku_x2.jpg": other})
self.assertEqual(
text_discriminated_siblings({"sku_x1": ["sku_x1.jpg"], "sku_x2": ["sku_x2.jpg"]}, root), []
)
def test_two_views_of_one_sku_are_not_siblings(self):
"""The same-SKU duplicate rule already covers these; this flag is cross-SKU."""
root = self.make_source({"sku_x1.jpg": artwork(), "sku_x1_2.jpg": artwork()})
self.assertEqual(
text_discriminated_siblings({"sku_x1": ["sku_x1.jpg", "sku_x1_2.jpg"]}, root), []
)
class PyramidFloorTests(unittest.TestCase):
"""The l09 defect: a 305 px view whose scores dwarf every other reference.
Refused, not reported. Reporting it is what let the defect ship - the view
enrolled anyway, because a line in a catalog README is not a gate.
"""
def make_source(self, height):
root = tempfile.mkdtemp(prefix="floor-")
image = cv2.resize(artwork(), (300, height), interpolation=cv2.INTER_NEAREST)
cv2.imwrite(os.path.join(root, "sku_x1.jpg"), image)
return root
def test_a_view_under_the_floor_is_refused(self):
kept, refused = audit(self.make_source(305), min_views=1)
self.assertEqual(kept, {}, refused)
self.assertEqual(len(refused), 1, refused)
self.assertIn("305px", refused[0]["reason"])
self.assertIn("pyramid floor", refused[0]["reason"])
def test_the_floor_is_exact(self):
"""799 refused, 800 kept: the boundary is the number, not a rounding."""
under = audit(self.make_source(PYRAMID_FLOOR - 1), min_views=1)
self.assertEqual(under[0], {}, under[1])
at = audit(self.make_source(PYRAMID_FLOOR), min_views=1)
self.assertEqual(at[0], {"sku_x1": ["sku_x1.jpg"]}, at[1])
self.assertEqual(at[1], [])
def test_a_full_size_view_still_enrolls(self):
kept, refused = audit(self.make_source(1000), min_views=1)
self.assertEqual(kept, {"sku_x1": ["sku_x1.jpg"]}, refused)
self.assertEqual(refused, [])
class NameOcrTests(unittest.TestCase):
"""The wrong-product gate: a packshot's OCR must explain its catalog row.
Pinned to the measured distribution, not to invented numbers. The reads are
what the vision backend returned for the shipped packshots and for the
rejected Kewpie 450g; recording them keeps the test on the pure function
instead of on Vision's recall, which moves with the macOS version.
"""
# Worst correct packshot in stock-lopia-clean: the l11 read carries
# "キューピー ハーフ" and "400g" but never "マヨネーズ".
WORST_CORRECT = (
"キューピー ハーフマヨネーズ 400g",
"2 キューピーハーフ New！ 卵黄ならではのコクとうま味 "
"※写真はイメージです。 カロリー 50%ット 当社マヨネーズ比 400g",
0.9167,
)
# The recorded l09/l10 mix-up: Kewpie Mayonnaise 450g, filed under an
# Ajinomoto row. This is its best score against any of the 12 catalog names.
WRONG_PRODUCT = (
"キューピー からしマヨネーズ 200g",
"キューピーマヨネーズ KEWPIE MAYONNAISE ® 450g",
0.5833,
)
def test_a_correct_packshot_clears_the_gate(self):
name, text, measured = self.WORST_CORRECT
self.assertAlmostEqual(name_ocr_similarity(name, text), measured, places=3)
self.assertGreaterEqual(name_ocr_similarity(name, text), MIN_NAME_OCR)
def test_the_wrong_size_packshot_is_refused(self):
"""The check that exists to stop the l09/l10 mix-up keeps its teeth."""
name, text, measured = self.WRONG_PRODUCT
self.assertAlmostEqual(name_ocr_similarity(name, text), measured, places=3)
self.assertLess(name_ocr_similarity(name, text), MIN_NAME_OCR)
def test_the_threshold_sits_between_the_two_measured_bands(self):
"""The recalibration pin. Lower it and the wrong product enrolls."""
worst_correct = name_ocr_similarity(*self.WORST_CORRECT[:2])
best_wrong = name_ocr_similarity(*self.WRONG_PRODUCT[:2])
self.assertGreater(worst_correct, best_wrong, "the bands overlap; no threshold separates them")
self.assertGreaterEqual(MIN_NAME_OCR, best_wrong)
self.assertLessEqual(MIN_NAME_OCR, worst_correct)
self.assertGreaterEqual(MIN_NAME_OCR - best_wrong, 0.1, "no headroom below the wrong-product ceiling")
self.assertGreaterEqual(worst_correct - MIN_NAME_OCR, 0.1, "no headroom above the worst correct packshot")
def test_a_maker_prefix_gap_does_not_admit_a_wrong_row(self):
"""l07: its product tokens all read, its maker and size do not.
The catalog names l07 "丸和油脂 ラッキーマヨネーズ 200g" and the packshot
reads "ラッキー マヨネーズ 調味料（アミノ酸）不使用" - every product bigram,
none of the maker or the size. The 0.5714 it scores is inside the
wrong-product band and stays refused: the gap is a name-token problem,
not a threshold to move. Scoping the check to the product tokens would
admit it; that is a change to this gate, not a reason to lower it.
"""
score = name_ocr_similarity("丸和油脂 ラッキーマヨネーズ 200g",
"ラッキー マヨネーズ 調味料（アミノ酸）不使用")
self.assertAlmostEqual(score, 0.5714, places=3)
self.assertLess(score, MIN_NAME_OCR)
def test_audit_refuses_a_packshot_the_name_never_supports(self):
"""l04 at 0.00: the upright read is three stray characters."""
root = tempfile.mkdtemp(prefix="name-ocr-")
cv2.imwrite(os.path.join(root, "sku_x1.jpg"),
cv2.resize(artwork(), (300, 800), interpolation=cv2.INTER_NEAREST))
class FixedOcr:
name = "fixed"
def read(self, image):
class Line:
text = "明太手屋さんが つくつ た コ"
class Read:
lines = [Line()]
return Read()
kept, refused = audit(root, min_views=1,
names={"sku_x1": "やまや めんたいマヨネーズタイプ 200g"},
ocr_engine=FixedOcr())
self.assertEqual(kept, {}, refused)
self.assertEqual(len(refused), 1, refused)
self.assertIn("name/OCR", refused[0]["reason"])
class CompletenessVerdictTests(unittest.TestCase):
"""The enrollment protocol's completeness rule, as a verdict."""
def test_without_a_shelf_list_the_verdict_is_unknown(self):
"""Not a guess: the rule needs the shelf's SKU list to mean anything."""
verdict = live_ready_verdict({"sku_a": ["a.jpg"]}, {"sku_a": "A"}, None)
self.assertIsNone(verdict["live_ready"])
self.assertIn("shelf", verdict["reason"])
def test_every_shelf_sku_enrolled_is_live_ready(self):
verdict = live_ready_verdict({"sku_a": ["a.jpg"]}, {"sku_a": "A"}, {"sku_a"})
self.assertTrue(verdict["live_ready"])
def test_a_text_only_row_satisfies_the_rule(self):
"""A CSV row with no image still names and vetoes from its text."""
verdict = live_ready_verdict({"sku_a": ["a.jpg"]}, {"sku_a": "A", "sku_b": "B"}, {"sku_a", "sku_b"})
self.assertTrue(verdict["live_ready"])
def test_a_shelf_sku_absent_from_the_catalog_fails(self):
verdict = live_ready_verdict({"sku_a": ["a.jpg"]}, {"sku_a": "A"}, {"sku_a", "sku_z"})
self.assertFalse(verdict["live_ready"])
self.assertEqual(verdict["missing_from_catalog"], ["sku_z"])
class ShelfTagListTests(unittest.TestCase):
def write(self, text):
path = os.path.join(tempfile.mkdtemp(prefix="tags-"), "tags.json")
with open(path, "w", encoding="utf-8") as handle:
handle.write(text)
return path
def test_a_json_list_is_read(self):
self.assertEqual(load_shelf_skus(self.write('["sku_a", "sku_b"]')), {"sku_a", "sku_b"})
def test_one_sku_per_line_is_read(self):
self.assertEqual(load_shelf_skus(self.write("sku_a\nsku_b\n")), {"sku_a", "sku_b"})
def test_no_path_means_no_list(self):
self.assertIsNone(load_shelf_skus(None))
def test_a_json_object_is_refused(self):
with self.assertRaises(RebuildError):
load_shelf_skus(self.write('{"sku_a": 1}'))
class BadShelfListFailsBeforeTheSwapTests(unittest.TestCase):
"""A bad --shelf-tags must not leave a replaced catalog behind.
The list is read before write_catalog runs, so a malformed file or a missing
path fails with the tool's own message and the destination is untouched. Read
after the swap, the same error would escape as a traceback over a catalog
that had already been replaced.
"""
def make_source(self):
root = tempfile.mkdtemp(prefix="guard-src-")
cv2.imwrite(os.path.join(root, "sku_x1.jpg"), artwork())
with open(os.path.join(root, "stock.csv"), "w", encoding="utf-8") as handle:
handle.write("SKU_ID,product_name,price,number_in,number_out,number_on\n")
handle.write("sku_x1,Alpha,100,1,0,1\n")
return root
def run_tool(self, shelf_tags):
from rebuild_catalog import main
source = self.make_source()
output = os.path.join(tempfile.mkdtemp(prefix="guard-out-"), "catalog")
with self.assertRaises(SystemExit) as caught:
main(["--source", source, "--output", output, "--csv",
os.path.join(source, "stock.csv"), "--min-views", "1",
"--ocr", "none", "--shelf-tags", shelf_tags])
return str(caught.exception), output
def test_a_malformed_list_fails_cleanly_and_writes_nothing(self):
path = os.path.join(tempfile.mkdtemp(prefix="bad-tags-"), "tags.json")
with open(path, "w", encoding="utf-8") as handle:
handle.write('{"sku_x1": 1}')
message, output = self.run_tool(path)
self.assertIn("expected a JSON list", message)
self.assertFalse(os.path.exists(output), "the catalog was replaced before the check")
def test_a_missing_list_fails_cleanly_and_writes_nothing(self):
message, output = self.run_tool("/tmp/definitely-not-a-shelf-list.json")
self.assertIn("No such file", message)
self.assertFalse(os.path.exists(output), "the catalog was replaced before the check")
class OrientationTests(unittest.TestCase):
"""A packshot whose identifying text is not horizontal.
The やまや めんたいマヨネーズ packshot reads its brand only when rotated;
the upright pass returns stray kana. Checking one orientation refused a
correct image, which is the worst failure this check has - it rejects good
data while claiming to validate it.
"""
class FakeOcr:
"""Reads a fixed string per rotation.
Rotations are identified by aspect ratio, not by exact shape: the
caller upscales before OCR, so the fake sees scaled dimensions. The
upright and 180-degree passes keep the source's orientation (wide),
the two 90-degree passes swap it (tall). A corner marker cannot tell
them apart, because `cv2.rotate` carries the corner value through.
"""
name = "fake"
def __init__(self, by_orientation):
self.by_orientation = by_orientation
def read(self, image):
class Line:
def __init__(self, text):
self.text = text
class Read:
def __init__(self, text):
self.lines = [Line(part) for part in text.split("|") if part]
height, width = image.shape[:2]
key = "wide" if width > height else "tall"
return Read(self.by_orientation.get(key, ""))
def marked(self, height, width):
return np.zeros((height, width, 3), dtype="uint8")
def test_a_rotated_read_wins_when_it_matches_the_name(self):
"""The upright pass reads noise; the rotated pass reads the product."""
from rebuild_catalog import best_orientation_text
engine = self.FakeOcr({"wide": "つくつ|た", "tall": "明太手屋さんが|マヨネーズ"})
text = best_orientation_text(engine, self.marked(40, 60), "やまや めんたいマヨネーズタイプ 200g")
self.assertIn("明太手屋さんが", text)
def test_the_upright_read_wins_when_it_matches(self):
from rebuild_catalog import best_orientation_text
engine = self.FakeOcr({"wide": "ピュアセレクト|マヨネーズ|450g", "tall": "ノイズ"})
text = best_orientation_text(engine, self.marked(40, 60), "ピュアセレクト マヨネーズ 450g")
self.assertIn("ピュアセレクト", text)
def test_without_a_name_the_longest_read_wins(self):
from rebuild_catalog import best_orientation_text
engine = self.FakeOcr({"wide": "ab", "tall": "abcdefgh"})
self.assertEqual(best_orientation_text(engine, self.marked(40, 60)), "abcdefgh")
def test_audit_keeps_a_view_whose_name_only_reads_rotated(self):
"""The regression: `audit` must try every orientation before refusing.
This is the boundary that matters. `best_orientation_text` passing its
own unit test proves nothing about enrollment - the bug was that
`audit` called a single-orientation read, so a correct packshot whose
brand text runs vertically was refused as a name/OCR mismatch.
"""
from rebuild_catalog import audit
root = tempfile.mkdtemp(prefix="orient-")
# Wider than tall, so the upright pass is the "wide" read (noise) and
# only a 90-degree rotation produces the "tall" read that matches.
cv2.imwrite(os.path.join(root, "sku_x1.jpg"), self.marked(800, 1000))
engine = self.FakeOcr({"wide": "つくつ|た", "tall": "キユーピー|エッグケア|305g"})
kept, refused = audit(root, min_views=1, names={"sku_x1": "キューピー エッグケア 305g"},
ocr_engine=engine)
self.assertEqual(kept, {"sku_x1": ["sku_x1.jpg"]}, refused)
self.assertEqual(refused, [])
def test_audit_still_refuses_a_view_the_name_never_matches(self):
"""The check must keep its teeth: a wrong product is still refused."""
from rebuild_catalog import audit
root = tempfile.mkdtemp(prefix="orient-bad-")
cv2.imwrite(os.path.join(root, "sku_x1.jpg"), self.marked(800, 300))
engine = self.FakeOcr({"wide": "キューピー|マヨネーズ", "tall": "キューピー|マヨネーズ"})
kept, refused = audit(root, min_views=1, names={"sku_x1": "やまや めんたいマヨネーズタイプ 200g"},
ocr_engine=engine)
self.assertEqual(kept, {})
self.assertEqual(len(refused), 1, refused)
self.assertIn("name/OCR", refused[0]["reason"])

363
recognization/code/test_reconcile.py
View file

@ -1,363 +0,0 @@
"""
Contract tests for reconciliation and the compliance rule family.
Pure functions over evidence, so these run without OCR, weights, or a camera.
The rules that matter most here are the fail-closed ones: unreadable evidence
must never become a violation.
"""
from __future__ import annotations
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from reconcile import ( # noqa: E402
ComplianceSettings,
ComplianceTracker,
RECONCILED_CONFIRMED,
RECONCILED_MISMATCH,
RECONCILED_NO_TAG,
RECONCILED_TAG_ONLY,
RECONCILED_UNVERIFIED,
RULE_MISSING_TAG,
RULE_MISSING_TAG_GROUP,
RULE_ORPHAN_TAG,
RULE_PRICE_MISMATCH,
RULE_UNREADABLE_TAGS,
evaluate_compliance,
longest_common_substring_ratio,
normalize_name,
reconcile,
summarize_findings,
)
from tags import TAG_STATE_ABSENT, TAG_STATE_READABLE, TAG_STATE_UNREADABLE # noqa: E402
def accepted(sku="sku_003"):
return {"state": "recognized", "sku_id": sku, "identity_level": "accepted", "confidence": 0.9}
def tag(state=TAG_STATE_READABLE, **overrides):
payload = {
"id": "tag_1",
"box": [100, 300, 200, 340],
"state": state,
"price": None,
"regular_price": None,
"name": None,
"barcode": None,
"flags": [],
"confidence": 0.8,
"observations": 3,
"settled": True,
}
payload.update(overrides)
return payload
def product(price=168.0, name="Tropicana エッセンシャルズ マルチビタミン", jan=None):
payload = {"sku_id": "sku_003", "product_name": name, "price": price}
if jan:
payload["jan"] = jan
return payload
class NameMatchingTests(unittest.TestCase):
def test_normalization_strips_noise(self):
self.assertEqual(normalize_name("Ｔｒｏｐｉｃａｎａ エッセンシャルズ"), "tropicanaエッセンシャルズ")
self.assertEqual(normalize_name("キュービー<310g>"), "キュービー")
def test_lcs_ratio(self):
self.assertEqual(longest_common_substring_ratio("tropicana", "tropicana"), 1.0)
self.assertGreater(longest_common_substring_ratio("tropicanaエッセンシャルズ", "tropicana"), 0.5)
self.assertEqual(longest_common_substring_ratio("", "x"), 0.0)
class ReconcileTests(unittest.TestCase):
def test_price_agreement_confirms(self):
result = reconcile(accepted(), tag(price=168), product(price=168.0))
self.assertEqual(result["state"], RECONCILED_CONFIRMED)
self.assertTrue(any(signal["kind"] == "price" and signal["agrees"] for signal in result["signals"]))
def test_regular_price_agreement_confirms(self):
result = reconcile(accepted(), tag(price=140, regular_price=168), product(price=168.0))
self.assertEqual(result["state"], RECONCILED_CONFIRMED)
def test_price_disagreement_is_a_mismatch_with_both_values(self):
result = reconcile(accepted(), tag(price=269), product(price=168.0))
self.assertEqual(result["state"], RECONCILED_MISMATCH)
self.assertIn("269", result["message"])
self.assertIn("168", result["message"])
def test_name_agreement_confirms_even_without_a_price(self):
result = reconcile(accepted(), tag(name="Tropicana エッセンシャルズ"), product(price=None, name="Tropicana エッセンシャルズ マルチビタミン"))
self.assertEqual(result["state"], RECONCILED_CONFIRMED)
def test_barcode_disagreement_is_a_mismatch(self):
result = reconcile(accepted(), tag(price=168, barcode="4902105000001"),
product(price=168.0, jan="4902105000002"))
self.assertEqual(result["state"], RECONCILED_MISMATCH)
def test_barcode_agreement_confirms(self):
result = reconcile(accepted(), tag(price=None, barcode="4902105000001"),
product(price=None, jan="4902105000001"))
self.assertEqual(result["state"], RECONCILED_CONFIRMED)
def test_tag_read_but_nothing_comparable(self):
result = reconcile(accepted(), tag(name="コーヒー"), product(price=None, name=None))
self.assertEqual(result["state"], RECONCILED_UNVERIFIED)
def test_unreadable_tag_is_unverified_not_missing(self):
result = reconcile(accepted(), tag(state=TAG_STATE_UNREADABLE), product())
self.assertEqual(result["state"], RECONCILED_UNVERIFIED)
def test_absent_tag(self):
result = reconcile(accepted(), tag(state=TAG_STATE_ABSENT), product())
self.assertEqual(result["state"], RECONCILED_NO_TAG)
def test_readable_tag_without_a_product(self):
result = reconcile(None, tag(price=269), None)
self.assertEqual(result["state"], RECONCILED_TAG_ONLY)
def test_nothing_to_reconcile_returns_none(self):
self.assertIsNone(reconcile(None, None, None))
self.assertIsNone(reconcile({"state": "unconfirmed"}, tag(state=TAG_STATE_ABSENT), product()))
# --------------------------------------------------------------------------- compliance
def band(row_index, box):
class Band:
pass
instance = Band()
instance.row_index = row_index
instance.box = box
return instance
def track(track_id, box, sku="sku_003", state="recognized"):
return {
"track_id": track_id,
"box": box,
"raw_box": box,
"identity": {"state": state, "sku_id": sku if state == "recognized" else None},
}
# Wide enough that separate products can sit apart in a row.
BANDS = [band(1, [0, 300, 3000, 340]), band(2, [0, 600, 3000, 640])]
class ComplianceTests(unittest.TestCase):
def test_missing_tag_on_a_covered_row_is_a_violation(self):
# Four separate products (wide gaps), three with tag evidence - 75%, above
# the 70% bar - and one with none.
tracks = [
track(index, [10 + index * 300, 100, 140 + index * 300, 300], sku=f"sku_{index}")
for index in range(4)
]
tags = {
0: tag(state=TAG_STATE_READABLE, price=168, id="tag_a"),
1: tag(state=TAG_STATE_READABLE, price=178, id="tag_b"),
2: tag(state=TAG_STATE_ABSENT),
3: tag(state=TAG_STATE_READABLE, price=188, id="tag_c"),
}
findings = evaluate_compliance(tracks, tags, [tags[0], tags[1], tags[3]], BANDS)
rules = [finding.rule for finding in findings]
self.assertIn(RULE_MISSING_TAG, rules)
missing = [finding for finding in findings if finding.rule == RULE_MISSING_TAG][0]
self.assertEqual(missing.count, 1, "one finding per row, counting the products")
def test_a_row_with_poor_tag_coverage_is_not_evaluated(self):
"""The real failure mode: our detector found tags for only a few of many
facings. Claiming the rest are missing would be a false violation."""
tracks = [track(index, [10 + index * 300, 100, 90 + index * 300, 300]) for index in range(6)]
tags = {index: tag(state=TAG_STATE_ABSENT) for index in range(6)}
tags[0] = tag(state=TAG_STATE_READABLE, price=168, id="tag_a")
findings = evaluate_compliance(tracks, tags, [tags[0]], BANDS)
self.assertEqual([finding.rule for finding in findings], [RULE_UNREADABLE_TAGS])
self.assertEqual(findings[0].severity, "hint")
self.assertEqual(summarize_findings(findings)["violations"], 0)
def test_unreadable_row_yields_a_hint_never_a_violation(self):
"""The demo image case: tags exist but cannot be read. Nothing may be claimed."""
tracks = [
track(1, [10, 100, 90, 300]),
track(2, [110, 100, 190, 300]),
track(3, [210, 100, 290, 300]),
]
tags = {
1: tag(state=TAG_STATE_UNREADABLE, price=None, id="tag_1"),
2: tag(state=TAG_STATE_UNREADABLE, price=None, id="tag_2"),
3: tag(state=TAG_STATE_UNREADABLE, price=None, id="tag_3"),
}
findings = evaluate_compliance(tracks, tags, [tags[1], tags[2], tags[3]], BANDS)
self.assertEqual([finding.rule for finding in findings], [RULE_UNREADABLE_TAGS])
self.assertEqual(findings[0].severity, "hint")
self.assertEqual(summarize_findings(findings)["violations"], 0)
def test_a_row_with_no_tag_evidence_at_all_is_silent(self):
tracks = [track(1, [10, 100, 90, 300])]
tags = {1: tag(state=TAG_STATE_ABSENT)}
findings = evaluate_compliance(tracks, tags, [], BANDS)
self.assertEqual(findings, [])
def test_group_of_consecutive_facings_without_tags(self):
# Coverage counts *products* with any tag evidence, so the row needs enough
# of it to be judged at all (bar is 70%: 7 of 10 products).
tracks = [
track(index, [10 + index * 300, 100, 70 + index * 300, 300], sku=f"sku_{index}")
for index in range(10)
]
tags = {index: tag(state=TAG_STATE_READABLE, price=168 + index, id=f"tag_{index}") for index in range(10)}
for absent in (2, 3, 4):
tags[absent] = tag(state=TAG_STATE_ABSENT, id=None)
findings = evaluate_compliance(tracks, tags, [t for t in tags.values() if t["id"]], BANDS)
groups = [finding for finding in findings if finding.rule == RULE_MISSING_TAG_GROUP]
self.assertEqual(len(groups), 1)
self.assertEqual(groups[0].count, 3)
self.assertEqual(groups[0].row_index, 1)
self.assertIn("3 products", groups[0].detail)
def test_a_product_whose_tag_sits_elsewhere_in_the_row_is_not_missing(self):
"""Measured false positive: a run of ピュアセレクト facings was split by
geometry, and the piece was reported as missing the tag that was sitting
further along its own run. Identity and the row's tags both guard it."""
tracks = [
track(0, [10, 100, 160, 300], sku="sku_a"),
track(1, [170, 100, 320, 300], sku="sku_a"),
track(2, [330, 100, 460, 300], sku="sku_a"),
]
tags = {
0: tag(state=TAG_STATE_READABLE, price=209, id="tag_a"),
1: tag(state=TAG_STATE_ABSENT),
2: tag(state=TAG_STATE_ABSENT),
}
findings = evaluate_compliance(tracks, tags, [tags[0]], BANDS)
self.assertEqual(findings, [], "the product's tag is on its own run")
def test_two_missing_products_are_not_a_group(self):
tracks = [
track(index, [10 + index * 300, 100, 140 + index * 300, 300], sku=f"sku_{index}")
for index in range(4)
]
tags = {
0: tag(state=TAG_STATE_READABLE, price=168, id="tag_a"),
1: tag(state=TAG_STATE_ABSENT),
2: tag(state=TAG_STATE_READABLE, price=188, id="tag_b"),
3: tag(state=TAG_STATE_READABLE, price=198, id="tag_c"),
}
findings = evaluate_compliance(tracks, tags, [tags[0], tags[2], tags[3]], BANDS)
self.assertNotIn(RULE_MISSING_TAG_GROUP, [finding.rule for finding in findings])
missing = [finding for finding in findings if finding.rule == RULE_MISSING_TAG]
self.assertEqual(len(missing), 1)
self.assertEqual(missing[0].count, 1)
def test_facings_of_one_product_share_a_tag(self):
"""The measured domain rule: this store prints one tag per product, not per
facing - a row carries ~16 facings but only ~4 tags. Counting facings would
accuse a perfectly labelled shelf."""
# Four facings of the same product (touching, same size) under one tag.
tracks = [track(index, [10 + index * 80, 100, 90 + index * 80, 300]) for index in range(4)]
tags = {
0: tag(state=TAG_STATE_READABLE, price=269, id="tag_a"),
1: tag(state=TAG_STATE_ABSENT),
2: tag(state=TAG_STATE_ABSENT),
3: tag(state=TAG_STATE_ABSENT),
}
findings = evaluate_compliance(tracks, tags, [tags[0]], BANDS)
self.assertEqual(findings, [], "one tag covers the whole product: nothing is wrong")
def test_orphan_tag(self):
tracks = [track(1, [10, 100, 90, 300])]
tags = {1: tag(state=TAG_STATE_READABLE, price=168, id="tag_a")}
all_tags = [tags[1], tag(state=TAG_STATE_READABLE, price=999, id="tag_lonely", box=[300, 600, 400, 640])]
findings = evaluate_compliance(tracks, tags, all_tags, BANDS)
orphans = [finding for finding in findings if finding.rule == RULE_ORPHAN_TAG]
self.assertEqual(len(orphans), 1)
self.assertEqual(orphans[0].box, [300, 600, 400, 640])
def test_price_mismatch_becomes_a_finding(self):
# A reportable mismatch carries an anchor: the tag's JAN was read, so we
# know which tag we are comparing.
tracks = [track(1, [10, 100, 90, 300])]
# The anchor is the tag's own arithmetic agreeing, not a price field.
tags = {1: tag(state=TAG_STATE_READABLE, price=269, flags=["tax_consistent"])}
reconciliations = {1: reconcile(accepted(), tags[1], product(price=168.0))}
findings = evaluate_compliance(tracks, tags, [tags[1]], BANDS, reconciliations)
mismatches = [finding for finding in findings if finding.rule == RULE_PRICE_MISMATCH]
self.assertEqual(len(mismatches), 1)
self.assertIn("269", mismatches[0].detail)
def test_a_mismatch_without_an_anchor_is_only_a_hint(self):
"""Measured noise: a wrong identity plus a garbled tag read (name match
0.11, no JAN) produced a confident "tag says 90, catalog says 188" that was
really two bad reads disagreeing. Without an anchor it is a hint."""
tracks = [track(1, [10, 100, 90, 300])]
tags = {1: tag(state=TAG_STATE_READABLE, price=90, name="ラスお要をおいしくだべおまにすくg)")}
reconciliations = {1: reconcile(accepted(), tags[1], product(price=188.0))}
findings = evaluate_compliance(tracks, tags, [tags[1]], BANDS, reconciliations)
self.assertNotIn(RULE_PRICE_MISMATCH, [finding.rule for finding in findings])
hints = [finding for finding in findings if finding.severity == "hint"]
self.assertTrue(hints)
self.assertEqual(summarize_findings(findings)["violations"], 0)
def test_summary_counts_only_violations(self):
findings = [
evaluate_compliance(
[track(1, [10, 100, 90, 300])],
{1: tag(state=TAG_STATE_READABLE, price=269, flags=["tax_consistent"])},
[tag(state=TAG_STATE_READABLE, price=269, flags=["tax_consistent"])],
BANDS,
{1: reconcile(accepted(), tag(price=269), product(price=168.0))},
)
][0]
summary = summarize_findings(findings)
self.assertGreaterEqual(summary["violations"], 1)
self.assertIn(RULE_PRICE_MISMATCH, summary["by_rule"])
class StabilityTests(unittest.TestCase):
def test_findings_need_to_persist(self):
tracker = ComplianceTracker(ComplianceSettings(stability_frames=3, stability_window=5))
from reconcile import Finding
finding = Finding(rule=RULE_MISSING_TAG, severity="violation", row_index=1, box=[0, 0, 10, 10])
first = tracker.update([finding])
self.assertEqual(len(first), 0, "a single frame must not publish a violation")
tracker.update([finding])
stable = tracker.update([finding])
self.assertEqual(len(stable), 1)
def test_a_transient_finding_never_publishes(self):
"""The rule is windowed: a finding must appear in 3 of the last 5 frames.
Two sightings spread over six frames is a flicker, not a violation."""
tracker = ComplianceTracker(ComplianceSettings(stability_frames=3, stability_window=5))
from reconcile import Finding
finding = Finding(rule=RULE_MISSING_TAG, severity="violation", row_index=1, box=[0, 0, 10, 10])
published = 0
for index in range(6):
published += len(tracker.update([finding] if index in (0, 2) else []))
self.assertEqual(published, 0)
def test_a_finding_present_in_three_of_five_frames_publishes(self):
tracker = ComplianceTracker(ComplianceSettings(stability_frames=3, stability_window=5))
from reconcile import Finding
finding = Finding(rule=RULE_MISSING_TAG, severity="violation", row_index=1, box=[0, 0, 10, 10])
published = 0
for index in range(5):
published += len(tracker.update([finding] if index in (0, 2, 4) else []))
self.assertGreater(published, 0)
if __name__ == "__main__":
unittest.main()

96
recognization/code/test_record.py
View file

@ -1,96 +0,0 @@
"""Tests for recording a live session as data to label."""
from __future__ import annotations
import json
import os
import shutil
import sys
import tempfile
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from record import FrameRecorder, labelling_candidates # noqa: E402
def a_result(sku=None, flags=None):
return {
"summary": {"in_view": 1, "recognized": 1 if sku else 0},
"tracks": [
{
"box": [10, 100, 100, 300],
"identity": {"state": "recognized", "sku_id": sku} if sku else {"state": "unconfirmed"},
"tag_identity": {"sku_id": sku, "score": 0.7} if sku else {},
"price_tag": {"state": "readable", "price": 269, "flags": flags or []},
}
],
}
class RecorderTests(unittest.TestCase):
def setUp(self):
self.root = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.root, ignore_errors=True)
def test_frames_and_an_index_are_written(self):
recorder = FrameRecorder(self.root, every=1)
for frame_id in range(3):
recorder.write("session-a", frame_id, b"\xff\xd8jpeg", a_result("sku_l05"))
directory = os.path.join(self.root, "session-a")
self.assertEqual(sorted(os.listdir(directory)), ["000000.jpg", "000001.jpg", "000002.jpg", "index.jsonl"])
lines = [json.loads(line) for line in open(os.path.join(directory, "index.jsonl"), encoding="utf-8")]
self.assertEqual([line["frame"] for line in lines], [0, 1, 2])
self.assertEqual(lines[0]["tracks"][0]["tag_identity"]["sku_id"], "sku_l05")
def test_the_active_backend_is_written_on_each_line(self):
recorder = FrameRecorder(self.root, every=1, identity_backend="gallery")
recorder.write("session-d", 0, b"x", a_result("sku_l05"))
recorder.write("session-d", 1, b"y", a_result())
with open(os.path.join(self.root, "session-d", "index.jsonl"), encoding="utf-8") as handle:
lines = [json.loads(line) for line in handle]
self.assertEqual([line["identity_backend"] for line in lines], ["gallery", "gallery"])
def test_an_unspecified_backend_is_omitted(self):
recorder = FrameRecorder(self.root, every=1)
recorder.write("session-e", 0, b"x", a_result())
with open(os.path.join(self.root, "session-e", "index.jsonl"), encoding="utf-8") as handle:
line = json.loads(handle.readline())
self.assertNotIn("identity_backend", line)
def test_sampling_keeps_the_frame_numbering_visible(self):
recorder = FrameRecorder(self.root, every=3)
for frame_id in range(9):
recorder.write("session-b", frame_id, b"x", a_result())
lines = [json.loads(line) for line in open(os.path.join(self.root, "session-b", "index.jsonl"), encoding="utf-8")]
self.assertEqual([line["frame"] for line in lines], [2, 5, 8],
"every third frame, recorded under its own number")
def test_sessions_are_separate(self):
recorder = FrameRecorder(self.root, every=1)
recorder.write("one", 0, b"a", a_result())
recorder.write("two", 0, b"b", a_result())
self.assertTrue(os.path.exists(os.path.join(self.root, "one", "000000.jpg")))
self.assertTrue(os.path.exists(os.path.join(self.root, "two", "000000.jpg")))
def test_a_session_is_capped(self):
recorder = FrameRecorder(self.root, every=1, max_frames_per_session=2)
for frame_id in range(5):
recorder.write("capped", frame_id, b"x", a_result())
self.assertEqual(recorder.stats()["frames_written"], 2)
def test_labelling_candidates_come_from_the_tag_channel(self):
recorder = FrameRecorder(self.root, every=1)
recorder.write("session-c", 0, b"x", a_result("sku_l05", ["tax_consistent"]))
recorder.write("session-c", 1, b"x", a_result())
seeds = labelling_candidates(os.path.join(self.root, "session-c", "index.jsonl"))
self.assertEqual(len(seeds), 1, "only the tag-named facing is a seed")
self.assertEqual(seeds[0]["sku_id"], "sku_l05")
self.assertTrue(seeds[0]["corroborated"])
self.assertEqual(seeds[0]["tag_price"], 269)
if __name__ == "__main__":
unittest.main()

137
recognization/code/test_score_stream.py
View file

@ -1,137 +0,0 @@
"""
Tests for the G1 scorer.
The gate has two halves - the new lane covers at least as much as Van, and the
lower bound of the paired-bootstrap difference clears -2 pp - and the second
half is only computable if a scored row still says which facing it matched.
These tests pin that pairing and the bound it produces.
"""
from __future__ import annotations
import os
import sys
import tempfile
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from score_stream import ( # noqa: E402
G1_BOUND_MIN,
correct_facings,
paired_bootstrap,
score,
)
def facing(index, box, label="sku_a"):
return {"index": index, "box": list(box), "label": label}
def track(track_id, box, sku="sku_a"):
return {"track_id": track_id, "sku_id": sku, "label_box": list(box)}
def report(rows, windows=None):
"""A score report shaped the way `score` returns one, without the boxes."""
return {"rows": rows, "windows": windows}
def correct(track_id, matched):
return {"track_id": track_id, "predicted": "sku_a", "facing": matched,
"truth": "sku_a", "iou": 1.0, "outcome": "correct"}
def grid(count):
return [facing(i, [i * 20, 0, i * 20 + 10, 10]) for i in range(count)]
class ScoreRowTests(unittest.TestCase):
def setUp(self):
self.stock = tempfile.TemporaryDirectory()
open(os.path.join(self.stock.name, "sku_a.jpg"), "wb").close()
self.addCleanup(self.stock.cleanup)
def test_rows_carry_the_matched_facing(self):
facings = [facing(7, [0, 0, 100, 100]), facing(8, [900, 0, 1000, 100])]
rows = score([track(1, [0, 0, 100, 100]), track(2, [400, 400, 500, 500])],
facings, self.stock.name, 0.5)["rows"]
by_track = {row["track_id"]: row for row in rows}
self.assertEqual(by_track[1]["facing"], 7)
self.assertEqual(by_track[1]["outcome"], "correct")
# A track that hit nothing is unpaired, and says so instead of guessing.
self.assertIsNone(by_track[2]["facing"])
self.assertEqual(by_track[2]["outcome"], "unmatched")
def test_correct_facings_counts_a_facing_once(self):
rows = [correct(1, 7), correct(2, 7), correct(3, 9)]
self.assertEqual(correct_facings(rows), {7, 9})
class PairedBootstrapTests(unittest.TestCase):
"""27 facings is a small sample, so the pairing is what carries the gate."""
def test_identical_systems_sit_at_zero_everywhere(self):
rows = [correct(1, i) for i in range(10)]
result = paired_bootstrap(report(rows), report(rows), [])
self.assertEqual(result["difference"], 0.0)
self.assertEqual(result["lower_95"], 0.0)
self.assertEqual(result["upper_95"], 0.0)
self.assertEqual(result["van"], result["new"])
def test_a_superset_of_vans_hits_can_never_bound_below_zero(self):
# The image6 shape: the new lane gets everything Van got, plus a facing.
van = [correct(i, i) for i in range(17)]
new = van + [correct(99, 17)]
result = paired_bootstrap(report(van), report(new), grid(27))
self.assertEqual(result["facings"], 27)
self.assertEqual(result["van"], 17)
self.assertEqual(result["new"], 18)
self.assertAlmostEqual(result["difference"], 1 / 27, places=4)
# No resample of these 27 can put a facing in the new run that Van's run
# did not already have, so the 2.5th percentile is a hard zero. A bound
# printed as +0.0 pp is the floor of this pairing, not a margin.
self.assertEqual(result["lower_95"], 0.0)
self.assertGreaterEqual(result["lower_95"], G1_BOUND_MIN)
def test_a_swapped_facing_spreads_across_zero(self):
van = [correct(i, i) for i in range(17)]
new = van[:16] + [correct(99, 17)]
result = paired_bootstrap(report(van), report(new), grid(27))
self.assertEqual(result["difference"], 0.0)
self.assertLess(result["lower_95"], G1_BOUND_MIN)
self.assertGreater(result["upper_95"], 0.0)
def test_both_runs_are_graded_on_what_either_window_reached(self):
# Two facings at opposite edges; each run panned to one of them. Grading
# on the union keeps both on the same facings; the intersection would
# score each run on its own pan and the pairing would be empty.
facings = [facing(0, [0, 0, 100, 100]), facing(1, [900, 0, 1000, 100])]
left, right = [0, 0, 500, 1000], [500, 0, 1000, 1000]
result = paired_bootstrap(report([correct(1, 0)], [left]),
report([correct(2, 1)], [right]), facings)
self.assertEqual(result["facings"], 2)
self.assertEqual((result["van"], result["new"]), (1, 1))
self.assertEqual(result["difference"], 0.0)
def test_the_bound_repeats_for_the_same_seed(self):
van = [correct(i, i) for i in range(17)]
new = van[:5] + [correct(90 + i, 20 + i) for i in range(9)]
facings = grid(27)
first = paired_bootstrap(report(van), report(new), facings)
self.assertEqual(first, paired_bootstrap(report(van), report(new), facings))
def test_no_reachable_facing_scores_zero_rather_than_dividing_by_zero(self):
# Both runs panned to a corner the only facing is not in: the pairing
# set is empty, and a rate is not a number there.
elsewhere = [facing(0, [0, 0, 10, 10])]
corner = [[500, 500, 600, 600]]
result = paired_bootstrap(report([], corner), report([], corner), elsewhere)
self.assertEqual(result["facings"], 0)
self.assertEqual(result["lower_95"], 0.0)
if __name__ == "__main__":
unittest.main()

125
recognization/code/test_shelf_map.py
View file

@ -1,125 +0,0 @@
"""
Tests for the shelf map: registration, merging, and evidence pooling.
The sign convention is tested explicitly, because getting it backwards is silent:
the map still produces boxes, they just land in the wrong place, and a shelf of
~60 facings becomes 857. That happened during development.
"""
from __future__ import annotations
import os
import sys
import unittest
import numpy as np
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(current_dir)
if os.path.exists(os.path.join(project_root, "stock")):
os.chdir(project_root)
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from shelf_map import Facing, ShelfMap, _iou # noqa: E402
def a_frame(shift_x=0, shift_y=0, size=(160, 240)):
"""A textured frame, optionally a shifted copy of another."""
height, width = size
rng = np.random.RandomState(7)
base = rng.randint(0, 255, (height * 2, width * 2, 3), dtype=np.uint8)
return base[height - shift_y:height * 2 - shift_y, width - shift_x:width * 2 - shift_x]
def a_track(track_id, box, sku=None, score=None, tag_sku=None):
track = {"track_id": track_id, "box": list(box)}
if sku:
track["identity"] = {"state": "recognized", "sku_id": sku, "score": score or 1.0}
else:
track["identity"] = {"state": "unconfirmed"}
if tag_sku:
track["tag_identity"] = {"sku_id": tag_sku, "score": 0.7}
return track
class IouTests(unittest.TestCase):
def test_overlap_maths(self):
self.assertAlmostEqual(_iou([0, 0, 10, 10], [0, 0, 10, 10]), 1.0)
self.assertEqual(_iou([0, 0, 10, 10], [20, 20, 30, 30]), 0.0)
class RegistrationTests(unittest.TestCase):
def test_the_frame_origin_moves_opposite_to_the_content(self):
"""The sign that matters: content moving left means the camera moved right.
Getting this backwards walks the map the wrong way. It is silent - boxes
still appear - and it turned a shelf of ~60 facings into 857.
"""
shelf = ShelfMap()
shelf.register(a_frame())
shift = shelf.register(a_frame(shift_x=10)) # content moved right 10
self.assertIsNotNone(shift)
self.assertAlmostEqual(shift[0], 10, delta=1.0)
# Content moving right means the camera moved LEFT, so the frame's origin
# sits 10 px further back along the shelf.
shelf.update(0, a_frame(), [a_track(1, [0, 0, 20, 20])])
offset_before = list(shelf._offset)
shelf.update(1, a_frame(shift_x=10), [a_track(2, [0, 0, 20, 20])])
self.assertLess(shelf._offset[0], offset_before[0] + 1,
"the offset must follow the camera, not the content")
def test_a_first_frame_has_no_shift(self):
shelf = ShelfMap()
self.assertEqual(shelf.register(a_frame()), (0.0, 0.0))
class MergingTests(unittest.TestCase):
def test_a_facing_seen_twice_is_one_facing(self):
shelf = ShelfMap()
shelf.update(0, a_frame(), [a_track(1, [10, 10, 110, 200])])
shelf.update(1, a_frame(), [a_track(2, [12, 11, 112, 201])])
self.assertEqual(len(shelf.facings()), 1)
self.assertEqual(shelf.stats()["merges"], 1)
def test_two_neighbouring_facings_stay_separate(self):
shelf = ShelfMap()
shelf.update(0, a_frame(), [a_track(1, [10, 10, 100, 200]), a_track(2, [110, 10, 200, 200])])
self.assertEqual(len(shelf.facings()), 2)
def test_identity_votes_pool_across_track_ids(self):
"""The point of the map: one facing, many transient track ids, one answer."""
shelf = ShelfMap()
for index, (track_id, sku) in enumerate([(1, "sku_a"), (2, "sku_a"), (3, "sku_a"), (4, "sku_b")]):
shelf.update(index, a_frame(), [a_track(track_id, [10, 10, 110, 200], sku=sku)])
facing = shelf.facings()[0]
self.assertEqual(facing.observations, 4)
self.assertEqual(len(facing.seen_track_ids), 4, "four tracks, one facing")
sku_id, agreement = facing.best_identity()
self.assertEqual(sku_id, "sku_a")
self.assertAlmostEqual(agreement, 0.75)
def test_tag_evidence_pools_separately_from_packaging(self):
shelf = ShelfMap()
shelf.update(0, a_frame(), [a_track(1, [10, 10, 110, 200], sku="sku_a", tag_sku="sku_a")])
shelf.update(1, a_frame(), [a_track(2, [10, 10, 110, 200], tag_sku="sku_a")])
facing = shelf.facings()[0]
self.assertEqual(facing.best_identity()[0], "sku_a")
self.assertEqual(facing.best_tag_identity()[0], "sku_a")
self.assertEqual(len(facing.tag_readings), 0, "no readable tag payload was attached")
class PayloadTests(unittest.TestCase):
def test_a_facing_reports_its_pooled_evidence(self):
facing = Facing(facing_id="face_1", shelf_box=[0, 0, 10, 10])
facing.vote_identity("sku_a", 2.0)
facing.vote_identity("sku_b", 1.0)
payload = facing.as_payload()
self.assertEqual(payload["sku_id"], "sku_a")
self.assertAlmostEqual(payload["identity_agreement"], 2 / 3, places=3)
self.assertIsNone(payload["tag_sku_id"])
if __name__ == "__main__":
unittest.main()

91
recognization/code/test_suite.py Normal file
View file

@ -0,0 +1,91 @@
"""
Unit and Integration Test Suite
Validates InventoryManager, ShelfProductDetector, ProductFeatureMatcher, and Pipeline.
"""
import os
import sys
import cv2
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from inventory import InventoryManager
from feature_matcher import ProductFeatureMatcher
from detector import ShelfProductDetector
from pipeline import ShelfRecognitionPipeline
def test_inventory():
print("[TEST] Testing InventoryManager...")
inv = InventoryManager("stock")
assert len(inv.products) >= 10, f"Expected products >= 10 in stock.csv, found {len(inv.products)}"
p1 = inv.get_product("sku_001")
assert p1 is not None, "sku_001 not found"
assert p1["product_name"] == "明治 おいしい牛乳"
assert p1["price"] == 258.0
assert p1["number_on"] == 73
print(" -> InventoryManager test: PASSED")
def test_feature_matcher():
print("[TEST] Testing ProductFeatureMatcher...")
matcher = ProductFeatureMatcher("stock")
assert len(matcher.db.templates) >= 7, f"Expected templates >= 7, got {len(matcher.db.templates)}"
for method in ["hybrid", "orb", "brisk"]:
tmpl_img = matcher.db.templates["sku_001"]["image"]
res = matcher.classify_crop(tmpl_img, method=method)
assert res["status"] == "success", f"Method {method} failed"
assert res["best_sku"] == "sku_001", f"Expected sku_001 with {method}, got {res['best_sku']}"
print(f" -> {method.upper()} classification test: PASSED (Score: {res['score']}, Conf: {res['confidence']})")
def test_detector():
print("[TEST] Testing ShelfProductDetector...")
detector = ShelfProductDetector("runs/detect/train/weights/best.pt")
sample_shelf = "runs/detect/train/train_batch0.jpg"
res = detector.detect(sample_shelf)
assert res["status"] == "success", "Detection failed"
assert res["total_detected"] > 0, "No boxes detected"
print(f" -> Detector test: PASSED ({res['total_detected']} items localized)")
first_box = res["boxes"][0]
cx, cy = first_box["center"]
hit = detector.find_box_at_point(cx, cy, res["boxes"])
assert hit is not None, "Hit test failed on center coordinate"
assert hit["id"] == first_box["id"], "Hit test returned incorrect box"
print(" -> Spatial hit-test: PASSED")
def test_pipeline():
print("[TEST] Testing End-to-End Pipeline...")
pipeline = ShelfRecognitionPipeline(
weights_path="runs/detect/train/weights/best.pt",
stock_dir="stock"
)
init_res = pipeline.set_active_shelf_image("runs/detect/train/train_batch0.jpg")
assert init_res["status"] == "success"
assert len(init_res["boxes"]) > 0
first_box = init_res["boxes"][0]
cx, cy = first_box["center"]
click_res = pipeline.process_click(cx, cy, method="hybrid")
assert click_res["status"] == "success", f"Click processing failed: {click_res.get('message')}"
assert "inventory" in click_res
assert click_res["inventory"] is not None
print(f" -> End-to-End Pipeline test: PASSED (Matched: {click_res['classification']['matched_sku']} - {click_res['inventory']['product_name']})")
if __name__ == "__main__":
print("=" * 60)
print("STARTING TEST SUITE")
print("=" * 60)
test_inventory()
test_feature_matcher()
test_detector()
test_pipeline()
print("=" * 60)
print("ALL TESTS PASSED SUCCESSFULLY!")
print("=" * 60)

239
recognization/code/test_tag_cards.py
View file

@ -1,239 +0,0 @@
"""
Tests for price tags as cards with fields.
Two layers: the pure logic (grouping, layout priors, the tax relation) with
synthetic lines, and one real-model test against a shelf photograph whose tags
were read by eye, so the parser is measured against the printed truth.
"""
from __future__ import annotations
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(current_dir)
if os.path.exists(os.path.join(project_root, "stock")):
os.chdir(project_root)
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from tag_cards import ( # noqa: E402
OcrLine,
best_tag_rows,
card_from_lines,
cluster_rows,
group_lines_into_cards,
parse_tag_fields,
)
def line(text, x1, y1, x2, y2, confidence=0.9):
return OcrLine(text=text, box=(x1, y1, x2, y2), confidence=confidence)
def a_tag(name="キューピー ゼロ ノンコレステロール＜310g＞", regular="357", price="269",
unit="290.52", discount="2.4割引", origin=(0, 0)):
"""A tag's printed lines, laid out the way a real one is."""
ox, oy = origin
return [
line("毎日お買得", ox + 20, oy + 5, ox + 150, oy + 22, 0.5),
line(name, ox + 10, oy + 26, ox + 240, oy + 44, 0.3),
line(regular, ox + 12, oy + 48, ox + 46, oy + 62, 0.3),
line(discount, ox + 60, oy + 50, ox + 140, oy + 64, 0.3),
line(price, ox + 30, oy + 60, ox + 130, oy + 96, 0.6), # the big print
line(unit, ox + 140, oy + 66, ox + 210, oy + 80, 0.5),
]
class GroupingTests(unittest.TestCase):
def test_a_row_of_tags_becomes_separate_cards(self):
lines = a_tag(origin=(0, 0)) + a_tag(origin=(300, 0), regular="400", price="379")
rows = best_tag_rows(lines)
self.assertEqual(len(rows), 1, "one rail line of tags")
cards = group_lines_into_cards(rows[0])
self.assertEqual(len(cards), 2, "the gap between tags separates them")
def test_packaging_text_above_the_rail_is_its_own_row(self):
# Product text sits above the rail; a tag row must be chosen on its own.
product = [line("キユーピー マヨネーズ 450g", 100, 0, 320, 18, 0.3)]
lines = product + a_tag(origin=(0, 60))
rows = cluster_rows(lines)
self.assertEqual(len(rows), 2)
chosen = best_tag_rows(lines, keep=1)
self.assertEqual(len(chosen), 1)
self.assertTrue(all(item.box[1] >= 60 for item in chosen[0]),
"the tag row wins, not the packaging row")
class ParsingTests(unittest.TestCase):
def test_the_tax_relation_recovers_a_price_the_big_digits_got_wrong(self):
# Measured failure: the stylised big digits read as 262/252/258 while the
# small clean number read 290.52. 290.52 / 1.08 = 269 exactly.
fields = parse_tag_fields(a_tag(price="262"))
self.assertEqual(fields.price, 269)
self.assertEqual(fields.price_source, "derived")
self.assertIn("tax_pair_consistent", fields.flags)
self.assertIn("digit_read_disagrees", fields.flags)
self.assertTrue(fields.corroborated())
def test_an_agreeing_pair_is_reported_as_measured(self):
fields = parse_tag_fields(a_tag(price="269"))
self.assertEqual(fields.price, 269)
self.assertEqual(fields.price_source, "measured")
self.assertIn("tax_consistent", fields.flags)
def test_the_small_number_is_the_regular_price_not_a_code(self):
"""Corrected: the number beside the sale price is the pre-discount price.
Checked against the manufacturer, this product's JAN is 4901577092381, whose
last three digits are 381 - not the 357 the tag prints. The discount confirms
it: (357-269)/357 = 24.6 %, printed as 2.4割引.
"""
fields = parse_tag_fields(a_tag(price="269", regular="357"))
self.assertEqual(fields.regular_price, 357)
self.assertEqual(fields.price, 269)
self.assertIn("discount_consistent", fields.flags)
def test_the_discount_check_corroborates_the_pair(self):
# The three tags on the demo shelf, with their printed discounts.
# Each fixture carries the tax-included price that belongs with it.
for regular, price, unit, discount in ((357, 269, "290.52", "2.4割引"),
(400, 379, "409.32", "0.5割引"),
(248, 199, "214.92", "1.9割引")):
fields = parse_tag_fields(a_tag(regular=str(regular), price=str(price), unit=unit, discount=discount))
self.assertIn("discount_consistent", fields.flags, f"{regular}/{price} {discount}")
self.assertTrue(fields.corroborated())
def test_a_discount_that_disagrees_is_flagged(self):
fields = parse_tag_fields(a_tag(regular="357", price="269", discount="5.0割引"))
self.assertIn("discount_mismatch", fields.flags)
def test_a_package_size_is_not_a_price(self):
fields = parse_tag_fields([
line("キューピー ゼロ ノンコレステロール", 10, 10, 240, 30, 0.3),
line("<310g>", 200, 12, 250, 30, 0.3),
line("357", 12, 48, 46, 62, 0.3),
line("269", 30, 60, 130, 96, 0.6),
])
self.assertEqual(fields.price, 269)
self.assertNotEqual(fields.price, 310)
def test_an_implausible_printed_discount_is_replaced_by_the_derived_one(self):
self.assertEqual(parse_tag_fields(a_tag()).discount, 2.4)
# 12.4割引 is not a real rate. Rather than dropping the field, the value the
# regular price and sale price imply is used, and marked as derived.
fields = parse_tag_fields(a_tag(discount="12.4割引"))
self.assertEqual(fields.discount, 2.4)
self.assertIn("discount_derived", fields.flags)
def test_the_name_is_the_longest_text_line(self):
fields = parse_tag_fields(a_tag())
self.assertIn("ノンコレステロール", fields.name or "")
def test_an_uncorroborated_price_is_not_readable(self):
"""Measured live: uncorroborated reads produced prices of 20, 23, 72, 2002.
Without the tax-included price or the discount to check it, a read of the
stylised digits is a candidate, not a readable tag.
"""
fields = parse_tag_fields([
line("キューピー ゼロ ノンコレステロール", 10, 10, 240, 30, 0.3),
line("269", 30, 60, 130, 96, 0.6),
])
self.assertEqual(fields.price, 269)
self.assertFalse(fields.corroborated())
self.assertFalse(fields.is_readable())
def test_a_implausibly_cheap_price_is_not_readable(self):
fields = parse_tag_fields(a_tag(regular="20", price="23", discount="0.1割引", unit="24.84"))
self.assertLess(fields.price, 50)
self.assertFalse(fields.is_readable())
self.assertIn("price_implausible", fields.flags)
def test_a_tag_without_a_price_is_not_readable(self):
fields = parse_tag_fields([line("毎日お買得", 0, 0, 100, 20, 0.5)])
self.assertFalse(fields.is_readable())
self.assertIn("no_price", fields.flags)
def test_the_card_box_is_the_union_of_its_lines(self):
# The box wraps the printed lines (with a little padding), so it lines up
# with the tag instead of with the rail band it was found in.
card = card_from_lines(a_tag(origin=(50, 100)))
self.assertLessEqual(card.box[0], 60) # the leftmost line starts at ox+10
self.assertLessEqual(card.box[1], 105)
self.assertGreaterEqual(card.box[2], 260)
self.assertGreaterEqual(card.box[3], 190)
class RealShelfTagTests(unittest.TestCase):
"""One measured row of a real LOPIA shelf (add/image6.jpg), read by eye."""
# (x centre, jan, price, tax-included price, discount, name fragment)
# (x centre, regular price, sale price, tax-included price, discount, name)
EXPECTED = [
(175, 357, 269, 290.52, 2.4, "ノンコレステロール"),
(555, 400, 379, 409.32, 0.5, None),
(975, 248, 199, 214.92, 1.9, "ラッキーマヨネーズ"),
]
def test_a_real_rail_reads_its_tags(self):
import cv2
from ocr import VisionOcr
from tag_cards import read_card_lines
image_path = "add/image6.jpg"
if not os.path.exists(image_path):
self.skipTest("demo image not present")
engine = VisionOcr()
if not engine.available():
self.skipTest("no OCR backend on this host")
image = cv2.imread(image_path)
band = image[630:800]
seed_lines = [
OcrLine(text=item.text, box=(item.box[0], item.box[1] + 630, item.box[2], item.box[3] + 630),
confidence=item.confidence)
for item in engine.read(band).lines if item.text.strip()
]
rows = best_tag_rows(seed_lines, keep=1)
self.assertTrue(rows, "the rail must yield a tag row")
cards = []
for group in group_lines_into_cards(rows[0]):
seed = card_from_lines(group)
pooled = read_card_lines(engine, image, seed.box)
card = card_from_lines(pooled) if pooled else seed
card.box = seed.box
cards.append(card)
cards = [card for card in cards if card.fields.price or card.fields.regular_price]
self.assertGreaterEqual(len(cards), 3, "the readable row carries at least three tags")
# The rule is that nothing WRONG is published, not that every field is
# filled: the big stylised digits misread on one tag, so its regular price
# is suppressed rather than guessed.
correct = 0
suppressed = 0
for centre_x, regular, price, unit, discount, name in self.EXPECTED:
card = min(cards, key=lambda c: abs(c.centre_x - centre_x))
fields = card.fields
if fields.regular_price is None:
suppressed += 1
elif fields.regular_price == regular:
correct += 1
if fields.price == price:
correct += 1
if fields.unit_price == unit:
correct += 1
if fields.discount == discount:
correct += 1
if name is None or (fields.name and name in fields.name):
correct += 1
self.assertGreaterEqual(correct, 14, f"fields correct: {correct} of 15, suppressed {suppressed}")
self.assertLessEqual(suppressed, 1, "at most one field may be withheld")
if __name__ == "__main__":
unittest.main()

121
recognization/code/test_tag_identity.py
View file

@ -1,121 +0,0 @@
"""
Tests for identity from the shelf's own price tags.
The measured case these encode: the store's printed label names the product in
clean text, one tag per product, and on the demo shelf the packaging matcher
agreed with it only 69 % of the time where both spoke - so the tag wins a
disagreement, and the disagreement is recorded rather than hidden.
"""
from __future__ import annotations
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(current_dir)
if os.path.exists(os.path.join(project_root, "stock")):
os.chdir(project_root)
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from reconcile import ( # noqa: E402
TAG_STATE_ABSENT,
TAG_STATE_READABLE,
best_catalog_match,
cluster_into_rows,
identities_from_tags,
normalize_name,
)
CATALOG = {
"sku_l05": {"sku_id": "sku_l05", "product_name": "キューピー ゼロ ノンコレステロール 310g", "price": 269},
"sku_l07": {"sku_id": "sku_l07", "product_name": "丸和油脂 ラッキーマヨネーズ 200g", "price": 199},
}
def facing(track_id, box, sku=None):
identity = {"state": "recognized", "sku_id": sku} if sku else {"state": "unconfirmed"}
return {"track_id": track_id, "box": box, "identity": identity}
def tag(box, name=None, state=TAG_STATE_READABLE):
return {"state": state, "box": box, "name": name, "regular_price": None, "price": None,
"barcode": None, "flags": []}
class NameMatchTests(unittest.TestCase):
def test_an_ocr_name_still_matches_the_catalog(self):
# Measured OCR output against the catalog entry it should find.
sku, score = best_catalog_match(
"キュービーゼロ ノンコレステロールく310g＞",
{k: normalize_name(v["product_name"]) for k, v in CATALOG.items()},
)
self.assertEqual(sku, "sku_l05")
self.assertGreater(score, 0.5)
def test_unrelated_text_matches_nothing(self):
sku, _ = best_catalog_match(
"ラス BEをさいしく使べるまにてく。",
{k: normalize_name(v["product_name"]) for k, v in CATALOG.items()},
)
self.assertIsNone(sku, "a garbled read must not be forced onto a product")
class AssignmentTests(unittest.TestCase):
def test_a_facing_is_named_by_the_tag_under_its_row(self):
# A row of four facings, one tag under it, one product.
tracks = [facing(index, [10 + index * 90, 100, 100 + index * 90, 300]) for index in range(4)]
tags = [tag([40, 320, 260, 380], name="キューピー ゼロ ノンコレステロール 310g")]
found = identities_from_tags(tracks, {}, CATALOG, all_tags=tags)
self.assertEqual(len(found), 4, "one tag covers the whole run of facings")
self.assertTrue(all(info["sku_id"] == "sku_l05" for info in found.values()))
def test_a_distant_facing_does_not_inherit_a_tag(self):
"""A product whose own tag was not detected must not take its neighbour's.
Without a reach limit the nearest readable tag names it, which is how a
facing several columns away can acquire the wrong product.
"""
tracks = [
facing(0, [10, 100, 100, 300]),
facing(1, [1600, 100, 1690, 300]), # far along the same row
]
tags = [tag([20, 320, 180, 380], name="キューピー ゼロ ノンコレステロール 310g")]
found = identities_from_tags(tracks, {}, CATALOG, all_tags=tags)
self.assertIn(0, found)
self.assertNotIn(1, found, "out of reach of that tag")
def test_the_nearest_tag_wins_along_a_row(self):
# Two products side by side, one tag each.
tracks = [
facing(0, [10, 100, 100, 300]),
facing(1, [110, 100, 200, 300]),
facing(2, [700, 100, 790, 300]),
facing(3, [800, 100, 890, 300]),
]
tags = [
tag([20, 320, 180, 380], name="キューピー ゼロ ノンコレステロール 310g"),
tag([710, 320, 870, 380], name="丸和油脂 ラッキーマヨネーズ 200g"),
]
found = identities_from_tags(tracks, {}, CATALOG, all_tags=tags)
self.assertEqual(found[0]["sku_id"], "sku_l05")
self.assertEqual(found[1]["sku_id"], "sku_l05")
self.assertEqual(found[2]["sku_id"], "sku_l07")
self.assertEqual(found[3]["sku_id"], "sku_l07")
def test_an_unreadable_tag_names_nothing(self):
tracks = [facing(0, [10, 100, 100, 300])]
tags = [tag([20, 320, 180, 380], name=None, state="unreadable")]
self.assertEqual(identities_from_tags(tracks, {}, CATALOG, all_tags=tags), {})
def test_rows_are_clustered_before_assignment(self):
# The same column on two shelves must not become one product.
tracks = [facing(0, [10, 100, 100, 300]), facing(1, [10, 600, 100, 800])]
rows = cluster_into_rows(tracks)
self.assertEqual(len(rows), 2)
if __name__ == "__main__":
unittest.main()

110
recognization/code/test_tracking.py
View file

@ -1,110 +0,0 @@
"""
Contract tests for session tracking.
The important one is ID isolation: ultralytics mints track ids from a
process-global counter and resets it when a tracker is constructed, so two live
clients would otherwise be able to hand each other's cached labels to the wrong
product.
"""
from __future__ import annotations
import os
import sys
import unittest
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from tracking import SessionTracker # noqa: E402
def pan(tracker, start_x, frames=8, step=10, width=60, height=80, y=100):
ids = []
for index in range(frames):
x = start_x + index * step
tracks = tracker.update([[x, y, x + width, y + height]], [0.9])
ids.append(tracks[0].track_id if tracks else None)
return ids
class IdIsolationTests(unittest.TestCase):
def test_sessions_do_not_share_ids(self):
first = SessionTracker()
second = SessionTracker()
first_ids = pan(first, 100)
second_ids = pan(second, 100)
self.assertEqual(len(set(first_ids)), 1, "a panning box keeps one id")
self.assertEqual(len(set(second_ids)), 1, "a panning box keeps one id")
self.assertEqual(first_ids[0], 1)
self.assertEqual(second_ids[0], 1, "each session starts its own numbering")
def test_constructing_a_tracker_does_not_reset_a_running_session(self):
"""ultralytics resets the global id counter in BYTETracker.__init__; a
second client connecting mid-scan must not renumber the first client's
tracks (its cached labels are keyed by id)."""
first = SessionTracker()
ids = pan(first, 100, frames=6, step=5)
self.assertEqual(len(set(ids)), 1)
second = SessionTracker() # a second client connects
continued = pan(first, 100 + 6 * 5, frames=6, step=5)
self.assertEqual(set(continued), {ids[0]}, "ids must keep counting, not restart")
self.assertEqual(pan(second, 400)[0], 1, "the new session still starts at 1")
def test_two_sessions_track_the_same_product_independently(self):
first = SessionTracker()
second = SessionTracker()
first_ids = pan(first, 100)
second_ids = pan(second, 100)
self.assertEqual(first_ids, second_ids, "same motion, same ids per session")
# ... but they are separate objects: dropping one does not touch the other
for _ in range(10):
first.update([], [])
self.assertEqual(len(set(pan(second, 100 + 8 * 10, frames=3, step=5))), 1)
def test_two_boxes_get_distinct_ids(self):
tracker = SessionTracker()
tracks = tracker.update([[10, 10, 70, 90], [200, 10, 260, 90]], [0.9, 0.8])
self.assertEqual(len(tracks), 2)
self.assertEqual(len({track.track_id for track in tracks}), 2)
class TrackLifecycleTests(unittest.TestCase):
def test_pan_continuity(self):
tracker = SessionTracker()
ids = pan(tracker, 50, frames=10, step=12)
self.assertEqual(len(set(ids)), 1)
def test_absent_track_is_dropped_and_gets_a_new_id(self):
tracker = SessionTracker(buffer=5)
first_ids = pan(tracker, 50, frames=2)
for _ in range(8):
tracker.update([], [])
for _ in range(3): # a re-detected object is unconfirmed for a frame or two
tracks = tracker.update([[50, 100, 110, 180]], [0.9])
self.assertTrue(tracks, "the object is tracked again")
self.assertNotEqual(tracks[0].track_id, first_ids[0], "a dropped track must not resume its old id")
def test_detection_index_maps_back_to_the_input_box(self):
tracker = SessionTracker()
tracks = tracker.update(
[[10, 10, 70, 90], [200, 10, 260, 90], [400, 10, 460, 90]], [0.9, 0.85, 0.95]
)
indices = sorted(track.det_index for track in tracks)
self.assertEqual(indices, [0, 1, 2])
def test_reset_starts_numbering_over_without_touching_others(self):
other = SessionTracker()
other_ids = pan(other, 100, frames=6, step=5)
tracker = SessionTracker()
pan(tracker, 100, frames=6, step=5)
tracker.reset()
after = pan(tracker, 300)
self.assertEqual(set(after), {1}, "reset restarts this session's numbering")
continued = pan(other, 100 + 6 * 5, frames=4, step=5)
self.assertEqual(set(continued), {other_ids[0]}, "the other session keeps counting")
if __name__ == "__main__":
unittest.main()

204
recognization/code/tracking.py
View file

@ -1,204 +0,0 @@
"""
Session-scoped multi-object tracking for live recognition.
Wraps ultralytics' BYTETracker (Kalman prediction + two-stage association) with
one critical fix: ultralytics mints track IDs from a PROCESS-GLOBAL counter
(BaseTrack._count), and BYTETracker.__init__ resets that counter. With several
live sessions that means one client's tracker creation re-issues IDs already in
use by another client, and a cached label can land on the wrong product.
SessionTracker gives every tracker instance its own counter and never touches
the global one. The rest of the algorithm is used unmodified.
Pinned to ultralytics 8.4.153 (see requirements.txt): this module touches
tracker internals.
"""
from __future__ import annotations
import itertools
import os
import sys
import threading
from typing import Any, List, Optional, Sequence
import numpy as np
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
sys.path.insert(0, current_dir)
from ultralytics.trackers.byte_tracker import BYTETracker, STrack
from ultralytics.utils import YAML, IterableSimpleNamespace
_BYTETRACK_YAML = os.path.join(
os.path.dirname(os.path.abspath(__import__("ultralytics").__file__)),
"cfg", "trackers", "bytetrack.yaml",
)
DEFAULT_TRACKER_CONFIG = {
"tracker_type": "bytetrack",
"track_high_thresh": 0.25,
"track_low_thresh": 0.1,
"new_track_thresh": 0.25,
"track_buffer": 30,
"match_thresh": 0.8,
"fuse_score": True,
}
def _default_args() -> IterableSimpleNamespace:
try:
return IterableSimpleNamespace(**YAML.load(_BYTETRACK_YAML))
except Exception:
return IterableSimpleNamespace(**DEFAULT_TRACKER_CONFIG)
class _Detections:
"""Duck-typed stand-in for ultralytics Results: only what BYTETracker reads.
Must NOT expose `xywhr`; the tracker uses its presence to pick rotated-box
handling.
"""
__slots__ = ("xywh", "conf", "cls")
def __init__(self, boxes_xyxy: np.ndarray, scores: np.ndarray):
boxes = np.asarray(boxes_xyxy, dtype=np.float32).reshape(-1, 4)
scores = np.asarray(scores, dtype=np.float32).reshape(-1)
widths = boxes[:, 2] - boxes[:, 0]
heights = boxes[:, 3] - boxes[:, 1]
self.xywh = np.stack(
[boxes[:, 0] + widths / 2.0, boxes[:, 1] + heights / 2.0, widths, heights], axis=1
).astype(np.float32)
self.conf = scores
self.cls = np.zeros(len(scores), dtype=np.float32)
def __len__(self) -> int:
return len(self.conf)
def __getitem__(self, mask: Any) -> "_Detections":
mask = np.asarray(mask)
subset = _Detections.__new__(_Detections)
subset.xywh = self.xywh[mask]
subset.conf = self.conf[mask]
subset.cls = self.cls[mask]
return subset
class _SessionByteTrack(BYTETracker):
"""BYTETracker with a per-instance ID counter."""
def __init__(self, args: IterableSimpleNamespace):
self._id_counter = itertools.count(1)
self._id_lock = threading.Lock()
# super().__init__ calls self.reset_id(); the override below only touches
# this instance's counter, so the global one is never disturbed.
super().__init__(args)
self.track_class = self._make_track_class()
def _next_id(self) -> int:
with self._id_lock:
return next(self._id_counter)
def _make_track_class(self):
tracker = self
class SessionSTrack(STrack):
@staticmethod
def next_id() -> int:
return tracker._next_id()
SessionSTrack.__name__ = "SessionSTrack"
return SessionSTrack
def reset_id(self) -> None: # type: ignore[override]
"""Reset only this tracker's counter; never BaseTrack._count."""
with self._id_lock:
self._id_counter = itertools.count(1)
self.track_class = self._make_track_class()
class TrackOut:
"""One tracked object for a frame."""
__slots__ = ("track_id", "box", "score", "det_index", "low_score")
def __init__(
self,
track_id: int,
box: List[int],
score: float,
det_index: Optional[int],
*,
low_score: bool = False,
):
self.track_id = int(track_id)
self.box = [int(v) for v in box]
self.score = float(score)
self.det_index = None if det_index is None or det_index < 0 else int(det_index)
# ByteTrack's second pass keeps an id alive on a low-score box. That box
# is not a new measurement: callers must not crop or re-label from it.
self.low_score = bool(low_score)
def __repr__(self) -> str: # pragma: no cover - debug helper
return (
f"TrackOut(id={self.track_id}, box={self.box}, score={self.score:.3f}, "
f"det={self.det_index}, low={self.low_score})"
)
class SessionTracker:
"""Per-session tracker. Used by one session at a time, under its lock."""
def __init__(
self,
*,
high: float = 0.25,
low: float = 0.1,
new: float = 0.25,
buffer: int = 30,
match: float = 0.8,
fuse: bool = True,
):
args = _default_args()
args.track_high_thresh = float(high)
args.track_low_thresh = float(low)
args.new_track_thresh = float(new)
args.track_buffer = int(buffer)
args.match_thresh = float(match)
args.fuse_score = bool(fuse)
self.args = args
self._tracker = _SessionByteTrack(args)
def update(
self,
boxes_xyxy: Sequence[Sequence[int]],
scores: Sequence[float],
) -> List[TrackOut]:
boxes = np.asarray(boxes_xyxy, dtype=np.float32).reshape(-1, 4)
scores_array = np.asarray(scores, dtype=np.float32).reshape(-1)
if len(boxes) != len(scores_array):
raise ValueError("boxes and scores must have the same length")
detections = _Detections(boxes, scores_array)
rows = self._tracker.update(detections)
outputs: List[TrackOut] = []
if rows is None or len(rows) == 0:
return outputs
high = float(self.args.track_high_thresh)
for row in np.asarray(rows):
# [x1, y1, x2, y2, track_id, score, cls, idx]
score = float(row[5])
outputs.append(
TrackOut(
track_id=int(row[4]),
box=[int(round(row[0])), int(round(row[1])), int(round(row[2])), int(round(row[3]))],
score=score,
det_index=int(row[7]) if len(row) > 7 else None,
low_score=score < high,
)
)
return outputs
def reset(self) -> None:
self._tracker.reset()

268
recognization/code/transport.py
View file

@ -1,268 +0,0 @@
"""
Request bounds and the LAN token check.
A client can declare a tiny Content-Length and then keep writing, so the
declared length is a hint and the read itself stops at the cap. The token
never travels in a query string: proxies and access logs keep those.
"""
from __future__ import annotations
import hmac
import ipaddress
import os
import shutil
import signal
import socket
import subprocess
from typing import Any, Dict, Optional
LOOPBACK_HOSTS = frozenset({"127.0.0.1", "::1", "localhost"})
def is_loopback_host(host: Optional[str]) -> bool:
text = (host or "").strip().lower()
if text.startswith("[") and text.endswith("]"):
text = text[1:-1]
if "%" in text: # fe80::1%lo0
text = text.split("%", 1)[0]
if text in LOOPBACK_HOSTS:
return True
try:
return ipaddress.ip_address(text).is_loopback
except ValueError:
return False
def require_token_for_bind(host: Optional[str], token: Optional[str]) -> None:
"""A non-loopback bind with no token is reachable by anyone on that network."""
if token or is_loopback_host(host):
return
raise SystemExit(
f"refusing to bind {host} without DORAYAKI_API_TOKEN "
"(loopback is the only bind that may run with auth off)"
)
def token_matches(presented: Optional[str], token: Optional[str]) -> bool:
if not presented or not token:
return False
# compare_digest raises when the lengths differ. Fold the length in so a
# short guess neither throws nor returns early.
return hmac.compare_digest(
len(presented).to_bytes(8, "big") + presented.encode(),
len(token).to_bytes(8, "big") + token.encode(),
)
def bearer_matches(authorization: Optional[str], token: str) -> bool:
if not authorization or not token:
return False
scheme, _, presented = authorization.partition(" ")
if scheme.lower() != "bearer" or not presented or " " in presented:
return False
return token_matches(presented, token)
def declared_length(header: Optional[str]) -> Optional[int]:
if header is None or header == "":
return None
try:
length = int(header)
except (TypeError, ValueError):
return None
return length if length >= 0 else None
async def read_bounded(source, max_bytes: int) -> bytes:
"""Stop once the cap is passed. The caller rejects the oversize body.
A request stream is async-iterable. An UploadFile is not: it has read(n).
"""
out = bytearray()
read = getattr(source, "read", None)
if read is not None and not hasattr(source, "__aiter__"):
while True:
chunk = await read(65536)
if not chunk:
break
out += chunk
if len(out) > max_bytes:
break
return bytes(out)
async for chunk in source:
if not chunk:
continue
out += chunk
if len(out) > max_bytes:
break
return bytes(out)
class BodyLimit:
"""Stop the ASGI receive once the body passes the cap.
A handler that checks Content-Length, or reads request.body(), runs after
the server has already buffered the stream. This counts bytes at receive
and returns 413 without handing the rest to the app. Content-Length is
not trusted: the stream is what counts, and reading stops at the cap.
If the app already started the response, the connection is aborted
instead of sending a second http.response.start. WebSocket frames are
not HTTP bodies and are left alone.
"""
def __init__(self, app: Any, max_bytes: int) -> None:
self.app = app
self.max_bytes = int(max_bytes)
async def __call__(self, scope, receive, send) -> None:
if scope.get("type") != "http":
await self.app(scope, receive, send)
return
class TooLarge(Exception):
pass
seen = 0
started = False
async def limited() -> Dict[str, Any]:
nonlocal seen
message = await receive()
if message.get("type") != "http.request":
return message
body = message.get("body") or b""
seen += len(body)
if seen > self.max_bytes:
# The app must not see this chunk or any later one.
raise TooLarge
return message
async def tracked(message: Dict[str, Any]) -> None:
nonlocal started
if message.get("type") == "http.response.start":
started = True
await send(message)
try:
await self.app(scope, limited, tracked)
except TooLarge:
if started:
# The status line is already on the wire. Returning leaves the
# response unfinished so the server closes the connection
# instead of accepting a second http.response.start.
return
await send(
{
"type": "http.response.start",
"status": 413,
"headers": [(b"content-type", b"application/json")],
}
)
await send(
{
"type": "http.response.body",
"body": b'{"error":{"code":"PAYLOAD_TOO_LARGE","message":"request body too large"}}',
}
)
SERVICE_TYPE = "_dorayaki._tcp"
def _lan_address() -> Optional[str]:
"""Address a phone on this network can open. Loopback is not one.
The demo launcher uses `ipconfig getifaddr en0` then `en1`. A UDP connect
is the fallback when those interfaces are absent.
"""
if shutil.which("ipconfig"):
for iface in ("en0", "en1"):
try:
out = subprocess.check_output(
["ipconfig", "getifaddr", iface], stderr=subprocess.DEVNULL, text=True
).strip()
except (subprocess.CalledProcessError, OSError):
continue
if out and not is_loopback_host(out):
return out
try:
probe = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
probe.connect(("192.0.2.1", 9))
found = probe.getsockname()[0]
finally:
probe.close()
except OSError:
return None
if not found or is_loopback_host(found):
return None
return found
def advertise_host(bind_host: Optional[str]) -> Optional[str]:
"""The address clients should browse. A loopback bind is not advertised.
A wildcard bind (0.0.0.0, ::) has no address of its own, so the record
uses the LAN address the demo launcher would hand a phone.
"""
text = (bind_host or "").strip()
if text.startswith("[") and text.endswith("]"):
text = text[1:-1]
if is_loopback_host(text):
return None
if text in ("", "0.0.0.0", "::"):
return _lan_address()
try:
parsed = ipaddress.ip_address(text)
except ValueError:
return None
if parsed.is_loopback or parsed.is_unspecified:
return None if parsed.is_loopback else _lan_address()
return str(parsed)
class ServiceAdvertiser:
"""Register `_dorayaki._tcp` with the system mDNS daemon and drop it on close.
ponytail: macOS `dns-sd` only. A Linux host needs `avahi-publish` wired the
same way; do not add a Python mDNS package for this one record.
"""
def __init__(self, host: str, port: int, name: str = "Dorayaki") -> None:
self.host = host
self.port = int(port)
self.name = name
self._process: Optional[subprocess.Popen] = None
def start(self) -> bool:
if self._process is not None:
return True
binary = shutil.which("dns-sd")
if not binary:
return False
self._process = subprocess.Popen(
[binary, "-P", self.name, SERVICE_TYPE, "local", str(self.port), self.host, self.host],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
return True
def close(self) -> None:
process = self._process
self._process = None
if process is None or process.poll() is not None:
return
try:
os.killpg(process.pid, signal.SIGTERM)
except ProcessLookupError:
return
try:
process.wait(timeout=2)
except subprocess.TimeoutExpired:
try:
os.killpg(process.pid, signal.SIGKILL)
except ProcessLookupError:
return
process.wait(timeout=2)

1175
recognization/code/web/app.js
View file

File diff suppressed because it is too large Load diff

179
recognization/code/web/index.html
View file

@ -1,179 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Dorayaki - Smart Supermarket Shelf AI</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><ellipse cx='50' cy='55' rx='44' ry='24' fill='%23f59e0b'/><ellipse cx='50' cy='45' rx='42' ry='20' fill='%23b45309'/></svg>">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link href="https://fonts.googleapis.com/css2?family=Plus+Jakarta+Sans:wght@400;500;600;700;800&family=Noto+Sans+JP:wght@400;500;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="styles.css">
</head>
<body>
<header>
<div class="brand">
<svg class="brand-logo-svg" viewBox="0 0 100 100" xmlns="http://www.w3.org/2000/svg">
<defs>
<linearGradient id="pancakeTop" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" stop-color="#f59e0b"/>
<stop offset="100%" stop-color="#b45309"/>
</linearGradient>
<linearGradient id="ankoFill" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" stop-color="#450a0a"/>
<stop offset="100%" stop-color="#2d0606"/>
</linearGradient>
<linearGradient id="pancakeBottom" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" stop-color="#d97706"/>
<stop offset="100%" stop-color="#92400e"/>
</linearGradient>
</defs>
<ellipse cx="50" cy="62" rx="42" ry="18" fill="url(#pancakeBottom)" />
<ellipse cx="50" cy="53" rx="43" ry="14" fill="url(#ankoFill)" />
<ellipse cx="50" cy="42" rx="44" ry="22" fill="url(#pancakeTop)" />
<ellipse cx="44" cy="35" rx="20" ry="8" fill="#fde68a" opacity="0.4" />
<path d="M 52,18 C 65,12 72,25 65,30 C 58,28 54,23 52,18 Z" fill="#22c55e" />
</svg>
<div>
<div class="brand-title">Dorayaki <span class="brand-jp">どら焼き</span></div>
<div class="brand-sub" id="t-subtitle">Smart Supermarket Shelf AI &amp; Inventory Recognition</div>
</div>
</div>
<div class="header-actions">
<span class="conn-pill" id="connPill" data-state="idle"><span class="dot"></span><span id="connText">Offline</span></span>
<button class="lang-switch-btn" id="langBtn" type="button">
<span>🌐</span> <span id="langText">日本語 / JA</span>
</button>
</div>
</header>
<main>
<section class="stage-panel">
<div class="toolbar">
<div class="toolbar-group">
<div class="segmented" id="modeControl">
<button class="seg-btn active" data-mode="live" type="button" id="t-mode-live">Live</button>
<button class="seg-btn" data-mode="photo" type="button" id="t-mode-photo">Photo</button>
</div>
<div class="segmented" id="methodControl">
<button class="seg-btn active" data-method="hybrid" type="button">Hybrid</button>
<button class="seg-btn" data-method="surf" type="button">SURF</button>
<button class="seg-btn" data-method="orb" type="button">ORB</button>
<button class="seg-btn" data-method="brisk" type="button">BRISK</button>
</div>
</div>
<div class="toolbar-group" id="liveControls">
<select id="sourceSelect" class="select" title="Camera or video file"></select>
<button class="tool-btn primary" id="startBtn" type="button" id-label="start">▶ <span id="t-start">Start</span></button>
<button class="tool-btn" id="stopBtn" type="button" disabled>■ <span id="t-stop">Stop</span></button>
<div class="hud" id="hud" title="Click for detail">--</div>
</div>
<div class="toolbar-group" id="photoControls" hidden>
<select id="sampleSelect" class="select"></select>
<button class="tool-btn" id="captureBtn" type="button">📷 <span id="t-capture">Take photo</span></button>
<button class="tool-btn" id="uploadBtn" type="button">📁 <span id="t-upload">Upload image</span></button>
<input type="file" id="fileInput" accept="image/*" hidden>
</div>
<div class="toolbar-group">
<button class="tool-btn danger" id="emptyBtn" type="button">🔴 <span id="t-empty">Empty spaces</span></button>
</div>
</div>
<div class="stage" id="stage">
<video id="video" playsinline muted></video>
<img id="photo" alt="" hidden>
<canvas id="overlay"></canvas>
<div class="stage-empty" id="stageEmpty">
<div class="stage-empty-icon">📷</div>
<div id="t-stage-empty">Start the live scan, or open Photo mode to inspect a shelf image.</div>
</div>
<div class="hint-banner" id="hintBanner" hidden></div>
<div class="compliance-strip" id="complianceStrip" hidden></div>
</div>
</section>
<aside class="side-panel">
<div class="card">
<div class="card-header">
<span id="t-card-title">Product details</span>
<span class="status-badge badge-neutral" id="statusBadge"></span>
</div>
<div class="placeholder" id="cardPlaceholder">
<div class="placeholder-icon">🏷️</div>
<span id="t-card-placeholder">Tap a product on the shelf to see its details.</span>
</div>
<div class="card-body" id="cardBody" hidden>
<div class="product-hero">
<img id="productThumb" class="product-thumb" alt="" hidden>
<div class="product-details">
<span class="product-sku" id="productSku" hidden>—</span>
<div class="product-name" id="productName"></div>
<div class="product-price" id="productPrice"></div>
</div>
</div>
<div class="candidates" id="candidates" hidden></div>
<div class="stock-grid">
<div class="stock-metric">
<div class="metric-title" id="t-metric-on">On shelf</div>
<div class="metric-val" id="metricOn"></div>
</div>
<div class="stock-metric">
<div class="metric-title" id="t-metric-out">Sold / dispatched</div>
<div class="metric-val" id="metricOut"></div>
</div>
<div class="stock-metric">
<div class="metric-title" id="t-metric-receipt">Received</div>
<div class="metric-val small" id="metricReceipt"></div>
</div>
<div class="stock-metric">
<div class="metric-title" id="t-metric-expiry">Expires</div>
<div class="metric-val small warn" id="metricExpiry"></div>
</div>
</div>
<div class="ocr-section" id="ocrSection" hidden>
<div class="ocr-header">
<span id="t-ocr-title">Price tag</span>
<span class="recon-chip" id="reconChip"></span>
</div>
<div class="ocr-rows" id="ocrRows"></div>
</div>
<button class="action-btn" id="dispenseBtn" type="button">🛒 <span id="t-dispense">Dispense 1 unit</span></button>
</div>
</div>
<div class="card">
<div class="card-header">
<span id="t-match-title">Feature matching</span>
<span class="score-label" id="scoreLabel"></span>
</div>
<div class="match-box" id="matchBox">
<span class="match-placeholder" id="t-match-placeholder">Tap a product, then “Show matching”.</span>
</div>
<button class="tool-btn wide" id="showMatchBtn" type="button" disabled>🔍 <span id="t-show-match">Show matching</span></button>
</div>
<div class="card">
<div class="card-header">
<span id="t-inview-title">On this photo</span>
<span class="score-label" id="inViewCount">0</span>
</div>
<div class="inview-list" id="inViewList">
<div class="placeholder small" id="t-inview-placeholder">Recognized products appear here.</div>
</div>
</div>
</aside>
</main>
<div class="modal-backdrop" id="cameraModal">
<div class="modal-window">
<div class="modal-header">
<div class="modal-title">📷 <span id="t-camera-title">Take shelf photo</span></div>
<button class="close-btn" id="cameraClose" type="button">&times;</button>
</div>
<div class="video-box"><video id="cameraVideo" autoplay playsinline></video></div>
<button class="action-btn" id="snapshotBtn" type="button">📸 <span id="t-snapshot">Capture</span></button>
</div>
</div>
<script type="module" src="app.js"></script>
</body>
</html>

253
recognization/code/web/styles.css
View file

@ -1,253 +0,0 @@
:root {
--bg-body: #f4fbf6;
--bg-card: #ffffff;
--primary-green: #22c55e;
--primary-dark: #15803d;
--primary-light: #dcfce7;
--primary-subtle: #f0fdf4;
--border-color: #d1fae5;
--border-strong: #86efac;
--text-main: #0f172a;
--text-muted: #64748b;
--amber: #f59e0b;
--amber-soft: #fef3c7;
--danger: #ef4444;
--danger-soft: #fee2e2;
--slate-soft: #e2e8f0;
--night: #0f172a;
--shadow-soft: 0 4px 20px -2px rgba(34, 197, 94, 0.08), 0 2px 6px -1px rgba(0, 0, 0, 0.04);
--radius-lg: 18px;
--radius-md: 12px;
--radius-sm: 8px;
}
* { box-sizing: border-box; margin: 0; padding: 0; }
[hidden] { display: none !important; }
body {
background-color: var(--bg-body);
color: var(--text-main);
min-height: 100vh;
display: flex;
flex-direction: column;
font-family: 'Plus Jakarta Sans', 'Noto Sans JP', sans-serif;
}
header {
background: var(--bg-card);
border-bottom: 1px solid var(--border-color);
padding: 14px 32px;
display: flex;
justify-content: space-between;
align-items: center;
position: sticky;
top: 0;
z-index: 100;
}
.brand { display: flex; align-items: center; gap: 14px; }
.brand-logo-svg { width: 44px; height: 44px; filter: drop-shadow(0 2px 6px rgba(34, 197, 94, 0.25)); }
.brand-title { font-size: 20px; font-weight: 800; color: #14532d; letter-spacing: -0.5px; display: flex; align-items: center; gap: 6px; }
.brand-jp { font-size: 13px; font-weight: 600; color: #16a34a; background: var(--primary-light); padding: 2px 8px; border-radius: 6px; }
.brand-sub { font-size: 12px; color: var(--text-muted); font-weight: 500; }
.header-actions { display: flex; align-items: center; gap: 12px; }
.lang-switch-btn {
background: var(--primary-subtle);
border: 1px solid var(--border-color);
color: var(--primary-dark);
padding: 6px 14px;
border-radius: 999px;
font-size: 12px;
font-weight: 700;
cursor: pointer;
display: flex;
align-items: center;
gap: 6px;
font-family: inherit;
}
.lang-switch-btn:hover { background: var(--primary-light); border-color: var(--primary-green); }
.conn-pill {
display: flex; align-items: center; gap: 6px;
font-size: 11px; font-weight: 700; color: #166534;
background: var(--primary-subtle); border: 1px solid var(--border-color);
padding: 5px 12px; border-radius: 999px;
}
.conn-pill .dot { width: 7px; height: 7px; border-radius: 50%; background: var(--text-muted); }
.conn-pill[data-state="live"] .dot { background: var(--primary-green); box-shadow: 0 0 6px var(--primary-green); }
.conn-pill[data-state="photo"] .dot { background: #38bdf8; box-shadow: 0 0 6px #38bdf8; }
.conn-pill[data-state="error"] .dot { background: var(--danger); box-shadow: 0 0 6px var(--danger); }
.conn-pill[data-state="connecting"] .dot { background: var(--amber); box-shadow: 0 0 6px var(--amber); }
main {
flex: 1;
padding: 20px 32px;
display: grid;
grid-template-columns: 1fr 420px;
gap: 24px;
align-items: start;
}
.stage-panel {
background: var(--bg-card);
border: 1px solid var(--border-color);
border-radius: var(--radius-lg);
padding: 18px;
display: flex;
flex-direction: column;
gap: 14px;
box-shadow: var(--shadow-soft);
}
.toolbar { display: flex; justify-content: space-between; align-items: center; flex-wrap: wrap; gap: 10px; padding-bottom: 12px; border-bottom: 1px solid var(--border-color); }
.toolbar-group { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; }
.segmented { display: flex; background: var(--primary-subtle); border: 1px solid var(--border-color); border-radius: 10px; padding: 3px; gap: 2px; }
.seg-btn { background: transparent; border: none; color: var(--text-muted); padding: 6px 12px; border-radius: 8px; font-size: 12px; font-weight: 700; cursor: pointer; font-family: inherit; }
.seg-btn.active { background: var(--primary-green); color: white; box-shadow: 0 2px 8px rgba(34, 197, 94, 0.3); }
.tool-btn {
background: var(--bg-card); border: 1px solid var(--border-color); color: var(--primary-dark);
padding: 7px 14px; border-radius: 10px; font-size: 12px; font-weight: 700; cursor: pointer;
display: inline-flex; align-items: center; gap: 6px; font-family: inherit;
}
.tool-btn:hover:not(:disabled) { background: var(--primary-subtle); border-color: var(--primary-green); }
.tool-btn:disabled { opacity: 0.45; cursor: default; }
.tool-btn.primary { background: var(--primary-green); border-color: var(--primary-green); color: white; }
.tool-btn.danger { background: var(--danger-soft); border-color: #fca5a5; color: #b91c1c; }
.tool-btn.wide { width: 100%; justify-content: center; }
.select { background: var(--bg-card); color: var(--text-main); border: 1px solid var(--border-color); padding: 7px 12px; border-radius: 10px; font-size: 12px; font-weight: 600; font-family: inherit; max-width: 220px; }
.hud { font-size: 11px; font-weight: 700; color: #166534; background: var(--primary-subtle); border: 1px solid var(--border-color); padding: 6px 12px; border-radius: 999px; cursor: pointer; font-variant-numeric: tabular-nums; }
.hud[data-detail="1"] { border-radius: 10px; }
.stage {
position: relative;
background: var(--night);
border-radius: 14px;
overflow: hidden;
height: 70vh;
min-height: 420px;
display: flex;
align-items: center;
justify-content: center;
}
.stage video, .stage img { max-width: 100%; max-height: 100%; display: block; }
/* The canvas is the same contained box as the media it annotates: its backing
store is the media's own pixel size, so server coordinates map 1:1 and the
element rect (used for click mapping) is exactly the visible picture. */
.stage canvas {
position: absolute; left: 50%; top: 50%; transform: translate(-50%, -50%);
max-width: 100%; max-height: 100%;
}
.stage-empty { position: absolute; inset: 0; display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 10px; color: #cbd5e1; font-size: 13px; text-align: center; padding: 24px; }
.stage-empty-icon { font-size: 34px; }
.hint-banner {
position: absolute; bottom: 16px; left: 50%; transform: translateX(-50%);
background: rgba(255, 255, 255, 0.95); backdrop-filter: blur(8px);
border: 1px solid var(--primary-green); color: #14532d;
padding: 8px 20px; border-radius: 999px; font-size: 12px; font-weight: 700;
box-shadow: 0 4px 16px rgba(34, 197, 94, 0.2); pointer-events: none;
}
.hint-banner[data-kind="warning"] { border-color: var(--amber); color: #92400e; }
.hint-banner[data-kind="danger"] { border-color: var(--danger); color: #991b1b; }
.side-panel { display: flex; flex-direction: column; gap: 16px; }
.card { background: var(--bg-card); border: 1px solid var(--border-color); border-radius: var(--radius-lg); padding: 18px; display: flex; flex-direction: column; gap: 14px; box-shadow: var(--shadow-soft); }
.card-header { font-size: 12px; font-weight: 800; color: #166534; text-transform: uppercase; letter-spacing: 0.6px; display: flex; justify-content: space-between; align-items: center; padding-bottom: 8px; border-bottom: 1px solid var(--border-color); }
.score-label { font-size: 12px; color: #16a34a; font-weight: 700; text-transform: none; letter-spacing: 0; }
.placeholder { text-align: center; padding: 32px 10px; color: var(--text-muted); font-size: 13px; display: flex; flex-direction: column; gap: 8px; align-items: center; }
.placeholder.small { padding: 16px 8px; font-size: 12px; }
.placeholder-icon { font-size: 32px; }
.card-body { display: flex; flex-direction: column; gap: 14px; }
.product-hero { display: flex; gap: 14px; align-items: center; }
.product-thumb { width: 92px; height: 116px; border-radius: var(--radius-md); object-fit: cover; border: 2px solid var(--primary-light); background: #f8fafc; }
.product-details { flex: 1; display: flex; flex-direction: column; gap: 6px; min-width: 0; }
.product-name { font-size: 16px; font-weight: 800; line-height: 1.3; }
.product-sku { font-size: 11px; color: #16a34a; font-family: ui-monospace, monospace; background: var(--primary-light); padding: 2px 6px; border-radius: 4px; width: fit-content; font-weight: 700; }
.product-price { font-size: 24px; font-weight: 800; color: var(--primary-dark); }
.candidates { display: flex; flex-direction: column; gap: 6px; background: var(--amber-soft); border: 1px solid #fcd34d; border-radius: var(--radius-md); padding: 10px 12px; }
.candidates-title { font-size: 11px; font-weight: 800; color: #92400e; text-transform: uppercase; letter-spacing: 0.5px; }
.candidate-row { display: flex; justify-content: space-between; font-size: 12px; color: #78350f; font-weight: 600; }
.stock-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 10px; }
.stock-metric { background: var(--primary-subtle); border: 1px solid var(--border-color); border-radius: var(--radius-md); padding: 10px 14px; }
.metric-title { font-size: 11px; color: var(--text-muted); margin-bottom: 4px; font-weight: 600; }
.metric-val { font-size: 16px; font-weight: 800; color: var(--primary-dark); }
.metric-val.small { font-size: 13px; }
.metric-val.warn { color: #b45309; }
.action-btn {
width: 100%; padding: 12px; border: none; border-radius: var(--radius-md);
background: linear-gradient(135deg, #22c55e, #16a34a); color: white;
font-weight: 800; font-size: 14px; cursor: pointer; font-family: inherit;
display: inline-flex; align-items: center; justify-content: center; gap: 8px;
box-shadow: 0 4px 14px rgba(34, 197, 94, 0.3);
}
.action-btn:hover { transform: translateY(-1px); }
.match-box { width: 100%; min-height: 150px; background: var(--night); border-radius: var(--radius-md); border: 1px solid var(--border-color); display: flex; align-items: center; justify-content: center; overflow: hidden; }
.match-box img { width: 100%; height: 100%; object-fit: contain; }
.match-placeholder { color: #94a3b8; font-size: 12px; padding: 20px; text-align: center; }
.inview-list { display: flex; flex-direction: column; gap: 8px; max-height: 320px; overflow-y: auto; }
.inview-row { display: flex; align-items: center; gap: 10px; padding: 8px; border-radius: var(--radius-md); cursor: pointer; }
.inview-row:hover { background: var(--primary-subtle); }
.inview-thumb { width: 34px; height: 44px; border-radius: 6px; object-fit: cover; border: 1px solid var(--border-color); background: #f8fafc; }
.inview-text { flex: 1; min-width: 0; }
.inview-name { font-size: 12px; font-weight: 700; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.inview-meta { font-size: 11px; color: var(--text-muted); }
.inview-facings { font-size: 12px; font-weight: 800; color: var(--primary-dark); }
.status-badge { font-size: 11px; padding: 3px 10px; border-radius: 999px; font-weight: 700; text-transform: none; letter-spacing: 0; }
.badge-neutral { background: var(--slate-soft); color: var(--text-muted); }
.badge-success { background: var(--primary-light); color: var(--primary-dark); }
.badge-warning { background: var(--amber-soft); color: #b45309; }
.badge-danger { background: var(--danger-soft); color: #b91c1c; }
.modal-backdrop { display: none; position: fixed; inset: 0; background: rgba(15, 23, 42, 0.6); backdrop-filter: blur(4px); z-index: 1000; align-items: center; justify-content: center; }
.modal-backdrop.open { display: flex; }
.modal-window { background: #fff; border-radius: 20px; width: 90%; max-width: 600px; padding: 22px; display: flex; flex-direction: column; gap: 14px; }
.modal-header { display: flex; justify-content: space-between; align-items: center; }
.modal-title { font-size: 17px; font-weight: 800; color: #14532d; }
.close-btn { background: none; border: none; font-size: 24px; color: var(--text-muted); cursor: pointer; }
.video-box { width: 100%; height: 340px; background: #000; border-radius: var(--radius-md); overflow: hidden; }
.video-box video { width: 100%; height: 100%; object-fit: cover; }
@media (max-width: 1024px) {
main { grid-template-columns: 1fr; padding: 16px; }
.stage { height: 56vh; min-height: 320px; }
}
.ocr-section { border-top: 1px solid var(--border-color); padding-top: 12px; display: flex; flex-direction: column; gap: 8px; }
.ocr-header { display: flex; justify-content: space-between; align-items: center; font-size: 11px; font-weight: 800; color: #0369a1; text-transform: uppercase; letter-spacing: 0.5px; }
.recon-chip { font-size: 10px; font-weight: 800; padding: 2px 8px; border-radius: 999px; text-transform: none; letter-spacing: 0; }
.recon-chip[data-state="confirmed"] { background: var(--primary-light); color: var(--primary-dark); }
.recon-chip[data-state="mismatch"] { background: var(--danger-soft); color: #b91c1c; }
.recon-chip[data-state="unverified"], .recon-chip[data-state="no_tag"] { background: var(--slate-soft); color: var(--text-muted); }
.recon-chip[data-state="tag_only"] { background: #e0f2fe; color: #0369a1; }
.ocr-rows { display: flex; flex-direction: column; gap: 4px; }
.ocr-row { display: flex; justify-content: space-between; gap: 10px; font-size: 12px; }
.ocr-row .ocr-key { color: var(--text-muted); font-weight: 600; }
.ocr-row .ocr-val { color: var(--text-main); font-weight: 700; text-align: right; }
.ocr-row .ocr-val.bad { color: #b91c1c; }
.compliance-strip {
position: absolute; top: 14px; left: 14px; right: 14px;
background: rgba(255, 255, 255, 0.94); backdrop-filter: blur(8px);
border: 1px solid #fca5a5; border-radius: 12px; padding: 8px 14px;
font-size: 12px; font-weight: 700; color: #b91c1c; display: flex; gap: 12px; flex-wrap: wrap;
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.08); pointer-events: none;
}
.compliance-strip[data-kind="hint"] { border-color: var(--amber); color: #92400e; }
.compliance-strip .compliance-item { display: inline-flex; align-items: center; gap: 6px; }

34
recognization/data/README.md
View file

@ -1,34 +0,0 @@
# Recognition data
Everything the identity work needs to be reproducible: sourced packshots,
evaluation runs, and the audits that produced them. Nothing here is ephemeral;
a run that matters is written here, not to a temp directory.
## `packshots/source/`
Front packshots as downloaded, before enrollment. `ATTRIBUTION.json` records the
source URL, the product as printed, and the manufacturer for each. `ocr-reads.json`
holds the OCR text each image produced, so a name/OCR refusal can be re-checked
without re-running Vision.
`rebuild_catalog.py` reads this directory and writes a catalog:
.venv/bin/python code/rebuild_catalog.py --source data/packshots/source \
--output stock-lopia-clean --csv stock-lopia/stock.csv --min-views 1 --replace
## `evaluations/`
One directory per session. Each holds the raw artifacts a claim was computed
from, so a number in DECISION.md can be traced back to the run that produced it.
| Directory | Contents |
|---|---|
| `stream/` | WebSocket track dumps from `stream_client.py`, scored by `score_stream.py`. |
| `still/` | `/v1/recognize` responses for the still-image path. |
| `catalog-audit/` | `rebuild_catalog.py --json` reports. |
| `scratch/` | Intermediate images (rotations, crops) kept because they document a finding. |
Scoring a stream dump:
.venv/bin/python code/score_stream.py data/evaluations/<session>/stream/<file>.json \
--labels eval/labels/image6.json --stock-dir stock-lopia-clean

76
recognization/data/evaluations/2026-09-29-session/catalog-audit/enroll-2026-09-29.json
View file

@ -1,76 +0,0 @@
{
"source": "data/packshots/source",
"output": "stock-lopia-clean",
"min_width": 120,
"min_height": 160,
"min_views": 1,
"duplicate_cosine": 0.98,
"min_name_ocr": 0.7,
"ocr": "vision",
"text_only_skus": [
"sku_l04",
"sku_l07",
"sku_l08",
"sku_l12"
],
"text_discriminated_siblings": [
{
"sku": "sku_l09",
"view": "sku_l09.jpg",
"sibling_sku": "sku_l10",
"sibling_view": "sku_l10.jpg",
"cosine": 0.9817,
"name_similarity": 0.333
}
],
"below_pyramid_floor": [],
"completeness": {
"live_ready": null,
"reason": "no --shelf-tags given; the completeness rule needs the shelf's SKU list",
"enrolled": 8,
"text_only": [
"sku_l04",
"sku_l07",
"sku_l08",
"sku_l12"
]
},
"skus": {
"sku_l01": [
"sku_l01.jpg"
],
"sku_l02": [
"sku_l02.jpg"
],
"sku_l03": [
"sku_l03.jpg"
],
"sku_l05": [
"sku_l05.jpg"
],
"sku_l06": [
"sku_l06.jpg"
],
"sku_l09": [
"sku_l09.jpg"
],
"sku_l10": [
"sku_l10.jpg"
],
"sku_l11": [
"sku_l11.jpg"
]
},
"refused": [
{
"sku": "sku_l04",
"view": "sku_l04.jpg",
"reason": "name/OCR 0.00 below 0.70: 明太手屋さんが つくつ た コ"
},
{
"sku": "sku_l07",
"view": "sku_l07.jpg",
"reason": "name/OCR 0.57 below 0.70: ラッキー マヨネーズ 調味料（アミノ酸）不使用"
}
]
}

221
recognization/data/evaluations/2026-09-29-session/catalog-audit/eval-after-l10.json
View file

@ -1,221 +0,0 @@
{
"image": "add/image6.jpg",
"stock_dir": "stock-lopia-clean",
"labels": "eval/labels/image6.json",
"provenance": {
"catalog_hash": "3f1653d9d962073af96121250006b9b6d86113556797bbee65de68b60e87a64d",
"labels_hash": "5aabf029921ced767d4e14a18e8bccb55812dd974852c56e0931a98fa10366cf",
"image_hash": "a08b89ba37d27c9d5a81bcd56bcd042e6baf54d2f182bbaaf47e97f14c9cd525",
"code_revision": "68061635d9fef3097609945bfd9ac1935791b398"
},
"facings": 111,
"leakage": {
"excluded": 0,
"labels": []
},
"hub_share_limit": 0.3,
"enrollment": {
"catalog_skus": [
"sku_l01",
"sku_l02",
"sku_l03",
"sku_l04",
"sku_l05",
"sku_l06",
"sku_l07",
"sku_l08",
"sku_l09",
"sku_l10",
"sku_l11",
"sku_l12"
],
"enrolled_skus": [
"sku_l01",
"sku_l02",
"sku_l03",
"sku_l05",
"sku_l06",
"sku_l09",
"sku_l10",
"sku_l11"
],
"unenrolled_skus": {
"sku_l07": 6,
"sku_l04": 1,
"sku_l08": 1
},
"facings_without_enrollment": 8,
"scored_facings": 109
},
"results": {
"full": [
{
"lane": "local",
"accept": 2.0,
"margin": 0.15,
"in_catalog": 61,
"out_catalog": 48,
"enrolled": 53,
"correct": 48,
"wrong": 9,
"rejected": 52,
"catalog_ceiling": 0.8689,
"identity_ceiling": 0.9245,
"decision_ceiling": 0.8113,
"end_to_end_coverage": 0.7869,
"false_accepts": 9,
"unknown_false_accepts": 8,
"unenrolled_false_accepts": 0,
"false_accept_rate": 0.1667,
"near_unknown_false_accept_rate": 0.0,
"precision": 0.8421,
"max_sku_share": 0.2281,
"hub_sku": "sku_l09",
"hub_alarm": false,
"histogram": {
"sku_l09": 13,
"sku_l11": 10,
"sku_l10": 9,
"sku_l01": 5,
"sku_l03": 5,
"sku_l07": 5,
"sku_l06": 4,
"sku_l02": 3,
"sku_l05": 3
},
"per_sku": {
"sku_l01": {
"correct": 5,
"total": 5
},
"sku_l02": {
"correct": 2,
"total": 4
},
"sku_l03": {
"correct": 4,
"total": 4
},
"sku_l04": {
"correct": 0,
"total": 1
},
"sku_l05": {
"correct": 3,
"total": 3
},
"sku_l06": {
"correct": 4,
"total": 4
},
"sku_l07": {
"correct": 5,
"total": 6
},
"sku_l08": {
"correct": 0,
"total": 1
},
"sku_l09": {
"correct": 11,
"total": 13
},
"sku_l10": {
"correct": 9,
"total": 11
},
"sku_l11": {
"correct": 5,
"total": 9
}
},
"ms_per_crop": 26.7
}
],
"live": [
{
"lane": "local",
"accept": 2.0,
"margin": 0.15,
"in_catalog": 61,
"out_catalog": 48,
"enrolled": 53,
"correct": 41,
"wrong": 4,
"rejected": 64,
"catalog_ceiling": 0.8689,
"identity_ceiling": 0.8491,
"decision_ceiling": 0.6604,
"end_to_end_coverage": 0.6721,
"false_accepts": 4,
"unknown_false_accepts": 3,
"unenrolled_false_accepts": 0,
"false_accept_rate": 0.0625,
"near_unknown_false_accept_rate": 0.0,
"precision": 0.9111,
"max_sku_share": 0.2,
"hub_sku": "sku_l09",
"hub_alarm": false,
"histogram": {
"sku_l09": 9,
"sku_l10": 9,
"sku_l11": 7,
"sku_l07": 6,
"sku_l01": 4,
"sku_l03": 3,
"sku_l05": 3,
"sku_l06": 2,
"sku_l02": 1,
"sku_l04": 1
},
"per_sku": {
"sku_l01": {
"correct": 4,
"total": 5
},
"sku_l02": {
"correct": 1,
"total": 4
},
"sku_l03": {
"correct": 3,
"total": 4
},
"sku_l04": {
"correct": 1,
"total": 1
},
"sku_l05": {
"correct": 3,
"total": 3
},
"sku_l06": {
"correct": 2,
"total": 4
},
"sku_l07": {
"correct": 5,
"total": 6
},
"sku_l08": {
"correct": 0,
"total": 1
},
"sku_l09": {
"correct": 9,
"total": 13
},
"sku_l10": {
"correct": 9,
"total": 11
},
"sku_l11": {
"correct": 4,
"total": 9
}
},
"ms_per_crop": 20.9
}
]
}
}

70
recognization/data/evaluations/2026-09-29-session/catalog-audit/source-audit.json
View file

@ -1,70 +0,0 @@
{
"source": "data/packshots/source",
"output": "/tmp/lopia-audit",
"min_width": 120,
"min_height": 160,
"min_views": 1,
"duplicate_cosine": 0.98,
"min_name_ocr": 0.7,
"ocr": "vision",
"text_only_skus": [
"sku_l04",
"sku_l07",
"sku_l08",
"sku_l09",
"sku_l10",
"sku_l12"
],
"text_discriminated_siblings": [],
"below_pyramid_floor": [],
"completeness": {
"live_ready": null,
"reason": "no --shelf-tags given; the completeness rule needs the shelf's SKU list",
"enrolled": 6,
"text_only": [
"sku_l04",
"sku_l07",
"sku_l08",
"sku_l09",
"sku_l10",
"sku_l12"
]
},
"skus": {
"sku_l01": [
"sku_l01.jpg"
],
"sku_l02": [
"sku_l02.jpg"
],
"sku_l03": [
"sku_l03.jpg"
],
"sku_l05": [
"sku_l05.jpg"
],
"sku_l06": [
"sku_l06.jpg"
],
"sku_l11": [
"sku_l11.jpg"
]
},
"refused": [
{
"sku": "sku_l04",
"view": "sku_l04.jpg",
"reason": "name/OCR 0.00 below 0.70: 明太手屋さんが つくつ た コ"
},
{
"sku": "sku_l07",
"view": "sku_l07.jpg",
"reason": "name/OCR 0.57 below 0.70: ラッキー マヨネーズ 調味料（アミノ酸）不使用"
},
{
"sku": "sku_l09",
"view": "sku_l09.jpg",
"reason": "name/OCR 0.05 below 0.70: キューピーマヨネーズ KEWPIE MAYONNAISE ® 450g"
}
]
}

1461
recognization/data/evaluations/2026-09-29-session/pace-medians/result_1.json
View file

File diff suppressed because it is too large Load diff

1433
recognization/data/evaluations/2026-09-29-session/pace-medians/result_2.json
View file

File diff suppressed because it is too large Load diff

1261
recognization/data/evaluations/2026-09-29-session/pace-medians/result_3.json
View file

File diff suppressed because it is too large Load diff

1491
recognization/data/evaluations/2026-09-29-session/pace-medians/send_1.json
View file

File diff suppressed because it is too large Load diff

1559
recognization/data/evaluations/2026-09-29-session/pace-medians/send_2.json
View file

File diff suppressed because it is too large Load diff

1351
recognization/data/evaluations/2026-09-29-session/pace-medians/send_3.json
View file

File diff suppressed because it is too large Load diff

40
recognization/data/evaluations/2026-09-29-session/pace-medians/summary.json
View file

@ -1,40 +0,0 @@
{
"send": {
"coverage": [
0.63,
0.704,
0.667
],
"coverage_median": 0.667,
"precision": [
0.773,
0.884,
0.973
],
"precision_median": 0.884,
"wrong": [
0,
0,
0
]
},
"result": {
"coverage": [
0.63,
0.63,
0.704
],
"coverage_median": 0.63,
"precision": [
0.857,
0.825,
0.9
],
"precision_median": 0.857,
"wrong": [
0,
0,
0
]
}
}

27
recognization/data/evaluations/2026-09-29-session/scratch/fa_detail.py
View file

@ -1,27 +0,0 @@
"""Which facings are falsely accepted, and as what."""
import sys, cv2
sys.path.insert(0, "code")
from eval_identity import LocalLaneAdapter, load_labels, at_live_resolution, leaked, UNKNOWN, crop_of
from identity_lanes import DecisionPolicy
payload = load_labels("eval/labels/image6.json")
image = cv2.imread("add/image6.jpg")
facings = [f for f in payload["facings"] if f["label"] != "exclude"]
facings = [f for f in facings if leaked(image, f["box"], "stock-lopia-clean") is None]
enrolled = {"sku_l01","sku_l02","sku_l03","sku_l05","sku_l06","sku_l09","sku_l10","sku_l11"}
for live in (False, True):
picture = at_live_resolution(image) if live else image
scaled = facings if not live else [
{**f, "box": [int(round(v * picture.shape[1] / image.shape[1])) for v in f["box"]]}
for f in facings]
lane = LocalLaneAdapter("stock-lopia-clean")
print(f"=== {'live' if live else 'full'} ===")
for f in scaled:
crop = crop_of(picture, f["box"])
if crop.size == 0: continue
ranked = lane.rank(crop)
d = lane.decide(crop, ranked, accept=2.0, margin=0.0, scale="local")
ans = d.get("best_sku") if d.get("identity_level") == "accepted" else None
if ans and ans != f["label"]:
print(f" #{f['index']:>3} truth={f['label']:<9} answered={ans:<9} enrolled={f['label'] in enrolled}")

54
recognization/data/evaluations/2026-09-29-session/scratch/fa_text.py
View file

@ -1,54 +0,0 @@
"""What the falsely-accepted crops actually read, and what the text lane scores.
The advisory says crops 82-97 are Kewpie Mayonnaise 450g - a product the
catalog does not hold - and that no catalog row can therefore win the text
ranking, so the veto at DecisionPolicy.decide never fires. This prints the
reads and the scores that claim depends on.
"""
import csv
import sys
import cv2
import numpy as np
sys.path.insert(0, "code")
from identity_lanes import TextLane, carries_identity # noqa: E402
from ocr import build_ocr_engine # noqa: E402
INDICES = [73, 82, 83, 84, 86, 94, 95, 96, 97]
rows = []
with open("stock-lopia-clean/stock.csv", encoding="utf-8") as handle:
for record in csv.reader(handle):
if len(record) >= 2 and record[0].startswith("sku_"):
rows.append({"sku_id": record[0], "product_name": record[1]})
engine = build_ocr_engine("vision")
lane = TextLane.from_catalog(rows, "stock-lopia-clean", ocr_engine=engine)
labels = {f["index"]: f for f in __import__("json").load(open("eval/labels/image6.json"))["facings"]}
image = cv2.imread("add/image6.jpg")
tiles = []
for index in INDICES:
facing = labels[index]
x1, y1, x2, y2 = facing["box"]
crop = image[y1:y2, x1:x2]
read = lane.read(crop)
scores = lane.scores_from_text(read)
top = sorted(scores.items(), key=lambda kv: -kv[1])[:3]
print(f"#{index:>3} truth={facing['label']:<9} carries={carries_identity(read)}")
print(f" read: {read[:110]}")
print(f" text top: {[(s, round(v, 3)) for s, v in top]}")
height, width = crop.shape[:2]
scale = 260.0 / max(1, height)
resized = cv2.resize(crop, (max(1, int(width * scale)), 260), interpolation=cv2.INTER_CUBIC)
canvas = np.full((300, 220, 3), 255, np.uint8)
canvas[30:290, : min(resized.shape[1], 220)] = resized[:, :220]
cv2.putText(canvas, str(index), (6, 22), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)
tiles.append(canvas)
grid = np.vstack([np.hstack(tiles[:5]), np.hstack(tiles[5:])])
cv2.imwrite("data/evaluations/2026-09-29-session/scratch/fa_tiles.jpg", grid)
print("tile:", grid.shape)

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l04_label.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 240 KiB

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l04_rot270.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 255 KiB

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l10_facing59.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 39 KiB

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l10_shelf_tag.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 117 KiB

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l11_shelf_tag.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 307 KiB

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l11_tag_half0.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 281 KiB

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l11_tag_half1.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 263 KiB

BIN
recognization/data/evaluations/2026-09-29-session/scratch/l11_tag_zoom.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 360 KiB

71
recognization/data/evaluations/2026-09-29-session/scratch/size_gate.py
View file

@ -1,71 +0,0 @@
"""Mass-only size contradiction, with full reads.
The first pass counted every quantity token, including percentages, so a crop
reading "カロリー 50% カット" looked like a 50 g package. Only mass units can
contradict a package size, so this restricts to g/kg and prints the whole read
for every accepted facing that carries one.
"""
import csv
import json
import re
import sys
import cv2
sys.path.insert(0, "code")
from eval_identity import LocalLaneAdapter, at_live_resolution, crop_of, leaked # noqa: E402
from identity_lanes import TextLane, normalize_text # noqa: E402
from ocr import build_ocr_engine # noqa: E402
MASS = re.compile(r"(\d+(?:\.\d+)?)\s*(KG|G)", re.IGNORECASE)
# Package sizes this product class actually comes in. Below it are OCR fragments
# of a longer number ("20" from 200); above it are concatenations ("5210").
PLAUSIBLE = (100.0, 2000.0)
def masses(text: str):
return {float(number) for number, _unit in MASS.findall(normalize_text(text))}
rows = []
with open("stock-lopia-clean/stock.csv", encoding="utf-8") as handle:
for record in csv.reader(handle):
if len(record) >= 2 and record[0].startswith("sku_"):
rows.append({"sku_id": record[0], "product_name": record[1]})
names = {row["sku_id"]: row["product_name"] for row in rows}
engine = build_ocr_engine("vision")
text_lane = TextLane.from_catalog(rows, "stock-lopia-clean", ocr_engine=engine)
local_lane = LocalLaneAdapter("stock-lopia-clean")
payload = json.load(open("eval/labels/image6.json"))
image = cv2.imread("add/image6.jpg")
facings = [f for f in payload["facings"] if f["label"] != "exclude"]
facings = [f for f in facings if leaked(image, f["box"], "stock-lopia-clean") is None]
for live in (False, True):
picture = at_live_resolution(image) if live else image
scaled = facings if not live else [
{**f, "box": [int(round(v * picture.shape[1] / image.shape[1])) for v in f["box"]]}
for f in facings]
print(f"=== {'live' if live else 'full'} ===")
for facing in scaled:
crop = crop_of(picture, facing["box"])
if crop.size == 0:
continue
ranked = local_lane.rank(crop)
decision = local_lane.decide(crop, ranked, accept=2.0, margin=0.0, scale="local")
if decision.get("identity_level") != "accepted":
continue
answer = decision.get("best_sku")
read = text_lane.read(crop)
read_masses = masses(read)
name_masses = masses(names.get(answer, ""))
if not read_masses or not name_masses:
continue
plausible = {m for m in read_masses if PLAUSIBLE[0] <= m <= PLAUSIBLE[1]}
veto = bool(plausible) and not (read_masses & name_masses)
correct = "correct" if answer == facing["label"] else "WRONG"
print(f" #{facing['index']:>3} {'VETO' if veto else 'keep'} {correct:<7} "
f"answer={answer:<9} read_m={sorted(read_masses)} name_m={sorted(name_masses)}")
print(f" read: {read[:120]}")

BIN
recognization/data/evaluations/2026-09-29-session/scratch/unknown_row.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 238 KiB

28
recognization/data/evaluations/2026-09-29-session/server-smoke/final-summary.json
View file

@ -1,28 +0,0 @@
{
"endpoint": "/v1/recognize",
"image": "add/image6.jpg",
"catalog": "stock-lopia-clean",
"gates": "LOCAL_ACCEPT=4.0 TEXT_MIN_SHARED=4",
"still": {
"detections": 111,
"named": 44,
"matched": 44,
"correct": 42,
"wrong": 2,
"precision": 0.955
},
"stream": {
"seconds": 20,
"pan": 12,
"tracks_seen": 165,
"tracks_labelled": 62,
"label_changes": 0,
"unexplained_label_changes": 0,
"facings_named": "17/27",
"precision": 0.84,
"wrong": 0,
"unenrolled_false_accepts": 0,
"unknown_false_accepts": 8
},
"note": "Final state with every catalog correction and both calibrated accept lines live."
}

24
recognization/data/evaluations/2026-09-29-session/server-smoke/gate-summary.json
View file

@ -1,24 +0,0 @@
{
"endpoint": "/v1/recognize",
"image": "add/image6.jpg",
"catalog": "stock-lopia-clean",
"gates": "LOCAL_ACCEPT=4.0 TEXT_MIN_SHARED=4",
"detections": 111,
"named": 44,
"matched": 44,
"correct": 42,
"wrong": 2,
"precision": 0.955,
"wrong_detail": [
"sku_l07 vs unknown",
"sku_l03 vs unknown"
],
"previous": {
"gates": "2.0/2",
"named": 53,
"correct": 48,
"wrong": 5,
"precision": 0.906
},
"note": "Live still endpoint, box-IoU matched to the corrected labels. The stricter gates trade 9 names for 3 wrong labels."
}

1
recognization/data/evaluations/2026-09-29-session/server-smoke/recognize-after-l12-rename.json
View file

File diff suppressed because one or more lines are too long

1
recognization/data/evaluations/2026-09-29-session/server-smoke/recognize-after-size-gate.json
View file

File diff suppressed because one or more lines are too long

1
recognization/data/evaluations/2026-09-29-session/server-smoke/recognize-final-gated.json
View file

File diff suppressed because one or more lines are too long

1
recognization/data/evaluations/2026-09-29-session/server-smoke/recognize-final.json
View file

File diff suppressed because one or more lines are too long

1
recognization/data/evaluations/2026-09-29-session/server-smoke/recognize-gate-calibrated.json
View file

File diff suppressed because one or more lines are too long

1
recognization/data/evaluations/2026-09-29-session/server-smoke/recognize-verified.json
View file

File diff suppressed because one or more lines are too long

23
recognization/data/evaluations/2026-09-29-session/server-smoke/summary.json
View file

@ -1,23 +0,0 @@
{
"endpoint": "/v1/recognize",
"image": "add/image6.jpg",
"catalog": "stock-lopia-clean",
"catalog_version": "b862c55f4623",
"detections": 111,
"named": 53,
"named_without_product_info": 0,
"view_suffixed_ids": [],
"elapsed_s": 2.38,
"by_sku": {
"sku_l10": 9,
"sku_l11": 5,
"sku_l01": 5,
"sku_l06": 4,
"sku_l09": 13,
"sku_l07": 6,
"sku_l02": 3,
"sku_l03": 5,
"sku_l05": 3
},
"note": "Final state: l06 current package enrolled (4/4 facings named, was 1/4 at live), l10/l11 400g, l12 renamed. l12 not named: no enrolled view."
}

1631
recognization/data/evaluations/2026-09-29-session/still/final_0.json
View file

File diff suppressed because it is too large Load diff

Some files were not shown because too many files have changed in this diff Show more