feat(identity): local-feature matcher plus text lane as the default backend
Replaces the DINO gallery as the recognition path with the measured design from docs/research/2026-09-28-recognition-architecture/DECISION.md: RootSIFT over a per-SKU reference pyramid, similarity-constrained RANSAC and a hue check, fused with an OCR text lane that vetoes a match whose package text names a different catalog row and answers on its own when geometry finds nothing. - identity_lanes.py: LocalLane, TextLane, DecisionPolicy. The text lane builds each row's signature from the catalog name plus its packshot OCR, so a row with no enrolled image is still answerable. - engine.py: every backend groups scores by canonical SKU, so no view-suffixed id reaches the wire and a name and price always resolve. decide_crop fuses the lanes; score_crop_lanes feeds the live path. Both lanes are built in warm_up (non-fatally) so the first frame is not the slow one, and the text lane shares the engine's OCR lock. - live.py: decides per crop on the lane's own rank order. The lane's order comes from its rank key and its value is only the gate, so re-sorting by value named a different SKU than the lane chose; averaging in an empty observation also diluted a good confidence below the accept line. - identity.py: a `local` score scale whose gate is one absolute line, because the value is already a margin over the runner-up. - app.py, run-demo-server.sh: `--identity local` is the default; the demo server no longer starts on the leaked `stock-lopia` catalog. - eval_live.py: the replay vocabulary accepts `local`, so a recording from the shipped server can be scored. - README.md: documents the three backends and what each is for. Measured on the corrected image6 labels at live resolution: 36/61 in-catalog facings named at 95% precision, 0 of 48 unknowns accepted, 11 labels sourced from the text lane alone. Through the real server: /v1/recognize 90% precision, /v1/stream 100% precision, no view-suffixed ids on either.
This commit is contained in:
parent
25dbf132e7
commit
aaed5d388f
44 changed files with 7847 additions and 60 deletions
|
|
@ -0,0 +1,53 @@
|
|||
# Evidence scripts (2026-09-28 audit)
|
||||
|
||||
Read-only against product code: every script imports from `code/` and writes
|
||||
only into this folder. Run from `recognization/` with `.venv/bin/python`.
|
||||
|
||||
`image6_v2.json` is the corrected image6 label set (26 of 111 v1 labels
|
||||
changed; every row carries `v1_label`, `source`, and a `note` where changed).
|
||||
`image6-v1-labels-overlay.jpg` shows the v1 labels on the shelf; green = catalog
|
||||
label, red = unknown. `sheet.py` regenerates per-SKU contact sheets.
|
||||
|
||||
| Step | Command (prefix `E=docs/research/2026-09-28-recognition-architecture/evidence`) | Produces |
|
||||
|---|---|---|
|
||||
| Original matcher, both catalogs, full/live/live-with-unscaled-boxes | `.venv/bin/python $E/diag.py --stock stock-lopia-clean --labels $E/image6_v2.json [--live] [--bug] --out $E/run_stock-lopia-clean_<full\|live\|livebug>.pkl` (repeat with `--stock stock-lopia`) | `run_*.pkl` |
|
||||
| Score them under v1 or v2 labels | `.venv/bin/python $E/analyze.py <labels.json> $E/run_*.pkl` | identity top-1, argmax, current gate |
|
||||
| Matcher variants | `.venv/bin/python $E/variants.py --stock stock-lopia-clean --labels $E/image6_v2.json [--live] --scales 200,400 --root --geom affine --out $E/v_ms_root_aff_<full\|live>.pkl` (also `--tmpl-h 250/400`, no flags = raw) | `v_*.pkl` |
|
||||
| Variant ranking + open-set frontier | `.venv/bin/python $E/an2.py $E $(basename -a $E/v_*.pkl)` (`LABELS=image6_v2_lfo.json` for the leave-facing-out runs) | correct at 100/95/90 % precision |
|
||||
| Leave-facing-out catalog | `.venv/bin/python $E/lfo_setup.py`, then `variants.py --stock $E/cat_<pack\|full> --labels $E/image6_v2_lfo.json --scales 200,400 --root --geom affine [--live] --out $E/lfo_cat_<pack\|full>_<full\|live>.pkl` | `lfo_*.pkl` |
|
||||
| Unrelated-shelf distractors | `YOLO_OFFLINE=1 .venv/bin/python $E/distract.py stock-lopia-clean $E/cat_full` | `distract.pkl` |
|
||||
| Catalog growth, exhaustive vs descriptor index | `YOLO_OFFLINE=1 .venv/bin/python $E/scale.py` | stdout |
|
||||
| DINO retrieval under growth | `YOLO_OFFLINE=1 .venv/bin/python $E/dino_scale.py` | stdout |
|
||||
| OCR read-rate | `.venv/bin/python $E/ocr_exp.py` | `ocr.pkl` |
|
||||
| OCR signatures, then fusion | `.venv/bin/python $E/ocr_fuse.py && .venv/bin/python $E/fuse_eval.py` | `sig.pkl`, stdout |
|
||||
| Per-SKU contact sheets + the labelled shelf | `.venv/bin/python $E/sheet.py <outdir>` | `gt_<label>.jpg`, `shelf_labels.jpg` |
|
||||
|
||||
Timings in DECISION.md come from `scale.py` (single process). The other
|
||||
scripts ran up to ten processes at once, so their ms figures are relative only.
|
||||
|
||||
## Verified 2026-09-28
|
||||
|
||||
Every command above was run from `recognization/` and produced its listed
|
||||
output. Two rows were wrong as written:
|
||||
|
||||
- The table said `python`; the `python` on this machine's PATH has no `cv2`, so
|
||||
every row failed on import. It is `.venv/bin/python`.
|
||||
- `an2.py` takes the folder and then file names *inside* it, so `v_*.pkl`
|
||||
expanded in the working directory, where those files do not live. It is
|
||||
`$(basename -a $E/v_*.pkl)`.
|
||||
|
||||
`sheet.py` was missing from the table. It takes its output directory as a
|
||||
positional argument, not a flag: `sheet.py --help` reads `--help` as a
|
||||
directory name, writes nothing, and still exits 0. `analyze.py` and `an2.py`
|
||||
with no arguments raise `IndexError`; neither validates its own usage.
|
||||
|
||||
v1 labels are no longer a file on disk — `eval/labels/image6.json` is the v2
|
||||
set. To score a run under v1, rebuild it from the `v1_label` field:
|
||||
|
||||
.venv/bin/python -c "import json;d=json.load(open('$E/image6_v2.json'));\
|
||||
[f.update(label=f.pop('v1_label')) for f in d['facings']];\
|
||||
json.dump(d,open('/tmp/image6_v1.json','w'),ensure_ascii=False)"
|
||||
|
||||
That reconstruction reproduces DECISION.md's v1 numbers exactly: `analyze.py`
|
||||
gives ungated top-1 26/35 at both resolutions and the current gate 24/35 full,
|
||||
23/35 live (§2 row 15, F-01).
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
import pickle, json, sys, numpy as np, collections
|
||||
import os; S = sys.argv[1]; L = {f['index']: f['label'] for f in json.load(open(f'{S}/' + os.environ.get('LABELS','image6_v2.json')))['facings']}
|
||||
RANK = {
|
||||
'inl': lambda e: e['inliers'],
|
||||
'inl*col': lambda e: e['inliers'] * (0.25 + e['color']),
|
||||
}
|
||||
def conf_feats(ranked):
|
||||
(k1, e1), (k2, e2) = ranked[0], ranked[1]
|
||||
i1, i2 = e1['inliers'], e2['inliers']
|
||||
return {
|
||||
'inl': i1,
|
||||
'margin': i1 - i2,
|
||||
'ratio': 1 - i2 / max(i1, 1),
|
||||
'inl*ratio': i1 * (1 - i2 / max(i1, 1)),
|
||||
'cover': e1['cover'],
|
||||
'inl*ratio*qcov': i1 * (1 - i2 / max(i1, 1)) * np.sqrt(e1.get('qcover', 0)),
|
||||
'inl*ratio*col': i1 * (1 - i2 / max(i1, 1)) * e1['color'],
|
||||
}
|
||||
def evaluate(path, rank='inl'):
|
||||
r = pickle.load(open(path, 'rb')); E = set(r['enrolled'])
|
||||
rows = [x for x in r['rows'] if L[x['index']] != 'exclude']
|
||||
n_in = sum(L[x['index']] != 'unknown' for x in rows); n_en = sum(L[x['index']] in E for x in rows)
|
||||
recs = []; top1 = 0
|
||||
for x in rows:
|
||||
t = L[x['index']]
|
||||
if not x['ev']: recs.append((t, None, {k: 0 for k in conf_feats([('a', dict(inliers=0,cover=0,color=0)),('b',dict(inliers=0,cover=0,color=0))])})); continue
|
||||
ranked = sorted(x['ev'].items(), key=lambda kv: -RANK[rank](kv[1]))
|
||||
b = ranked[0][0] if ranked[0][1]['inliers'] > 0 else None
|
||||
if t in E and b == t: top1 += 1
|
||||
recs.append((t, b, conf_feats(ranked)))
|
||||
out = dict(top1=f'{top1}/{n_en}', argmax_e2e=f'{top1}/{n_in}')
|
||||
for f in ['inl', 'margin', 'ratio', 'inl*ratio', 'inl*ratio*qcov', 'inl*ratio*col']:
|
||||
pts = sorted(((c[f], t == b and b is not None) for t, b, c in recs if b), key=lambda p: -p[0])
|
||||
best = {0.9: 0, 0.95: 0, 1.0: 0}; c = a = 0; i = 0
|
||||
while i < len(pts):
|
||||
j = i
|
||||
while j < len(pts) and pts[j][0] == pts[i][0]: c += pts[j][1]; a += 1; j += 1
|
||||
for p in best:
|
||||
if c / a >= p: best[p] = max(best[p], c)
|
||||
i = j
|
||||
out[f] = '/'.join(str(best[p]) for p in (1.0, 0.95, 0.9))
|
||||
return out
|
||||
if __name__ == '__main__':
|
||||
print('feature cells = correct accepted at precision 100%/95%/90% (enrolled in-catalog n=42, all in-catalog n=61, unknown n=48)')
|
||||
for p in sys.argv[2:]:
|
||||
for rk in ['inl', 'inl*col']:
|
||||
print(p.split('/')[-1].replace('.pkl',''), rk, json.dumps(evaluate(f'{S}/{p}', rk)))
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
import pickle, json, sys, collections, numpy as np
|
||||
sys.path.insert(0,'code')
|
||||
from identity import decide_identity
|
||||
def load_labels(p): return {f['index']: f['label'] for f in json.load(open(p))['facings']}
|
||||
def feats(sc):
|
||||
r = sorted(sc.items(), key=lambda kv: -kv[1])
|
||||
if not r: return None, 0.0, 0.0
|
||||
top = r[0][1]; run = r[1][1] if len(r) > 1 else 0.0
|
||||
return r[0][0], top, (top-run)/max(top,1.0)
|
||||
def analyze(run, labels, verbose=False):
|
||||
E = set(run['enrolled']); rows = [x for x in run['rows'] if labels[x['index']] != 'exclude']
|
||||
T = lambda x: labels[x['index']]
|
||||
inc = [x for x in rows if T(x) != 'unknown']; ince = [x for x in inc if T(x) in E]
|
||||
unk = [x for x in rows if T(x) == 'unknown']
|
||||
# identity ceiling: ungated argmax on enrolled in-catalog
|
||||
ranks = []
|
||||
for x in ince:
|
||||
order = [k for k, _ in sorted(x['scores'].items(), key=lambda kv: -kv[1])]
|
||||
ranks.append(order.index(T(x))+1 if T(x) in order and x['scores'].get(T(x),0)>0 else 99)
|
||||
top1 = sum(r == 1 for r in ranks)
|
||||
# Van contract: argmax always
|
||||
arg_correct = top1; answered = sum(1 for x in rows if x['scores'] and max(x['scores'].values())>0)
|
||||
# current decision rule
|
||||
acc = collections.Counter(); c = w = wu = wi = 0
|
||||
for x in rows:
|
||||
d = decide_identity(x['scores'], method='hybrid')
|
||||
b = d['best_sku'] if d['identity_level']=='accepted' else None
|
||||
if not b: continue
|
||||
acc[b]+=1
|
||||
if b == T(x): c += 1
|
||||
else:
|
||||
w += 1
|
||||
if T(x)=='unknown': wu+=1
|
||||
else: wi+=1
|
||||
hub = max(acc.values())/sum(acc.values()) if acc else 0
|
||||
out = dict(n=len(rows), in_catalog=len(inc), enrolled_in=len(ince), unknown=len(unk),
|
||||
identity_top1=f"{top1}/{len(ince)}={top1/max(1,len(ince)):.0%}",
|
||||
argmax_e2e=f"{arg_correct}/{len(inc)}={arg_correct/max(1,len(inc)):.0%}",
|
||||
argmax_precision=f"{arg_correct}/{answered}={arg_correct/max(1,answered):.0%}",
|
||||
gated=f"correct {c} wrong {w} (unk {wu}, in-cat {wi}) cov_enr {c/max(1,len(ince)):.0%} prec {c/max(1,c+w):.0%} hub {hub:.0%}",
|
||||
ms=round(float(np.median([x['ms'] for x in run['rows'] if x['ms']>0] or [0])),1))
|
||||
return out, ranks
|
||||
def sweep(run, labels, feature='margin'):
|
||||
"""Pareto of (correct accepted, precision) over a single threshold on a feature."""
|
||||
E=set(run['enrolled']); rows=[x for x in run['rows'] if labels[x['index']]!='exclude']
|
||||
pts=[]
|
||||
for x in rows:
|
||||
b,top,mar=feats(x['scores']); t=labels[x['index']]
|
||||
v={'margin':mar,'top':top,'both':min(mar/0.3, top/40)}[feature]
|
||||
pts.append((v, b==t and b is not None, b))
|
||||
pts.sort(key=lambda p:-p[0]); res=[]; c=a=0
|
||||
for v,ok,b in pts:
|
||||
a+=1; c+=ok; res.append((v,c,a,c/a))
|
||||
return res
|
||||
if __name__=='__main__':
|
||||
labels=load_labels(sys.argv[1])
|
||||
for p in sys.argv[2:]:
|
||||
r=pickle.load(open(p,'rb')); o,ranks=analyze(r,labels)
|
||||
print(p.split('/')[-1], json.dumps(o, ensure_ascii=False))
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
"""Diagnostic identity harness. Logs full per-facing score vectors so every
|
||||
ceiling (catalog, identity rank, decision) is separable. Read-only on product code."""
|
||||
import sys, os, json, time, argparse, pickle
|
||||
sys.path.insert(0, 'code')
|
||||
import cv2, numpy as np
|
||||
from feature_matcher import ProductFeatureMatcher
|
||||
|
||||
def sku_of(name):
|
||||
p = name.split('_'); return '_'.join(p[:2])
|
||||
|
||||
def crops(image_path, labels, live, bug=False, long_side=1280):
|
||||
im = cv2.imread(image_path); f = 1.0
|
||||
if live:
|
||||
h, w = im.shape[:2]; f = long_side / max(h, w)
|
||||
im = cv2.resize(im, (int(w*f), int(h*f)), interpolation=cv2.INTER_AREA)
|
||||
out = []
|
||||
for fc in labels['facings']:
|
||||
b = fc['box'] if bug else [int(round(v*f)) for v in fc['box']]
|
||||
x1, y1, x2, y2 = b
|
||||
out.append((fc, im[max(0,y1):y2, max(0,x1):x2]))
|
||||
return out
|
||||
|
||||
def run(stock, image, labels, live, bug, method):
|
||||
m = ProductFeatureMatcher(stock_dir=stock)
|
||||
rows = []
|
||||
for fc, c in crops(image, labels, live, bug):
|
||||
if c.size == 0 or min(c.shape[:2]) < 8: rows.append(dict(index=fc["index"], truth=fc["label"], shape=c.shape[:2], scores={}, templ={}, sub={}, ms=0.0)); continue
|
||||
t = time.perf_counter(); s, det = m.score_crop(c, method=method); dt = time.perf_counter()-t
|
||||
g = {}
|
||||
for k, v in s.items(): g[sku_of(k)] = max(g.get(sku_of(k), 0.0), v)
|
||||
sub = {k: det[k].get('sub_details') for k in det}
|
||||
rows.append(dict(index=fc['index'], truth=fc['label'], shape=c.shape[:2], scores=g, templ=s, sub=sub, ms=dt*1000))
|
||||
return dict(stock=stock, live=live, bug=bug, method=method, enrolled=sorted({sku_of(k) for k in m.db.templates}), rows=rows)
|
||||
|
||||
if __name__ == '__main__':
|
||||
ap = argparse.ArgumentParser(); ap.add_argument('--stock'); ap.add_argument('--labels'); ap.add_argument('--live', action='store_true')
|
||||
ap.add_argument('--bug', action='store_true'); ap.add_argument('--method', default='hybrid'); ap.add_argument('--out')
|
||||
a = ap.parse_args(); L = json.load(open(a.labels))
|
||||
r = run(a.stock, 'add/image6.jpg', L, a.live, a.bug, a.method)
|
||||
pickle.dump(r, open(a.out, 'wb'))
|
||||
print(a.out, len(r['rows']), 'rows', 'median ms', round(float(np.median([x['ms'] for x in r['rows']])),1))
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
import sys,os,glob,json,time,numpy as np,cv2,random
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__)); os.environ['YOLO_OFFLINE']='1'
|
||||
from gallery import ProductGallery
|
||||
from detector import ShelfProductDetector
|
||||
from diag import crops, sku_of
|
||||
from PIL import Image
|
||||
S=os.path.dirname(__file__); random.seed(0)
|
||||
g=ProductGallery('stock-lopia-clean',['sku_l01','sku_l02','sku_l03','sku_l05','sku_l06','sku_l09','sku_l11']); assert g.load()
|
||||
refs=[(sku_of(os.path.basename(p)[:-4]),g.embed(cv2.imread(p))) for p in sorted(glob.glob('stock-lopia-clean/*.jpg'))]
|
||||
det=ShelfProductDetector(); dis=[]
|
||||
for p in sorted(glob.glob('add/*')):
|
||||
if 'image6' in p: continue
|
||||
im=cv2.cvtColor(np.array(Image.open(p).convert('RGB')),cv2.COLOR_RGB2BGR)
|
||||
for j,b in enumerate(det.detect(im).get('boxes',[])):
|
||||
x1,y1,x2,y2=[int(v) for v in b['box']]; c=im[max(0,y1):y2,max(0,x1):x2]
|
||||
if min(c.shape[:2])>=40: dis.append((f'd{j}_{os.path.basename(p)}',c))
|
||||
random.shuffle(dis)
|
||||
D=[(k,g.embed(c)) for k,c in dis]
|
||||
L=json.load(open(f'{S}/image6_v2.json')); E={k for k,_ in refs}
|
||||
Q=[(fc['label'],g.embed(c)) for fc,c in crops('add/image6.jpg',L,True) if fc['label'] in E]
|
||||
t=time.perf_counter(); _=g.embed(Q and dis[0][1]); print('embed ms',round((time.perf_counter()-t)*1000,1))
|
||||
for N in [0,50,200,len(D)]:
|
||||
R=refs+D[:N]; M=np.stack([e for _,e in R]); names=[k for k,_ in R]
|
||||
rec={1:0,5:0,10:0,20:0}
|
||||
for truth,q in Q:
|
||||
s=M@q; order=[]
|
||||
for i in np.argsort(-s):
|
||||
if names[i] not in order: order.append(names[i])
|
||||
for K in rec: rec[K]+= truth in order[:K]
|
||||
print(f'skus={len(set(names)):4d} DINO recall@1/5/10/20 '+'/'.join(str(rec[K]) for K in rec)+f' of {len(Q)}')
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
import sys,os,glob,pickle,numpy as np,cv2,json
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__))
|
||||
os.environ['YOLO_OFFLINE']='1'
|
||||
from detector import ShelfProductDetector
|
||||
from variants import Matcher
|
||||
from PIL import Image
|
||||
det=ShelfProductDetector()
|
||||
res={}
|
||||
for cat in sys.argv[1:]:
|
||||
name=os.path.basename(cat); m=Matcher(cat,root=True,geom='affine',scales=(200,400)); rows=[]
|
||||
for p in sorted(glob.glob('add/*')):
|
||||
if 'image6' in p: continue
|
||||
im=cv2.cvtColor(np.array(Image.open(p).convert('RGB')),cv2.COLOR_RGB2BGR)
|
||||
for b in det.detect(im).get('boxes',[]):
|
||||
x1,y1,x2,y2=[int(v) for v in b['box']]; c=im[max(0,y1):y2,max(0,x1):x2]
|
||||
if c.size==0 or min(c.shape[:2])<20: continue
|
||||
rows.append(dict(img=p,ev=m.score(c)))
|
||||
res[name]=rows; print(name,len(rows),'distractor crops')
|
||||
pickle.dump(res,open(f'{os.path.dirname(__file__)}/distract.pkl','wb'))
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
import sys,os,json,pickle,csv,numpy as np
|
||||
S=os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0,S)
|
||||
from ocr_fuse import norm,sim # noqa (module runs OCR of packshots on import; cheap)
|
||||
sig=pickle.load(open(f'{S}/sig.pkl','rb')); O=pickle.load(open(f'{S}/ocr.pkl','rb'))
|
||||
L={f['index']:f['label'] for f in json.load(open(f'{S}/image6_v2.json'))['facings']}
|
||||
def conf(ev):
|
||||
r=sorted(ev.items(),key=lambda kv:-kv[1]['inliers']*(0.25+kv[1]['color']))
|
||||
(k1,e1),(k2,e2)=r[0],r[1]; i1,i2=e1['inliers'],e2['inliers']
|
||||
return k1, i1*(1-i2/max(i1,1))*e1['color']
|
||||
for res,up,run in [('full',1,'v_ms_root_aff_full.pkl'),('live',2,'v_ms_root_aff_live.pkl')]:
|
||||
ocr={r['index']:r['text'] for r in O[res] if r['up']==up}
|
||||
M={x['index']:x for x in pickle.load(open(f'{S}/{run}','rb'))['rows']}
|
||||
n_in=sum(1 for t in L.values() if t not in ('unknown','exclude'))
|
||||
# OCR-alone identity among ALL catalog SKUs
|
||||
ok=tot=0
|
||||
for i,t in L.items():
|
||||
if t in ('unknown','exclude') or i not in ocr: continue
|
||||
tx=ocr[i];
|
||||
if len(norm(tx))<4: continue
|
||||
s={k:sim(tx,v) for k,v in sig.items()}; tot+=1; ok+= max(s,key=s.get)==t
|
||||
print(f'== {res}: OCR-alone top-1 over all 12 catalog SKUs, on facings with text: {ok}/{tot}')
|
||||
for delta in [None,0.0,0.1,0.2]:
|
||||
line=[]
|
||||
for tau in [0.5,1,2,3]:
|
||||
c=a=0
|
||||
for i,t in L.items():
|
||||
if t=='exclude' or i not in M or not M[i]['ev']: continue
|
||||
k,cf=conf(M[i]['ev'])
|
||||
if cf<tau: continue
|
||||
if delta is not None and len(norm(ocr.get(i,'')))>=4:
|
||||
s={kk:sim(ocr[i],v) for kk,v in sig.items()}; j=max(s,key=s.get)
|
||||
if j!=k and s[j]-s[k]>delta: continue # text names a different catalog product: veto
|
||||
a+=1; c+= k==t
|
||||
line.append(f'tau{tau}: {c}/{n_in} p={c/max(a,1):.0%}')
|
||||
print(' veto' if delta is not None else ' matcher only', delta, ' | '.join(line))
|
||||
print('== union: matcher (veto) OR OCR-only identity')
|
||||
for res,up,run in [('full',1,'v_ms_root_aff_full.pkl'),('live',2,'v_ms_root_aff_live.pkl')]:
|
||||
ocr={r['index']:r['text'] for r in O[res] if r['up']==up}
|
||||
M={x['index']:x for x in pickle.load(open(f'{S}/{run}','rb'))['rows']}
|
||||
n_in=sum(1 for t in L.values() if t not in ('unknown','exclude'))
|
||||
for tau in [0.5,1,2]:
|
||||
for sg,mu in [(0.5,0.15),(0.6,0.2),(0.7,0.25)]:
|
||||
c=a=0; fa_unk=0; bysrc={'m':0,'o':0}
|
||||
for i,t in L.items():
|
||||
if t=='exclude' or i not in M: continue
|
||||
tx=ocr.get(i,''); s={kk:sim(tx,v) for kk,v in sig.items()} if len(norm(tx))>=4 else {}
|
||||
j=max(s,key=s.get) if s else None
|
||||
ans=None
|
||||
if M[i]['ev']:
|
||||
k,cf=conf(M[i]['ev'])
|
||||
if cf>=tau and not (j and j!=k and s[j]-s[k]>0): ans=k; src='m'
|
||||
if ans is None and s:
|
||||
r=sorted(s.values(),reverse=True)
|
||||
if r[0]>=sg and r[0]-r[1]>=mu: ans=j; src='o'
|
||||
if ans:
|
||||
a+=1; c+= ans==t; bysrc[src]+= ans==t; fa_unk+= t=='unknown'
|
||||
print(f' {res} tau{tau} ocr>={sg} m{mu}: correct {c}/{n_in} ({c/n_in:.0%}) precision {c/max(a,1):.0%} accepted-unknown {fa_unk}/48 from {bysrc}')
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
File diff suppressed because it is too large
Load diff
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,22 @@
|
|||
"""Build the leave-facing-out catalogs: cat_pack (packshots only) and cat_full
|
||||
(packshots + the widest facing of each in-catalog SKU as a shelf view), and
|
||||
image6_v2_lfo.json with those reference facings excluded from scoring.
|
||||
Same-photo references make this an optimistic proxy, not a held-out result."""
|
||||
import json, cv2, os, shutil, collections
|
||||
E = os.path.dirname(os.path.abspath(__file__))
|
||||
L = json.load(open(f'{E}/image6_v2.json')); im = cv2.imread('add/image6.jpg')
|
||||
by = collections.defaultdict(list)
|
||||
for f in L['facings']:
|
||||
if f['label'] not in ('unknown', 'exclude'): by[f['label']].append(f)
|
||||
refs = {k: max(fs, key=lambda f: f['box'][2] - f['box'][0])['index'] for k, fs in by.items() if len(fs) >= 2}
|
||||
for cat in ('cat_pack', 'cat_full'):
|
||||
d = f'{E}/{cat}'; shutil.rmtree(d, ignore_errors=True); os.makedirs(d)
|
||||
for p in os.listdir('stock-lopia-clean'):
|
||||
if p.endswith('.jpg'): shutil.copy(f'stock-lopia-clean/{p}', d)
|
||||
if cat == 'cat_full':
|
||||
for k, i in refs.items():
|
||||
x1, y1, x2, y2 = L['facings'][i]['box']; cv2.imwrite(f'{d}/{k}_shelf.jpg', im[y1:y2, x1:x2])
|
||||
for f in L['facings']:
|
||||
if f['index'] in refs.values(): f['label'] = 'exclude'
|
||||
json.dump(L, open(f'{E}/image6_v2_lfo.json', 'w'), ensure_ascii=False)
|
||||
print('shelf references', refs)
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
import sys,os,json,pickle,numpy as np,cv2,time
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__))
|
||||
from ocr import VisionOcr
|
||||
from diag import crops
|
||||
S=os.path.dirname(__file__)
|
||||
L=json.load(open(f'{S}/image6_v2.json'))
|
||||
# distinguishing tokens per SKU (any hit counts); includes siblings for near-unknowns
|
||||
KEY={'sku_l01':['エッグケア','卵を使って'],'sku_l02':['アマニ'],'sku_l03':['からし','Spicy'],'sku_l04':['めんたい'],
|
||||
'sku_l05':['ゼロ','ノンコレ'],'sku_l06':['フィッテ','内臓脂肪'],'sku_l07':['ラッキー'],'sku_l08':['PERI','ペリ'],
|
||||
'sku_l09':['コクうま','65'],'sku_l10':['ピュアセレクト','マヨネーズ'],'sku_l11':['ハーフ','1/2'],'kewpie450':['KEWPIE','マヨネーズ']}
|
||||
o=VisionOcr(languages=['ja-JP','en-US']); out={}
|
||||
for res in ['full','live']:
|
||||
rows=[]
|
||||
for fc,c in crops('add/image6.jpg',L,res=='live'):
|
||||
if fc['label']=='exclude' or c.size==0 or min(c.shape[:2])<8: continue
|
||||
for up in [1,2]:
|
||||
cc=cv2.resize(c,None,fx=up,fy=up,interpolation=cv2.INTER_CUBIC) if up>1 else c
|
||||
t=time.perf_counter(); r=o.read(cc); ms=(time.perf_counter()-t)*1000
|
||||
txt=' '.join(l.text for l in r.lines)
|
||||
rows.append(dict(index=fc['index'],truth=fc['label'],up=up,text=txt,ms=ms))
|
||||
out[res]=rows
|
||||
pickle.dump(out,open(f'{S}/ocr.pkl','wb'))
|
||||
for res,rows in out.items():
|
||||
for up in [1,2]:
|
||||
rr=[r for r in rows if r['up']==up]; tr=[r for r in rr if r['truth'] in KEY]
|
||||
hit=sum(any(k in r['text'] for k in KEY[r['truth']]) for r in tr)
|
||||
print(res,'up',up,f'in-catalog key-token hit {hit}/{len(tr)}','median ms',round(np.median([r['ms'] for r in rr]),1))
|
||||
for r in out['live']:
|
||||
if r['up']==2 and r['truth']!='unknown': print(r['index'],r['truth'],'|',r['text'][:80])
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
import sys,os,json,pickle,csv,unicodedata,numpy as np,cv2,glob
|
||||
sys.path.insert(0,'code'); from ocr import VisionOcr
|
||||
S=os.path.dirname(__file__)
|
||||
def norm(s):
|
||||
s=unicodedata.normalize('NFKC',s).upper()
|
||||
tr=str.maketrans('ッャュョァィゥェォヮヵヶユ','ツヤユヨアイウエオワカケユ'); s=s.translate(tr)
|
||||
s=s.replace('キユーピー','キユピ').replace('キューピー','キユピ')
|
||||
return ''.join(ch for ch in s if ch.isalnum() or ch=='%')
|
||||
def grams(s): return {s[i:i+2] for i in range(len(s)-1)}
|
||||
def sim(a,b):
|
||||
A,B=grams(norm(a)),grams(norm(b))
|
||||
return len(A&B)/max(1,len(A)) if A else 0.0 # share of OCR bigrams explained by signature
|
||||
# leak-free signatures: catalog names (all SKUs) + OCR of enrolled packshots
|
||||
names={r['SKU_ID']:r['product_name'] for r in csv.DictReader(open('stock-lopia/stock.csv'))}
|
||||
if __name__=='__main__':
|
||||
o=VisionOcr(languages=['ja-JP','en-US']); sig={}
|
||||
for k,n in names.items():
|
||||
sig[k]=n
|
||||
p=f'stock-lopia-clean/{k}.jpg'
|
||||
if os.path.exists(p): sig[k]+=' '+' '.join(l.text for l in o.read(cv2.imread(p)).lines)
|
||||
for k,v in sig.items(): print(k, v[:90])
|
||||
pickle.dump(sig,open(f'{S}/sig.pkl','wb'))
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
"""Catalog growth: packshots + N distractor references from unrelated shelves.
|
||||
Compares exhaustive local matching vs a global RootSIFT descriptor index
|
||||
(vote -> shortlist K -> geometric verification). No DINO pass."""
|
||||
import sys,os,glob,json,time,numpy as np,cv2,random
|
||||
sys.path.insert(0,'code'); sys.path.insert(0,os.path.dirname(__file__)); os.environ['YOLO_OFFLINE']='1'
|
||||
from detector import ShelfProductDetector
|
||||
from diag import crops, sku_of
|
||||
from PIL import Image
|
||||
S=os.path.dirname(__file__); random.seed(0)
|
||||
sift=cv2.SIFT_create(nfeatures=2000)
|
||||
def root(d):
|
||||
if d is None: return None
|
||||
d=d/(np.abs(d).sum(1,keepdims=True)+1e-7); return np.sqrt(d).astype(np.float32)
|
||||
def feat(img,h):
|
||||
if h and img.shape[0]>h: img=cv2.resize(img,(max(1,int(img.shape[1]*h/img.shape[0])),h),interpolation=cv2.INTER_AREA)
|
||||
g=cv2.cvtColor(img,cv2.COLOR_BGR2GRAY); kp,d=sift.detectAndCompute(g,None); return kp,root(d),g.shape
|
||||
# references
|
||||
refs=[]
|
||||
for p in sorted(glob.glob('stock-lopia-clean/*.jpg')):
|
||||
img=cv2.imread(p); k=sku_of(os.path.basename(p)[:-4])
|
||||
for h in (200,400): kp,d,sh=feat(img,h); refs.append((k,kp,d,sh))
|
||||
det=ShelfProductDetector(); dis=[]
|
||||
for p in sorted(glob.glob('add/*')):
|
||||
if 'image6' in p: continue
|
||||
im=cv2.cvtColor(np.array(Image.open(p).convert('RGB')),cv2.COLOR_RGB2BGR)
|
||||
for j,b in enumerate(det.detect(im).get('boxes',[])):
|
||||
x1,y1,x2,y2=[int(v) for v in b['box']]; c=im[max(0,y1):y2,max(0,x1):x2]
|
||||
if min(c.shape[:2])>=40: dis.append((f'd_{os.path.basename(p)}_{j}',c))
|
||||
random.shuffle(dis); print('distractor refs available',len(dis))
|
||||
L=json.load(open(f'{S}/image6_v2.json')); Q=[(fc,c) for fc,c in crops('add/image6.jpg',L,True) if fc['label']!='exclude' and min(c.shape[:2])>=8]
|
||||
Qf=[(fc['label'],)+feat(c,None) for fc,c in Q]
|
||||
bf=cv2.BFMatcher(cv2.NORM_L2)
|
||||
def verify(kq,dq,r):
|
||||
k,kp,d,sh=r
|
||||
if dq is None or d is None or len(d)<2 or len(dq)<2: return 0
|
||||
good=[m for m,n in (p for p in bf.knnMatch(dq,d,k=2) if len(p)==2) if m.distance<0.75*n.distance]
|
||||
if len(good)<4: return 0
|
||||
src=np.float32([kq[m.queryIdx].pt for m in good]); dst=np.float32([kp[m.trainIdx].pt for m in good])
|
||||
H,mask=cv2.estimateAffinePartial2D(src,dst,method=cv2.RANSAC,ransacReprojThreshold=0.02*max(sh))
|
||||
return int(mask.sum()) if mask is not None else 0
|
||||
E={r[0] for r in refs}
|
||||
for N in [0,50,200,len(dis)]:
|
||||
R=refs+[(k,)+feat(c,None) for k,c in dis[:N]]
|
||||
# exhaustive
|
||||
t=time.perf_counter(); ok=0; fa=0; n=0
|
||||
best_all=[]
|
||||
for truth,kq,dq,_ in Qf:
|
||||
sc={}
|
||||
for r in R: sc[r[0]]=max(sc.get(r[0],0),verify(kq,dq,r))
|
||||
b=max(sc,key=sc.get); best_all.append(sc)
|
||||
if truth in E: n+=1; ok+= b==truth
|
||||
ex_ms=(time.perf_counter()-t)*1000/len(Qf)
|
||||
# descriptor index
|
||||
D=np.vstack([r[2] for r in R if r[2] is not None]); owner=np.concatenate([[i]*len(r[2]) for i,r in enumerate(R) if r[2] is not None])
|
||||
t0=time.perf_counter(); fl=cv2.FlannBasedMatcher(dict(algorithm=1,trees=4),dict(checks=64)); fl.add([D]); fl.train(); build=time.perf_counter()-t0
|
||||
rec={5:0,10:0,20:0}; t=time.perf_counter(); ok_idx=0
|
||||
for truth,kq,dq,_ in Qf:
|
||||
votes={}
|
||||
if dq is not None and len(dq)>=2:
|
||||
for m,nn in (p for p in fl.knnMatch(dq,k=2) if len(p)==2):
|
||||
if m.distance<0.8*nn.distance:
|
||||
k=R[owner[m.trainIdx]][0]; votes[k]=votes.get(k,0)+1
|
||||
order=sorted(votes,key=lambda k:-votes[k])
|
||||
if truth in E:
|
||||
for K in rec: rec[K]+= truth in order[:K]
|
||||
short=set(order[:10]); sc={}
|
||||
for r in R:
|
||||
if r[0] in short: sc[r[0]]=max(sc.get(r[0],0),verify(kq,dq,r))
|
||||
if truth in E and sc: ok_idx+= max(sc,key=sc.get)==truth
|
||||
ix_ms=(time.perf_counter()-t)*1000/len(Qf)
|
||||
print(f'refs={len(R):4d} skus={len({r[0] for r in R}):4d} | exhaustive top1 {ok}/{n} {ex_ms:6.1f} ms/crop | index recall@5/10/20 {rec[5]}/{rec[10]}/{rec[20]} of {n}, index+verify(K=10) top1 {ok_idx}/{n} {ix_ms:6.1f} ms/crop, build {build*1000:.0f} ms, {len(D)} descriptors')
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
import json,cv2,numpy as np,sys,glob,os
|
||||
d=json.load(open('eval/labels/image6.json'));im=cv2.imread('add/image6.jpg')
|
||||
S=sys.argv[1]
|
||||
H=300
|
||||
def tile(img,txt,col=(0,0,0)):
|
||||
h,w=img.shape[:2];s=H/h;img=cv2.resize(img,(max(1,int(w*s)),H))
|
||||
pad=np.full((H+30,img.shape[1]+6,3),255,np.uint8);pad[30:,3:3+img.shape[1]]=img
|
||||
cv2.putText(pad,txt,(2,20),cv2.FONT_HERSHEY_SIMPLEX,0.5,col,1);return pad
|
||||
groups={}
|
||||
for f in d['facings']:
|
||||
x1,y1,x2,y2=f['box'];c=im[y1:y2,x1:x2]
|
||||
groups.setdefault(f['label'],[]).append(tile(c,f"#{f['index']}"))
|
||||
for lab,ts in groups.items():
|
||||
ref=glob.glob(f'stock-lopia-clean/{lab}.jpg')
|
||||
row=([tile(cv2.imread(ref[0]),'REF',(0,0,255))] if ref else [])+ts
|
||||
rows=[row[i:i+12] for i in range(0,len(row),12)]
|
||||
out=[]
|
||||
for r in rows:
|
||||
w=sum(t.shape[1] for t in r);canvas=np.full((H+30,max(w,10),3),255,np.uint8);x=0
|
||||
for t in r: canvas[:,x:x+t.shape[1]]=t;x+=t.shape[1]
|
||||
out.append(canvas)
|
||||
W=max(o.shape[1] for o in out);out=[np.pad(o,((0,0),(0,W-o.shape[1]),(0,0)),constant_values=255) for o in out]
|
||||
cv2.imwrite(f'{S}/gt_{lab}.jpg',np.vstack(out))
|
||||
vis=im.copy()
|
||||
for f in d['facings']:
|
||||
x1,y1,x2,y2=f['box'];c=(0,0,255) if f['label']=='unknown' else (0,200,0)
|
||||
cv2.rectangle(vis,(x1,y1),(x2,y2),c,2);cv2.putText(vis,f"{f['index']}:{f['label'].replace('sku_','')}",(x1+2,y1+15),cv2.FONT_HERSHEY_SIMPLEX,0.45,c,1)
|
||||
cv2.imwrite(f'{S}/shelf_labels.jpg',vis)
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
"""Matcher variants. Each variant returns per-SKU evidence dicts so ranking and
|
||||
open-set features can be analysed separately."""
|
||||
import sys, os, json, time, glob, pickle, argparse
|
||||
import cv2, numpy as np
|
||||
sys.path.insert(0, os.path.dirname(__file__)); from diag import crops, sku_of
|
||||
|
||||
class Matcher:
|
||||
def __init__(self, stock, tmpl_h=None, root=False, geom='homog', nfeat=2000, scales=(None,)):
|
||||
self.sift = cv2.SIFT_create(nfeatures=nfeat); self.root = root; self.geom = geom
|
||||
self.T = []
|
||||
for p in sorted(glob.glob(f'{stock}/*.jp*g')+glob.glob(f'{stock}/*.png')):
|
||||
img = cv2.imread(p); name = os.path.splitext(os.path.basename(p))[0]
|
||||
for h in scales:
|
||||
im = img if h is None else cv2.resize(img, (max(1,int(img.shape[1]*h/img.shape[0])), h), interpolation=cv2.INTER_AREA)
|
||||
g = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY); kp, des = self.sift.detectAndCompute(g, None)
|
||||
des = self._root(des)
|
||||
hsv = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)
|
||||
self.T.append(dict(name=name, sku=sku_of(name), kp=kp, des=des, shape=g.shape, hist=self._hist(hsv)))
|
||||
self.bf = cv2.BFMatcher(cv2.NORM_L2)
|
||||
def _root(self, des):
|
||||
if des is None or not self.root: return des
|
||||
des = des / (np.abs(des).sum(1, keepdims=True) + 1e-7); return np.sqrt(des).astype(np.float32)
|
||||
@staticmethod
|
||||
def _hist(hsv):
|
||||
m = (hsv[...,1] > 60) & (hsv[...,2] > 50) # chromatic pixels only
|
||||
h = cv2.calcHist([hsv], [0], m.astype(np.uint8), [18], [0, 180]).ravel()
|
||||
return h / (h.sum() + 1e-6)
|
||||
def score(self, crop):
|
||||
g = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY); kq, dq = self.sift.detectAndCompute(g, None); dq = self._root(dq)
|
||||
hq = self._hist(cv2.cvtColor(crop, cv2.COLOR_BGR2HSV))
|
||||
ev = {}
|
||||
for t in self.T:
|
||||
e = dict(inliers=0, cover=0.0, color=float(np.minimum(hq, t['hist']).sum()), nq=len(kq))
|
||||
if dq is not None and t['des'] is not None and len(dq) >= 2 and len(t['des']) >= 2:
|
||||
good = [m for m, n in (p for p in self.bf.knnMatch(dq, t['des'], k=2) if len(p) == 2) if m.distance < 0.75*n.distance]
|
||||
if len(good) >= 4:
|
||||
src = np.float32([kq[m.queryIdx].pt for m in good]); dst = np.float32([t['kp'][m.trainIdx].pt for m in good])
|
||||
if self.geom == 'homog':
|
||||
H, mask = cv2.findHomography(src, dst, cv2.RANSAC, 0.02*max(t['shape']))
|
||||
else:
|
||||
H, mask = cv2.estimateAffinePartial2D(src, dst, method=cv2.RANSAC, ransacReprojThreshold=0.02*max(t['shape']))
|
||||
if mask is not None:
|
||||
inl = mask.ravel().astype(bool); n = int(inl.sum())
|
||||
if n >= 3:
|
||||
pts = dst[inl]; hull = cv2.convexHull(pts)
|
||||
e['inliers'] = n; e['cover'] = float(cv2.contourArea(hull) / (t['shape'][0]*t['shape'][1]))
|
||||
# crop-side coverage too: does the match span the query?
|
||||
qh = cv2.convexHull(src[inl]); e['qcover'] = float(cv2.contourArea(qh)/(g.shape[0]*g.shape[1]))
|
||||
k = t['sku']
|
||||
if k not in ev or e['inliers'] > ev[k]['inliers']: ev[k] = e
|
||||
return ev
|
||||
|
||||
if __name__ == '__main__':
|
||||
ap = argparse.ArgumentParser(); ap.add_argument('--stock'); ap.add_argument('--labels'); ap.add_argument('--live', action='store_true')
|
||||
ap.add_argument('--tmpl-h', type=int, default=None); ap.add_argument('--root', action='store_true'); ap.add_argument('--geom', default='homog')
|
||||
ap.add_argument('--scales', default=''); ap.add_argument('--out')
|
||||
a = ap.parse_args(); L = json.load(open(a.labels))
|
||||
scales = tuple(int(s) for s in a.scales.split(',')) if a.scales else ((a.tmpl_h,) if a.tmpl_h else (None,))
|
||||
t0 = time.perf_counter(); m = Matcher(a.stock, root=a.root, geom=a.geom, scales=scales); init = time.perf_counter()-t0
|
||||
rows = []
|
||||
for fc, c in crops('add/image6.jpg', L, a.live):
|
||||
if c.size == 0 or min(c.shape[:2]) < 8: rows.append(dict(index=fc['index'], ev={}, ms=0)); continue
|
||||
t = time.perf_counter(); ev = m.score(c); rows.append(dict(index=fc['index'], ev=ev, ms=(time.perf_counter()-t)*1000, shape=c.shape[:2]))
|
||||
pickle.dump(dict(rows=rows, enrolled=sorted({t['sku'] for t in m.T}), init_s=init, args=vars(a)), open(a.out, 'wb'))
|
||||
print(a.out, 'init', round(init,2), 'median ms', round(float(np.median([r['ms'] for r in rows if r['ms']])),1))
|
||||
Loading…
Add table
Add a link
Reference in a new issue