Make the catalog hash honest, share the text helpers, drop the duplicate labels

`catalog_hash` hashed the CSV and the view *names*, so replacing
`sku_l09.jpg` with a different product's packshot left the hash unchanged and
a report could claim the same catalog while every l09 decision in it had
moved. It hashes each view's bytes now.

`identity_lanes` carried its own `normalize_text` and `bigrams`, near-copies
of the ones in `rebuild_catalog` that had already drifted (the lane's table
folded ユ to itself, the catalog's did not). The lane imports the catalog's
now, so one read scores the same either way.

`evidence/image6_v2.json` was byte-identical to `eval/labels/image6.json`; the
evidence scripts and the decision document point at the canonical file and the
copy is gone. `score_crop`'s docstring now states the contract its callers
already depend on: the dict's insertion order is the lane's ranking, and
re-sorting by value names a different SKU than the lane chose.

Tests: the lane is shared by the classify pool, so a concurrent-ranking test
pins that one BFMatcher does not cross crops under eight threads.
This commit is contained in:
tngo 2026-09-29 00:48:03 +09:00
parent fc49ba60e6
commit b6c87c45ea
36 changed files with 89 additions and 1524 deletions

View file

@ -22,7 +22,7 @@ detector). The hybrid lane also reports shortlist recall, because a true SKU
outside the shortlist is a hard ceiling no verifier can lift.
python code/eval_identity.py \
--labels docs/research/2026-09-28-recognition-architecture/evidence/image6_v2.json \
--labels eval/labels/image6.json \
--image add/image6.jpg --stock-dir stock-lopia-clean --lanes matcher --live
Labels are {"image", "facings": [{"box", "label"}]} with "unknown" for facings
@ -70,7 +70,7 @@ LIVE_LONG_SIDE = 1280
# Pixel equality is not a leak test: a template re-cut from the same photo with
# a different box passes a same-shape comparison. A 64x64 grayscale signature
# catches most of those, but it is a heuristic, not a proof — measured on
# image6_v2 against stock-lopia, re-cuts in the realistic 0.94..1.06 box band
# the corrected labels against stock-lopia, re-cuts in the realistic 0.94..1.06 box band
# land 7..37 MAD from their facing's signature while the closest pair of
# genuinely distinct same-SKU facings is 23.1, so the distributions overlap and
# no threshold is clean. 20.0 catches 81% of that band with zero false
@ -116,12 +116,13 @@ def file_hash(path: str) -> Optional[str]:
def catalog_hash(stock_dir: str) -> Optional[str]:
"""SHA-256 over the catalog's CSV bytes and its image names.
"""SHA-256 over the catalog's CSV bytes and each view's bytes.
Image bytes are not hashed: a re-encoded view of the same product is the
same catalog, and hashing every packshot would make the number move for
reasons that do not change a single decision. A changed row or a different
set of views does change it.
The view bytes are hashed, not just their names. Names alone were not
enough: replacing `sku_l09.jpg` with a different product's packshot left
the hash unchanged, so a report could claim the same catalog while every
l09 decision in it had moved. Re-encoding the same image does move the
hash, which is the price of not being able to lie about what was scored.
"""
digest = hashlib.sha256()
csv_path = os.path.join(stock_dir, "stock.csv")
@ -137,8 +138,14 @@ def catalog_hash(stock_dir: str) -> Optional[str]:
)
except OSError:
names = []
digest.update(b"\n")
digest.update("\n".join(names).encode("utf-8"))
for name in names:
digest.update(b"\n")
digest.update(name.encode("utf-8"))
try:
with open(os.path.join(stock_dir, name), "rb") as handle:
digest.update(handle.read())
except OSError:
digest.update(b"unreadable")
return digest.hexdigest()
@ -557,7 +564,7 @@ def shortlist_recall(
shortlist already dropped. The denominator is enrolled facings only — a
facing whose SKU has no reference cannot be retrieved by any shortlist, so
counting it here reports a retrieval failure that is really a catalog gap.
Measured on image6_v2, that distinction is 67% vs 98% at top-5.
Measured on the corrected labels, that distinction is 67% vs 98% at top-5.
"""
hits = total = 0
misses: List[Dict[str, Any]] = []