tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

fix(eval): confirm a leak signature before calling it one, and score external sets in code

Browse source 
Enrolling a packshot from an external set is normal - it is how F-12 was
closed - and it silently turns one of those rows into training data. The
external sets were scored by an ad-hoc script that never checked, so
sku_l06_1.jpg was being reported as a held-out correct accept while being
byte-identical to the enrolled view.

code/eval_external.py now scores those sets with the same leaked() check the
shelf evaluator runs, and refuses a row that has since been enrolled.

That check needed fixing first. Its 64x64 signature fires on two different
photographs of the same product: the l09 rear label from a retailer page sits
19.5 from the enrolled front packshot, under the 20.0 line, while being a
different photograph (resized MAD 26.3). A signature hit is now confirmed by
correlation - a genuine re-cut, re-encode or rescale correlates at 0.99+, that
pair at 0.28 - so the two signals together are decisive where either alone is
not. Verified against both directions: identical, q55 re-encode and 700px
rescale are caught; the different-photo pair is not.

Also adds the van backend (always-argmax, no gate) so the stream comparison in
DECISION.md is reproducible rather than asserted, with two tests pinning that
it answers where the shipped gate abstains and still refuses an empty score set.

External result on the corrected basis: 19 scored, 12 correct, 7 abstain,
0 false accepts, 1 leak excluded. Suite 307 -> 309.
This commit is contained in:
kiyo9w 2026-09-29 03:29:56 +09:00
parent dfdaef4e61
commit e25fcda67d
9 changed files with 252 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

34
recognization/data/external-eval/other-mayo/eval_result.json
View file

@ -7,75 +7,77 @@
"file": "sku_l04_1.jpg",
"truth": "sku_l04",
"best": null,
"conf": 0.0,
"confidence": 0.0,
"status": "rejected",
"source": null,
"verdict": "ABSTAIN"
"verdict": "abstain"
},
{
"file": "sku_l04_2.jpg",
"truth": "sku_l04",
"best": null,
"conf": 0.0,
"confidence": 0.0,
"status": "unknown",
"source": null,
"verdict": "ABSTAIN"
"verdict": "abstain"
},
{
"file": "sku_l04_3.jpg",
"truth": "sku_l04",
"best": null,
"conf": 0.3294,
"confidence": 0.3294,
"status": "ambiguous",
"source": null,
"verdict": "ABSTAIN"
"verdict": "abstain"
},
{
"file": "sku_l04_4.jpg",
"truth": "sku_l04",
"best": null,
"conf": -1.6712,
"confidence": -1.6712,
"status": "unknown",
"source": null,
"verdict": "ABSTAIN"
"verdict": "abstain"
},
{
"file": "sku_l07_1.jpg",
"truth": "sku_l07",
"best": "sku_l07",
"conf": 0.0,
"confidence": 0.0,
"status": "success",
"source": "text",
"verdict": "OK"
"verdict": "correct"
},
{
"file": "sku_l07_2.jpg",
"truth": "sku_l07",
"best": "sku_l07",
"conf": -0.5503,
"confidence": -0.5503,
"status": "success",
"source": "text",
"verdict": "OK"
"verdict": "correct"
},
{
"file": "sku_l09_1.jpg",
"truth": "sku_l09",
"best": "sku_l09",
"conf": 27.9546,
"confidence": 27.9546,
"status": "success",
"source": "local",
"verdict": "OK"
"verdict": "correct"
},
{
"file": "sku_l09_2.jpg",
"truth": "sku_l09",
"best": "sku_l09",
"conf": 0.975,
"confidence": 0.975,
"status": "success",
"source": "text",
"verdict": "OK"
"verdict": "correct"
}
],
"scored": 8,
"leaked": 0,
"correct": 4,
"abstain": 4,
"wrong": 0