要点
- Property Finder found 96.8% of eval labels inherited from the incumbent vision model’s predictions—structurally disadvantaging alternatives.
- After expert re-labeling of 356 UAE listing images, Llama 4 Maverick 17B reached Micro F1 0.80 with tuned prompts and ~86% lower estimated cost.
- Method: independent labels, multi-label outputs, Micro/Macro F1, and prompt search treated as a scored optimization loop through v27.
A failed model swap is not always a weaker challenger. Property Finder’s joint write-up with the AWS Generative AI Innovation Center found 96.8% of evaluation labels came from the incumbent vision model’s own predictions—so the answer key was “how much like the incumbent.”
Thesis: for vision QA migrations, the scarce asset is unbiased human labels. Prompt optimization only matters after the judge stops being captured by the incumbent.
[1]The MENA real-estate app scores about 11 quality attributes per listing image. Moving to a cheaper Bedrock option looked impossible against old labels. Experts freshly annotated 356 UAE images (256 for development; full set for final compare), with an Egypt set as a regional check. They switched to multi-label outputs and used Micro F1 as primary metric, with Macro F1 and per-label precision/recall for rare classes.
[1]Prompts iterated from v0 to v27, later with MIPROv2-style automation. On the 356-image eval set, Llama 4 Maverick 17B Instruct reached Micro F1 0.80 versus 0.56 for proprietary model A under its tuned prompt; on the development set the lift versus model A was reported as 0.55→0.81. Some attributes hit high exact-match (rotation 100%, watermark 94%); room size lagged at 65%. Estimated cost ~$2.35 per 1,000 images (~86% lower) and 40–60% lower latency in their monitoring.
[1]Steelman: n=356 is thin for rare classes; single-expert labels carry subjectivity; cost figures are regional estimates in a co-authored AWS post.
Under-disclosed: identity of proprietary model A, full confusion matrices, Egypt results. Confidence high on the bias mechanism; medium on portable exact F1 deltas.
[1]Any team treating production-model self-labels as gold is running a circular eval. As model catalogs widen, a small unbiased gold set becomes the asset that decides whether cheaper models are even visible.
[1]Watch whether teams drop “agreement with incumbent” as a primary metric, and whether prompt search still helps after the gold set is refreshed. Prompt-only gains on a captured key can look like progress while fitting old bias.
[1]