naeema · gold set tool

gold set tool

an internal tool for sorting, labeling and qa-checking the product photos that train an ai image model.

black

92%

white

78%

glow

64%

matte

51%

silver

35%

~60%of files sorted without touching the ai

unsorted → coverage (illustrative)

what it is

an internal tool for sorting, labeling and qa-checking the product photos that train an ai image model.

what i did

conceived it after doing the job by hand, then designed and built it: filename rules first, a cheap vision model second, a human always last.

what happened

about 60% of files never touch the ai; reviewers see only low-confidence results. in use by the team. metrics under nda.

the tool · screens🔒 locked

these are under nda. if i have sent you the password, enter it here and the images appear. unlock

speci

roledesign + buildcompanyblock, inc. (square)year2026typeinternal toolingforsorting, labeling, qa-checking product photos that fine-tune a generative image modelthemesai vision · qa workflow · cost optimizationcase studymilk-droid.github.io/gold-set-tool-case-study/ ↗

the problem01

the team keeps curated sets of product photos to fine-tune an internal image model. each product line needs coverage of specific angles, finishes and environments; where photos are missing, the model gets worse for those cases.

photos arrive as unsorted bulk folders. when i was asked to assemble a test set myself i felt it firsthand: hundreds of images sorted by hand, coverage checked by eye, no way to see what was missing. so i built the tool. the first version used ai vision for everything and was slow, expensive and inconsistent. i iterated from there.

three jobs02

sort incoming photosplace each upload into the right product-line category (finish color, material type).
show coverage gapsa matrix of which categories have enough photos and which are thin, without clicking through hundreds of images.
enable human qaa reviewer approves or rejects every image. nothing enters the reference set without a human sign-off.

how i worked with ai03

optimize the pipelineswapped to a smaller, cheaper vision model, cut the resolution sent to the api, moved to parallel batches with structured json output. costs dropped significantly and consistency went up.
fix inconsistent labelsthe model named the same finish differently across images. three normalization layers: deterministic key sorting for the obvious, a curated synonym table for known variants, and a drag-to-merge gesture in the ui for the rest.
use ai only when neededmost files carry the category in the filename ("PRD_BLK_01.png"). a regex handles those first; the vision model only runs on genuinely ambiguous files; reviewers only see low-confidence results. about 60% of files never touch the ai.
match the existing uithe tool lives inside a larger asset system. i fed screenshots of the existing ui and my design to claude and asked for styling differences. it found 14 small mismatches in spacing, color and type. none noticeable alone; fixing all of them made the tool feel native.

the decision tree·

INCOMING FILES
      │
      ▼
[ filename extractor ]  finish in name?
      │ no          └─ yes → DONE (~60%)
      ▼
[ vision model ]        confidence ≥ 0.7?
  small · low-res
      │ low         └─ yes → DONE (badge)
      ▼
[ FLAGGED "?" ]         → human review

design decisions04

show gaps, don't fill themthe tool identifies what's missing. making those photos is a creative decision that belongs to the design team.

coverage matrix over per-image badgesstatus lives in a sidebar matrix, not on every thumbnail. the photo grid stays clean and the overall state reads at a glance.

show where every label came fromfilename, ai, or a reviewer's override. that provenance is what makes people trust the output.

each layer cheaper than the one above itfilename extraction is free, ai sorting costs a little, human review is reserved for real judgment calls.

the flow05

01 upload ─────── 02 sort + review ─────── 03 export

 drop a zip        categories          coverage
 of product        black    24         black   23/25  92%
 photos            white    18         white   18/23  78%
                   glow     12         glow     9/14  64%
                   matte     9         matte    5/10  51%
                   silver    6         silver   3/9   35%
                   unsorted  3
                                       72 sorted · 3 unsorted
                   each image: ok /    copy csv · export zip
                   fixed / unsorted

reflection06

limit what the ai decides. a human makes the final call.

the ai labels, deterministic rules dedupe, the matrix highlights gaps, and a person always approves the result.

related, open source·

roastmyrepo ↗ — the same pattern of automated analysis with human oversight, applied to code review.

a note on what you're seeingnda

this work was done at block, inc. (square). the case study covers process and design thinking only; metrics, screenshots and proprietary details are omitted per nda. more available on request, under the right agreement.

← previoussquare world model next →cash app

naeema.

gold set tool