Vision AI for Ecommerce: Tagging, Search, and QC That Actually Ship

Your product photos already contain most of the data your catalogue is missing.

Colour, material, pattern, neckline, the fact that the box contains three of something rather than one. All of it sits in images that until recently were opaque to every system you own.

Multimodal models changed that, and the ecommerce use cases are unusually well-defined: an image goes in, structured data comes out, and the data has a schema you already own. This post covers six workflows that work today, what each costs, and the validation that separates a demo from something you would let write to a live store.

Who This Is For

• Store owners and merchandisers with thousands of SKUs and thin product data
• Ecommerce developers building against Shopify or WooCommerce APIs
• Ops leads who handle returns, review moderation, or marketplace compliance
• Agencies running catalogues for multiple clients

What You Will Need

• Product images at a reasonable resolution, ideally the originals rather than thumbnails
• Your attribute taxonomy. Shopify’s Standard Product Taxonomy publishes around 10,000 categories and 1,000 attributes as data files; Google’s product taxonomy works too. This matters more than the model choice.
• API access to write back: Shopify Admin GraphQL, WooCommerce REST, or your PIM
• A hand-labelled evaluation set, 100 to 300 SKUs per category. Without it you are guessing.

That last item is where most projects skip a step and pay for it later.

The Pattern

Every workflow below is the same six stages. The interesting engineering is in stages three and five, not in the model call.

flowchart TD
  I[Product image] --> M[Vision model + JSON schema]
  M --> V[Validate against taxonomy]
  V --> C{Confidence}
  C -->|High| W[Write to store as suggested]
  C -->|Low| Q[Review queue]
  Q --> H[Human decides]
  H --> F[Edit becomes a few-shot example]
  W --> L[Live after approval]

Two design choices carry the whole thing. Constrain the output to your taxonomy rather than accepting free text, so every value is checkable. And ask the model for its evidence, not just its answer, so a reviewer can see whether “100% cotton” came from a visible label or from the model’s impression of the texture.

Shopify runs this shape at a scale worth knowing about. Their engineering team published the progression from LLaVA to Llama 3.2 to Qwen2-VL 7B, producing more than 30 million classification and attribute predictions a day against their open taxonomy. The lesson is not the model name, which will change. It is that they treat classification as a taxonomy problem with a model attached.

1. Catalogue Tagging and Attribute Extraction

The highest-value workflow, because thin attributes suppress you everywhere at once: on-site filters, Google Shopping, and now the agentic surfaces where ChatGPT and Gemini answer shopping questions from your feed.

Amazon reports its generative listing tools fill more than 70 percent of required product attributes, with sellers seeing a 40 percent lift in listing quality scores. Akeneo’s PIM extracts colours, materials, dimensions, and certifications from images and PDFs. Neither publishes per-attribute precision, which tells you what to do: measure your own.

The technique that makes the difference is enum constraint. Give the model the allowed values for that category and an explicit escape hatch.

from anthropic import Anthropic
import base64, json

client = Anthropic()

def attribute_schema(allowed: dict) -> dict:
  """Build a per-category schema from your taxonomy's allowed values."""
  props = {}
  for name, values in allowed.items():
    props[name] = {
      "type": "object",
      "properties": {
        # "unknown" gives the model an out instead of a plausible guess
        "value": {"type": "string", "enum": [*values, "unknown"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1},
        "evidence": {"type": "string", "enum": ["visible_label", "clearly_visible", "inferred"]},
      },
      "required": ["value", "confidence", "evidence"],
    }
  return {"type": "object", "properties": props, "required": list(props)}


def extract(image_path: str, title: str, allowed: dict) -> dict:
  image = base64.standard_b64encode(open(image_path, "rb").read()).decode()
  response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    system=(
      "Extract product attributes from the image. Use 'unknown' when the image "
      "does not show it. Never infer fibre content from texture alone."
    ),
    messages=[{
      "role": "user",
      "content": [
        # Image before text: the documented ordering for best results
        {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": image}},
        {"type": "text", "content": f"Product title: {title}"},
      ],
    }],
    output_config={"format": {"type": "json_schema", "schema": attribute_schema(allowed)}},
  )
  return json.loads(response.content[0].text)


def route(result: dict) -> tuple[dict, dict]:
  """Split into auto-write and review-queue halves."""
  auto, review = {}, {}
  for field, data in result.items():
    if data["value"] == "unknown":
      continue
    target = auto if data["confidence"] >= 0.8 and data["evidence"] != "inferred" else review
    target[field] = data
  return auto, review

The evidence field does more work than the confidence score. Self-reported confidence is poorly calibrated across every model family; “did you actually see this or are you guessing” is a question models answer more honestly, and it gives your reviewer something to check.

Write results as suggested values first. On Shopify that means category metafields; the promotion to live happens after review. Bulk writes go through the same cost-aware GraphQL patterns as any other catalogue job, which cost limits and bulk operations covers in detail.

2. Descriptions and Alt Text

Two outputs from one call, with very different risk profiles.

Descriptions are marketing copy and the failure mode is embarrassment. Alt text is an accessibility obligation and the failure mode is legal. The European Accessibility Act has applied since June 2025, and EN 301 549 pulls in WCAG 2.1 AA, which means meaningful alt text on product images. In the US there were more than five thousand digital accessibility lawsuits in 2025, roughly 70 percent of them against ecommerce.

The rule that keeps descriptions honest: generate from the attributes, not from the image. Run extraction first, review it, then write copy from the approved attribute set with the image as context only. A model looking at a photo will happily describe brushed cotton that is actually polyester. A model working from a reviewed attribute list cannot.

Alt text is different because it describes what is visible rather than what is true about the product. Keep it under 125 characters, lead with the product, include the view: “Navy cotton crew-neck t-shirt, front view, flat lay.” Decorative images get empty alt, not a description.

If you already generate copy in bulk, the vision step slots in ahead of it; generating product descriptions as a bulk pipeline covers the batching, rate limits, and rollback that any catalogue-wide write needs.

3. Visual Search

Photo in, matching SKUs out. The consumer behaviour is already there: Google Lens handles around 20 billion visual searches a month with roughly a fifth carrying shopping intent, and Pinterest’s Lens does about 1.5 billion visual queries a month.

The architecture is not a language model at all. It is embeddings.

flowchart LR
  P[Query photo] --> S[Segment the item]
  S --> E[Image embedding]
  E --> A[ANN search over catalogue]
  A --> R[Rerank with stock and price]
  R --> K[SKU results]

Every catalogue image becomes a vector, the query photo becomes a vector, and you retrieve nearest neighbours. Two details separate a working implementation from a frustrating one.

Segment before you embed. A lifestyle shot embeds the model, the beach, and the bag together. Run a promptable segmenter such as SAM 3 with the noun phrase for the category, crop to the product, then embed. Your vector then describes the thing you sell.

Use ecommerce-tuned embeddings. Generic CLIP is a weak product retriever. Marqo’s ecommerce models report roughly 18 percent better MRR than a strong SigLIP baseline and far more against older multimodal embeddings, and their fashion variant claims up to 57 percent over FashionCLIP. Then rerank with the things a vector does not know: stock, size availability, margin.

Be sceptical of the lift figures. Visual search vendors quote conversion improvements from 27 percent to 177 percent, all vendor-published, none from a controlled independent test. Run it as an A/B on your own traffic before you believe any number.

4. Image Quality Control and Compliance

The workflow that pays for itself quietly, because rejected listings cost sales invisibly.

Marketplace image rules are specific and mechanical: Amazon’s main image wants pure white at 255,255,255 with the product filling at least 85 percent of the frame and no accessories that are not in the box. Google Merchant Center wants at least 500 by 500 pixels, no watermarks, logos, text or price overlays, and a unique image per variant.

Those are rubrics, and rubrics are exactly what a vision model is good at.

RUBRIC = """Check this product image against marketplace rules and return JSON:
{"white_background": bool, "product_fills_85_percent": bool, "has_text_overlay": bool,
 "has_watermark": bool, "extra_props_present": bool, "verdict": "pass"|"fail"|"unsure",
 "reasons": [str]}
Judge only what is visible. Use "unsure" when the image is ambiguous."""

Run a cheap classifier first for anything involving safety, then the rubric call, then a three-way split: auto-approve clear passes, auto-reject clear failures, and queue everything marked unsure. Zalando published a production multimodal LLM judge for search quality that matches human annotators at roughly a thousandth of the cost, evaluating twenty thousand pairs in about twenty minutes. The pattern generalises.

Two hard limits to design around. Models cannot tell you whether an image is AI-generated; the documentation says so explicitly. And counting is approximate, so a rubric asking “how many items in this multipack” will be wrong often enough to matter. Use metadata and provenance for the first, and a human for the second.

5. Returns and Damage Triage

Customer uploads a photo of a damaged item. The model extracts what is visible; your rules decide what happens.

Intercom’s Fin handles this in production: it converts images into structured text within the conversation, evaluates return eligibility against the merchant’s own procedure, and asks for a better photo when the image is unusable. Amazon’s Project P.I. runs the industrial version, with imaging tunnels checking millions of items a month for damage, wrong colour, and wrong size before they ship.

The design rule is the same as everywhere else in agent work: extraction is automatic, the decision is a rule, and the money has a threshold. Under a value ceiling with clear damage, refund automatically. Above it, or when the photo is ambiguous, escalate with the extraction attached so the human starts from a summary rather than a photo.

One new fraud vector deserves a mention: AI-generated damage photos. Layer your defences with reverse-image matching, EXIF and provenance checks, and repeat-claimant signals, because the model itself cannot detect synthetic imagery.

6. Product Photography and Try-On

The generative half, and the half where the legal surface is real.

Background removal and scene generation genuinely work. Zalando cut imagery production from six to eight weeks down to three to four days with roughly a 90 percent cost reduction. Open tooling is mature: BiRefNet under MIT handles hair and thin straps well, and rembg wraps it behind a one-line CLI.

Virtual try-on is more marketing than measurement. Google shut its standalone try-on app less than a year after launching it and folded the capability into Search. IKEA’s room visualiser has published no engagement metrics. Wayfair’s first attempt warped architecture and invented doors. Every “20 to 40 percent fewer returns” figure I could find traces back to a vendor selling try-on. Treat those as hypotheses, not results.

If you do generate images, three non-negotiables:

• Keep the provenance metadata. Google Merchant Center requires AI-generated images to retain their IPTC DigitalSourceType tags and treats stripping them as a violation.
• Disclose where required. Amazon asks for a keyword marking photorealistic synthetic people, Etsy requires disclosure in the listing, and EU AI Act transparency rules applying from August 2026 require machine-readable marking of synthetic output.
• Never generate the product itself. Change the background, the lighting, the scene. The moment a generated image adds an accessory or alters the item, you have a returns problem and a “not as described” problem.

Costs

Image understanding is cheap. Image generation is not.

Batch APIs halve the extraction numbers, and fashion catalogues carry four to six images per SKU, so multiply accordingly. The practical consequence: run a cheap model over everything and an expensive one only over what the cheap model flags. A small local model such as Florence-2 or Moondream can do the first pass at effectively zero marginal cost if you have a GPU or a recent Mac, which is the argument in running open models locally.

Failure Modes to Avoid

• Free-text tags instead of taxonomy values. “Dark blue”, “navy”, and “midnight” are three filters nobody can use. Constrain to enums or do not bother.
• Trusting fibre content read off a photo. Materials are the most-hallucinated attribute because texture is genuinely ambiguous. Require a visible label or mark it unknown.
• Counting from images. Pack sizes, piece counts, and set contents are exactly where vision models are documented to be approximate.
• Writing straight to live. Suggested, reviewed, live. A wrong attribute now propagates to Google Shopping and to every AI shopping assistant reading your feed.
• Stripping metadata to pass a check. It is a policy violation on Google, and from August 2026 a regulatory one in the EU.
• Building on a single vendor’s product API. Google deprecated Vertex AI Vision with an end of life in September 2026, Rekognition closed batch moderation to new customers, and Google killed its try-on app. Put an adapter in front of whatever you choose.
• Skipping the eval set. Per-attribute precision and recall on 200 labelled SKUs per category, re-run on every prompt and model change. Overall accuracy hides the one attribute that is wrong half the time.

Before and After

What to Build First

1. Export 200 SKUs from your worst-covered category and hand-label their attributes. This is the boring step that makes everything after it measurable.
2. Build the schema from your taxonomy’s allowed values, with unknown included.
3. Run a cheap model over the 200 and score per-attribute precision. Anything below 90 percent stays in the review queue permanently.
4. Wire the write path as suggested values, never straight to live.
5. Add the compliance rubric next; it is the same call shape and it prevents a different kind of loss.
6. Only then look at visual search or generated imagery, both of which are bigger projects with softer evidence.

Final Take

Vision AI in ecommerce is not one capability, it is a set of small, checkable jobs that happen to share an input.

The ones that pay reliably are unglamorous: attributes filled in, alt text written, images checked before they are rejected, damage claims summarised. The ones with the loudest marketing, try-on and generated lifestyle scenes, have the least published evidence and the most regulatory surface.

Start where the output has a schema and the schema has a right answer. That is where a model can be measured, and anything you can measure, you can trust.