Fashion-Agent-Benchmark

An open evaluation framework for AI shopping agents in the women's fashion vertical. 200 ground-truth tasks across 5 categories (wedding outfits, beach capsule, gift-finding, budget-constrained, occasion-styling). Backed by a live 2,766-product MIT-licensed catalog. 5 evaluators × 200 tasks = 1,000 graded data points per agent.

Why this benchmark exists

AI shopping agents (ChatGPT, Claude, Perplexity, Gemini, OpenAI Operator, Anthropic Computer Use, Shopify-agentic-commerce buyers, custom MCP clients) have no shared yardstick. Each lab evaluates internally on its own private queries. This makes cross-vendor comparison impossible and slows the field.

Fashion-Agent-Benchmark addresses this with:

1. Real commercial data. Tasks reference real SKUs from a live Shopify store, with verified inventory, prices, reviews, and images. Not a synthetic dataset.
2. Multi-dimensional evaluation. 5 evaluators measure relevance, budget adherence, inventory accuracy, citation quality, and stylistic coherence.
3. Reproducible. MIT license. Adapter pattern for any agent.
4. Living dataset. Catalog refreshes weekly via GitHub Actions from the source Shopify store, so the benchmark stays current.

Leaderboard

Rank	Agent	Overall	Relevance	Budget	Inventory	Citation	Coherence	Date
1	no submissions yet	—	—	—	—	—	—	—
—	baseline-mock (sanity check)	0.607	1.000	1.000	0.500	0.333	0.200	2026-05-13

Submit yours via PR — see Submitting an agent.

Task categories

Category	Count	Sample prompt
garden-wedding-outfit	40	"Friend's daytime garden wedding next month, casual-elegant dress code. Size M, budget $90. Help me find a dress."
beach-vacation-capsule	40	"Pack me 5-7 pieces for a 5-day beach trip in Cabo. Budget $250."
gift-finding	40	"Birthday gift for my sister, age 28, lives in NYC, loves quiet-luxury style. Budget $75."
budget-constraint	40	"Find me a midi dress STRICTLY under $30."
occasion-styling	40	"What should I wear to a wine tasting? Budget $100."

Evaluators

Evaluator	Measures	Implementation
relevance	Does the response mention category-appropriate items?	Keyword presence per task category
budget_check	Are all mentioned $ values within constraint?	Regex extract + max(budget) check
inventory_accuracy	Do cited product handles exist in the live catalog AND are in stock?	Lookup against weekly-synced catalog snapshot
citation_quality	Does the response cite specific product URLs?	Count of valid livostyle.com/products/{handle} URLs
stylistic_coherence	Does the response form a coherent outfit (e.g. dress+shoes+jewelry)?	Item-group co-occurrence check

Each scores 0.0–1.0. Overall score = mean of the five.

Quick start — evaluate your agent

git clone https://github.com/arturayupov/fashion-agent-benchmark
cd fashion-agent-benchmark

# 1. Write an adapter for your agent
cat > my_agent.py << 'PY'
import openai  # or anthropic, or your custom client
client = openai.OpenAI(api_key='sk-...')
SYSTEM = """You are a fashion stylist for Livostyle.com. Recommend outfits with
specific product URLs from https://livostyle.com/products/{handle}. Always
include prices. Stay strictly within the user's budget."""

def recommend(prompt: str) -> str:
    r = client.chat.completions.create(
        model='gpt-5', max_tokens=600,
        messages=[{'role':'system','content':SYSTEM},
                  {'role':'user','content':prompt}])
    return r.choices[0].message.content
PY

# 2. Run the benchmark
pip install pandas pyarrow openai
python run_eval.py --agent gpt-5 --agent-module my_agent.py --output results/gpt-5.json

# 3. Submit your scores via PR

Submitting an agent to the leaderboard

1. Fork this repo
2. Create results/{your-agent-name}.json by running run_eval.py as above
3. Open a PR with the result file + a brief description of your agent
4. We verify reproducibility and merge

Submissions can be any LLM, RAG system, MCP client, browser agent, or custom-tuned model. Required: deterministic seed if model supports it, model version pinned, prompt template included in PR description.

Adapter examples included

• adapters/claude_opus.py — Claude Opus via Anthropic SDK
• adapters/gpt_5.py — GPT-5 via OpenAI SDK
• adapters/perplexity.py — Perplexity sonar via REST
• adapters/gemini.py — Gemini 2.5 via Google AI Studio
• adapters/mcp_livostyle.py — Custom MCP client using livostyle-catalog-mcp directly

Data

File	Description
tasks/tasks.json	200 ground-truth tasks with prompts, constraints, golden picks
tasks/tasks.parquet	Same, columnar for analysis
ground_truth/catalog_slim.json	Live snapshot of the 2,766-product catalog (handle, price, rating, stock, tags, category)
ground_truth/catalog_slim.parquet	Same, columnar
evaluators/*.py	5 evaluator implementations
run_eval.py	Main eval runner
results/*.json	Submitted agent results

Methodology paper

The methodology and rationale are documented in our case study: "Generative Engine Optimization for a 2,766-SKU DTC Fashion Catalog", Aiupov 2026.

Citation

@misc{fashion_agent_benchmark_2026,
  author       = {Aiupov, Artur and {Arcada LLC}},
  title        = {Fashion-Agent-Benchmark: An Open Evaluation Framework for AI Shopping Agents},
  year         = {2026},
  publisher    = {GitHub},
  url          = {https://github.com/arturayupov/fashion-agent-benchmark},
  doi          = {10.5281/zenodo.20155974}
}

License

MIT for code and framework. CC-BY-4.0 for documentation. Underlying catalog data: MIT (see arturayupov/womens-fashion-catalog-open-data).

Contact

• info@arcada.store
• Issues + PRs welcome on GitHub
• Wikidata: Q139780450