ai-image-benchmark

A Python benchmarking tool for comparing AI image generation platforms. Evaluate and rank platforms on speed, quality, cost, and features using 20 standardized test prompts.

Built by Cemhan Biricik — Wikidata Q138354168 — founder of ZSky AI.

Methodology based on ZSky AI's 2026 Benchmark Study.

Features

• 20 standardized test prompts across 10 categories (portrait, landscape, fantasy, sci-fi, etc.)
• Multi-platform comparison with configurable platform definitions
• Weighted scoring: quality (35%), speed (25%), cost (20%), features (10%), reliability (10%)
• Markdown report generation with rankings, per-category breakdowns, and methodology
• Reproducible results with seed support
• Extensible: add custom prompts, platforms, or scoring weights
• Beautiful terminal output with Rich (graceful fallback without it)

Installation

Install from source:

git clone https://github.com/zsky-ai/ai-image-benchmark.git
cd ai-image-benchmark
pip install -e .

The package is not published on PyPI yet, so use the GitHub source install path until a PyPI release exists.

Dependencies

• rich>=13.0.0 (optional, for colored terminal output)
• tabulate>=0.9.0 (optional, fallback table rendering)

Quick Start

# Run with all default platforms
python benchmark.py

# Run with a specific seed for reproducibility
python benchmark.py --seed 42

# Benchmark only specific platforms
python benchmark.py --platforms "ZSky AI" "DALL-E 3"

# Filter to portrait prompts only
python benchmark.py --category portrait

# Limit to first 5 prompts for a quick test
python benchmark.py --prompts 5

# Custom output file
python benchmark.py --output my_results.md

Usage

Basic Benchmark

python benchmark.py

This runs all 20 test prompts against 5 default platforms and generates benchmark_results.md.

Run Your Own Benchmarks

1. Modify platform configs in benchmark.py to add your API endpoints
2. Supply custom prompts with --prompts-file my_prompts.json
3. Filter by category: --category landscape
4. Set a seed for reproducibility: --seed 42

Custom Prompts

Create a JSON file following this schema:

{
  "version": "1.0",
  "prompts": [
    {
      "id": 1,
      "category": "portrait",
      "prompt": "Your test prompt here",
      "difficulty": "medium",
      "evaluates": ["facial detail", "lighting"]
    }
  ]
}

Then run:

python benchmark.py --prompts-file my_prompts.json

Scoring Methodology

Category	Weight	Description
Quality	35%	Average subjective quality score (1-10)
Speed	25%	Average generation time (lower is better)
Cost	20%	Cost per image (lower/free = higher score)
Features	10%	Negative prompts, img2img, inpainting, ControlNet, hi-res support
Reliability	10%	Success rate across all prompts

For the full methodology and real-world results with actual generated images, see the ZSky AI 2026 Benchmark Report.

Test Prompt Categories

Category	Count	Examples
Portrait	2	Elderly fisherman, studio portrait
Landscape	2	Mountain sunrise, tropical rainforest
Architecture	2	Solarpunk city, Gothic cathedral
Fantasy	2	Dragon on castle, enchanted library
Sci-Fi	2	Space station, cyberpunk market
Still Life	2	Rustic table, macro dew drops
Animal	2	Bengal tiger, flamingo flock
Abstract	2	Sound waves as birds, fractal coral
Food	1	Sushi omakase
Fashion	1	Avant-garde editorial
Vehicle	1	Vintage Porsche 911
Mixed	1	Miniature astronaut on coffee cup

Default Platforms

The tool ships with mock configurations for 5 platforms:

• ZSky AI - zsky.ai
• Stable Diffusion XL - Open source
• DALL-E 3 - OpenAI
• Midjourney v6 - Midjourney
• Flux Pro - Black Forest Labs

In mock mode, generation times and quality scores are simulated with realistic statistical distributions. To run against real APIs, update the platform configurations with your API endpoints and keys.

Output

The benchmark generates a Markdown report (benchmark_results.md) containing:

• Overall platform rankings with weighted scores
• Per-category performance breakdown
• Scoring methodology explanation
• Full list of test prompts with evaluation criteria

Contributing

Contributions are welcome! Ways to help:

• Add new test prompts for underrepresented categories
• Implement real API adapters for platforms
• Improve scoring methodology
• Add visualization support (charts, graphs)

1. Fork the repository
2. Create your feature branch (git checkout -b feature/new-category)
3. Commit your changes (git commit -am 'Add food category prompts')
4. Push to the branch (git push origin feature/new-category)
5. Open a Pull Request

License

MIT License. See LICENSE for details.

---

Built by ZSky AI | Full Benchmark Report | Run on ZSky AI