Question 1

How are these AI benchmarks actually scored?

Accepted Answer

Each benchmark uses the exact same prompt across every tested model, runs 3-5 times with temperature 0.7 (where configurable), and scores each output on 5 task-specific dimensions (1-10 scale). We publish the raw prompt, per-model notes, cost per run, and a final verdict. Editorial-grade, not academic lab-grade.

Question 2

Why do you also publish a "price-performance" winner?

Accepted Answer

Because benchmark leaderboards without cost context mislead. Midjourney v6.1 wins on cinematic portraits — but it's 4¢/image. Gemini Imagen 3 is free and scores 7-8/10 on the same task. For a creator running 1,000 generations a month, the free runner-up matters more than the paid winner. Both get called out.

Question 3

How often are benchmarks re-run?

Accepted Answer

When a model gets a major version bump (Midjourney v6 → v7, Claude 3.5 → 4, Gemini 1.5 → 2.0). We also re-run quarterly against the current best-in-class so the leaderboard doesn't go stale. The "Last run" date on each benchmark page shows when it was last tested.

Question 4

Can I reproduce these benchmarks myself?

Accepted Answer

Yes — that's the point of publishing the exact prompt. Every benchmark page has a copy-paste prompt. Run it on the same models with temperature 0.7 and you'll get within one point of our scores on most dimensions. For deterministic workloads (code generation), you'll get nearly identical results.

Question 5

Why Llama 3.3 70B over other open-source models?

Accepted Answer

For the Prompt Optimizer backend, Llama 3.3 70B via Cloudflare Workers AI hits the sweet spot: free tier is generous (10k neurons/day = ~3,000-6,000 optimizations), latency is 2-8s per call, and output quality on structured JSON tasks is on par with GPT-4o-mini. If CF Workers AI hits a quota, we fall back to Gemini 2.0 Flash free tier.

Question 6

Will you add more benchmarks?

Accepted Answer

Yes — current coverage is 4 LLM, 5 image, 1 video. Planned additions: code generation (Python, TypeScript, Rust), long-document reasoning, tool-use agents, multilingual translation quality, audio generation (ElevenLabs vs Play.ai vs OpenAI TTS), and image editing (Photoshop Firefly vs Krea vs DALL-E inpaint).

AI Model Benchmarks

LLM / Text (4 benchmarks)

Coding — Python function

Writing — Blog opener

Analysis — PDF summary

Writing — Short-form

Image Generation (5 benchmarks)

Image — Cinematic portrait

Image — Anime

Image — Logo mark

Image — Product photo

Image — YouTube thumbnail

Video Generation (1 benchmarks)

Video — 5s text-to-video

Why these benchmarks matter

Frequently asked questions

Related features

Prompt Search

Prompt Playground

AI Workflows