Benchmarks¶
How well do different agent harnesses, models and voices make videos with manimvox? Each benchmark run gives one prompt to one agent, records what the agent did and what it cost, and measures the result. Voices are compared on the same episode, so only the voice changes.
Every run's records are in benchmarks/runs/: the agent's project, its session summary, the per-line logs and the scores. You can check every number below, or run the benchmark yourself.
Runs¶
| Run | Date | What it compares |
|---|---|---|
| UPI payments: three voices | 2026-10-23 | Sarvam AI, Gnani.ai and ElevenLabs, on one episode made by Claude Code with Sonnet 5.5 |
Leaderboard¶
Every run across all examples. Each gallery example with runs also shows its own leaderboard on its page. Compare rows from different examples with care: the prompts differ.
Voice¶
Ranked by the listener's order within each run, then word error rate, then crest factor (higher is less compressed, cleaner audio). Each voice read the same lines as the others in its run.
| # | Voice | Word errors | Words a minute | Crest factor | Time per line | Cost | Listener | Run |
|---|---|---|---|---|---|---|---|---|
| 1 | Gnani.ai, Timbre v3.0, Avinashgnani:timbre-v3.0/Avinash@en-IN |
0.6% | 171 | 20.2 dB | 3.97 s | $0.029 | the most realistic; clear, but slow | 2026-10-23 |
| 2 | ElevenLabs, Eleven v3, Brianelevenlabs:eleven_v3/nPczCjzI2devNBz1zQrb |
0.0% | 169 | 18.0 dB | 2.96 s | $0.18 | the best pacing; less noise and distortion than Sarvam; sounds AI-generated | 2026-10-23 |
| 3 | Sarvam AI, Bulbul v3, shubhsarvam:shubh |
2.2% | 215 | 13.5 dB | 1.36 s | $0.029 | sounds AI-generated; not clear, background noise; good pace for a short video | 2026-10-23 |
Harness¶
Ranked by result, then cost, then time.
| # | Harness | Model | Example | Result | Time | Turns | Cost | Run |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Code 2.1.294 | claude-sonnet-5-5 |
How UPI payments work | video made | 3 min 3 s | 21 | $0.40 | 2026-10-23 |
Model¶
Every run with each model, ranked by the share that made a video, then mean cost.
| # | Model | Runs | Videos made | Mean cost | Mean time | Harnesses |
|---|---|---|---|---|---|---|
| 1 | claude-sonnet-5-5 |
1 | 100% | $0.40 | 3 min 3 s | Claude Code |
Each example's own leaderboard is on its gallery page: How UPI payments work.
Methodology¶
What a run controls¶
- One prompt and one agent. The agent starts in an empty folder, with manimvox and its skills installed and no other instructions. It runs headless, so it can't ask questions, and the prompt states the permissions it would otherwise ask for (web access, the voice).
- One episode for every voice. The agent makes the episode with the default offline voice. Its project is then copied once per voice and rendered with
manimvox render --voice <voice>, unchanged. The script, the pictures, the speed (1.0) and the pronunciation list are the same, so only the voice differs. - Each provider's newest general model, matched on language and gender. Other settings use the provider's defaults.
- A cold cache. No line is synthesised before the measured render.
What is recorded¶
- The agent session: the harness's own JSON transcript (
claude -p --output-format stream-json) gives the model, version, time, turns, tokens and cost. That cost is the harness's list-price figure. - Every narration line: a small
sitecustomize.pyonPYTHONPATH(benchmarks/harness) wraps each voice engine'ssynthduring the render. It logs the text, the wall time of the call, the number of HTTP attempts and any error, and keeps a copy of the audio the service returned. manimvox itself runs unchanged.
What is measured¶
Every score comes from the returned audio or the log, computed by benchmarks/voice_score.py.
| Criterion | How it is computed |
|---|---|
| Reliability | Lines that failed, HTTP attempts (one per line means no retries), and manimvox's own audio check after the render. |
| Latency | Wall time of each synth call, including network and retries: mean, median and maximum. The real-time factor is that time divided by the length of the audio. |
| Cost | Characters sent, multiplied by the provider's published per-character price on the day of the run. If there is no published price, the charge shown in the account's usage dashboard. A service that bills in credits is converted at the price per credit of its cheapest plan that allows commercial use, and the page names the plan. Every cost is shown in US dollars; another currency is converted at the mid-market rate on the day of the run, and the run's page gives the rate. |
| Intelligibility | Each line is transcribed by a local speech recogniser (faster-whisper small.en, int8, beam 5) and compared with the text sent, as a word error rate. Both sides are normalised first: lower case, no punctuation or apostrophes, numbers as words, and acronyms joined whether written or spelt out (NPCI, N P C I and N-P-C-I all match). Every remaining error is listed on the run's page and checked by hand. |
| Pace | Words divided by speaking time (each line's audio minus its leading and trailing silence). manimvox's skills aim for 150 to 175 words a minute. |
| Pauses | Silences of 0.35 s or more inside a line, where silence is a 20 ms frame below −40 dBFS. |
| Loudness | RMS level over all lines, its spread between lines, the peak level, and clipped samples, measured on the audio as the service returned it. manimvox then levels every line to loudness in series.json (−16 LUFS by default), so the finished video's loudness is the same whichever voice is used. |
| Noise and distortion | The noise floor (each line's 10th-percentile frame level) and how far the speech sits above it; the crest factor (peak over RMS; a low value means heavy compression, which sounds harsh); and the final video's peak and clipping, to tell the service's audio from manimvox's mix. Two more numbers describe the sound without ranking it: the energy above 7 kHz (brightness, from consonants, breath or hiss) and the spectral flatness of each line's quietest frames. Both are high for a crisp, breathy voice as well as for a hissy one. |
| Intonation | Median pitch and its spread in semitones over voiced frames (librosa pYIN). This is a rough measure of how much the pitch varies, not of whether it sounds right. |
| Listening | Notes by a person who watched every video, and their order of the voices, which the leaderboard ranks on first: clarity, noise, pace, accent and pronunciation. Measurements can't replace this. In the first run, the listener heard noise the word error rate missed, and found the voice with the best measurements synthetic. |
| Picture | Frames taken at the same narration line in each video are compared, to check that the animation still follows the voice. |
Limitations¶
- One run is one sample. Network time varies, and both renders ran at the same time on one machine. Treat latency differences under a factor of two as noise.
- Speech recognition is a proxy. A recogniser trained mostly on American and British English can mishear a correct Indian-English pronunciation, and can guess right through a wrong one. The errors are listed so you can listen to them yourself.
- One voice per provider. Each service has many voices and models. A run compares the two voices it names, not the providers.
- The agent's video was reviewed, not scored. Model and harness comparisons need several prompts and several runs each. The table above starts that record.
Run a benchmark¶
The steps are in benchmarks/README.md. Results for another harness, model or voice are welcome as a pull request that adds a folder to benchmarks/runs/.