Run benchmarks
Measure your models on your hardware so the router has real numbers. Five commands, two flavours, one JSON output the router picks up automatically.
Two modes
| Mode | Command | Time per model | When |
|---|---|---|---|
| Quick | python -m benchmarks --quick | ~4 minutes | After first install, after pulling a new model |
| Full | python -m benchmarks | ~15 minutes | Before relying on routing for important workloads |
Targeted runs
# single model, all categories
python -m benchmarks --model qwen2.5-coder:7b
# single category, all installed models
python -m benchmarks --category reasoning
# combine
python -m benchmarks --model deepseek-r1:7b --category code_generation
Where the results land
Results go to benchmarks/results/capability_map.json. The router reads this on every prompt, so changes take effect immediately on the next request.
Gap analysis
After a full run, see where your stack has holes:
python -m benchmarks --gaps
Output lists categories where no model scores above 0.7 and suggests models to install.
History
python -m benchmarks --history
Show previous runs, useful for catching regressions after a model upgrade.
Benchmarks load and unload models, which means high RAM pressure. On 16 GB Apple Silicon plan for the machine being effectively unavailable during a full sweep.