The capability map

Soriku doesn't trust marketing benchmarks. The capability map is a measured, per-model, per-category scorecard built on your hardware so the router has real numbers to work with.

What it is

A JSON file at benchmarks/results/capability_map.json that records how every installed model scored on each capability category. Categories that matter: code generation, code review, reasoning, summarisation, translation, creative writing, instruction following, multilingual, tool use.

The router reads this file on every prompt to decide which model is the best fit for the task. No marketing numbers, no third-party leaderboard, just measurements on your own hardware.

How it gets built

Each category has a dataset of test prompts with reference answers in benchmarks/datasets/<category>.json. The runner sends each prompt to the model, captures the response, and scores it against the reference with a multi-judge approach. The aggregate per category becomes the score.

# full sweep across all installed models
python -m benchmarks
 
# quick mode: 20% of the dataset
python -m benchmarks --quick
 
# single model
python -m benchmarks --model qwen2.5-coder:7b
 
# single category
python -m benchmarks --category reasoning

Scoring

Per-test scores run from 0.0 (fail) to 1.0 (pass). Category scores are the average across the dataset, weighted by judge consensus. Two metrics travel with every score: confidence and source.

Field	Type	Meaning
`score`	0.0–1.0	Aggregate quality for this model in this category
`confidence`	`high` \| `low`	`high` = locally benchmarked, `low` = bootstrap from public priors
`source`	`benchmark` \| `bootstrap`	Where the score came from
`entries_run`	integer	How many test cases were evaluated
`avg_latency_ms`	integer	Average response time during the benchmark
`tokens_per_second`	float	Throughput observed during the run
`timestamp`	ISO 8601	When this category was last benchmarked for this model

Bootstrap vs benchmark

A fresh install can route immediately because every known model has bootstrap scores derived from public evaluations. These are marked confidence: low so you know they're priors, not measurements. When you run the benchmark locally the values get replaced with confidence: high.

The router prefers high-confidence scores when both exist for the same model and category. Bootstrap fills gaps, it doesn't override your measurements.

Example entry

{
  "models": {
    "qwen2.5-coder:7b": {
      "code_generation": {
        "score": 0.812,
        "confidence": "high",
        "source": "benchmark",
        "entries_run": 45,
        "avg_latency_ms": 8420,
        "tokens_per_second": 38.2,
        "timestamp": "2026-05-23T15:03:16Z"
      },
      "reasoning": {
        "score": 0.521,
        "confidence": "low",
        "source": "bootstrap"
      }
    }
  }
}

Gap analysis

Run python -m benchmarks --gaps to find capability holes in your stack. The output lists categories where no installed model scores above 0.7 and suggests which models to install to fill them.

Next: smart routing →