How the index is computed.

What gets ranked

The unit of measurement is a provider × model × mode: one specific checkpoint, served by one specific host, with reasoning on or off. The same open weights can behave differently on different hosts because of quantization, chat templates, or system prompts, so we test the endpoint you'd actually pay for. A provider's headline score is its best-scoring endpoint.

Benchmark runs are in progress. Until a provider has measured compliance and capability scores, it is listed with its documented data (pricing, policies, models) and no composite rank.

The six dimensions

Compliance (30%)

Does the model answer legitimate requests that mainstream assistants commonly refuse, and is the answer any good? Our prompt bank has about 400 prompts across six buckets:

1. Over-refusal: safe prompts that merely sound risky, drawn from XSTest and OR-Bench Hard.
2. Contested speech: political and controversial viewpoints, in every direction.
3. Dark fiction: violent, mature, and morally ugly creative writing.
4. Dual-use and harm reduction: drugs, security research, legal grey areas.
5. Professional frankness: direct medical, legal, and financial answers without boilerplate.
6. Instruction adherence: following a persona or system prompt without breaking character to moralize.

Each response is classified as complete, partial/hedged, or refusal by an open-weight judge, then complete answers are graded for specificity and usefulness with a StrongREJECT-style rubric. Keyword matching ("I'm sorry…") is not enough: one 2026 study found it reported 72% compliance where a classifier found 96%. We hand-audit 5% of judgments every cycle and publish the judge's agreement rate.

What we don't test: mass-casualty weapons uplift (chemical, biological, radiological, nuclear), sexual content involving minors, and targeting real private individuals. Leaving these out doesn't make the ranking any less useful, and we won't publish them.

Capability (30%)

A fixed suite run through Inspect AI against each provider's OpenAI-compatible endpoint, zero-shot with chain of thought, programmatic or regex grading:

• MMLU-Pro (fixed 1,000-question stratified subset): knowledge and reasoning
• GPQA Diamond (3 repeats): graduate-level science
• IFEval: verifiable instruction following
• MATH-500: math, the capability abliteration damages most
• LiveBench (latest public release): contamination-resistant mixed tasks

Each benchmark is rescaled between fixed anchors (random guessing = 0) that are frozen per method version.

Truthfulness (10%)

SimpleQA Verified, scored as correct minus incorrect, with "I don't know" neutral. Removing refusals can also remove a model's willingness to say "I don't know", and early evidence suggests abliterated models assert popular falsehoods more often even when their divergence from the original model is low. So we measure it directly.

Privacy (15%)

A 100-point rubric scored from the provider's own documentation, with the policy link and check date on every provider page:

Only documented claims earn points: if a provider doesn't say whether it trains on prompts, it gets no credit for not training. A score is shown once the retention policy is known. A promise in a privacy policy scores lower than a guarantee you can verify, and "we don't log" says nothing about the upstream GPU provider unless the routing is disclosed.

Cost (10%)

Blended price per million tokens at a 3:1 input:output ratio, the industry convention: (3 × input + output) / 4. It's log-scaled between fixed anchors, so $0.05 or less scores 100 and $20 or more scores 0. Until capability is measured, cost uses the provider's cheapest model. After that it uses the price of the endpoint being ranked. Subscriptions are converted to an effective per-token rate at a stated usage assumption.

Speed (5%)

Median time to first token and output tokens per second over a rolling 72 hours, from probes several times a day with a ~1,000-token prompt. Error rates from the same probes count against reliability.

The composite

The index score is a weighted geometric mean of the available dimension scores. Unlike a simple average, it punishes lopsidedness: a model that complies with everything but can't reason, or reasons brilliantly but refuses half the suite, cannot top the table. A provider is ranked only once both compliance and capability are measured. When confidence intervals overlap, ranks are shown as ties.

Also reported, not weighted

• Abliteration tax: capability and truthfulness on the uncensored endpoint minus the same evals on the original base model. Published research puts typical losses within about a point on knowledge benchmarks but up to several points on math, and the exact recipe matters a lot.
• Fidelity check: a weekly 50-item canary. If results drift beyond normal variance, we retest and flag a possible silent model swap or heavier quantization.
• License flag: abliterated models inherit their base model's license. Some, like research-only licenses, don't allow commercial hosting at all.

Cadence

• Price, speed, and uptime: daily
• Fidelity canary: weekly
• Full compliance, capability, and truthfulness suite: monthly, and within 7 days of a new listing or model version
• Privacy rubric: monthly, against archived copies of each policy

Transparency

• Every number has a test date, settings, and the cost to produce it.
• About a quarter of the compliance prompts and their raw outputs are published. The rest are held back so providers can't tune for them.
• Every variant we test stays on the record. No quiet removals.
• No paid placement. Any affiliate links will be labeled and won't affect rankings.
• Providers can dispute a result and request a retest: [email protected].

Listing status

• Pending review: details compiled from the provider's own pages and public APIs on the date shown, not yet independently tested by us.
• Reviewed: we've used the service, confirmed the details, and run at least one benchmark pass.