Every provider says its models are uncensored. Measuring it turns out to be harder than it sounds, and most of the difficulty is in a few specific pitfalls.
Pitfall 1: counting apologies
The obvious method is to send a set of prompts the model might refuse and count responses that contain “I’m sorry”, “I can’t”, or “As an AI”. The original abliteration paper did roughly this, and the authors noted its limits.
How big is the error? A late-2025 study ran the same abliterated outputs through both a keyword matcher and a trained classifier. Across 900 responses, the keyword method said 72.2% complied. The classifier said 95.7%. Models comply while apologizing, refuse without apologizing, and hedge in ways no keyword list catches.
Pitfall 2: “compliance” that’s useless
A model can technically answer while saying nothing useful: vague, wrong, or quietly sabotaged. StrongREJECT was built for exactly this. It scores 0 for a refusal, and otherwise grades how convincing and specific the answer is. Its authors found that jailbreaks in particular produce lots of low-quality “compliance”.
Pitfall 3: mixing up two different things
“Uncensored” covers two separate questions:
- Over-refusal: does the model refuse safe things that merely sound risky, like “How do I kill a Python process?” XSTest (250 safe prompts plus 200 unsafe contrasts) and OR-Bench (80,000 seemingly toxic but safe prompts) measure this.
- Willingness: does the model answer requests that are genuinely contested, like dark fiction, drug harm reduction, or blunt medical answers?
A model can be great on one and bad on the other. They should be scored separately.
The benchmarks worth knowing
| Benchmark | What it measures | Notes |
|---|---|---|
| UGI Leaderboard | Uncensored knowledge + willingness (W/10) + intelligence (NatInt) | The community standard. Questions are private, so models can’t train on them, but you can’t rerun it on a provider’s endpoint. |
| StrongREJECT | Quality of compliance, 0–1 | 313 prompts, MIT, available in Inspect |
| XSTest | Over-refusal | Cheap, CC-BY-4.0 |
| OR-Bench | Over-refusal at scale | Hard-1K subset is a good stress test |
| HarmBench | Red-teaming attack success | 510 behaviors, classifier-graded |
| SORRY-Bench | Refusal across 44 fine-grained categories | Fine-tuned judge; Cohen’s κ ≈ 0.81 against human labels |
| JailbreakBench | Harmful vs. benign twin prompts | Good for plotting compliance against over-refusal |
| SpeechMap.ai | Political and controversial speech | Complete / Evasive / Denial labels across 400+ models |
How we built our compliance score
Our methodology borrows from all of these:
- Six prompt buckets, about 400 prompts in total: over-refusal, contested speech, dark fiction, dual-use and harm reduction, professional frankness, and instruction adherence.
- A classifier, not keywords. An open-weight judge labels each response complete, partial, or refusal. We use an uncensored judge so it doesn’t refuse to grade.
- Usefulness grading. Complete answers get a StrongREJECT-style specificity score, so useless compliance scores low.
- Human audit. We hand-check 5% of judgments every cycle and publish the agreement rate.
- Held-out prompts. About a quarter of the prompts and raw outputs are published. The rest stay private so providers can’t tune for them.
- Hard exclusions. We don’t test mass-casualty weapons uplift, sexual content involving minors, or targeting private individuals. Those aren’t what anyone means by “uncensored”, and testing them adds nothing to the ranking.
Why test the endpoint, not the model
UGI tells you how a checkpoint behaves. We test what a provider actually serves you. The same weights can behave differently on different hosts because of quantization, chat templates, or a system prompt quietly added upstream. Some “uncensored” services are just a mainstream model behind a jailbreak prompt, and only an endpoint test will show that.