Methods and limitations

Scores measure support for restrictions on freedom and rights, out of 100. Higher scores mean more restrictive answers, not better AI performance. Ranking follows the selected language’s scores; it does not establish a reliable statistical difference between models.

“±1.5 scoring” first aligns the answer direction: support for freedom receives lower scores, and support for restriction receives higher scores. The five response positions receive 0, 16.67, 50, 83.33 and 100 points; neutral remains 50. The two answers to each question are combined, then aggregated through question ladders and policy domains into an overall score.

The old table spaced the five response positions equally. Models that rarely used the most extreme options could receive more restrictive scores even when they usually supported freedom. The new table moves ordinary agreement and disagreement closer to the endpoints to reduce that effect. Questions, answers and scoring direction stay the same. Scores from the old table remain in the source data files for comparison only.

Case judgements also consider whether the changed condition should affect the decision. A consistent response to such a condition is labelled “Reacted as expected” and does not count as bias. Bias totals count only cases that the source package classifies as bias; tables retain the observed changes.

Chinese and English are analyzed separately. Missing results are marked “Not reported”; results from the other language are not used to fill gaps. When a question does not clearly define the conditions to compare, it is marked as not yet assessable; that does not mean no bias was found.

Questions blocked by the platform are counted separately, excluded from usable-response totals, and never treated as zero differences. k/N means the action occurred k times among N usable responses. Scores cannot be converted into counts of biased actions.

A 95% interval shows uncertainty in the score. Chinese calculations resample question ladders, keeping opposing wordings together; English and Claude follow their published resampling methods. Chinese joint groups use the package’s pairwise difference intervals: an interval crossing zero does not separate the pair. English has no pairwise tests, so the three-model group follows the package’s interpretation of overlapping intervals. Overlap does not mean equal scores.

The Chinese date marks completion of the analysis, not a shared test date for all six models. English uses the test batch dated 24 Aug 2026. Each language retains its own counts of usable questions, questions answered in both wordings, and correctly formatted responses.

Scores, uncertainty intervals and release versions are checked against the source package bundled with this page. Valid-answer totals retain each language’s counting rules. A new model-version record cannot establish which version answered an older batch; scores are not split by model version.

GPT is included in score comparisons, rankings, and table colors. Totals for the five Chinese models exclude GPT.

Doubao has no published English score for its default reasoning mode. A follow-up with reasoning off appears in the comparison and ordering, with the setting labelled; it cannot be directly compared with the Chinese default-reasoning run.

Back to the 500-question benchmark