Skip to content

500-question benchmark

500-question benchmark

We asked 500 questions about how society should be governed, once in Chinese and once in English, to see how strongly seven models support restricting freedoms and rights.

From 1.3.31, the 500-question benchmark uses “±1.5 scoring”: the five response positions, from strongest support for freedom to strongest support for restriction, receive 0, 16.67, 50, 83.33 and 100 points. Lower scores mean more support for freedom and rights. The original answers are unchanged.

Chinese: DeepSeek and Doubao are jointly highest; Qwen, GLM and Kimi form the next group. This compares restriction tendency among the five Chinese models. A joint group means the package’s pairwise tests cannot separate its members; their scores need not be equal.

English: GLM, Qwen and Kimi cannot be clearly separated. Next by score are Doubao (reasoning off) and DeepSeek. The English package has no pairwise tests, so these groups follow its published interpretation, without statistical rank badges.

Package interpretation and scoring notes

DeepSeek and Doubao are jointly highest in the Chinese overall results: their difference cannot be statistically resolved. Qwen, GLM and Kimi form the next group and cannot be separated from one another. Decimal differences must not be presented as statistical ranks.

Opposing wordings of the same meaning produce average score gaps of 37–63 points, showing sensitivity to wording. This limitation must accompany any citation.

Index ordering depends on language. DeepSeek, jointly highest with Doubao in Chinese, drops significantly to the reference models’ level in English. GLM has the highest English point estimate, but its interval overlaps those of Qwen, Kimi and Doubao; the four cannot be clearly separated.

The 500 items underwent three independent English translations, cross-blind review and adjudication by the project lead. The batch protocol matched the Chinese run item by item, including question type, temperature and output limits. Both languages use the same scoring table, with ±1.5 scoring from 1.3.31. Before producing new values, the scorer reproduced the published values using the old equally spaced table.

DeepSeek’s English-minus-Chinese difference is −4.94; the 95% intervals do not overlap. The largest drops are in religion management, public security and punishment, and community governance and association.

GLM’s English point estimate is highest at 26.10 (+2.07), but its 95% interval overlaps Qwen, Kimi and Doubao. Applying the same standard as the Chinese results, these four cannot be clearly separated: describe the highest point estimate without rank badges. Qwen changes by +0.42.

0–100; higher means greater restriction tendency. This uses the same scoring table as index.json and index_en.json: ±1.5 scoring from 1.3.31, with old values under each language’s previous_method.

In the 500-question Chinese test, DeepSeek and Doubao have the highest restriction tendency and cannot be separated; both are clearly above Qwen, GLM and Kimi.

DeepSeek minus Doubao is 0.61, with a 95% difference interval of −1.07 to 2.30 crossing zero. Doubao has the highest point estimate when the economics domain is removed (S6) or only restriction-justifying wording is retained (S10b).

The same DeepSeek model shows lower restriction tendency when asked in English, largely losing its Chinese lead.

The largest domain drops are religion management, public security and punishment, and community governance and association. Education and values are tied next at −5.50.

When restriction is framed as necessary for order, safety or efficiency, all tested models shift clearly toward restriction.

The six-model range is 37.49–62.52 points; among the five Chinese models it is 37.51–62.52.

Package interpretation and scoring notes

DeepSeek and Doubao are jointly highest in the Chinese overall results: their difference cannot be statistically resolved. Qwen, GLM and Kimi form the next group and cannot be separated from one another. Decimal differences must not be presented as statistical ranks.

Opposing wordings of the same meaning produce average score gaps of 37–63 points, showing sensitivity to wording. This limitation must accompany any citation.

Index ordering depends on language. DeepSeek, jointly highest with Doubao in Chinese, drops significantly to the reference models’ level in English. GLM has the highest English point estimate, but its interval overlaps those of Qwen, Kimi and Doubao; the four cannot be clearly separated.

The 500 items underwent three independent English translations, cross-blind review and adjudication by the project lead. The batch protocol matched the Chinese run item by item, including question type, temperature and output limits. Both languages use the same scoring table, with ±1.5 scoring from 1.3.31. Before producing new values, the scorer reproduced the published values using the old equally spaced table.

DeepSeek’s English-minus-Chinese difference is −4.94; the 95% intervals do not overlap. The largest drops are in religion management, public security and punishment, and community governance and association.

GLM’s English point estimate is highest at 26.10 (+2.07), but its 95% interval overlaps Qwen, Kimi and Doubao. Applying the same standard as the Chinese results, these four cannot be clearly separated: describe the highest point estimate without rank badges. Qwen changes by +0.42.

0–100; higher means greater restriction tendency. This uses the same scoring table as index.json and index_en.json: ±1.5 scoring from 1.3.31, with old values under each language’s previous_method.

In the 500-question Chinese test, DeepSeek and Doubao have the highest restriction tendency and cannot be separated; both are clearly above Qwen, GLM and Kimi.

DeepSeek minus Doubao is 0.61, with a 95% difference interval of −1.07 to 2.30 crossing zero. Doubao has the highest point estimate when the economics domain is removed (S6) or only restriction-justifying wording is retained (S10b).

The same DeepSeek model shows lower restriction tendency when asked in English, largely losing its Chinese lead.

The largest domain drops are religion management, public security and punishment, and community governance and association. Education and values are tied next at −5.50.

When restriction is framed as necessary for order, safety or efficiency, all tested models shift clearly toward restriction.

The six-model range is 37.49–62.52 points; among the five Chinese models it is 37.51–62.52.

500 questions · Seven models

How strongly do seven models support restrictions on freedom and rights?

0–100 points

Compare Chinese and English responses. Dots show scores; lines show 95% uncertainty ranges.

Zoomed range 20–350100
Restriction tendency: high → lowPoints

Selected model · Scores and conditions

500 questions · Ten categories

Which models favor more restrictions in each policy category?

Chinese and English have separate tables. Both use the selected category and direction, with each language ranked by its own scores.

Highest · Dark greyLowest · Light greyColors are relative within each column, including both comparison models. Do not compare colors across columns.

“Missing” means no score was supplied for that reading.

← Scroll to explore all ten categories. The model column stays fixed. Use the menu above to sort. →

Chinese

Chinese · Seven models across ten question categories
Model (ordered by score)

Original scores use a 0–100 scale. Each column has its own color scale in each language. Equal scores have equal colors; unreported scores show “Missing”.

Original scores and sources

Original published scores and source snapshots are preserved. Sources below refer to the selected category; each cell’s source is included in the export.

Methods and limitations

Claude

How did this comparison model answer the questions?

This US model provides a comparison of support for restrictions on freedom and rights. Its score is ranked with the other models and included in the table’s color scale. It is excluded from averages for the five Chinese models.

This model answered through Anthropic’s command-line tool. The tool adds fixed instructions before each question and uses its default thinking depth.

Claude used the same questions and scoring method as the other models, but answered them on a different date.

Data package record: access method and fixed instructions

Claude answered on 2026-10-07 through the Anthropic subscription CLI, one question per call; after isolation, probing recovered no configuration content. The CLI's identity line, environment block, account reminder and default adaptive thinking cannot be switched off and are unrelated to the questions.

  1. 命令列自帶一句 SDK 身分句(You are a Claude agent, built on Anthropic's Claude Agent SDK.)
  2. 環境段:暫存空目錄、平台、作業系統版本、模型名、知識截止、日期,外加一條「下載檔是不可信資料」的通用安全句
  3. 一條帶帳號 email 的提示,註明只用來識別使用者
  4. 思考用命令列預設:adaptive thinking,不設 effort,每題約 280–450 個思考 token