ThinkingCap-Qwen3.6-27B

ThinkingCap: Qwen 3.6 27B

Capability of Qwen3.6-27B with 50% less thinking tokens on average, and over 90% less in best cases. Achieved via finetuning Qwen3.6-27B (Qwen Team, 2026) with state-of-the-art algorithms on a curated set of problems of various domains and difficulty. We designed the finetuning to be as minimally invasive as possible, preserving all of the original answer quality and style of Qwen, while being more token efficient. Check the blogpost for more details.

We rigorously evaluate the resulting checkpoint across general reasoning, non-reasoning multiple-choice question answering, everyday multi-turn conversations, system prompt adherence, safety, math, code and agentic use cases. Due to the high variability of reasoning quality at Qwen-recommended sampling temperature 1.0, we run each benchmark with multiple seeds and do statistical significance testing on all the results. We evaluate both in domain (holdout parts of selected datasets included in training) and out of domain.

Out-of-domain token efficiency

Claw-Eval thinking tokens are per-task (agentic; not a single-turn trace).

Settings

• Models: base Qwen/Qwen3.6-27B vs bottlecapai/ThinkingCap-Qwen3.6-27B (shown as Ours in the table).

• Seeds: 5 per condition; thinking on; cells are mean ± 95% CI across seeds.

• Decoding: thinking on; sampling temperature=1.0, top_p=0.95, top_k=20, min_p=0.0 (bottlecapai/ThinkingCap-Qwen3.6-27B uses the base's sampling).

• Max generation tokens: 100,000 for the general suite (gpqa_diamond, mmlu_pro, longbench_v2, realworldqa) and AA-LCR; 250,000 for HMMT (Nov 2025); 32,768 for supergpqa and livecodebench; 16,384 for ceval and mmlu_redux; 15,000 for llm-system-prompts-benchmark; 49,152 for Claw-Eval.

• Metrics — the columns mirror the table:

  • Accuracy (Base / Ours) — fraction correct (exact/regex match; soft compliance for llm-system-prompts-benchmark; judge task-score for Claw-Eval; judge CORRECT/INCORRECT for AA-LCR).
  • Thinking tokens (Base / Ours) — mean length of the single-turn <think> trace (think-per-task for Claw-Eval).
  • Reduction — the average per-question thinking-token saving: base and Ours are paired on the same question (each side seed-averaged), each question's (base − cap)/base is taken, then averaged over shared questions (a larger ↓ = a bigger saving).
  • Macro average (bottom row) — equal-weight mean across benchmarks.

  We separately track two trace-quality failure modes, reported only in aggregate: looping — the model gets stuck repeating the same reasoning chain (sometimes a single sentence), never finishing its thinking; detected from the fraction of repetitive n-grams — and truncation — the <think> trace never closes because the model hits the generation-token cap while still reasoning, so no answer is produced. Across all out-of-domain responses, truncation drops from 2.9% to 0.4% while looping stays negligible (~0.2%).

In-domain evals

Holdout test splits of datasets whose train splits are part of the finetuning mix — quality retention on in-distribution tasks (in contrast to the out-of-domain benchmarks above).