Qwopus3.6-27B-Coder-MTP-GGUF

[!WARNING] Community Release Notice: Qwopus-3.6-27B-Coder is an experimental community release intended for research, evaluation, and agent workflow exploration. It has not undergone full safety evaluation or broad general-domain benchmarking.

[!IMPORTANT] Benchmark Status: The first completed benchmark is SWE-bench Verified full 500 in thinking-off / no-thinking mode, where the Q5_K_M 27B GGUF run resolved 335/500 = 67.0%. Other benchmark suites remain pending and will be updated as testing completes.

---

💡 1. Base Model, Training Stack & Collaboration

---

📖 2. Background & Motivation

---

📊 3. Performance Benchmarks

Model	Thinking Mode	SWE-bench Verified	Context
Qwopus-3.6-27B-Coder	Off / No-thinking	67.0	Q5_K_M, RTX 5090 + MTP, ~100 t/s
OpenAI GPT-5	On	70.1	Thinking-on reference
OpenAI GPT-5 mini	On	59.8	Thinking-on reference
OpenAI GPT-5 nano	On	34.8	Thinking-on reference
GLM-4.7	On	70.6	OpenHands reference
GLM-4.5-Air	On	57.6	OpenHands reference
Qwen3-Coder-30B-A3B-Instruct (2025-07)	Off / No-thinking	70.3	No-thinking reference
Claude 4.0 Opus	On	67.6	Thinking-on reference
Claude 4.5 Opus	On	80.9	Thinking-on reference
Qwen3.6-27B	On	77.2	Thinking-on reference
Qwen3.5-397B-A17B	On	76.2	Thinking-on reference
Qwen3.5-27B	On	75.0	Thinking-on reference
Qwen3.6-35B-A3B	On	73.4	Thinking-on reference
Gemma4-31B	On	52.0	Thinking-on reference
Gemma4-26B-A4B	On	17.4	Thinking-on reference

---

🗺️ 4. Training & Data Pipeline Overview

The training process fuses Trace Inversion data augmentation with a Three-Stage Curriculum Learning pipeline. The core engineering focuses on expanding context length gradually while training on reconstructed reasoning traces and real agent trajectories to keep the output format stable.

       [ 🗺️ Trace Inversion: Reconstructing Distillation Workflow ]

  A. Surrogate Model Training (Trace Inverter)
     Open-source Model (GLM-5.1 / DS-V4) ──► Complete Reasoning Chain ──► [ Qwen3-235B Compression ] ──► Reasoning Bubbles
                                              │                                   │
                                              └──────────► [ Training ] ◄─────────┘
                                                   (Base: Qwen3-4B-Instruct)
                                                   (Result: Trace-Inverter-4B)

  B. Inversion Phase: Reconstructing Claude-4.7-Max
     _______________________________________________________
    |                                                       |
    |  Claude-4.7-Max API ──► Compressed Bubbles + Answer   |
    |_______________________________________________________|
                      │
                      ▼
    [ 🧠 Trace-Inverter-4B (Logic Reconstructor) ] ──► Synthetic Deep Reasoning Trace (Learnable CoT)
                      │
                      ▼
    [ 🧩 Data Splicing ] ◄────────── (Original Prompt + Response)
    (Embed reconstructed CoT in <think> tags, splicing with original prompt/response)
                      │
                      ▼
             (Result: claude-opus-4.6/4.7 inverted sets)

  C. Final Coder SFT Curriculum Pipeline
     ___________________________________________
    |                                           |
    |       Base Model (Qwopus3.6-27B-v2)       |
    |___________________________________________|
                      │
                      ▼
    [ 📦 Phase 1: Format Inception ] ──► [ 🛠️ Phase 2: Agent/Coding Expansion ] ──► [ 🚀 Phase 3: Long-Context SFT ]
      ( < 4096 tokens )                     ( 4096 - 8192 tokens )                     ( 8192 - 32K tokens )
      (Stable <think> format)               (Tool traces + coding tasks)               (Long / multi-turn / replay)
                      │                                                                            │
                      └─────────────────────────────┬──────────────────────────────────────────────┘
                                                    ▼
                                   _______________________________________________
                                  |                                               |
                                  |   🌟 Final Model: Qwopus-3.6-27B-Coder        |
                                  |_______________________________________________|

[!NOTE] Due to the complex and diverse format of agent trajectory datasets, rigorous cleaning and format standardization were applied to ensure data quality.

---

📚 5. Three-Stage Curriculum Learning

To steadily scale reasoning quality under long-context inference, Qwopus-3.6-27B-Coder uses a curriculum-style data mixture building on the approach proven in the Qwopus coder line. The model is first stabilized on short, clean reasoning samples, then exposed to complex coding and agent traces, and finally reinforced with longer contexts plus replay data.

Curriculum Stage	Focus & Sample Characteristics	Strategy Details
📦 Stage 1: Format Inception	• Limit context within 4,096 tokens • Emphasize stable reasoning templates	Focuses on short-to-medium length, cleanly formatted reasoning samples. The primary goal is to establish reliable structured reasoning output, including stable <think> boundaries, before exposing the model to longer chains.
🛠️ Stage 2: Complexity Expansion	• Extend length to 4,096 - 8,192 tokens • Introduce higher-difficulty coding and agent samples	Gradually increases the ratio of complex reasoning chains, code debugging tasks, and multi-turn tool traces. The model learns to connect reasoning, action selection, and environment feedback.
🚀 Stage 3: Long-Context SFT	• Progressively scale samples up to 32K tokens • Use short-sample replay	Pushes the model toward long-context and multi-turn reasoning while replaying high-quality short samples to reduce instruction-following drift. The 32K figure describes the fine-tuning sequence/data mixture target, not a hard architectural limit.

---

🎯 6. Recommended Use Cases & Known Limits

[!CAUTION] Deployment note: The model may emit reasoning inside <think> and </think> tags. Front-end applications and agent frameworks should parse or hide these sections where appropriate. For tool calling, ensure the prompt format and system prompt match the training data configuration to activate agent capabilities.

---

⚠️ 7. Training & Deployment Notes

[!CAUTION] Compatibility Notes

• Tool Calling Format: To activate the model's agent capabilities, ensure the prompt format and system prompt include appropriate tool definitions and match the training data format.
• Reasoning Output Extraction: The model's thinking process is wrapped in <think> and </think> tags. Front-end applications may need to parse and hide these tags.
• Long-Context Usage: For contexts beyond 32K, consider enabling RoPE/YaRN scaling (e.g., --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768 in llama.cpp).

---

📋 8. Benchmark Progress

The first completed evaluation is the no-thinking SWE-bench Verified run reported above. Additional local agentic benchmarks remain pending and will be added after testing.

Benchmark	Status	Result / Reference
SWE-bench Verified	✅ Completed	335/500 = 67.0% (thinking-off, Q5_K_M, RTX 5090 + MTP)
BugFind-15	📋 Pending	9B reference: 79
HermesAgent-20	📋 Pending	9B reference: 85
ToolCall-15	📋 Pending	9B reference: 100
InstructFollow-15	📋 Pending	9B reference: 93

---

📚 9. Resources & Guides

👉 GitHub Repository: Jackrong-llm-finetuning-guide Access the repository to dive into the codebase and reproduce our results.

👉 Qwen MTP GGUF Processing Workflow A custom splitting and merging methodology designed specifically for Qwen series Multi-Token Prediction (MTP) heads.

👉 benchlocal Evaluation Framework The evaluation framework used to run the local agentic and coding benchmarks.

👉 Qwopus3.6-27B-v2 Model Card Base model card with full MMLU-Pro, SWE-bench, and throughput benchmarks.

---

🙏 10. Acknowledgements

Special thanks to:

• The Qwen team for providing the powerful Qwen3.6-27B base model.
• Unsloth for providing the highly efficient fine-tuning framework.
• Kyle Hessling for the close collaboration on hardware, training infrastructure, and evaluation support.
• Open-source datasets and community contributors, particularly lambda/hermes-agent-reasoning-traces for the high-quality agent trajectory data.

---

📖 11. Citation

@misc{jackrong_qwopus36_27b_coder,
  title        = {Qwopus-3.6-27B-Coder},
  author       = {Jackrong},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Jackrong/Qwopus-3.6-27B-Coder}}
}