GLM-5.2-colibri-int4
jlnsrk's INT4-quantized GLM-5.2 MoE variant with expert streaming for CPU-based bilingual inference.
Base model
Model Description
Pre-converted weights for colibrì — the pure-C engine that runs GLM-5.2 (744B MoE) on a consumer machine with ~25 GB of RAM by streaming routed experts from disk.
This is the output of colibrì's coli convert (convert_fp8_to_int4.py --ebits 4 --io-bits 8, including the MTP head for native speculative decoding), uploaded so you don't have to download the 756 GB FP8 checkpoint and spend a day converting it.
⚠️ This is NOT a GGUF / AWQ / GPTQ / MLX model. It is colibrì's own container: for each quantized weight, name (U8, packed int4 nibbles) + name.qs (F32 per-row scales), quantized with math bit-identical to the engine's C kernels. It only works with the colibrì engine.
Usage
# get the engine
git clone https://github.com/JustVugg/colibri && cd colibri/c && ./setup.sh
# download this repo to a FAST local disk (NVMe, ext4 — never a network/9p mount)
hf download jlnsrk/GLM-5.2-colibri-int4 --local-dir /nvme/glm52_i4
# chat (RAM budget, expert cache and MTP auto-detected)
COLI_MODEL=/nvme/glm52_i4 ./coli chat
Requirements: Linux (or WSL2), gcc + OpenMP, AVX2, ≥16 GB RAM, ~400 GB free NVMe.
What's inside
| file | contents |
|---|---|
out-*.safetensors |
dense weights (attention/MLA, shared experts, embeddings) + 21,504 routed experts, int4 per-row scales; router/norms kept F32 |
| MTP shard | GLM-5.2's multi-token-prediction head (layer 78) — enables lossless speculative decoding (~2 tok/forward) |
config.json, tokenizer*.json, generation_config.json |
copied from the base repo |
Conversion: FP8 (e4m3, 128×128 block scales) → f32 → int4 with np.rint matching the engine's lrintf — token-identical to converting locally.
Provenance & license
Converted from zai-org/GLM-5.2-FP8 (MIT). This derivative is likewise MIT. Conversion performed with colibrì's official converter, unmodified.
Sign up to read complete case studies, access detailed metrics, and unlock all use cases.
Sign up to read complete case studies, access detailed metrics, and unlock all use cases.