How Trillion Labs Cuts LLM Training Time 7x with NVIDIA NeMo Curator

Trillion Labs is a Korean AI startup dedicated to building sovereign large language models for the Korean language. Its mission is to close the gap between English-dominant foundation models and the needs of Korean public sector organizations and enterprises, which require LLMs that understand Korean linguistic nuance, government terminology, and cultural context. Building high-quality Korean LLMs at scale requires curation pipelines capable of processing datasets exceeding 2 trillion tokens — volumes that expose every inefficiency in traditional CPU-based workflows.

The core problem was data pipeline throughput. Deduplication and shuffling operations on datasets at this scale took 24 hours per run on CPU infrastructure. Iteration cycles became prohibitively slow: every experiment in model architecture or data composition required waiting nearly a full day to simply complete preprocessing. This bottleneck made it impossible to move quickly on model development, creating a compounding disadvantage relative to resource-rich competitors working on high-resource language models.

Trilion Labs deployed NVIDIA NeMo Curator, a GPU-accelerated data curation library, running on a cluster of 8x H100 GPUs with DASK for parallel and distributed processing. NeMo Curator’s GPU acceleration applied to the most compute-intensive steps — exact and fuzzy deduplication, quality filtering, and data shuffling — across a curated dataset of 100 billion high-quality Korean tokens. NVIDIA NeMo provided the broader framework for model training and evaluation downstream.

The processing speedup was immediate and dramatic. Curation pipelines that previously ran for 24 hours completed in 3.4 hours on the 8x H100 configuration — a 7x improvement in throughput. Energy and compute costs dropped up to 10x compared to CPU-based approaches. The curated 100-billion-token Korean dataset produced a measurable 5% accuracy improvement in Korean language tasks, validating the quality of GPU-accelerated curation against the brute-force CPU alternative. Co-founder Jason Park summarized the bottleneck that NeMo Curator removed: “Deduplication is one of the most time-consuming processes when handling very large datasets. The time saved from NeMo Curator GPU acceleration was the most significant benefit.”

For Trillion Labs, the operational gain extends beyond raw speed. Faster iteration cycles mean faster experiments, faster model releases, and a faster path to competitive Korean LLMs that Korean public sector institutions can adopt without routing sensitive data through foreign infrastructure. The company is continuing to scale its sovereign AI work, with NeMo Curator’s GPU pipeline as the foundation for processing the next generation of Korean training datasets.