How Kantar Worldpanel Uses Databricks to Generate Market Insights Faster

Kantar Worldpanel’s business depends on knowing precisely what products consumers purchased and when. At the core of their data pipeline is a process that links descriptions from paper receipts to standard product barcode names—a step that determines which buying signals reach client dashboards and, ultimately, what business decisions manufacturers and retailers make.

For years, manual coding teams handled this linkage, creating a bottleneck that limited throughput and absorbed engineering resources on routine, repetitive work. Legacy systems were inflexible, required specialized programming skills that were increasingly hard to hire for, and could not scale to meet growing client demand. The data science team knew that generative AI could close this gap, but lacked a platform to experiment systematically across multiple model families.

Kantar Worldpanel selected the Databricks Data Intelligence Platform as a unified environment for AI and machine learning experimentation. The team used MLflow to track experiments and manage the model lifecycle, evaluated GPT-4, GPT-3.5, Llama, and Mistral side by side within the same platform, and adopted Databricks AI Search to compare receipt and reference product descriptions at scale. Unity Catalog provided secure, cross-team data sharing throughout the process.

By running head-to-head evaluations inside Databricks Marketplace, the team identified GPT-4 as the highest-accuracy option at 94%, then used it to automatically generate a training dataset of 120,000 receipt-to-barcode pairs in just a few hours. That synthetic dataset now trains a smaller, cost-effective production model served through Databricks’ Agent Bricks infrastructure. The result is a pipeline that previously required sustained manual effort now running with minimal human intervention.

The shift represents more than operational efficiency. Manual coding teams can redirect their attention to edge cases and discrepant results rather than bulk labeling. Engineers no longer need to configure clusters for each experiment—data scientists spin up environments in one or two lines of code and get results immediately. Kantar Worldpanel describes the receipt-to-barcode proof of concept as the first of many GenAI use cases it plans to productionize on Databricks, with further applications across its consumer data platform already in exploration.