How YipitData Uses Databricks Agent Bricks to Scale Data Intelligence 20x

YipitData sits at the intersection of financial data and machine intelligence: it converts billions of transaction records from credit card data, web-scraped receipts, and dozens of alternative data sources into actionable market intelligence for institutional investors. The company’s value proposition depends on reliably tagging each transaction to the correct merchant or company—matching messy, ambiguous vendor records to real businesses at a scale that keeps pace with daily data refreshes. As the company’s customer base and data volumes grew, the gap between what was possible and what was achievable became increasingly visible.

For years, YipitData relied on analysts who wrote and maintained hundreds of regular expression rules, supplemented by traditional NLP models. The approach worked on well-known patterns but broke down at the edges. Analysts effectively encoded their judgment into more than 500 regex statements, yet still missed the nuance and context that free-form text descriptions required. The real bottleneck was not computational—it was the irreplaceable judgment that experienced analysts applied instinctively to ambiguous cases. “We were bottlenecked by what an analyst could do in a day,” said Edward Goo, Head of Data Engineering at YipitData. “Even with hundreds of regex rules, you still miss nuance, context, and the judgment that analysts apply instinctively.”

The solution was to stop trying to encode analyst judgment in rules and instead give AI agents direct access to institutional data within the same governed platform. YipitData adopted Databricks Agent Bricks, using its Information Extraction capability to embed reasoning-based tagging directly into existing ETL pipelines via SQL batch inference. The key architectural decision was to keep AI enrichment inside the Databricks Data Intelligence Platform rather than exporting data to external services. Agents connect directly to enterprise data governed by Unity Catalog, inheriting the same permissions, lineage, and compliance guarantees as the underlying pipelines. For large-scale entity resolution, the team pairs Agent Bricks with Databricks Lakebase, a serverless PostgreSQL database that serves as the system of record for known companies. Retrieval logic uses Lakebase to narrow candidates from structured relationships before AI Search and Information Extraction agents evaluate unstructured text to make the final tagging decision.

The results arrived faster than the team expected. In a single quarter, the number of companies YipitData automatically tags and enriches grew from roughly 3,000 to more than 60,000. Processing 1 million records now takes about one hour, down from 24 hours. In one core use case, Agent Bricks achieved 92–95% tagging accuracy out of the box—meeting the quality bar required for customer-facing products. “Our analysts have a very high quality bar, and Agent Bricks met it faster than we expected,” said Edward. These gains directly enabled Signals, YipitData’s external market intelligence product for analyzing companies, competitors, and growth trends across private markets. The product exists at its current scale specifically because the enrichment pipeline can now operate at orders of magnitude beyond what was previously possible.

The impact extends beyond efficiency. Analysts who previously spent their days writing regex rules now focus on validating outputs, improving quality, and exploring new datasets. The same agent-driven framework is reusable across additional data sources, turning AI enrichment into a repeatable platform capability rather than a one-off project. Plans call for scaling Signals coverage from thousands to hundreds of thousands of companies, and the agentic architecture is what makes that trajectory plausible. “This wasn’t just an efficiency gain,” said Anup Segu, Chief Architect. “It changed what products we could realistically build.”