How BigID Uses Elasticsearch to Accelerate Data Queries 120x at Scale

BigID helps organizations gain a comprehensive view of their data scattered across cloud, SaaS, and on-premises environments. Its platform enables clients to discover sensitive data, address security vulnerabilities, simplify compliance adherence, and govern AI data effectively. As BigID’s customer base grew and data volumes scaled, maintaining the performance its customers required became increasingly difficult.

The core challenge was search. BigID’s existing data storage platform required frequent schema migrations and index additions just to keep pace with growing data volumes — a brittle approach that became harder to sustain at scale. Ad-hoc searches across diverse data fields were particularly problematic. For a large insurance company customer with billions of records, executing a complex query — such as identifying objects with specific attributes within a date range and data source — would traditionally have been inefficient and required pre-indexing that could rival the data size itself. Tomer Negbi, Director of Engineering at BigID, recognized that the company needed a fundamentally more scalable search architecture.

BigID migrated its data-driven modules to Elasticsearch running on Elastic Cloud with AWS as the cloud provider. The new architecture allowed customers to configure their data pipelines through BigID’s interface, specifying data sources and connectors, with scans prioritized by need. Elasticsearch handled the real-time indexing and analysis of unstructured data at any scale, enabling complex filtering and aggregation without pre-indexing penalties. Out-of-the-box Kibana dashboards surfaced key metrics, and customers could collaborate with BigID to build custom analytics tailored to their specific compliance or risk requirements.

The performance improvement was immediate and dramatic. Queries that previously took 20 minutes now completed in seconds — 120 times faster. Query timeouts on search, dashboard, and reports modules dropped to zero, replaced by consistently fast responses. This newfound reliability let BigID’s engineering team shift focus from building workarounds for performance issues to developing new features and capabilities.

BigID and Elastic work closely together, with BigID providing feedback on new releases and Elastic proactively anticipating product needs. Negbi describes the relationship as essential for a company where rapid innovation is a competitive requirement. BigID now has a platform capable of handling the most data-intensive enterprise use cases — enabling complex geographic, regulatory, and risk-based filtering — while remaining flexible enough to support the evolving demands of data security and compliance.