How BigID Uses Elasticsearch to Accelerate Data Queries 120x at Scale
BigID, a data security, privacy, compliance, and AI data management platform founded in 2016, deployed Elasticsearch on Elastic Cloud and AWS to overcome severe query performance degradation as its customer data volumes grew. By migrating its core data-driven modules to Elasticsearch, BigID cut query times from 20 minutes to seconds — a 120x speedup — eliminated all query timeouts on search, dashboard, and reporting modules, and built a foundation capable of handling billions of records with complex filtering and aggregation.
Impact
120x
Query speed improvement
-100%
Query timeout rate
Challenge
As BigID scaled, its existing data storage platform required frequent schema migrations and index additions to handle growing customer data volumes, crippling ad-hoc search performance. Complex queries on datasets with billions of records took up to 20 minutes and frequently timed out, degrading the customer experience and blocking engineering velocity.
Solution
BigID migrated its core data-driven modules to Elasticsearch on Elastic Cloud and AWS, enabling real-time indexed search and analysis of unstructured data at any scale — with customer-configurable pipelines, complex filtering and aggregation on billions of records, and Kibana dashboards for out-of-the-box and custom analytics.
Tools & Technologies
What Leaders Say
“Queries which previously took 20 minutes now run in a matter of seconds, 120 times faster — that’s a massive improvement.”
“Elastic Cloud’s scalability is perfectly aligned with our customers’ needs: handling vast data volumes with intricate details and enabling complex filtering and aggregation.”
Sign up to read complete case studies, access detailed metrics, and unlock all use cases.
Full Story
BigID helps organizations gain a comprehensive view of their data scattered across cloud, SaaS, and on-premises environments. Its platform enables clients to discover sensitive data, address security vulnerabilities, simplify compliance adherence, and govern AI data effectively. As BigID’s customer base grew and data volumes scaled, maintaining the performance its customers required became increasingly difficult.
The core challenge was search. BigID’s existing data storage platform required frequent schema migrations and index additions just to keep pace with growing data volumes — a brittle approach that became harder to sustain at scale. Ad-hoc searches across diverse data fields were particularly problematic. For a large insurance company customer with billions of records, executing a complex query — such as identifying objects with specific attributes within a date range and data source — would traditionally have been inefficient and required pre-indexing that could rival the data size itself. Tomer Negbi, Director of Engineering at BigID, recognized that the company needed a fundamentally more scalable search architecture.
BigID migrated its data-driven modules to Elasticsearch running on Elastic Cloud with AWS as the cloud provider. The new architecture allowed customers to configure their data pipelines through BigID’s interface, specifying data sources and connectors, with scans prioritized by need. Elasticsearch handled the real-time indexing and analysis of unstructured data at any scale, enabling complex filtering and aggregation without pre-indexing penalties. Out-of-the-box Kibana dashboards surfaced key metrics, and customers could collaborate with BigID to build custom analytics tailored to their specific compliance or risk requirements.
The performance improvement was immediate and dramatic. Queries that previously took 20 minutes now completed in seconds — 120 times faster. Query timeouts on search, dashboard, and reports modules dropped to zero, replaced by consistently fast responses. This newfound reliability let BigID’s engineering team shift focus from building workarounds for performance issues to developing new features and capabilities.
BigID and Elastic work closely together, with BigID providing feedback on new releases and Elastic proactively anticipating product needs. Negbi describes the relationship as essential for a company where rapid innovation is a competitive requirement. BigID now has a platform capable of handling the most data-intensive enterprise use cases — enabling complex geographic, regulatory, and risk-based filtering — while remaining flexible enough to support the evolving demands of data security and compliance.
Amazon Bedrock