How Edmunds Uses Databricks and GPT-4 to Automate Dealer Review Moderation

Edmunds processes more than 300 dealer quality-of-service reviews daily, and the accuracy of that content directly shapes which dealers prospective car buyers choose to trust. For years, a small moderation team manually evaluated every submission, checking whether each review pertained specifically to dealer service quality rather than the vehicle itself. The process could take up to 72 hours from submission to publication, limiting the freshness of information available to users and creating a bottleneck that grew with review volume.

The team's first attempt to automate moderation used a fine-tuned off-the-shelf model, but the results were insufficient. The rules for determining a valid dealer service review were complex, and fine-tuning alone could not capture the full range of edge cases. Technical Consultant Suresh Narasimhan described the conclusion plainly: the model needed all the rules expressed directly in the prompt rather than learned through training, and different foundation models handled those prompts with different accuracy and consistency. Without a way to efficiently compare model outputs, evaluating alternatives was slow and difficult.

The team turned to Databricks Model Serving, which consolidates third-party LLM providers alongside custom-served models inside a unified environment with shared permission management and rate controls. This made it straightforward to test GPT-4 against other models using the same prompt and evaluation framework. GPT-4 accessed through Databricks Model Serving with a detailed custom prompt proved most effective, and the system went into production. Alongside the moderation layer, Staff Engineer Sam Shuster led a migration from the team's existing Hive metastore to Databricks Unity Catalog, replacing coarse IAM-based access controls with fine-grained table and S3 permissions and adding automatic pipeline lineage tracking.

The automated moderation system now processes incoming reviews and issues accept or reject decisions in seconds. Turnaround time dropped from as long as three days to minutes, and the moderator headcount required to oversee the workflow fell to two. The team estimates three to five hours of moderator effort saved per week. The Unity Catalog migration delivered parallel improvements in auditing, compliance, and data discovery, and reduced the number of pipeline failures caused by undocumented upstream changes.

Edmunds is now extending the generative AI moderation approach beyond dealer quality reviews to cover all review categories on the platform. VP of Technology Greg Rokita sees Databricks as central to that expansion, describing the platform's unified treatment of data warehousing and AI modeling as essential for building systems that can reason across both historical data and forward-looking predictions without switching between separate infrastructure stacks.