How Notion Powers Precise AI Search for Millions with Cohere Rerank

Notion operates at a scale that makes search infrastructure a core product concern rather than a technical detail. The platform hosts collaborative wikis, project management systems, and connected knowledge bases for companies ranging from small startups to global enterprises — and as those organizations grow, so does the volume of documents, databases, and integrations they manage in Notion. When Notion AI launched, delivering accurate, fast answers to user queries across this heterogeneous body of content became a critical product requirement. The technical challenge was finding a reranking solution that could match the precision users expected from AI-native features.

Before integrating Cohere Rerank, Notion’s search pipeline faced a structural tension: embedding models were expensive to run at scale and required vector storage infrastructure, but simpler retrieval methods lacked the precision needed to surface the most relevant documents from large workspaces. The challenge was compounded by Notion’s global user base — more than half of whom work with multilingual content in EMEA and APAC markets — requiring a solution that could operate accurately across languages without separate models or additional engineering overhead.

Notion integrated Cohere Rerank directly into the search pipeline, placing it before the generative model processes any user query. The implementation used Amazon SageMaker for auto-scaling, allowing Notion to handle variable traffic loads without over-provisioning. A particularly important design choice: for workspaces with fewer than 1,000 documents — the majority of Notion’s customer base — Cohere Rerank allows Notion to bypass traditional embedding and vector search entirely, reducing both cost and system complexity while maintaining high answer quality. The same Rerank model also handles cross-source result combination, merging outputs from Slack, GitHub, and other connected tools into coherent, ranked answers.

The precision improvement over embedding models has been substantial. As software engineer Abhishek Modi explained, Cohere Rerank is dramatically more accurate than embedding approaches at scoping large document sets down to the most relevant subset — moving from a retrieval pool of 100,000 documents to 200 with much higher fidelity. Co-founder and CTO Simon Last described Rerank as essential to getting Notion AI Connectors out the door quickly, citing both speed and ongoing improvement as differentiating factors. The result: millions of Notion users have engaged with Notion AI features, with rapid month-on-month growth in usage and a meaningful contribution to company revenue.

Notion’s implementation illustrates how a consumer-grade AI experience depends on enterprise-grade retrieval infrastructure underneath. The decision to partner closely with Cohere — rather than relying on open-source alternatives that would require self-maintenance — reflects a strategic bet that rapid model improvement and collaborative support would be a durable competitive advantage as Notion AI scales.