Many organizations will be hesitant to make radical changes to their technology infrastructure and start from scratch. No perceptionFor version 3.0 of its productivity software (released in September), the company didn’t hesitate to rebuild from the ground up; He recognized that it was really necessary to support agentic AI at enterprise scale. While traditional AI-powered workflows involve clear, step-by-step instructions based on few-shot learning, AI agents driven by advanced reasoning models are thoughtful about tool definition, can recognize and understand what tools they have and plan next steps. “Instead of trying to recreate what we were building, we wanted to play to the strengths of the logic model,” Sarah Sachs, Notion’s head of AI modeling, told VentureBeat. “We rebuilt a new architecture because the workflows are different across agents.”
Rearranging so models can work autonomously
94% of the Forbes AI 50 companies have embraced this notion, it has a total of 100 million users and counts OpenAI, Cursor, Figma, Ramp, and Vercel among its customers. In the rapidly evolving AI landscape, the company identified the need to move beyond simple, task-based workflows toward goal-oriented reasoning systems that allow agents to autonomously select, organize, and execute tools in connected environments.
Very quickly, the logic models became “much better” at learning to use the tool and follow chain-of-thought (COT) instructions, Sachs said. This allows them to be “far more independent” and make multiple decisions within an agentic workflow. “We rebuilt our AI systems to deal with that," He said. From an engineering perspective, this means replacing rigid prompt-based flows with a unified orchestration model, Sacks explained. This core model is supported by modular sub-agents that search Notion and the web, query and add to the database and edit content. Each agent uses tools contextually; For example, they can decide whether to found Notion themselves or on another platform like Slack. The model will search sequentially until relevant information is found. For example, it can convert notes into proposals, create follow-up messages, track tasks, and make updates to knowledge bases. Sachs said that in Notion 2.0, the team focused on getting the AI to perform specific tasks, which required them to “think broadly” about how to motivate the model. However, with version 3.0, users can delegate tasks to agents, and agents can actually take actions and perform multiple tasks simultaneously. “We’ve redefined it to be self-selecting on the tool instead of a few-shotting, clearly indicating how to go through all these different scenarios,” Sachs explained. The aim is to ensure that everything interfaces with AI and “whatever you can do, your Notion agent can do too.”
Dichotomize to differentiate hallucinations
Notion’s philosophy of “better, faster, cheaper” drives a continuous iterative cycle that balances latency and accuracy through fine-grained vector embeddings and elastic search optimization. Sachs’ team employs a rigorous evaluation framework that combines deterministic tests, local language optimization, human-annotated data, and LLM-A-Judge, identifying anomalies and inaccuracies with model-based scoring. “By breaking down the assessment, we are able to identify where the problems come from, and that helps us isolate unnecessary hallucinations,” Sachs explained. Additionally, simplifying the architecture means it will be easier to make changes as the model and technology evolve. “We optimize latency and parallel thinking as much as possible,” Sachs said, which leads to “better accuracy.” The models are based on data from the web and Notion Connected Workspaces. Ultimately, Sachs pointed out, the investment in rebuilding its architecture has already provided notional returns in terms of capacity and faster rate of change. He added, “We are fully prepared to recreate this if we have the next success.”
Understanding Contextual Latency
When building and fine-tuning models, it’s important to understand that latency is subjective: AI should provide the most relevant information at the expense of speed, not necessarily the most. “You’d be surprised at how willing customers are to wait for things and not wait for things,” Sachs said. It’s an interesting experiment: How slow can you go before people abandon the model? For example, with pure navigational search, users may not be as patient; They want immediate answers. “If you ask, ‘What’s two and two,’ you don’t want to wait for your agent to search everywhere in Slack and JIRA,” Sachs explains. But the more time given, the more detailed the reasoning agent can become. For example, Notion can perform 20 minutes of autonomous work on hundreds of websites, files, and other content. Sachs explained that in these cases, users are more willing to wait; They allow models to execute in the background while they perform other tasks. “It’s a product question,” Sachs said. “How do we set user expectations from the UI? How do we gauge user expectations on latency?”
Notion is its biggest user
Notion understands the importance of using its own product – in fact, its employees are among its biggest power users. Sachs explained that teams have active sandboxes that generate training and evaluation data, as well as a “really active” thumbs-up-thumbs-down user feedback loop. Users don’t hesitate to say what they think should be improved or features they would like to see. Sachs emphasized that when a user declines an interaction, they are implicitly giving permission to a human annotator to analyze that interaction in a way that anonymizes them as much as possible. “As a company we’re using our own tools all day, every day, and so we get really fast feedback loops,” Sachs said. “We’re actually dogfooding our own product.” That said, it’s their own product that they’re making, Sachs said, so they understand that they can have the specs up to date when it comes to quality and functionality. Notion has been relied upon to balance this "very AI-savvy" Design partners who are given early access to new capabilities and provide critical feedback. Sacks emphasized that this is just as important as the internal prototype. “We’re all about experimenting out in the open, I think you’ll get a richer response,” Sachs said. “Because at the end of the day, if we just look at how Notion uses Notion, we’re not really giving our customers the best experience.” Equally important, continuous internal testing allows teams to evaluate progress and ensure that models are not lagging behind (when accuracy and performance decline over time). "Whatever you are doing remains faithful," Sachs explained. "You know your latency is within limits."
Many companies make the mistake of focusing too much on retrospective-focused Evans; Sachs explained that this makes it difficult for them to understand how and where they are improving. Notion treats evals as a "litmus test" Levels of development and forward-looking progress and observability and regression certification. “I think a big mistake a lot of companies make is mixing the two,” Sachs said. “We use them for both purposes; we just think about them really differently.”
Highlights from Notion’s journey
For enterprises, Notion can serve as a blueprint for responsibly and dynamically operating agentic AI in a connected, permissioned enterprise workspace. SAC’s tips for other tech leaders:
-
Don’t be afraid to rebuild when core capabilities change; Notion completely re-engineered its architecture to align with logic-based models.
-
Treat latency as contextual: optimize per use case rather than universal.
-
Ground all outputs in trusted, curated enterprise data to ensure accuracy and confidence. He advised: “Be prepared to make tough decisions. Be prepared to sit at the top of the line on what you’re developing to create the best product for your customers.”
