Match what your AI does with what users want to do

[MUSIC PLAYING] JANI CORTESINI: Hi, I'm Jani Cortesini, and I'm a strategic program manager in Google Core UX. Today, we're going to explore concepts from Google's people and AI guidebook that can help you translate user intent into tasks for your AI, create explanations that are useful and build trust, and turn errors into opportunities for user control. We'll deep dive into common challenges and provide recommendations, so you can start implementing people-first solutions right away. Let's get started. AI has fundamentally transformed how we build products. We've moved from predictable user journeys into open-ended scenarios. Our AI products are driven by natural language, simultaneously handle many modalities, and are now capable of taking action on a user's behalf. While this lets us support a long tail of use cases, we also become accountable for the uncertainty in the user experience that comes with it. This shift necessitates a new approach. Building trust is no longer a static, one-time achievement. Trust is now an ongoing critical process of continuous calibration. As product capabilities grow, user expectations can increase. An ideal trust level is where user expectations are aligned perfectly with product capabilities. Anything above is over-trust, and anything below is under-trust. The purple area is the zone where the continuous calibration takes place, but it is important to note that trust calibration rarely occurs in isolation. It happens collaboratively between system and user because trust can grow, but it can also regress. In reality, trust looks more like a sparkline, the purple line you see here, where users must experience both over- and under-trust in order to calibrate it, continuously aligning their expectations to actual product capabilities. Moreover, trust is also deeply influenced by users' past experiences, and their experiences today will shape their trust when you release your next AI update. To navigate this, we have a core philosophy. We must design AI with people not at people. So how do we do this and actually help users calibrate their trust? We can help them do so by defining what the AI actually does and matching it with what users want to do through the user experience. Our first step is to define how people can express their goals and determine the tasks the AI executes and how it does so. This is called Human-AI alignment. Broadly, this involves the user specifying what they want to do and how they want it done and then understanding and validating the AI's output and steering the AI iteratively until the user is satisfied. Then, we activate this using three foundational concepts and actions that drive our users' experience effectively matches what the AI does. We shape mental models, provide helpful explanations, and manage errors gracefully. Why is this important? For AI systems to be successful, it is critical to define the system's purpose based on a clear understanding of user needs. Human-AI alignment ensures that an AI model correctly infers a user's loose, casual language and turns it into specific, executable instructions that align with the user's true interests. This alignment covers both the task outcomes, the end result, and the process by which those outcomes are achieved. A failure to align can, for example, lead to reward hacking, where the model learns and executes a completely different goal, achieving the outcome in the wrong way. This risk is easily exacerbated when AI agents have the ability to take actions on a user's behalf. A foundational way to get started with Human-AI alignment is to clearly define what the AI system does by identifying its four core components. Let's look at what these are with some simple examples. The first one is the Primary Goal, the specific problem the AI should solve, for example, generating code. The second one is Sub-goals-- alternate goals, dependencies, or subproblems the user must address before or while addressing the primary goal, for example, picking a programming language. The next one is Underspecification, the parts of the problem or pieces of information that are frequently left out by users, for example, a developer forgetting to specify naming conventions they want the AI to use. And finally, we have Optimization, the varied ways to optimize the problem and understand how optimizing for one sub-goal might compromise another, for example, an AI system that optimizes for speed but at the expense of introducing inaccuracies. To make this highly practical, we're going to apply these concepts to a hypothetical enterprise product called Develocity, an integrated development environment that uses generative AI to help developers write, test, share, and manage their code. Develocity's models are trained on the organization's corpus, and it has AI coding assistants and agents designed to increase developer velocity while maintaining strict code consistency. So what is the primary goal for Develocity, the specific problem the AI should solve for the user? Well, an example could be efficiently generate code that fits within the organization's needs and style using organizationally approved resources. What are some sub-goals, again, alternate goals, dependencies or subproblems the user must address? In this case, sub-goal examples could be that the user needs to decide which programming language to use, set up storage resources, and import specific libraries. What could be underspecified? Which parts of the problem or pieces of information do users frequently leave out? For example, a user may underspecify the number of variables to initialize, specific naming conventions, or criteria for inclusive code. And finally, what are varied ways for Develocity AI to optimize the problem, and how could optimizing for one sub-goal compromise another? Here's an example. AI code generation can increase user efficiency. However, if optimized purely for speed, the AI might autocorrect errors without human oversight or use untested but inefficient methods that could inadvertently lead to data leaks. Now, despite best efforts, the system specification can inevitably clash with user expectations for a myriad of reasons, for example, because of the open endedness of user journeys and the myriad of paths available, the ambiguity of the language used to prompt the model and the different ways users can ask the AI to perform tasks, or even the inevitability of users bringing their own assumptions to the table and the system having no visibility into these. That is where the next three concepts and actions come in-- shaping mental models, providing helpful explanations, and managing errors gracefully. These are the foundations with which a system can connect and evolve with what users want to do. Let's look at each of these. A mental model is our starting point. It is the user's internal belief system about what the AI can do, can't do, and how it works. People form mental models of everything they interact with, including products, places, and people. Understanding mental models and incorporating ways to augment these can help users build an intuition for how they can leverage AI in products frequently and regularly. It is important to remember, however, that users only require a partial mental model or enough intuition to successfully use the product. They don't need to know precisely how it works. We also need to remember that mental models are dynamic and constantly shaped by external factors and experiences. A user's current mental model will not necessarily be their future one. Finally, people develop deep, resilient mental models through actual real world experience, which can override what they are explicitly taught. In other words, users learn effectively by experimenting and discovering the boundaries and limitations of the AI system themselves rather than through forced mechanisms such as onboarding or tutorials. This learning process through discovery is critical because mental models also have expectation setting power. They help us determine how much we might trust something and what value we might expect from it. And expectations can shape how users specify their intent to AI. If they over trust the AI, they might provide under-specified, vague prompts expecting the AI to read their mind. If they under trust the AI, it might end up inadvertently providing contradictory instructions or end up missing the exploratory power that your AI can offer them. Or if they expect human level understanding, a user might use language designed to motivate and describe the big picture. But if expectations are purely utilitarian and functional, they might provide more explicit, detailed instructions to AI for it to produce good results. Next, we have explanations, our bridge. These are in-the-moment cues and system feedback that can clarify how the AI makes decisions or responds to different user inputs. When the AI executes a specification or struggles to, explanations are how the system communicates its boundaries back to the user. Explanations actively help update and correct the user's mental model, ensuring their future inputs are better aligned with the system's true capabilities. And finally, we have errors, the ultimate stress test of your Human-AI alignment, the reality check. These are the moments when the system fails. The user makes a mistake, or the AI makes an incorrect assumption about what the user wants, the alignment errors we spoke about earlier. When an error occurs, you can use explanations to gracefully guide the user out of it. This is important because handled correctly, an error becomes a collaborative moment that actively improves the user's mental model and helps them calibrate their trust. To bring it all together, if you think of human and AI alignment, it is a way to ensure that what the AI actually does is actually what we want. This is a foundational step. However, this can clash with user expectations or users want to do with the system when using it. We design for this, shaping their mental models, providing helpful explanations and gracefully managing errors for ongoing improvement. Now, let's take all of these concepts and make them practical. We will do this again with Develocity, the hypothetical product that we introduced earlier. So let's start from how we can shape mental models. Imagine a scenario. You are rolling out Develocity to a new cohort of engineers. What is an important consideration here? A new user isn't static. They bring existing mental models, like traditional ID autocompletion, and will rapidly evolve into expert users, expecting complex, context-aware assistance. If you only design for a static persona, the product breaks down as they grow. We must design for this progression cycle by tackling the subsequent problems. Our first one is the blank slate problem, the start, the first use. The problem here is that we don't know what users will actually ask an open-ended AI coding assistant initially. It's safe to assume that users will intently explore the capabilities and limitations of your AI system with a set of goal to tasks. They'll refer back to their experiences of how other AI and non-AI tools perform these to decide how much they trust and value your AI. In this case, it becomes important to understand what the user wants, so you can help them create a good basis for understanding your product's full potential. For example, your AI could ask users clarification questions so it can act as a thought partner to refine a user's goals and understand their expectations. Once the task has been broken down sufficiently for your AI and the process the user wants is clear, you can help them express their intent better by proactively suggesting relevant and valuable tasks. For example, instead of a blank input, the Develocity AI panel could suggest actions based on the developer's current context, like find bugs in this function or generate unit tests, helping the user create a mental model of how to steer the AI. Inversely, you might use the inputs and responses to clarification questions to map out their actual initial needs, not your assumed ones. The next problem we have is that of old mental models, what users carry with them from before. Here's a situation. New employees expect Develocity to act like a standard predictive text, leading to frustration when it suggests entire architectural changes. Users expect to reuse or build on existing mental models from similar technology. Their past experience establishes their beliefs about what constitutes basic functionality and what generates meaningful value for them. Users will use their existing mental models to gauge if your AI deviates from their fundamental expectations. How can we solve for this? We need to explicitly onboard users into the product's way of operating. While an onboarding flow that shows users how something works seems concise and efficient, users invariably skip these. Tutorials stick a little more but only because users are purpose driven and looking for specific information. So taught mental models are often shallow and ephemeral. Users learn deeper, more resilient mental models through real-world experience, as we saw earlier. Therefore, we need to design systems that help users form a reasonable working mental model about how a task is executed in an agentic workflow and demonstrate if the benefits clearly outweigh the cost of introducing deviations to the user's task or desired process. Consider this case in which the user has provided an open-ended task-- create an application for a simulation, which may be executed in numerous ways. Here, we should empower the AI to translate high level intent into a prioritized plan that the user can refine. The AI may show users different approaches they might take, highlighting similarities, differences, and potential outputs. It's important to give them proper system cues and controls to compare options and make decisions. This allows the user to adjust their mental models of how the agent might work in practice until they're sufficiently able to decide which processes they can trust the AI to execute with minimal oversight. Finally, we have our third problem-- moving users from beginner to expert. As users move beyond early explorations, they start using your product's outcomes for decision making and real tasks with real consequences. For example, Develocity will need to factor in collaborative workflows and how AI might alter or shape expected interactions between multiple stakeholders. Users may need to unlearn old mental models and replace these with new ones. This is also a reason why switching costs can be high. In a high complexity tool like Develocity, a novice user may quickly become a power user, even if they may never become 100% true experts. They may need to participate in a new type of code review conversation that didn't exist before, and reviewers may need to be trained in AI-specific review techniques and new evaluation frameworks. When people get used to your product's AI capabilities, their interactions with your product will become more intentional. Here, how your product guides them will determine the benefits they experience. What this means is that as users increase the complexity of their tasks and expectations, they will inevitably find the boundaries of your system. Therefore, we have to acknowledge AI failures, offer alternatives, and ask for user feedback when the AI fails. But even when the AI succeeds, we should still acknowledge user queries, explain, and ask for feedback. This gives us insight into moments where people are swapping working mental models for better ones. As an example, we can see in the image that when a user enters a prompt that's out of scope, the AI response first acknowledges this boundary. This is followed by a contextually relevant and concise explanation. A list of alternative actions gives the user a way forward. To ease their frustration, users can additionally follow a link to documentation to learn more or provide meaningful feedback on the usefulness of response one. The constant cycle of casting aside and reframing our mental models means today's mental models may not be the ones of the future. But the metaphors, analogies and strategies used to guide users today set the stage for tomorrow. This leads into providing helpful explanations. Here's a scenario again. A developer is using Develocity for a complex refactor. What is important here? Trust is the ultimate currency of this interaction. Too little trust, and they abandon the tool. Too much trust, and they push broken code to production without verifying it. An under-trust is what we see in the image. Develocity functions as a standard code editor with all AI features turned off. The user has turned off AI features simply because it fails too much. For example, Develocity fails a couple of times earlier on because it doesn't have access to an internal API, and the developer decides the tool is useless, ignoring future valid suggestions. An over-trust example is what we see in this other image. Develocity generates a perfectly formatted, syntactically accurate block of code but hallucinates an inaccurate, deprecated, or non-existent package dependency. It may be so subtle that you might miss it if you weren't looking for it. Before jumping into specific examples, it is useful to consider how to make explanations as helpful as possible. In the moment, explanations are most helpful when contextually defined. There are three different types of explanation to help achieve this. We invite you to think about these in the context of your own AI product or feature. Our first one is proactive. These are explanations that describe how the AI arrived at an outcome in terms of broad cause and effect relationships to calibrate the user's intuition about AI behaviors. These often accompany the AI outcome, for example, in the form of a model's chain of reasoning, which describes how the model understands inputs and reasons about tasks. Influential feature explanations describe key factors or data types that most significantly influence an AI outcome. Such explanations attempt to answer what influenced the system into selecting outcome X, for example, letting the developer know that specifying skills for their agents will result in significantly better tool usage, not just in terms of what tools are used, but how they are used. Influential features are easily described through simple sentences or illustrations that generalize and make cause and effect relationships evident to the user. This, in turn, has the effect of aligning plausible, under the hood, explanations of how an AI arrived at an outcome with human intuition. It's worth noting that influential features are different from attribution, which involves being able to factually trace an AI output, the sources in the data. Contrastive explanations describe features and attributes that were or were not used when AI made one decision over another. A pure contrastive explanation attempts to answer why did the AI system select X over Y? A counterfactual explanation is a type of contrastive explanation that is more interventional, answering what is the smallest possible change to the input that would make the AI system select Y instead of X? While influential feature explanations help users intuit cause and effect relationships, contrastive explanations help them narrow down the space of these cause and effects, for example, comparing alternatives to the same input to establish how outputs might change without asking users to commit to the alternatives. Next, we have immersive explanations. These are particularly useful in scenarios where the user interacts with your AI system over time. These are effective where your user journey requires people to actively steer and help the AI system along. Immersive explanations can be both implicit and explicit but rely on human intelligence. Immersive explanations, therefore, draw their power from how well your AI feature or product has been designed. A poorly designed one is less likely to convey implicit explanations than a thoughtfully designed one. As users interact with your AI product, example-based explanations offer users relevant examples that illustrate when or why an AI behaves in a particular way. These examples come from user-engineered contexts for the AI or, where necessary, the model's training set or tuning set. By analyzing examples, users can independently deduce why the AI produced a specific output and determine how much to trust those results in their particular context. Even if inaccurate, these conclusions help people understand the general process by which the AI system produces outputs and validate whether the output can help them meet their objectives. Another way to explain AI systems and help users build more accurate mental models is by giving them the tools to experiment dynamically and ask what if questions. People often provide richer information about their context, detailed task descriptions, and more descriptive context, anticipating that the AI system will accommodate these details. This is most useful for steerable AI systems, where AI is collaborative. Intentionally designing experiences that allow users to engage with the AI on their own terms can lead to increased understanding and usability and better calibrated trust. Finally, we have metric-driven explanations. As the name suggests, these include model metrics and data sources when offering outcomes. They may be generated by the model itself, summarize different data sources, or may be computed after the model has produced outcomes. Throughout their product experience, users interact with data derived from two primary sources. The first is user-provided data, including user inputs, feedback, product logs, preferences, and history. The second encompasses data used to train or tune the AI system alongside any other signals the system utilizes as input features. Letting users know which parts of the data they have provided is used by the AI system, data which uses a given consent to capture, for example, prompts or path searches, can help users identify discrepancies between their own knowledge and that of the system and help the AI provide a better output. This transparency can prevent over-trust and undue reliance on AI outcomes, particularly when the user has more complete or current information than the system. Certain metrics can offer insight into why an AI system produced certain outcomes and establish clear thresholds of quality and confidence above an AI output is acceptable or relevant to the user. For metrics to be easily understood, they should align with user's mental models of your AI product. This can require you to translate a technical metric, either produced by the model or evaluated post-hoc, into an indicator or representation that is relevant to the task and meaningful to users when selecting from different AI outcomes. So we've looked at how people form and update their mental models and the different types of explanations that can help them do so. A user's mental model and the explanations we provide also play a role in how users recalibrate their trust after encountering an error. So let's go through practical ways to manage errors gracefully. Here's a situation. What happens if the model uses a tool incorrectly to perform a task and that leads to a failure state? Agents and agentic workflows are powerful because they excel at reasoning, being flexible, and handling ambiguity. But this comes at a higher development and cognitive cost. The tool failure state may present itself in a range of ways-- general systems, where it impacts how the whole system behaves; partial systems, where it affects a key element or specific feature; specific output, appearing in individual AI outcomes. What can we do about this? The first step is to identify if this is a possible system hierarchy error-- errors arising from using AI systems in which components of it may not be communicating with one another, and the hierarchy of tool use and operations may not be clear. In our scenario, the system should help users choose the right agent for the task. When adding tools for agents, make sure that each one has a clear description and a narrow, focused scope to minimize any agent confusion. You might even need to architect your AI feature to frequently declutter any low impact or unnecessary tools from the agent's context. Even in the best agentic systems, it's worthwhile to always remind the user to treat the AI's output as a starting point and to always verify the results. You can see in the image an example chain of thought description, in which Devlo is selecting from multiple tools with clear descriptions, a persisting message to users to verify results. Here is another scenario. The developer provides an ambiguous prompt or insufficient context that they think is sufficient, such as fix the whole thing. When we ask our human counterparts to perform tasks, we align on the motivations underlying the task, trusting them to use their creativity, judgment, and expertise to figure out the process. In contrast, we have managed deterministic AI systems by prescribing the specific actions-- the how-- especially if we don't yet trust them to navigate ambiguity or infer intent. In agentic workflows, it's reasonable for users to assume that the AI will figure out the how from a simple description of what the user wants to achieve. This description might even include the why, which, when read by a human, would empower iterative and collaborative explorations. Back to our scenario, the AI outcomes may have errors stemming from inaccurate user expectations that the system will understand their intent, regardless of AI capabilities. Perhaps the user has overestimated the capabilities and provided an input that is beyond the scope of what the AI system can do. Or they expect their input to be automatically corrected before being sent to the AI system. Unlike humans who can get overwhelmed, AI can handle a lot of instructions. The challenge lies in discovering the exact problem that users want solved and sometimes helping them discover it. Instead of guessing and failing silently, create your AI with explicit built in tripwires to force it to pause and seek feedback. For example, Develocity may respond with I see three potential data bugs in the current file. Did you mean the timeout issue, the indexing issue, or the connection string? Asking clarifying questions or co-creating plans with the user can turn a potential failure into a collaborative, trust-building moment. And in our final scenario, the AI agent gets stuck in a loop while partially completing a task but will gladly hallucinate and report that it's doing something productive. Spotting this type of behavior demands patience and can create enough frustration for the user to restart processes and lose vital progress. Consider that even with chain of thought reasoning, it is hard to verify the exact cause because such errors can stem from issues with models or the data used to train and tune them. Therefore, check if data used in training, tuning, and testing agentic workflows is actually representative of real use cases and comprehensive enough to capture the edge cases of your AI's capabilities. Despite adequate data, if AI outcomes continue to remain consistently incomplete or inaccurate, or if your AI system remains insensitive to inputs that users care about, perhaps the underlying training or tuning practices are inadequate. To solve for this, revisiting your training and tuning process may be a longer term approach. But this presents the opportunity to conduct human in the loop evaluations with users and domain experts who can provide real-world context on how your AI system should behave, especially when it loops or fails. More immediately, consider that a single, large instruction set can have a finality to it, and that may make workflows less iterative and harder to debug. Instead, design your system with tight feedback loops. Under the hood, break down large instructions into smaller ones, saving progress as it's made. Consider instituting a timeout at which your system offers users quick self-correction paths, while allowing the user to make targeted edits to the code. As in the image, Devlo's correct thinking trace and tool use is transparently shown to the user to help debugging. Crucially, the system should capture these targeted user edits as implicit feedback. Observations of tool usage patterns and successful steps further enrich implicit feedback. More explicit instructions like remember this for next time are also a form of high value feedback that can reduce repetition and help your AI adapt to the user's workflow style. By transforming these in-the-moment corrections into a feedback loop, the system can continuously fine tune the model to prevent these errors over time, improving the experience. So let's get ready to wrap up. The transition to frontier agentic AI is a fundamental shift in the relationship between humans and software. We are moving from building deterministic tools that execute commands to designing probabilistic systems that interpret and act on intent in powerful ways. In this new paradigm, the success of your product isn't measured solely by the intelligence of the underlying model but by the resilience of the Human-AI interaction. Trust through Human-AI alignment is the key for achieving this, defining what we want to do and how it is done. However, trust is not a switch that we flip on during onboarding and forget about. It is a spectrum that must be continuously calibrated by shaping mental models, providing helpful explanations, and managing errors gracefully. What the AI does needs to match what users want to do on an evolving basis. So let's leave you with four key takeaways from everything that we have explored. Human goals are inherently messy, contextual, and evolving. Do not rely on abstract system policies alone. Use Human-AI alignment to build a continuous translation layer, systematically mapping open-ended user desires into safe, optimized, and actionable constraints. Don't assume that users instinctively know how to collaborate with AI. It is our responsibility to shape their expectations. Move beyond static tutorials by designing adaptive interfaces that reveal the system's capabilities progressively. Scaffold the journey so users can safely graduate from simple, assisted tasks to complex, autonomous workflows. The greatest risks in AI are over-trust, for example, accidentally blindly accepting hallucinations, and under-trust, abandoning the tool entirely. Actively design against both. Introduce strategic friction, like well-timed explanations, confidence scores, or verification steps, forcing users to slow down and understand why a decision was made when the stakes are high. In probabilistic systems, edge cases and errors are inevitable. Rather than hiding them, treat them as your most valuable feedback loop. Design graceful fallbacks and intuitive correction mechanisms that capture the user's intent when the system fails. Every user correction is an opportunity to refine the AI specification and improve the product for everyone. In conclusion, that is why we must build AI with people, not at people. By designing for continuous calibration across AI systems and users, we stop treating users as passive consumers of AI outputs and actions and empower them as active co-creators. Start integrating all of this into your development cycles today. For practical tools, frameworks, and deeper explorations of these concepts, refer to the paired guidebook. [MUSIC PLAYING]