Case Studies

Every framework in this course was built backwards — by observing what worked and what failed in shipped products, then generalising. That means the most reliable way to internalise the frameworks is to run the process in reverse: start from a product, peel back the design decisions, and ask why each choice was made.

This lesson does that for five products across four categories. In each case we ask the same set of questions:

You do not need to have used each product deeply. The design patterns are visible from even a single session — and many decisions are evident from published case studies, help documentation, and onboarding flows.

The goal is not to produce verdicts about whether each product is good or bad. The goal is to see the decisions clearly enough that you can make better ones yourself.

The product category: AI writing assistants embedded inside a document editor — the model that Notion AI, Google Docs' "Help me write," and similar tools follow.

The problem being solved. Writers face two distinct problems: the blank page (getting started) and the stuck page (continuing when momentum stops). Both are caused by the same thing — converting a fuzzy intent into specific words is cognitively expensive. AI reduces that cost by producing a draft the writer can react to, which is faster than producing one from nothing.

UX pattern chosen: Inline suggestions. The AI lives inside the writer's existing workflow, not in a separate chat window. This is the right pattern here because context is the product: the AI reads what has already been written and continues it, or responds to a command that refers to selected text. Moving to a chat window would break the writing flow and force the writer to copy text back and forth.

Trust and transparency. Writing assistants handle a trust problem that other categories largely avoid: the output becomes the user's own work. Readers of the final document will not know which sentences were AI-generated. This creates two sub-problems:

1. The user needs to trust that the suggestion is accurate enough to use (low stakes for creative text, higher stakes for factual claims embedded in a business document).
2. The user needs to feel that accepting a suggestion is a genuine choice, not a subtle pressure.

Well-designed writing assistants solve both with visual separation: the suggested text appears in a distinct colour or fade, and it stays on-screen only until the user explicitly accepts, rejects, or replaces it. The visual boundary makes clear that the AI is offering, not deciding.

Error handling. Writing AI fails in two modes: factual hallucination (inventing a claim or figure) and stylistic mismatch (producing text that does not sound like the user). Factual hallucinations are managed by keeping human review responsibility explicit — the accept gesture signals that the writer has read and endorsed the text. Stylistic mismatches are managed by short suggestions (two to four sentences) rather than long autonomous passages, because a small mismatch is easy to fix and a page-length mismatch requires starting over.

Feedback loops. Acceptance rate — what percentage of generated suggestions the user accepts unedited — is the primary implicit signal. Products in this category also track edit distance after acceptance: how much does the user change an accepted suggestion? High edit distance means the suggestion was technically accepted but practically wrong. The healthy metric is: accepted and used without major revision.

What the best products get right that weaker ones miss. The strongest writing assistants show the suggestion in the right place at the right time — triggered by a pause in typing or an explicit command, never pushing text onto the screen uninvited. Uninvited text feels intrusive; invited text feels helpful. The same model capability, with different trigger design, produces completely different trust responses.

The product category: AI features built into an email client — the pattern demonstrated by Gmail's Smart Reply, Smart Compose, and Priority Inbox, and later by tools like Superhuman's AI summaries and draft generation.

The problem being solved. Email volume exceeds most people's capacity to process it. Two tasks consume disproportionate time: reading and routing (deciding which messages need a response, which need to wait, and which can be ignored) and composing (writing responses for messages that follow a predictable pattern). Both are rule-following under noise, which is a strong fit for AI.

UX patterns in use. Email AI products typically deploy two patterns simultaneously:

• Ambient intelligence for triage — the AI filters, prioritises, and surfaces messages without asking the user to do anything. The experience is that your inbox simply shows the right things first.
• Inline suggestions for composition — suggested replies and sentence completions appear as the user is reading or typing, without switching contexts.

This dual pattern creates a design tension: ambient AI is invisible by design, while inline AI is visible by design. Products that handle this well use ambient AI for low-consequence routing (ordering messages) and visible AI for consequential composition (words that will go to another person).

Trust challenges unique to email. Email has higher interpersonal stakes than a private document. An AI-generated reply carries your name and reaches another human who may adjust their relationship with you based on it. The trust problem is: users need to feel in control of their voice in a channel that is deeply personal.

Gmail's Smart Reply handles this with very short suggestions — three options, typically five words or fewer. A five-word suggestion is fast to evaluate and easy to edit. A full-paragraph suggestion requires careful review and creates editing work that undermines the time-saving premise. Length management is trust management here.

Error handling and degradation. Triage AI fails when it misprioritises — surfacing a low-urgency message prominently and burying an urgent one. The graceful degradation pattern here is to make the triage decision reversible and visible: users can see that messages have been sorted, they can move things manually, and the category labels ("Important," "Other") are always explicit. Invisible sorting that users cannot inspect or override destroys trust when it gets something wrong.

Priority Inbox's decision to show visible category labels rather than simply silently re-ordering was a significant trust design decision. It turned ambient AI into inspectable AI, which allowed users to calibrate their trust over time.

Feedback loops. Email triage products rely almost entirely on implicit feedback: opening a deprioritised message tells the system it got the importance wrong; deleting without opening a prioritised message tells it the same. No thumbs-up or thumbs-down required. The richness of email interaction data (open, reply, archive, delete, move) gives triage models an abundance of signal that most AI products do not have.

What transfers. The key lesson from email AI is that ambient intelligence only works if users can see enough to calibrate trust, even if they do not see everything. Full transparency and full invisibility are both worse than inspectable ambient AI.

The product category: AI code completion tools — the pattern established by GitHub Copilot and followed by Cursor, JetBrains AI, and similar tools embedded in development environments.

Why this case study matters for non-engineers. You do not need to write code to draw lessons from code completion AI. It is the most instrumented, most studied inline AI product category available, and the design decisions it made — particularly around trust, latency, and feedback — have become reference patterns for every other inline AI tool.

The problem being solved. Experienced engineers spend a surprising amount of time on predictable work: boilerplate structure, common patterns they have written dozens of times, and function signatures they can almost remember. This is not the thinking part of programming — it is the typing part. AI handles typing; engineers handle thinking.

UX pattern: Inline suggestions. Code completion is inline suggestions in their purest form. The suggestion appears at the cursor, in context, and accepts or declines with a single keystroke. No modal, no chat window, no output to copy.

Trust signals in code completion. Code carries real consequences — it executes. An incorrect writing suggestion is embarrassing; an incorrect code suggestion is a potential bug or security flaw. The trust design problem is acute.

GitHub Copilot's approach was to show the suggestion visually but never explain it, trusting engineers to read and verify code before accepting it. This works for an expert audience: engineers are trained to read code critically. The same approach would fail for an audience less equipped to verify the output.

The deeper trust signal is latency. Suggestions that appear within a few hundred milliseconds feel like a responsive tool; suggestions that take several seconds feel like waiting for an external service. Perceived responsiveness is a proxy for reliability. Products in this category invest heavily in latency optimisation precisely because speed affects trust independently of accuracy.

Error handling. Code completion fails in two ways that have driven significant design work:

1. Plausible but wrong suggestions — code that looks correct and compiles but has a subtle logical error or uses a deprecated API. The graceful degradation pattern is to show multiple alternatives (Copilot shows up to ten options with a keyboard shortcut) so the user is thinking comparatively rather than accepting the first thing they see.
2. Context-blind suggestions — the model does not understand the broader system the code lives in, so it proposes something that works in isolation but breaks something elsewhere. This is harder to handle with UX alone, and the honest answer is to set expectations: inline suggestions are line-level and function-level assists, not architectural ones.

Feedback loops and the acceptance rate metric. Acceptance rate — what percentage of shown suggestions the user accepts — became the primary health metric for code completion AI. But teams studying Copilot found that acceptance rate alone was misleading: a user who accepts every suggestion without review inflates the metric without receiving value. The more informative metric is acceptance plus retention: was the accepted code still present ten minutes later, or was it deleted?

Edit distance after acceptance, combined with whether accepted code survived to commit, gives a far richer picture of whether the AI is genuinely helping.

> Evaluate the AI feature in your product: if you could track only two implicit feedback signals, which two would most reliably indicate that the AI helped a user accomplish something real — not just that the user interacted with it?

What transfers. The code completion case study is a masterclass in metric design. The lesson is that engagement metrics (suggestions shown, suggestions accepted) measure how often users interact with AI; outcome metrics (accepted code retained, task completed faster) measure whether the AI produced value. Building toward outcome metrics from day one prevents teams from optimising for activity and calling it success.

The product category: AI-driven recommendation systems in consumer products — the pattern used by Spotify's Discover Weekly, Netflix's homepage recommendations, and similar features in streaming, e-commerce, and social platforms.

The problem being solved. Catalogue depth is simultaneously a product's greatest asset and its most difficult UX challenge. A catalogue with ten million songs cannot be meaningfully browsed. Recommendations replace browsing with surfacing: the AI presents a curated slice of the catalogue it predicts will match the individual user's taste.

UX pattern: Ambient intelligence. Recommendations appear without the user requesting them. The interface does not ask what you like; it observes what you do and adapts. This is ambient intelligence at its most complete — the AI is doing work continuously in the background, and the user experiences only the output.

Personalisation design. Spotify's Discover Weekly is worth examining in detail because it made an explicit design decision that most recommendation systems avoid: it presented personalised recommendations in a named, weekly playlist with a clear premise ("songs you haven't heard, matched to your taste"). This gave the ambient AI a visible identity.

The design consequences of this decision were significant:

• Users knew what to expect and when. A recommendation feed that updates unpredictably feels noisy; a weekly playlist feels like a delivery.
• Users could evaluate it as a unit. "Discover Weekly was good this week" or "it missed this time" was a natural response. This created social sharing behaviour and word-of-mouth that a generic feed cannot generate.
• The clear premise set calibrated expectations: these are new-to-you songs, not your favourites. Users did not mistake it for a playlist of things they already knew they liked.

The filter bubble problem. Recommendation AI that only surfaces content similar to what a user has already consumed creates a narrowing effect over time. A user whose listening history is dominated by one genre receives increasingly similar suggestions, missing content they might enjoy but would never discover. This is a design problem as much as a model problem.

Well-designed recommendation systems introduce deliberate diversity — a percentage of suggestions that sit at the edges of the user's observed taste rather than at its centre. Spotify's Discover Weekly reportedly allocated a fraction of each playlist to songs just outside the user's confirmed taste, surfaced because they were highly popular with users who otherwise resembled the listener. This breadth-vs-depth tradeoff is a product design decision, not an algorithmic inevitability.

Trust and the "why am I seeing this" question. Consumer recommendation products were among the first to face regulatory and user pressure to explain their recommendations. Netflix's star rating system (later changed to thumbs up/down) was an early attempt to make preferences explicit so users felt agency over what the system learned about them. The move to thumbs up/down was itself a UX decision: a five-star scale implied more precision than users actually had, and most people clustered ratings at extremes anyway.

What transfers. Recommendation systems demonstrate that ambient AI requires a visible identity to build sustained trust. Invisible helpfulness can feel magical at first and creepy or random later. Naming what the AI is doing, giving it a cadence, and explaining the premise once — without requiring explanation on every interaction — is the design pattern that scales.

The product category: AI features embedded in business intelligence and analytics tools that surface AI-generated observations about business data. The worked example here is Microsoft Power BI, which ships this pattern as a small family of features: Quick Insights scans a whole semantic model for patterns, the report Insights pane flags anomalies, trends and KPI outliers as you read a report, and Explain the increase / decrease explains a single change in a chart.

The problem being solved. Business analysts spend significant time on questions that have deterministic answers hiding in large datasets: Why did revenue drop in this region last quarter? Which customer segments are churning at an unusual rate? What is anomalous about this month's performance? These questions require pattern recognition across many variables simultaneously — a task AI can do quickly, but which produces outputs that are hard for non-technical users to trust or act on.

UX pattern: A hybrid. Enterprise analytics AI sits between ambient intelligence and a conversational interface. The most successful implementations surface a curated set of insights proactively (ambient), but let users drill in and ask follow-up questions (conversational). The Microsoft Human-AI Interaction guidelines describe this as "match relevant social norms": in a professional analytics context, the right tone is closer to a trusted analyst presenting findings than a chatbot answering questions.

The explainability problem. Enterprise analytics AI faces a trust obstacle more severe than any of the previous cases: the users are professional decision-makers who will be held accountable for the decisions they make based on AI output. "The AI said so" is not an acceptable justification for a strategic decision. Explainability — the ability to show the reasoning behind an AI-generated insight — is not a nice-to-have in this context; it is a prerequisite for adoption.

Power BI's Explain the increase / decrease shows the pattern at its simplest. The analyst right-clicks a data point, and Power BI compares it with the previous point, works out which categories changed their share of the total most, and returns a chart plus a short description of which categories most influenced the change. Microsoft's own example: sales rise 55% quarter on quarter, but Computers and Home Appliances grew 63% while TV and Audio grew only 23%, so those categories are what the explanation calls out. The analyst can switch the chart between waterfall, scatter, 100% stacked column and ribbon views, and add it to the report as an ordinary visual. Microsoft describes the method only "simplistically" and says the ranking uses "various heuristics" — so this is not deep transparency. What it does give the analyst is enough surface area to judge whether the explanation is plausible and check it against their own domain knowledge.

Error handling at enterprise scale. The consequences of AI errors in enterprise analytics are business decisions made on wrong data. The graceful degradation pattern here has three components:

1. Confidence levels made explicit. Showing not just the insight but how much of the data pattern supports it. Weak patterns should surface as tentative observations, not conclusions.
2. Drill-down to the underlying data. Every AI-generated insight should link directly to the raw data that produced it. This converts "trust me" into "verify me."
3. Acknowledgment of limitations. Tools that clearly document what data the AI had access to, what time period it analysed, and what variables it did not consider give users the context to evaluate whether the insight might be missing something critical.

Feedback loops. Enterprise analytics AI has access to a valuable but rarely used feedback signal: whether users act on the insight. If a sales team receives an AI-generated alert about customer churn risk and then takes no action, is that because the alert was wrong, or because they were unable to act? Distinguishing between "irrelevant insight" and "insight we could not act on" requires a lightweight feedback collection step that most tools do not include.

> You are designing the feedback mechanism for an enterprise analytics AI feature. What would you ask a user who dismissed an AI-generated insight without acting on it — in a way that takes less than five seconds to answer and still gives you signal about whether to improve the model or the user experience?

Personalisation in enterprise analytics. The most successful analytics AI tools learn what questions a specific analyst or team asks repeatedly and surface those dimensions more prominently. A sales operations analyst who always investigates pipeline coverage will benefit from an AI that leads with pipeline coverage anomalies. This is professional context personalisation — adapting not to personality or aesthetic preference but to job function and analytical style.

The privacy and governance tradeoff here is different from consumer personalisation. Enterprise users generally expect the tool to learn their role, and organisational data governance controls what data the AI can see in the first place. The personalisation questions are less about consent and more about role-based access and model transparency within the organisation.

Having examined five products across four categories, it is worth naming the patterns that appear regardless of product type. These are not universally obvious when reading about AI product design in the abstract — they become clear only when you look across cases.

Principle 1: Every successful AI feature has a clear problem statement that justifies the imperfection. Writing assistants are tolerated when they occasionally suggest the wrong word because the alternative — the blank page — is worse. Code completion is tolerated when it suggests wrong code because engineers review before accepting. Recommendation engines are tolerated when they miss because a curated selection from ten million items has always been better than unguided browsing. AI features that are introduced where a deterministic solution already works well get rejected — not because the AI is worse, but because imperfection is intolerable when perfection was previously available.

Principle 2: Trust is managed through visible boundaries, not confidence scores. Across all five cases, the dominant trust mechanism is visual or temporal separation between what the AI suggested and what the user accepted — ghost text, suggested labels, weekly playlist framing — not a numeric confidence percentage. Confidence scores require users to calibrate what 73% confidence means in context. Visual boundaries require only that users understand "this is provisional until I decide." Simpler trust mechanisms are more durable.

Principle 3: Error recovery must be easier than the original task. In every case, the product survives AI errors because correcting them costs less than doing the original task without AI. Accepting a writing suggestion and deleting it is one keypress. Moving a misprioritised email is two clicks. Dismissing a wrong code suggestion is one keystroke. If error recovery costs approach the cost of the task itself, users stop using the feature. Error cost is a design variable, not an accident.

Principle 4: Engagement metrics and outcome metrics diverge — and diverge more over time. Every case study revealed the same pattern: early teams measured engagement (suggestions shown, suggestions accepted, insights surfaced), later discovered that high engagement did not reliably predict user value, and invested in outcome metrics (retained code, better inbox response times, acted-on insights) only after the gap became obvious. Starting with outcome metrics is harder — they require more instrumentation and longer observation windows — but they are the only metrics that tell you whether the AI feature is actually doing its job.

Principle 5: The ambient-visible spectrum is a choice, not a default. Every product in this lesson made an explicit decision about how visible to make its AI. Gmail chose visible labels for sorting but invisible ranking; Spotify chose a named, weekly playlist rather than a continuously updating feed; Power BI presents Quick Insights as up to 32 separate cards, each a chart plus a short description, that the analyst can pin to a dashboard or run further insights on — and when its report Insights feature raises notifications unprompted, it stops showing them for the session if the user keeps dismissing them. The pattern from Google PAIR and the Microsoft Human-AI Interaction guidelines is consistent: make the AI's involvement visible enough to be trusted, invisible enough not to create cognitive overload. Neither extreme — full invisibility or full explanation — performs well.

Principle 6: Feedback loops that require effort have low participation; feedback loops embedded in natural behaviour are abundant. Explicit thumbs-up and thumbs-down collection works in some contexts (chat interfaces, where the interaction is already conversational) and fails in others (mid-task inline suggestions, where rating the suggestion interrupts the work). The cases that have the richest feedback signals are those where user behaviour naturally encodes preference: accepting or rejecting a suggestion, moving or replying to an email, acting or not acting on an insight. Designing for abundant implicit feedback from the first release is more valuable than investing in polished explicit feedback UI.

A final cross-cutting observation: all five products share a common launch sequencing that the course's prototyping and measurement lessons predict. They launched with narrower AI scope than the product eventually achieved, measured carefully, and expanded AI autonomy only after building a track record. The temptation to launch ambitious AI features is real; the products that built lasting user trust started more conservatively.