--- title: "3 Failures When Building AI Product Features" description: "PostHog moved fast on AI inside a product-analytics tool, then documented the three failures that keep teams shipping the wrong AI product features. Start from user problems that are both specific and valuable, be honest about whether your data and framing can beat a generalist..." publishDate: Thu Aug 27 2026 00:00:00 GMT+0000 (Coordinated Universal Time) author: Romain Penchenat language: en tags: ["ai product features","ai interaction patterns","llm quality","product design","product management"] canonical: https://blog.romain.garden/blog/building-ai-product-features-three-failures/ estimated-read-time-in-minutes: 7 --- PostHog moved fast on AI inside a product-analytics tool, then documented the three failures that keep teams shipping the wrong AI product features. Start from user problems that are both specific and valuable, be honest about whether your data and framing can beat a generalist model, and budget for maintenance from day one. PostHog turned product analytics into a tech-facing product, then treated AI as a core bet rather than a side experiment. They moved early, they watched what actually shipped, and they published the failures. That last part is the useful one. When a new technology shows up, the default instinct is to construct things *with* it. Real usage and the actual problems disappear. We already lived that with blockchain. We have been living it with AI. You ship fast, and you ship things that answer nothing. I am taking their three failures as a checklist for **AI product features**: how not to build the wrong solution, how to stay honest about quality, and how to stop pretending these features maintain themselves. *** ## Building AI Product Features Starts With Use Cases > Identify the user problems AI might solve. Do not start from AI and hunt for a problem afterward. The first failure is inverted product thinking. You want AI everywhere, so you think AI first. The filter that actually works is the opposite: look at the use cases already inside your product, then ask where a known capability could add value. You do need to know what the technology is good at. Typical capabilities worth mapping onto real jobs: - Filling long forms in a simpler way - Translating dense or technical configuration into language a user can handle - Drafting content on the user's behalf The point is not the list. It is whether those capabilities land on a problem someone already has. That takes a continuous, concrete habit: actually using the AI products that exist, forming an opinion on where they are relevant, and testing your own ideas with prompts before they become a roadmap item. The second half of that habit is learning the **AI interaction patterns** already in the market. Communicating change to users is always hard. Reusing a pattern people already know lowers the adoption tax. The more products have trained people on a given pattern, the more likely you are to get adoption too. Pattern literacy and problem literacy reinforce each other: once you know what is possible and how people already interact with it, you get better at spotting the jobs where AI can actually help. Then apply a two-part gate before you commit. A problem is worth an AI bet only if it is **specific** and **valuable**. Valuable means it clearly helps the user. If it does not, you burn money and you pull focus off work that would have added value. Specific matters even more with this technology. A generic problem puts you on a long project against ChatGPT, Claude, and Gemini — tools dedicated to generic tasks, against which you will have little extra value and a hard time differentiating. Skip that gate and you have already walked into the first failure. *Technology-first vs use-case-first* | | Technology-first | Use-case-first | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | Starting point | Want AI everywhere, then look for a place to put it — The same inversion that produced blockchain features nobody needed. | Start from existing product jobs, then map AI onto them — Identify use cases where a known capability could add value. | | What you ship | Things that answer nothing — Speed without a user problem. You built because the technology was available. | A capability that lands on a real user problem — Fill a long form, simplify technical configuration, or draft content the user would otherwise write. | | Interaction design | Invent a new way of working and hope people learn it — Change is already hard to communicate. A novel pattern adds adoption cost on top. | Reuse AI interaction patterns people already know — The more products have trained users on a pattern, the more likely you are to get adoption too. | | Decision gate | Ship because it is AI — A generic job puts you against ChatGPT, Claude, and Gemini with little extra value to add. | Ship only if the problem is specific and valuable — Valuable enough to help the user. Specific enough that a generalist tool cannot swallow it. | ## LLM Quality Comes From Context, Frame, and Monitoring > If you cannot do better than ChatGPT, Claude, or Gemini by default — or better than competitors on the same job — you do not have a feature. The second failure is wishful quality. The question to ask out loud is simple: can we actually beat a generalist tool, or beat our competitors, by putting AI on this problem? Everyone generating content from an LLM is standing on the same base: the ChatGPT, Claude, or Gemini API. Building your own model internally is rare, and few cases justify it. The difference is the prompt you send and the data sitting inside it. So the first quality question is: what information do you already have about the user that would let you generate content that is more relevant, better adapted, and faster than competitors and generalist tools? Identify those signals. Check which ones are actually useful. Check whether you have enough of them to be differentiating. If you do not, you are wrapping a generic completion. The second lever is the frame you put around the model: instructions, response strategies, and the output formats you allow — what gets displayed, what content comes back. That frame can make the experience more structured and more reliable than a raw prompt in ChatGPT. It also takes real internal work. Without that work, you should not expect to beat the default. The third lever is monitoring, and it is not optional. Shipping an AI-powered feature without logging what the user asked and what the model returned is a black box. You cannot tell if quality is good. You cannot tell how to improve it. Put that instrumentation in at MVP, even at proof of concept: see what people do with the feature, see what the model answers, and estimate real quality from that. None of this has to wait for engineering. Identifying the data you own is a product decision. Testing prompts can happen with no product at all, in Google AI Studio, ChatGPT, or Claude Playground. Those pre-development surfaces answer a simple question: is the information you have useful, and are the results good enough? Skip that realism filter and you can still pick a valuable problem and ship a weak answer. Time, energy, and money, for a result that is roughly neutral. **Quick check** When every competitor calls the same ChatGPT, Claude, or Gemini API, what actually creates differentiated quality? - Training your own model internally for every AI-powered feature. - The prompt you send, plus the user data you already have, plus the frame you set around the model. **(correct)** - Shipping without monitoring so the model stays unconstrained. Almost nobody trains an in-house model. Differentiation is the prompt, the user information you can ground it on, the output frame you enforce, and monitoring of questions and answers from the first prototype. ## AI Product Features Need a Maintenance Plan > Unlike most features you have shipped, AI-powered ones will not keep working without you. They cost money on every request, and the models underneath keep changing. The third failure is forgetting maintenance. Models are upgraded and deprecated continuously. You will have to migrate. Other operational issues pile on. The way to keep that cost from exploding is to stop every team from inventing its own island. Follow the companies that already built a shared AI foundation: a common technical base so model updates and migrations happen once, not N times. On the product and design side, ship shared guidelines the same way you ship a design system — a system for AI behavior and AI features, so usage stays recurrent and the logic stays consistent. Internally, keep knowledge moving: leads or experts involved across topics, so practices stay unified. Harmonized work is cheaper to maintain. That is the whole point. The other trade-off is speed versus power. A more capable model, especially one combined with other models, is usually slower. That is not always necessary, and latency hits perceived value. Sometimes you can generate the answer ahead of time so it feels instant. Sometimes you can use other techniques to reduce the feeling of waiting. Choosing the right model for the right task is how you make the wait — and the bill — feel worth it. Keep watching the market. A capability that was unrealistic a year ago can become realistic simply because models and their structure improved. That is as much a quality point as a maintenance point. Finally, track these features like any other feature, not only with AI-specific metrics. Adoption. Retention. Quantitative and qualitative feedback. The same instrumentation you would put on a standard capability belongs here, and it is often more critical because the run cost is higher. Iterate. Kill the ones whose metrics show you went the wrong way. Evolve from what is left. *** ## Final Thoughts The three failures stack. Build the wrong thing and quality is irrelevant. Ship quality you cannot defend and the maintenance bill pays for a feature nobody needed. Treat AI product features as product work: a specific, valuable user problem, a realistic quality bar grounded in your data and your frame, and an explicit plan for cost, models, and kill criteria. That is a usable bar. Hold it before the first prompt hits production. *** ## FAQs ### When does a problem actually make sense to solve with AI? When it is both specific and valuable. Valuable means it clearly helps the user. Specific means you are not competing with ChatGPT, Claude, or Gemini on a generic task where you have little extra value to add. ### What differentiates your AI feature if everyone uses the same LLM API? Almost nobody trains an in-house model. Differentiation comes from the prompt, the user information you already have, and the frame you set around outputs — instructions, response strategies, and what the model is allowed to return. ### When should you start monitoring an AI-powered feature? From the MVP or even the proof of concept. Without logging what the user asked and what the model returned, you are in a black box and cannot judge or improve real quality. ### Why is maintenance more expensive for AI product features than for standard features? Each request has a real run cost, and models are upgraded and deprecated continuously. Track adoption and retention like any feature, and kill the ones whose metrics show you went the wrong way.