We ship AI features for a living, and the most useful thing we can tell you is where they do not belong. The model is usually the last decision in a system, not the first, and treating it as the first is how you get a demo that impresses a room and an operation that quietly cannot trust its own outputs. What follows is drawn from work we have actually delivered: a multi-model AI workspace, a real-time voice agent, an advisory assistant, a public policy data index, document extraction inside a distribution platform. The pattern across all of them is the same. The AI is the visible part and the least of the engineering.
The unglamorous half comes first
Our data practice says it plainly: build reliable ingestion and a schema someone can reason about before any model is involved, and add machine learning only when there is a decision it demonstrably improves. That is sequencing, not modesty. A model reasons over whatever you feed it. If the ingestion is flaky, the freshness is unknown, and the schema is a pile of columns nobody can explain, then the model is reasoning over noise and doing it fluently, which is worse than doing it badly. Fluent wrongness is the specific failure the whole industry underprices.
So the question we ask before the interesting question is the dull one. Can a person open the warehouse, point at a number, and trace it back to the source system it came from? If not, no model saves you, and the budget should go on the pipeline. You cannot bolt trustworthy inputs onto an untrustworthy foundation after the fact, however good the model is.
An LLM is a component with a failure mode
The design question for any LLM feature is not what happens when it is right. It is what happens when it is wrong, because it will be, and all the interesting engineering lives in that branch. A traditional function has failure modes you can enumerate. An LLM has one that looks exactly like success: a confident, well-formed, plausible answer that is untrue. You cannot catch that with a try/catch. You catch it with product design.
In practice that means we design the surrounding system to assume the model is sometimes wrong, and to make wrongness cheap:
- Put a human between the model and any irreversible action. If the output triggers a payment, a legal commitment or a deletion, a person confirms it.
- Constrain the output to a shape you can validate. A model that must return structured data against a schema fails loudly when it drifts, instead of quietly when it invents a field.
- Keep an explicit, reviewable record of what the model was asked and what it produced, the same audit trail we build for any automated decision.
- Decide the blast radius before you decide the prompt. What is the worst single wrong answer this feature can emit, and who absorbs it?
Scope the assistant to drafts, not answers
The clearest example of designing for the wrong branch is an advisory assistant we built for a franchise services company. Its users are people buying into a franchise. Not lawyers, not accountants, often not working in English, usually without advice they can afford. The obvious product is an AI that answers their legal and financial questions. We deliberately did not build that.
Instead the assistant produces drafts for a professional to review, and the product says so plainly: a draft agreement is finalised by a lawyer. The reason is that the liability is asymmetric. When a draft is wrong, a reviewing professional catches it and the cost is a few minutes. When an answer a user acts on is wrong, the cost lands on the user, at the worst possible moment, in a domain where they cannot evaluate it. Same model, same accuracy, wildly different exposure depending on whether you framed the output as a draft or an answer. I watched one franchise buyer read a drafted clause on his phone, in his second language, nodding at it, and it struck me that he had no way in the world of knowing whether the thing he was nodding at was correct. The whole design exists to make sure it already had been read by someone who did.
An AI that appears to give legal advice is a liability, not a feature. The framing is the product decision, and the model barely enters into it.
This generalises. Wherever the user cannot check the output themselves and the consequences are real, scope the assistant to generate something a competent human reviews. The AI-assisted form completion in an admissions platform we built works the same way: it fills the application, and the student still owns the submit button.
In voice, latency is an architectural decision
We built a real-time inbound voice agent that answers calls for small businesses, switches language mid-call, and writes results back to calendar, CRM and ticketing so it produces bookings rather than transcripts. The hard constraint there is not intelligence. It is time to first token.
A caller experiences silence as a dropped call or a broken system. The whole value collapses if the agent is slower than a human picking up the phone, which makes latency a product requirement with a hard ceiling, not a metric you tune later. And that ceiling propagates backwards through every choice: which model, whether you stream, how you overlap speech recognition with reasoning and speech synthesis, where the compute physically sits relative to the caller, what you say in the first fraction of a second while the real answer is still forming. The first version we shipped got this backwards. We tuned the model for quality and left latency as a later problem, and later never arrives cheaply, so we rebuilt the pipeline around the time budget instead of the answer quality. A smarter model that replies half a second slower is the wrong model here, and we had to burn one to believe it.
The place a model does not help at all
We built a composite index scoring youth opportunity across more than 800 districts, from twenty-one government datasets. This is the kind of project people now reach for an LLM to solve, and it is worth being exact about why the model is close to irrelevant to the actual difficulty.
The twenty-one datasets share no common key, no common vintage, and no common district coding. One survey's district boundaries predate a reorganisation the next one already reflects. One is from one year, another from four years later. One names a district, another numbers it, and the numbering scheme has since changed. No model fixes this, because it is not a language problem. It is a question of ground truth that does not exist in the data and has to be established by decision. Which vintage do you treat as canonical? When two sources disagree about which district a block belongs to, who wins? Those are judgements a person makes and writes down, and the entire credibility of the index rests on them being inspectable rather than buried in a model's weights. Policymakers act on a methodology they can interrogate. They will not act on a black box, and they are right not to.
That is the general shape of where we do not use an LLM: reconciliation, joins, deterministic rules, anything where the requirement is that the answer be traceable and identical every time you run it. If a JOIN and a documented rule do the job, use the JOIN. It is faster, cheaper, testable, and it does not make things up on a slow afternoon.
Where extraction genuinely pays for itself
The clearest positive case is document extraction. In a B2B distribution platform we built, distributor onboarding involves identity documents that used to be couriered, scanned and keyed in by hand, a fat envelope working its way across someone's desk. Optical character recognition and structured extraction turned that into an upload. This pays for itself, and the reason is the same reason the advisory assistant is scoped to drafts: a human still verifies the output.
Extraction fits because it inverts the effort. Reading a field off a document and typing it is slow and error-prone for a person. Checking that a pre-filled field matches the document is fast and easy. The model does the tedious part, the human does the part they are actually good at, and the verification step means a wrong extraction is caught before it costs anything. The economics work whenever you can turn the human from a slow producer into a fast checker.
What changes as models get cheaper, and what does not
Cheaper, longer-context, faster models genuinely move some lines. Things that were marginal at last year's cost per token become obviously worth it. Longer context means more of the workspace-level state a multi-model workspace holds can go to the model directly, instead of being retrieved and stitched back together. Lower latency widens the range of what is viable in voice. These are real gains and we plan around them.
What does not move is the part that was never about the model. Twenty-one datasets with no common key still have no common key when the model is free. A wrong answer inside an irreversible action is still expensive when the model is fast. The liability of an assistant that answers instead of drafts does not fall because inference got cheaper. I keep coming back to that man on his phone, nodding at a clause in a language that was not his first, trusting it because it read smoothly. Everything we build is really an effort to make sure someone competent has already read the thing he is nodding at. The models got noticeably better in the months it took to write this. He is still nodding.