There is no shortage of articles explaining what generative AI is. Most of them define the term, list some tools, and leave you no better equipped than before. This one is aimed at the next step: the things that turn out to matter once you actually build something and put it in front of people.
Seven insights, each one a thing that surprises people the first time it bites.
First, the short definition
Generative AI refers to models that produce new content — text, images, audio, code — rather than only classifying or predicting a label. The current generation is dominated by large language models, which are trained to predict the next token in a sequence and turn out, at sufficient scale, to be startlingly general-purpose.
That is the whole definition. Everything genuinely difficult lies downstream of it.
1. The model is the easy part
Choosing a model takes an afternoon. Building the system around it takes months. Retrieval, context assembly, tool access, caching, fallbacks, evaluation, monitoring, cost control — that is where the engineering lives, and almost none of it is about the model itself.
This is the single most common miscalculation in project planning. Teams budget for model selection and integration, then discover that the remaining eighty per cent of the work has no name in their plan. If you take one thing from this article, take this: swapping to a better model rarely rescues a system whose surrounding architecture is wrong.
2. It does not know things — it predicts text
A language model has no database of facts it consults. It produces the most plausible continuation given everything in its context. Sometimes the most plausible continuation happens to be true, because true statements were common in training. Sometimes it is not.
This explains behaviour that otherwise looks bizarre. A model will invent a citation in exactly the format real citations take, because the format is what it learned. It will state a wrong figure with the same fluency as a right one, because fluency and accuracy are separate properties. Nothing in the machinery distinguishes them.
The practical rule that follows: never rely on the model’s own recall for anything that has to be correct. Put the fact in the context, or verify the output against something authoritative.
3. Context is a budget, not a memory
The context window is often described as the model’s memory. It is closer to a desk: everything the model can see at once, re-read from scratch on every request. It does not accumulate, and it is not free.
Two consequences follow. First, cost and latency grow with what you put in, which is why a long conversation gets progressively more expensive rather than staying flat. Second, more context is not automatically better: relevant material buried in a large volume of marginally related text gets less attention than the same material presented alone. Curating what goes in beats maximising it.
4. Retrieval solves one problem and introduces others
Retrieval-augmented generation — fetching relevant documents and placing them in context — is the standard answer to the knowledge problem, and it works. It is also where most production systems quietly break.
The failure is rarely the model. It is the retrieval step returning the wrong passages, or the right document chunked so that the answer is split across two pieces and neither is retrieved, or a filter silently emptying the result set. The model then answers confidently from whatever it was handed. We measured eight of these failure modes on a labelled corpus in why your RAG returns wrong answers, and in most of them the generation step was working exactly as intended.
5. Evaluation is the hard problem
In conventional software a test passes or fails. Here the output is open-ended, there is often no single correct answer, and quality is partly a matter of judgement. That makes evaluation genuinely difficult, and difficult things get skipped.
The result is teams shipping on vibes: someone tries ten prompts, the outputs look good, it goes live. Then a change to a prompt, a model version or a chunking parameter makes things worse in a way nobody detects for weeks, because there was never a baseline to compare against.
Building even a small labelled test set — fifty representative inputs with known-good outputs — puts you ahead of most teams. It converts “this feels worse” into a number, and it is the difference between engineering and guessing.
6. Cost and latency are design constraints
A prototype that costs a fraction of a cent per call looks free. Multiply by a hundred thousand daily users and it is a line item someone will ask you to defend.
The levers are well understood and worth designing in from the start: route easy requests to smaller models and reserve the expensive one for cases that need it; cache aggressively, since prompt caching can cut the cost of repeated context by an order of magnitude; trim what you send; batch what is not interactive. Running a capable open-weight model locally removes the per-token cost entirely for suitable workloads — our walkthrough on running a local agent on a single GPU covers where that trade-off makes sense.
7. Confident wrongness is a product problem
Ordinary software fails visibly: it throws an error, it returns nothing, the page breaks. Generative systems fail invisibly. They return something well-formed and plausible that happens to be wrong, and the user has no signal that anything went awry.
You cannot engineer this away entirely, which means the interface has to carry some of the load. Show sources so claims can be checked. Express uncertainty where the system has it. Keep a human in the loop wherever a wrong answer is expensive. Design the path for when it is wrong, because there will be one.
Teams that treat this as a product design question ship things people trust. Teams that treat it as a bug to be fixed with better prompting ship things people quietly stop using.
Where to start if you are building
Pick a narrow problem where being wrong is cheap and the output is checkable. Assemble a small evaluation set before you write the feature. Start with the simplest architecture that could work — often a single well-constructed prompt with the right context, no retrieval layer at all — and add machinery only when a measurement tells you to.
If you are preparing to discuss this in interviews rather than build it, the LLM system design interview framework works through these trade-offs in the format interviewers use.
Frequently asked questions
What is the difference between generative AI and machine learning?
Generative AI is a subset of machine learning. Machine learning covers any system that learns patterns from data, including classifiers and recommendation engines. Generative models specifically produce new content rather than a label or a score.
Do I need to train a model to build with generative AI?
Almost certainly not. The overwhelming majority of production systems use an existing model through an API or run open weights unchanged. Fine-tuning is a specialised tool for narrow style or format problems, not a default step.
What causes hallucination, and can it be eliminated?
It follows from insight two: the model generates plausible text, and plausibility is not truth. It can be substantially reduced by supplying facts in context and verifying outputs, but it cannot be eliminated by prompting alone. Systems that need correctness need a checking step outside the model.
Is a bigger context window always better?
No. It raises the ceiling on what you can supply, but filling it indiscriminately costs more, runs slower, and often produces worse answers than a smaller, well-curated context.
The short version
Generative AI is easy to demo and hard to make reliable, and the gap between those two states is where the actual discipline lives. Treat the model as a component, put the effort into retrieval quality and evaluation, design for the cases where it is wrong, and watch the cost curve from day one.
More in Generative AI. Last reviewed September 2026.
