What is inference?
Inference is the moment a model actually generates an answer to one specific question, separate from training, which is the earlier, one-time process of building the model in the first place.
Training happens once, or occasionally, ahead of time, and produces the model. Inference is what happens every single time someone asks that model something: it's the model running on new input to produce a response. The two are often confused because both involve the model 'processing' text, but only training changes the model's underlying patterns, inference just uses them.
Inference is also the point where any live activity happens: a tool being called, a page being fetched, a document being retrieved, all get folded into the answer at this stage, not baked in beforehand. And because a model can weigh several plausible next words rather than always picking the exact same one (see sampling-variance), the same question can get a slightly different answer at inference time even when nothing about the model itself, or what it retrieved, has changed.
Related
- Large language modelA large language model is a program trained on huge amounts of text to predict what word comes next, and that single skill is what lets it write, summarize, and answer questions.
- Tool useTool use is a model's ability to call an outside function mid-answer, such as a search, a calculator, or a code runner, and fold whatever it gets back into its response.
- Sampling varianceSampling variance is the run-to-run variation you get from asking a language model the same question more than once.
- Retrieval-augmented generation (RAG)Retrieval-augmented generation is the technique of retrieving relevant documents and giving them to a model as context before it answers.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.