Inference

Inference is what happens when a trained AI model processes your input and generates an output. If training is like a student studying for years, inference is like that student taking the exam. The learning phase is over; now the model is applying what it learned to answer your specific question.

Every single time you type a message into ChatGPT, Claude, or any AI tool and get a response, inference is happening behind the scenes. Your text gets converted into tokens, fed through the neural network layer by layer, and out comes the response, one token at a time. This process requires powerful computer chips (usually GPUs or specialized AI chips called TPUs) and happens on servers in data centers around the world.

Inference speed and cost are major considerations in the AI industry. Training a model is a one-time expense (albeit a massive one -- sometimes hundreds of millions of dollars). But inference happens billions of times per day across all users, so making it fast and cheap is crucial. This is why companies invest heavily in optimizing inference -- making models smaller (quantization), using more efficient hardware, and developing smarter serving infrastructure.

When AI companies talk about "cost per token" or "latency," they are talking about inference costs and speed. Smaller models run inference faster and cheaper, which is why companies often use smaller, specialized models for simple tasks and reserve the biggest, most expensive models for complex questions. The per-token pricing you see on platforms like OpenAI and Anthropic directly reflects the cost of running inference.