How Large Language Models Actually Work: A Visual, No-Hype Explainer
How Large Language Models Actually Work: A Visual, No-Hype Explainer
Every week, a new AI model drops and the internet loses its mind. "It can reason!" "It passed the bar exam!" "It wrote a novel!"
But ask someone how it actually works, and you usually get one of two answers: either a hand-wavy "it predicts the next word" or a wall of math that makes your eyes glaze over.
Neither is very useful. So let us build a real mental model - from the ground up - of what is happening inside ChatGPT, Claude, Gemini, and every other large language model. No PhD required. No hype. Just clarity.
Part 1: The Core Idea - Next-Token Prediction
At the most fundamental level, an LLM does one thing: given a sequence of text, predict what comes next.
That is it. Every impressive demo, every viral screenshot, every "AI is sentient" tweet - all of it traces back to this single operation performed billions of times.
How Prediction Becomes Conversation
When you type "What is the capital of France?" into ChatGPT, the model does not "know" the answer the way you do. Instead, it:
- 1Breaks your question into tokens (small pieces of text)
- 2Processes those tokens through a neural network
- 3Produces a probability distribution over every possible next token
- 4Picks the most likely one (say, "The")
- 5Appends "The" to the input, and repeats the process
- 6Picks "capital" ... then "of" ... then "France" ... then "is" ... then "Paris"
Each token is generated one at a time, left to right. The model never "sees" the full answer before it starts writing. It is improvising - but it is very good at improvising because it has seen trillions of examples of how text flows.
> Think of it like a jazz musician who has listened to every song ever recorded. They do not memorize songs - they internalize patterns so deeply that they can improvise something that sounds original but follows the rules of music.
Part 2: Tokens - How Machines Read Text
Before an LLM can process your text, it needs to convert it into numbers. This is called tokenization.
What Is a Token?
A token is not always a word. Common words like "the" or "and" are single tokens. Less common words get split into pieces:
- •"understanding" might become ["under", "standing"]
- •"tokenization" might become ["token", "ization"]
- •"FreeApexGears" might become ["Free", "Ap", "ex", "Ge", "ars"]
Most models use somewhere between 30,000 and 100,000 unique tokens in their vocabulary. This is learned during training - the tokenizer figures out which chunks of text appear most frequently and assigns each one a number.
Why Tokenization Matters
Tokenization is not just a technical detail - it directly affects model behavior:
- •Cost - API pricing is per token, not per word. A message with lots of unusual words costs more because each word becomes multiple tokens.
- •Context limits - When a model says it has a "128K context window," that means 128,000 tokens, not words. English text averages about 1.3 tokens per word, so 128K tokens is roughly 100,000 words.
- •Languages - English is very efficiently tokenized because training data is English-heavy. Languages like Japanese, Arabic, or Hindi often need 2 to 4 times more tokens to express the same meaning, making them more expensive and less efficient to process.
Part 3: The Transformer - The Engine Under the Hood
The architecture powering every modern LLM is called the Transformer, introduced in a landmark 2017 paper titled "Attention Is All You Need." Before transformers, we had recurrent neural networks (RNNs) that processed text one word at a time, like reading a book with your finger on each word. Transformers changed the game by processing all tokens simultaneously.
The Key Innovation: Self-Attention
The core mechanism is called self-attention, and it answers a crucial question: "When processing this token, how much should I pay attention to every other token in the input?"
Consider this sentence: "The cat sat on the mat because it was tired."
What does "it" refer to? The cat, obviously. But how does the model know? Through self-attention, the model calculates an "attention score" between "it" and every other token. The score between "it" and "cat" ends up being very high, while the score between "it" and "mat" is low.
This is not programmed - it is learned from seeing billions of examples where pronouns refer back to nouns.
Layers Upon Layers
A modern LLM is not one transformer - it is dozens stacked on top of each other:
- •GPT-4 is estimated to have ~120 layers
- •Claude uses a similar deep architecture
- •Llama 3 70B has 80 layers
Each layer refines the representation. Early layers handle syntax and grammar. Middle layers capture meaning and relationships. Late layers handle reasoning and output formatting.
> Imagine sending a rough draft through 80 rounds of increasingly skilled editors. The first few fix spelling. The middle ones restructure paragraphs. The last few polish the argument and tone.
Parameters: What "70 Billion Parameters" Actually Means
When you hear "Llama 3 has 70 billion parameters," those parameters are the numbers inside the neural network - the "weights" that determine how strongly different neurons connect to each other. Training an LLM means finding the right values for all of these parameters so that the model predicts text accurately.
- •7 billion parameters - can hold a conversation, but makes frequent mistakes
- •70 billion parameters - handles complex tasks well, strong reasoning
- •400+ billion parameters - frontier models like GPT-4 and Claude, capable of nuanced multi-step reasoning
More parameters means more capacity to store patterns, but it also means more compute, more memory, and more cost.
Part 4: Training - How Models Learn
Training an LLM happens in stages, each building on the last.
Stage 1: Pre-training
The model reads a massive dataset - think a significant portion of the public internet, plus books, code, scientific papers, and more. The objective is simple: predict the next token, over and over, trillions of times.
This stage is absurdly expensive:
- •GPT-4 reportedly cost over $100 million to train
- •Llama 3 405B used 30 million GPU hours
- •A single training run can consume enough electricity to power a small town for months
After pre-training, the model is incredibly knowledgeable but not very useful - it will ramble, complete your text in unexpected ways, and not follow instructions well.
Stage 2: Fine-tuning (Instruction Tuning)
The raw model gets refined on curated examples of "good" behavior - following instructions, answering questions helpfully, refusing harmful requests. This is where the model learns to be a chatbot rather than just a text-completion engine.
Stage 3: RLHF (Reinforcement Learning from Human Feedback)
Human raters compare different model outputs and rank them. The model learns to produce responses that humans prefer. This stage is what makes the difference between a model that technically answers correctly and one that answers helpfully.
The Knowledge Cutoff Problem
Because knowledge comes from training data, models have a knowledge cutoff - they do not know about events after their training data was collected. This is not a bug; it is a fundamental property of how these systems work. The model is not browsing the internet (unless explicitly given tools to do so). It is pattern-matching against its frozen training data.
Part 5: Hallucinations - Why Models Confidently Lie
This is the most important section. If you take one thing from this article, let it be this:
LLMs do not "know" things. They produce sequences of tokens that are statistically likely given the input.
When a model "hallucinates" - generating confident-sounding but factually wrong information - it is not malfunctioning. It is doing exactly what it was trained to do: producing plausible-sounding text. The problem is that "plausible-sounding" and "true" are not the same thing.
When Hallucinations Are Most Likely
- •Obscure topics - less training data means less reliable patterns
- •Specific numbers and dates - the model has no lookup table; it is guessing based on patterns
- •Recent events - anything after the knowledge cutoff is a blind spot
- •Confident-sounding questions - "What did Einstein say about quantum computing?" (He died in 1955, before the field existed - but the model might fabricate a plausible-sounding quote)
- •Multi-step reasoning chains - each step compounds the probability of error
How to Protect Yourself
- 1Never trust an LLM for critical facts without verification - especially medical, legal, or financial information
- 2Ask the model to cite sources - then actually check those sources (models frequently invent citations)
- 3Use specific, constrained prompts - "List the three largest cities in Brazil by 2023 population" is less hallucination-prone than "Tell me about Brazil"
- 4Leverage tools like search - models with web access can ground their responses in real-time data
Part 6: What Models Cannot Do (Yet)
Understanding limitations is just as important as understanding capabilities:
- •True reasoning - LLMs simulate reasoning through pattern matching. They can solve many problems that look like reasoning, but they lack the systematic, guaranteed logical deduction that formal methods provide.
- •Persistent memory - Each conversation starts fresh. The model does not remember you from yesterday (unless the application layer adds memory features).
- •Real-time knowledge - Without tool use, the model is frozen at its training cutoff.
- •Mathematical precision - Models approximate math through patterns rather than computation. They can get 87 times 23 wrong while writing a beautiful proof about number theory.
- •Self-awareness - Despite eloquent responses about their own "thoughts," LLMs have no inner experience, goals, or consciousness. They produce text that describes these things because such text exists in their training data.
Part 7: The Practical Takeaway
Now that you understand how LLMs work, here is how to use them better:
Be a Good Prompt Engineer
- •Provide context - the model's output quality is directly proportional to input quality
- •Be specific - vague prompts get vague outputs because the probability distribution is spread across many possible continuations
- •Use examples - showing the model what you want (few-shot prompting) constrains the output space dramatically
- •Iterate - treat the first response as a draft, not a final answer
Choose the Right Model for the Job
Not every task needs a frontier model. Use our AI Hub to explore and compare models:
- •Quick questions and brainstorming - smaller, faster models work fine
- •Complex analysis and writing - use the largest model you can afford
- •Code generation - coding-specialized models often outperform general ones
- •Creative work - experiment with multiple models; each has a different "voice"
Stay Informed
The field moves fast. Check out our AI News Digest to stay current without drowning in hype, and take the AI Knowledge Quiz to test your understanding.
> The best way to use AI is to understand what it actually is - not what the marketing says it is. You are now better equipped than 99% of people having opinions about AI on the internet. Use that knowledge wisely.