The mechanics under the API: tokens, context windows, sampling, and the knobs that actually change output.
Sign in to track your progress across sections.
17 articles
Listed inTokensHow LLMs Workon
Character and word counts disagree with real token counts, which is a problem because every limit you hit is counted in tokens. How byte pair encoding splits text, and how to budget a request so a response is not cut off mid-sentence.
Listed inHow LLMs WorkHow LLMs Workon
The transformer paper. Worth reading once even if you never train anything — everything downstream inherits its shape.
Listed inPrompt CachingHow LLMs Workon
How cache breakpoints work, what invalidates a prefix, and the latency and cost numbers to expect.
Listed inSampling ParametersHow LLMs Workon
Temperature and top-p change how a token is drawn, not what the model knows. And temperature 0 is not a reproducibility guarantee.
Listed inStreaming ResponsesHow LLMs Workon
The finish reason arrives late, and an error can land after a 200 with half an answer already on screen.
Listed inTokensHow LLMs Workon
The BPE paper — why tokenizers merge frequent byte pairs, and why your rare identifiers cost so much.
Listed inTokensHow LLMs Workon
The unit models actually read and bill for, and why character counts mislead you.
Listed inFine-tuningHow LLMs Workon
The documented benefits are about behavior and format rather than knowledge. Fine-tuning shapes how a model responds; retrieval is what gives it facts it did not train on.
Listed inPre-trained ModelsHow LLMs Workon
There are two cutoff dates, not one, and the gap between them is where a model is most confidently wrong. Plus which symptom each stage explains.
Listed inTop-KHow LLMs Workon
A fixed count applied to a distribution whose shape keeps changing. OpenAI never exposed it and Anthropic now returns a 400 for it.
Listed inTemperatureHow LLMs Workon
The randomness dial: what it does to the distribution, and sane defaults per task.
Listed inContext WindowHow LLMs Workon
The hard budget every prompt shares, and what gets dropped when you exceed it.
Listed inPrompt CachingHow LLMs Workon
Reusing a stable prefix across calls to cut latency and cost on long system prompts.
Listed inTypes of ModelsHow LLMs Workon
Base, instruction-tuned, reasoning, embedding, and multimodal models, and when each fits.
Listed inRepetition PenaltiesHow LLMs Workon
Frequency and presence penalties, and why they are the wrong fix for most loops.
Listed inHow LLMs WorkHow LLMs Workon
Next-token prediction, attention, and the path from raw text to a streamed response.
Listed inTop-PHow LLMs Workon
Nucleus sampling — trimming the tail by probability mass instead of by rank.