What the role is, how the models work, and the vocabulary everything else assumes.
Sign in to track your progress across sections.
47 articles
Listed inWhat is an AI Engineer?Foundationson
The day-to-day work is context plumbing, schema design, evaluation and cost control rather than model training. The prerequisite that carries most of the weight is ordinary software engineering.
Listed inWhat is an AI Engineer?Foundationson
A single support-reply feature, taken through the failures it hits in production: supplying a refund policy, grounding replies in help-center articles, structuring output for a ticket tool, and handling prompt injection in customer email. The role is the engineering around the model call.
Listed inInferenceFoundationson
What happens between your request and the answer: time to first token against total response time, why output length drives latency, where token costs accumulate, and the inference failures to handle in code.
Listed inCommon TerminologyFoundationson
The working vocabulary in one place, ordered so each term builds on the last, plus four confusions that cause real mistakes.
Listed inImpact on Product DevelopmentFoundationson
Because the same input can produce different output, testing, cost, latency and failure handling all change shape. A feature also inherits its model's published retirement date, which makes migration scheduled work rather than a surprise.
Listed inAI Engineer vs ML EngineerFoundationson
One role starts with data and produces a model; the other starts with a model and produces a system. Both use the word evaluation for different things, which is where the two talk past each other.
Listed inTrainingFoundationson
Pre-training, post-training, and RLHF — the parts you rarely do but always need to reason about.
Listed inLarge Language Models (LLMs)Foundationson
What goes into a request, what the model holds while answering, and what comes back. The context window is a shared budget for input and output, and some model limits can be fixed while others have to be designed around.
Listed inAI vs AGIFoundationson
Today's models have no memory between requests, a window that degrades well before its limit, a knowledge cutoff, and no way to check their own output. Your architecture is largely a response to those four, and blurring the line produces promises you cannot keep.
Listed inRoles and ResponsibilitiesFoundationson
The day-to-day surface area: prompts, retrieval, evals, latency budgets, and cost control.
Listed inTokensFoundationson
Character and word counts disagree with real token counts, which is a problem because every limit you hit is counted in tokens. How byte pair encoding splits text, and how to budget a request so a response is not cut off mid-sentence.
Listed inHow LLMs WorkFoundationson
The transformer paper. Worth reading once even if you never train anything — everything downstream inherits its shape.
Listed inPrompt CachingFoundationson
How cache breakpoints work, what invalidates a prefix, and the latency and cost numbers to expect.
Listed inSampling ParametersFoundationson
Temperature and top-p change how a token is drawn, not what the model knows. And temperature 0 is not a reproducibility guarantee.
Listed inStreaming ResponsesFoundationson
The finish reason arrives late, and an error can land after a 200 with half an answer already on screen.
Listed inTokensFoundationson
The BPE paper — why tokenizers merge frequent byte pairs, and why your rare identifiers cost so much.
Listed inTokensFoundationson
The unit models actually read and bill for, and why character counts mislead you.
Listed inFine-tuningFoundationson
The documented benefits are about behavior and format rather than knowledge. Fine-tuning shapes how a model responds; retrieval is what gives it facts it did not train on.
Listed inPre-trained ModelsFoundationson
There are two cutoff dates, not one, and the gap between them is where a model is most confidently wrong. Plus which symptom each stage explains.
Listed inTop-KFoundationson
A fixed count applied to a distribution whose shape keeps changing. OpenAI never exposed it and Anthropic now returns a 400 for it.
Listed inTemperatureFoundationson
The randomness dial: what it does to the distribution, and sane defaults per task.
Listed inContext WindowFoundationson
Input and output come out of one budget, so a request that fits can still fail partway through writing. Cached tokens change what you pay, not how much room they take.
Listed inPrompt CachingFoundationson
A cache read costs a tenth of base input on most Claude models, and a date in the system prompt switches the whole thing off. Below the per-model minimum, caching is skipped and no error is returned.
Listed inTypes of ModelsFoundationson
Base, instruction-tuned, reasoning, embedding and multimodal models answer different shapes of question, and reaching for the wrong kind costs more than picking the wrong member of the right one. Ask what shape the output is before asking which model is best.
Listed inRepetition PenaltiesFoundationson
`frequency_penalty` scales with how often a token has appeared and `presence_penalty` does not, so they answer different complaints. Both usually treat a symptom whose cause is an ambiguous prompt, a degrading context, or an agent with no stopping condition.
Listed inHow LLMs WorkFoundationson
Next-token prediction, attention, and the path from raw text to a streamed response.
Listed inTop-PFoundationson
Nucleus sampling — trimming the tail by probability mass instead of by rank.
Listed inGoogle GeminiFoundationson
Images, audio and video travel in the request and are billed as input tokens, at different per-second rates for audio and video. The same models are reachable through the Gemini Developer API and through Google Cloud, which differ in the platform around them rather than in capability.
Listed inSelf-Hosted ModelsFoundationson
Sizing a GPU for a self-hosted model means budgeting for more than the weights. The KV cache grows with concurrency, and exhausting it preempts in-flight requests that are then recomputed from the start.
Listed inHugging Face Inference SDKFoundationson
One client in front of hosted providers, dedicated endpoints and a local server, with a two-line diff from an OpenAI client.
Listed inOpenAI GPT & o-seriesFoundationson
Reasoning tokens are billed as output, invisible in the API, and share your context window. That explains both the bill and the truncation.
Listed inQwenFoundationson
Apache 2.0, over 100 languages, dense and mixture-of-experts across a wide size range. Why tokenizer coverage shows up in your bill.
Listed inModels on Hugging FaceFoundationson
A model card is a data structure. Read the license first, then what it was derived from, then whether that benchmark number has a source.
Listed inChoosing the Right ModelFoundationson
Compare models on capability, latency, cost per request, context window, maximum output, and where they can run. Price per million tokens is not cost per request, and twenty saved examples from your own application settle the choice faster than a leaderboard.
Listed inCohereFoundationson
Reranking scores the query and document together instead of comparing two vectors, which is why it goes second and not first.
Listed inOpenRouterFoundationson
One endpoint makes changing your mind cheap. The training opt-out governs the providers, not the router, and retention is not routed on at all.
Listed inMeta LlamaFoundationson
Downloadable weights come under a custom licence rather than Apache or MIT. It requires a "Built with Llama" notice, a model name beginning with Llama, and a separate agreement above 700 million monthly active users.
Listed inOllamaFoundationson
One command pulls a packaged build and serves it, which means a quantization was chosen for you. Compatible client surfaces let existing code point at it, and a single-user local runner is no evidence about serving traffic.
Listed inTransformers.jsFoundationson
Models run in the browser with no backend and no request leaving the page, which settles the privacy question outright. The cost nobody accounts for is the first load, where model size is page weight and quantization stops being optional.
Listed inAnthropic ClaudeFoundationson
Every model ID is a pinned snapshot, so nothing changes under a working system and improvements are an upgrade you perform. Extended thinking is not accepted on current models, and the tokenizer change at Opus 4.7 makes older capacity estimates wrong by about a quarter.
Listed inGemmaFoundationson
Google's open weights, taking text, audio and image input on hardware you own, which removes both the per-image token bill and the media leaving your network. Three variants do different jobs, and treating Gemma as one model misses most of it.
Listed inHugging Face TasksFoundationson
A task describes the shape of a model's API rather than its subject, so two models sharing one are interchangeable at the call site. That makes it the first filter and never the last, since the label says nothing about quality.
Listed inClosed vs Open Source ModelsFoundationson
Open weights is not open source, and the licence rules out more cases than the benchmark does. Data residency and a frozen model are the arguments that hold, while cost favours running it yourself only above a utilization line most workloads sit below.
Listed inDeepSeekFoundationson
Prices halve outside peak hours, and cache hits land at $0.003 per million input tokens, two orders of magnitude below the miss rate. The naming inverts the usual intuition: the cheaper model is the one that reasons.
Listed inHugging Face HubFoundationson
Every repository is git, which means you can pin a revision and most code does not. Loading a model by name gets whatever the main branch holds today, and the symptom is output drifting with no diff in your own repository.
Listed inMistralFoundationson
Apache 2.0 open weights from 3B to large, which is the difference that decides it for teams whose legal review stalled on Llama's conditions. Read the licence per model though: the catalogue mixes Apache, modified MIT, commercial and one that prohibits commercial use.
Listed inLM StudioFoundationson
What Ollama does from a terminal, in a desktop application, which is a different adoption curve for anyone comparing five models before committing. On Apple Silicon the MLX backend is the reason it comes up so often.