AI engineer
From Python and APIs through how language models work, RAG and agents to evaluation, security and deployment: building applications on top of LLMs.
Sign in to track how many of the 5 roadmap tasks you have done.
For programmers and computer science students who want to build applications on top of language models: not to train models, but to use them reliably in a product. The roadmap starts with engineering foundations and how models work, continues with prompt and context engineering, RAG and agents, and ends with evaluation, security and costs. By the end you will build and deploy an AI application that you can measure, secure and defend the cost of.
Steps
Step 1: Python and software engineering
external resourcesAn AI engineer is an engineer first: clean Python, types, tests, Git, virtual environments and a readable project structure. Models change every quarter, engineering habits stay. You are done when you have a project on GitHub with tests, a linter and a README that a stranger can install and run by following the instructions.
Resources
Step 2: APIs and web services
external resourcesMost AI applications are a service that accepts a request, calls a model and returns an answer. HTTP, REST, JSON, authentication, asynchronous calls and containers are the foundation everything else stands on. You are done when you build a small API in FastAPI, package it in Docker and can call it with a key from a client and from the command line.
Resources
Step 3: How language models work
Tokens, the context window, the transformer, pretraining and fine-tuning, temperature and sampling, and the limits that follow: hallucinations, sensitivity to wording, zero knowledge of anything that is not in the context. Without this mental model you will debug your application blindly. You are done when you can explain why a model "does not know what it does not know", estimate the cost of a request from the number of tokens, and explain what temperature 0 and 1 do.
Topics on the portal
Resources
Step 4: Prompt and context engineering
The system prompt, the structure of the instruction, examples (few-shot), an enforced output format and above all context management: what to give the model, what to leave out and how to arrange it so that it stays accurate even on long tasks. A prompt is code: version it and test it. You are done when you have prompts in a repository with a set of test inputs and can show how a prompt change changed the results.
Tasks
Resources
Step 5: Embeddings and RAG
external resourcesAn embedding turns text into a vector in which closeness means similarity of meaning; RAG (retrieval-augmented generation) finds relevant documents and puts them into the model's context instead of relying on its memory. The quality of RAG rests on document chunking, retrieval and reranking, not on the model. You are done when you build RAG over your own documents with a vector database and measure how often it returns the correct source.
Resources
Step 6: Agents and tools: function calling, MCP
external resourcesA model that can call functions stops being a chatbot and becomes an agent: it searches, reads, calculates, writes. Function calling, the MCP protocol for connecting tools, the plan, act, verify loop and clear boundaries on what the agent is allowed to do. You are done when you build an agent with two or three tools that completes a multi-step task, and can explain where you stop it when it gets stuck in a loop.
Resources
Step 7: Evaluation and observability
"It looks good" is not a metric. You need a set of test cases, automatic grading (exact matches, rubrics, a model as the judge) and, in operation, logging of every call: inputs, outputs, latency, costs. False positives and false negatives have different costs and you must know them. You are done when every change of prompt or model goes through an evaluation with a number and in production you can see which requests failed and why.
Tasks
Resources
Step 8: Security: prompt injection and data
Every text a model reads (an email, a web page, a document) can contain instructions, and the model cannot tell them apart from yours. Prompt injection, data leaks through tools, personal data in prompts and logs, limits on the agent's permissions. You are done when you can demonstrate prompt injection on your own application, have restrictions in place (permissions, confirmation of actions, filtering) and sensitive data is removed from the logs.
Tasks
Resources
Step 9: Deployment and costs
external resourcesLatency, price per token, rate limits, streaming, prompt caching, the choice between a large and a small model, local models for sensitive data. A well-designed application uses the smallest model that can handle the task and a larger one only where it makes sense. You are done when you can estimate the monthly cost of your application, reduce it with caching or a smaller model and measure that quality did not drop.
Resources