When you connect to a remote machine with SSH, the processes you launch are bound to your SSH session. That means when you disconnect, or the connection drops, the processes die together with the session.

A terminal multiplexer is a tool that detaches the processes you start from the SSH session, so they can keep running after the disconnect. Screen and tmux are the most well-known terminal multiplexers, but tmux is more modern and generally preferred among developers.

I’ve used tmux for ages, but with agents running 24/7 on my remote dev box, it no longer works for me. To replace it, I created aplexer. In this article I’ll tell you why.

In particular, we’ll cover:

• What tmux is and how it works

• What’s PTY and how it’s related to SSH connections

• OOM killers and cgroups

• Aplexer as the solution to my current problems with tmux

Let’s start!

Terminal multiplexer

The tmux README says:

tmux is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.

It’s a layer between you (your SSH session) and the processes you run.

After I ssh into my dev box, I can start a tmux session:

tmux new-session -s ai-shipping-labs

Under the hood, multiple things happen:

• My terminal starts a tmux client process

• The client connects to a tmux server

• If the server is not running, the command starts it

The client connects to the server through a socket. The server creates a session and a PTY for that session.

A PTY is a pseudo-terminal created by the kernel. It has two sides:

• the master (user side) - sends the input from the user to whatever is running in the terminal.

• the slave (shell side) - receives the input from the master and sends it to the shell or whatever program is running in the terminal.

When we start the standard terminal emulator app on any Linux, it creates a PTY. The master side connects to the terminal app, and the slave side connects to a shell (usually bash). It’s a bit more complicated than that but I won’t go into more details here.

When we start a process from a PTY, it gets attached to the shell. When the PTY stops, the shell stops too, and all the connected processes follow.

If we take the terminal app in Ubuntu, each tab in the app is a PTY. When I start a process in a tab, and then close that tab, the PTY closes too, and the process follows.

A similar thing happens when we use SSH. The SSH client on our computer connects to the SSH server (sshd) on the remote machine, and sshd creates a PTY. The master is on the sshd side, and the slave is on the shell side.

When SSH disconnects, sshd stops the PTY that was created for this session, and all the processes die with it.

Terminal multiplexers add one more hop: tmux server creates another PTY, and the shell that’s connected to that PTY is the parent for all these processes.

That’s why the processes that we start in tmux continue running - they are attached to tmux server’s PTY. Only the tmux client dies when the SSH connection drops.

There are multiple important organizational primitives in tmux:

• A pane is the basic unit of tmux. Each pane has its own PTY.

• A window has one or multiple panes.

• A session holds one or multiple windows.

• The tmux server can have many sessions.

When I create a new session with a command like this:

tmux new-session -s ai-shipping-labs

It creates a session with one window with one pane inside it, and this pane has a PTY. I can add a new window or split the current one horizontally or vertically into multiple panes, and each pane would be a separate PTY.

I usually don’t do that, though. I use tmux mostly for keeping my agents running when I disconnect, so I don’t need any window arrangement capabilities. For me it’s always one session - one PTY, and different agents are running in different sessions.

We can do a lot of things programmatically with tmux. We can create all these windows and panes using the tmux CLI, or we can send input to any of the panes as if it was typed by a human. In fact, the earlier screenshot with three panes was created by this script.

Because of that, with a bit of scripting, you can teach agents running in different tmux sessions to talk to each other.

Problems with tmux

I like tmux, but as I started running more and more agents, session management became more difficult.

I really struggle with the CLI. The commands are:

tmux new-session -s ai-shipping-labs
tmux list-sessions
tmux attach-session -t ai-shipping-labs

I always forget these commands. Typing all that, even with autocomplete, is always complicated. I never seem to remember what to type and need to look it up.

Also, you have to remember that -s is for new session and -t is for attach. To make it even more confusing, new-session also has the -t parameter, but it’s not the same as -s.

Eventually I solved this problem with tmuxctl - a wrapper around tmux.

I created an executable tmuxctl and an alias t for it to save typing time. With it, I can simply run

t -

It will:

• create a tmux session in the current directory

• name the session after the directory

• if a session already exists, attach to it

If I run just t without any arguments, I’ll see a numbered list of recent sessions:

IDX  SESSION               CREATED
1    codex                 2026-04-03 15:56:59
2    backend-worker        2026-04-03 15:22:10
3    docs                  2026-04-03 14:10:31

And if I want to connect to any particular session from that list, I simply run

t 2

That made the process more convenient and also saved a lot of time when jumping between sessions.

OOM and cgroups

But there’s another problem with tmux - its server is the single point of failure.

On my dev box, I run many things in parallel.

At the same time, it could be:

• Compiling a project in Rust (like Codex-ZCode bridge)

• Running Android emulator tests for PocketShell

• Running e2e tests with Playwright for AI Shipping Labs

If I’m unlucky and all these things run at the same time, my machine runs out of memory.

It’s usually not a problem for Android emulators or Playwright - they are simply killed when it happens. But if I run something like cargo clippy (a linter in Rust), it’s more dangerous.

OOM can not only kill the linter process, but also bring down the agent that’s running it, the shell that’s running the agent, the session that’s running the shell, and the tmux server too. When the tmux server dies, all the other sessions go with it.

So an OOM in Rust can wipe out all the tmux sessions on the machine. All of them.

An OOM kill doesn’t propagate from cargo to tmux. The kernel (”OOM killer”) decides who should receive SIGKILL when OOM happens. Usually it’s the process that caused the actual OOM. But sometimes it may decide to kill the entire memory cgroup where it’s working, which includes the process, the shell, and the tmux server (bringing down all the sessions too).

A cgroup (control group) is a container around a group of processes that Linux manages like a single unit. It’s used to limit the CPU, memory and other resources each group can use. If a process in the group exceeds its memory limit, the kernel will target the processes inside that group for SIGKILL.

So a natural solution is to run unsafe operations in an isolated cgroup, so the OOM killer won’t touch anything outside.

However, you can’t know in advance which process will cause the OOM collapse. It kept happening to me over and over again. When I run 20-30 sessions in parallel, it’s very frustrating.

Eventually, I decided to run each tmux session in its own cgroup. Since I was already using tmuxctl as a wrapper around tmux, I added support for cgroups there.

It worked well, but unfortunately it didn’t solve the main problem - the tmux server was still the single point of failure. Even with each session running in a cgroup, the server would die occasionally, bringing down all the sessions along with it.

After consulting Fable, we did the next logical thing: for each new tmux session start a separate server. So if one server dies, the rest keep running.

All that logic went into tmuxctl, which by that time became a Frankenstein monster, not just a simple wrapper around tmux CLI.

I got really tired of patching it up, and it felt like I was fighting tmux instead of using it. So I opened ChatGPT and asked “how difficult is it to write my own terminal multiplexer?”. It said it would be a few weeks of work. Then I turned on Pro mode and asked it to implement it. It thought for 10 minutes and gave me a first version written in Rust (which kind of worked).

I decided to call it “aplexer” which stands for “Agent Multiplexer”. “Amux” was already taken - I counted 3 products with this name, and none of them were doing what I needed. It’s very difficult to find a name that’s available on PyPI these days!

Aplexer’s initial requirements

What I needed from it:

• Simple commands to create sessions, list sessions, attach and detach

• No server - no single point of failure.

• Dealing with OOM errors without having to worry about cgroups

Plus I wanted to have the same features that tmux had:

• Working after ssh disconnects

• Attaching and detaching sessions

• Sending input to each session

• Seeing the history (scrolling up to see the previous messages from the agents)

The main focus was on running agents, so I also wanted:

• Seeing which folder (”workspace”) each session is running in

• Seeing what’s running inside each session - which agent

• Letting agents send messages to each other natively

I always follow the spec-driven development approach, so I discussed the requirements with ChatGPT, got the first specification out, and started developing it.

Eat your own dog food

I don’t know why this approach is called that. But the idea is to start using the tool you develop as soon as possible.

This approach works extremely well and forces you to find and fix all the inconvenient points.

That’s why the focus for v0 of aplexer was to let me start using it instead of tmux, and then iterate and polish all the rough edges.

The main acceptance criteria for v0 were:

• I can use it for running agents

• Agents run in detachable sessions that don’t stop after SSH disconnect

• OOM in one session doesn’t affect any other session

The first test that I implemented was causing OOM in one session and making sure the others weren’t affected. The OOM isolation test starts three sessions with 128 MB memory limits, exhausts the memory in session B, and checks that sessions A and C still respond to commands.

It took one week to have a stable version that works as v0, and then another couple of weeks to polish and add the features that I needed. (ChatGPT was right!)

I like short aliases, so I use a for aplexer in my terminal.

By now it has fully replaced tmux in my workflow, and I haven’t had a problem with OOM wiping out all my sessions since then.

Aplexer’s architecture

Each session in aplexer is independent of the others, and there’s no shared server. Instead, each session has a PTY worker that’s assigned only to that session.

When a new session is created, the aplexer client creates a worker, and the worker manages the PTY. Optionally, the process can be launched in a cgroup too.

Aplexer doesn’t need a central process for keeping track of all the sessions: it stores them in the filesystem.

I don’t need panes, windows and other things, so for me one session is one PTY.

But I still want to organize the sessions. I group sessions by workspace - the folder where the agents are running.

When I run a, I get a list of all the workspaces and a list of sessions in each.

Now if I want to attach to any of the sessions, I can type

a 7 2

This will attach to the workspace number 7 (~/git/dapier), session 2 (designer). I can refer to them by names too, but that’s too much to type.

Making it agent-aware

I also wanted aplexer to keep track of what’s running inside each session. It shows which agent is there, whether it is running or idle, and when it was last active.

If I want to start a Codex session with the tag “code-refactor” in the current directory, I simply type:

a - codex code-refactor

Of course, that’s all documented in the README.

Communication

In tmux my agents were already talking to each other, so I wanted to have the same functionality in aplexer too.

This is what it looks like:

a send code-refactor "Please review the current diff and report any bugs." --enter

And check what’s on the screen:

a capture code-refactor --screen

These commands send text as the prompt. But there’s also a message bus that the agents can use for asynchronous communication.

Within a session it looks like this:

a message send --to code-refactor "Implementation is ready. Please review the diff."
a message send --all "The tests pass. I’m ready for review."

The recipient can read its inbox, acknowledge it, and then reply when it’s ready:

a message inbox
a message ack <message-id>
a message reply <message-id> <message>

Aplexer and PocketShell

tmux is excellent at what it does, but it stopped working for me for running agents on my dev box. So I created aplexer. Aplexer is an agent multiplexer written in Rust that doesn’t require a single shared server. It groups my sessions in workspaces, and I can tag them and see their state.

If you run agents on a Linux dev box and want to try it, the README has installation instructions and the full command reference.

Now I rarely use aplexer directly - I mostly use it via PocketShell to manage my sessions. It’s an app that lets me connect to my dev box from my Android phone, from my laptop or from the web. It lists all the sessions there and lets me switch easily between them.

I started working on it in May and I like how it’s coming along. I’ll polish it a bit more and soon will write another blog post on how I use it for my work with agents, so you can try it too.

We had our first live session, where I went through the course structure, talked about projects, and answered questions from the participants.

In this post, I want to focus on what we discussed during the launch session.

In particular, I’ll cover:

• Course curriculum

• How the capstone project works

• How to come up with a project idea

• How I’ll help with the project scope

• How we’ll use coding agents

• Common questions

Registration is still open until next Monday, September 28, and there’s one buffer week to catch up if you’re joining late.

Join AI Engineering Buildcamp

Course Overview

I already wrote about the course in detail in my previous post. I covered the curriculum, homework projects, projects from the previous cohorts, workload, and other details in that post.

Let me do a quick recap:

• Week 1 is about the foundations. We prepare the environment, learn how to work with LLMs, and then focus on RAG and search.

• Week 2 is a buffer week. I want to make sure everyone has enough time to prepare the environment. If you’re joining a bit late, you will use this time to catch up.

• Week 3: agents. We go from regular RAG to agentic RAG, learn about tool calling, and implement the agent loop ourselves, and then use an agentic framework.

• Week 4: testing. We add tests to our agent and make sure it behaves as expected.

• Week 5: monitoring. We log all the interactions between the users and the agent.

• Week 6: evaluation. We create evaluation datasets, use LLM judges, and systematically evaluate the system.

• Weeks 7 and 8: focused project work.

• Week 9: peer review and demo day.

The course takes 9 weeks: 6 week of core work and 3 weeks of project work.

The Capstone Project

The project is the most important part of the course. I want everyone to finish with a project that you can add to your project portfolio, use at work or show it to potential clients.

For the capstone, you need to come up with a project idea and implement it end-to-end. It’ll be a RAG application, an agent, or something in between. You will cover it with tests, add monitoring, evaluate it and deploy to the cloud.

In our curriculum, weeks 7 and 8 are specifically dedicated to project work, but I want you to start working on your projects from week 1. This way, by the end of week 6 you have something working, and you can spend weeks 7 and 8 polishing it and finishing the missing parts.

I you already have an idea - that’s great, I’ll help you refine it.

But if you don’t, I’ll give you a project ideation process that you can follow to select a meaningful idea to work on. This approach uses design thinking to discover problems you can solve with AI.

Coding Agents

I also want to encourage you to use coding agents during the course.

During the lessons, we write code together. When I cover something in the coding sessions, I want you to understand how things work. For example we first implement the tool-calling loop ourselves before using any agent framework.

But once you understand the core concepts and the code from the lessons, you can give it to a coding agent:

Based on the code we created in the lesson, I want to create <YOUR PROJECT IDEA>. Please use this implementation as a reference and adapt it to my project.

It makes the project development much faster. Instead of starting from zero, the coding agent already has an example that it will use for your own project. You already understand the original implementation, so understanding the code for your project will be much easier.

This is how I recommend using coding agents: not as a replacement for learning, but as a tool that can get us from an idea to a working implementation faster.

See what people from Cohort 3 implemented:

Questions From the Launch Session

We had a few questions during the launch stream and I think they are quite interesting, so I wanted to cover them here too.

Is RAG Dead?

No, it’s not dead. On the contrary, according to my analysis for AI Engineering Field Guide, the majority of jobs require it. That’s why we start the course from RAG and cover it in detail.

The reasons we need it:

• The model doesn’t have access to proprietary data. If your company has an internal knowledge base, the model has never seen it. With RAG, you can search this knowledge base, retrieve the relevant information, and include it in the context.

• These knowledge bases are usually not that large. With models capable of processing 128 million tokens and more, we can fit the entire dataset into the context window. The RAG is still needed though: if you send the entire dataset to the model, it will have hard time figuring out what’s useful. But if you retrieve only the relevant pieces, you send less data. This makes it faster and cheaper, and it also gives the model more focused context.

Are We Going to Cover Fine-Tuning?

No. Fine-tuning is slow, expensive, and often not needed.

If we have our internal documentation, we can prepare a training dataset in a special format, take a model with open weights, rent a GPU, and spend a day or two fine-tuning it, so it actually learns from our data. Next time we use it, we don’t need to inject the context with RAG: the model already has our knowledge base internally in its weights.

There are a few problems with fine-tuning though:

• You need to have a lot of data and prepare it in a special way

• GPUs aren’t cheap and not always available

• It requires time to tune the model

• It’s difficult to enforce answering with the learned information instead of the general knowledge it had

• The worst part: you update the dataset by including one new page and you need to fine-tune again

Now compare it with RAG: we just update the documents in the search index and that’s it. These problems make fine-tuning impractical for the majority of use-cases. And my job analysis shows that most companies don’t actually require it.

That’s why we don’t cover it in the course.

Do We Need Multiple Agents?

In most cases you don’t. In many cases you don’t even need an agent, just RAG is enough.

I will help you understand when you need what, and scope your project in a way that it’s focused, manageable - and also possible to implement during the course.

We do have a few places where I cover multi-agent setup. For example, the Deep Research Agent uses multiple agents for different parts of the research process, or the Coding Agent that uses the plan-then-implement pattern.

But it’s not the focus of the course. It’s more important to first understand how to create one agent well, test it, add monitoring and evaluate it to make sure it works properly. And then, only if the problem actually requires more complexity, add it. (In many cases it doesn’t.)

Can I Use Another Technology Stack?

In the course we use OpenAI, Python, Pydantic AI and other technologies to implement agents.

But you don’t have to use these specific technologies in your capstone. If you want to build your project in TypeScript - that’s fine. You want to learn LangChain and use it instead of Pydantic AI - totally fine too. Anthropic, Gemini, or another provider work well instead of Open AI.

If you don’t have a strong reason to choose something else, I recommend using the same stack as in the course. It will be easier for me to help you.

But it’s not a requirement.

Do I Need an Expensive Coding Agent Subscription?

No. In fact, even a free subscription will be sufficient. There are a few places in the course where I use a coding agent to generate code - for example, for creating a streamlit tool for evals. But for that I use free Antigravity, so everyone can reproduce it. Some students prefer not to use any coding agent at all and it works too.

I’d still recommend investing at least $10 in getting an OpenCode subscription or $20 for Claude Code Pro or ChatGPT Plus, so you don’t need to worry about hitting the usage limits with the free subscriptions.

Also check my recent post about free and affordable coding agents:

What If I Can’t Attend the Live Sessions?

The course is designed to be asynchronous.

Each week we have a live office hour session. On these sessions we discuss questions, project ideas, architecture decisions, and other things that come up during the week.

If you can’t attend, it’s recorded and shared shortly after. You can also ask your questions in advance and we will cover them during the session.

Most of the content is already recorded as videos (with text articles along them), which you can watch whenever it’s convenient for you.

Can It Help me Transition into AI Engineering?

Yes.

In previous cohorts we had participants from different backgrounds: software engineers, DevOps engineers, data scientists, researchers, and others. For many people the course helped them start doing AI engineering work in their existing company. Others used what they learned when looking for a new role. Some use these skills for freelancing.

If finding an AI engineering job after this course is one of your goals, I recommend thinking about the capstone as a portfolio project from the beginning. Build something you can include in your CV.

If you need help with that, I’ll be very happy to support.

I always enjoy our launch streams and this one was not an exception. It’s going to be a lot of fun!

Join Cohort 4

Registration closes next Monday, September 28. Use code “SUBSTACK” to get 20% off.

Join Cohort 4

See you on Monday!

The model was recently released on OpenRouter, so I decided to give it a try.

Jev and TypeSafe AI

I saw the announcement of Jev on X a few days. It’s interesting because it doesn’t generate text. When I first read about it, I thought, “Huh? No text?” After seeing a few more posts, I realized that it could be useful for quick decision-making.

Jev is a very fast one-shot text classifier. You give it text, and it makes a decision. You don’t need to train or fine-tune it.

That makes it useful as a router:

• Should I send this request to a heavy model or a light one?

• Which agent should I select?

• Should I handle this request automatically or send it to a human?

TypeSafe calls Jev a “System One Model”. The name comes from “Thinking, Fast and Slow”. Yes, that’s this thick, long and boring book that I tried to read but gave up (I managed to finish the audio version though).

• System 1 is the fast, intuitive judgment your brain makes without effort.

• System 2 is slow, deliberate reasoning.

So, Jev is System 1, and LLMs are System 2.

Jev isn’t a replacement for LLMs. It’s a layer that sits in front of them.

You can also use it anywhere you previously relied on a cheap, fast LLM with structured output to make a decision.

For example, it can act as a guardrail. In my AI Shipping Labs workshop about guardrails, I used gpt-4o-mini as a quick classifier to decide whether to continue generation. Jev can replace that binary LLM classifier.

The API

You send Jev one request containing two things:

• a state (unstructured text or a JSON object describing the situation)

• a set of typed questions

Jev answers all questions in parallel and returns structured JSON with probability distributions.

There are three question types (TypeSafe calls them “primitives”):

• Noul: a yes/no question. Returns a single probability from 0 to 1. “Does this message request a refund?” might return 0.93. Think binary classifier.

• Choice: pick one option from a set of options. Returns the selected option, a probability distribution across all options, and a confidence score. Up to 255 options per question. Think multiclass classification.

• Score: rate the state on an ordered scale. Returns a weighted score, the distribution across levels, and confidence. Between 2 and 10 levels. Think ordinal classification.

Here’s a request that includes all three question types for one state:

{
  "state": "Help! My payouts have been failing for 3 days.",
  "model": "jev-latest",
  "questions": {
    "is_urgent": {
      "type": "noul",
      "instructions": "Does this convey urgency?"
    },
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Payments, invoicing, refunds",
        "technical": "Bugs, outages, integrations",
        "sales": "Pricing, upgrades, new accounts"
      }
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["Calm", "Frustrated", "Very angry"]
    }
  }
}

The state is the text that we analyze (it could also be a JSON object). Each entry in questions is one question with whatever key we choose. Inside, instructions is the question that the model needs to answer, and criteria describes the possible answers:

• a map of options for Choice

• an ordered list of levels for Score

• an optional explanation of yes and no for Noul.

Here’s what a response looks like:

{
  "model": "jev-1.13.0",
  "answers": {
    "is_urgent": { "type": "noul", "noul": 0.93 },
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.84, "technical": 0.15, "sales": 0.0, "other": 0.01 },
      "confidence": 0.6
    },
    "frustration": {
      "type": "score",
      "score": 1.3,
      "probabilities": { "0": 0.0, "1": 0.7, "2": 0.3 },
      "confidence": 0.54
    }
  }
}

The response contains one typed answer for each question. The noul value is the probability that the message is urgent, choice is the selected department, and score is the weighted frustration level.

We can now use these answers in our code:

urgency = answers["is_urgent"]["noul"]
department = answers["department"]["choice"]
department_confidence = answers["department"]["confidence"]

if urgency > 0.8 and department == "billing":
    escalate_to_billing()
elif department_confidence < 0.5:
    route_to_human()

Benchmarks

TypeSafe built a benchmark suite of four business workflows to show how it works:

• security incident triage

• agent trace observability

• invoice processing

• customer service

Each model ran through the same workflow harness, and the results were compared against a reference policy derived from the average judgments of GPT-6 Astra and Claude Fable 5.1.

A separate code-review benchmark by an independent developer compared Jev against Gemini Flash and Claude Fable on Python linting rules.

• Jev cost 45 times less than Gemini Flash and 274 times less than Claude Fable.

• Its median response time was 0.75 seconds (versus 3.59 seconds for Flash and 4.31 seconds for Fable).

• Jev scored 98% correctness versus 100% for both comparators.

Applications

I’ve seen a lot of posts on X about what people have built with Jev over the last few days.

Self-Driving Simulations

• A 2D top-down browser simulation where Jev acts as the driving classifier, receiving road state and making steering decisions. Source

• A 3D Three.js driving simulator. Jev receives a compact JSON snapshot of current speed, speed limit, next turn distance, nearby vehicles and pedestrians with relative positions and speeds, traffic signals, and predicted hazards. It returns a driving decision. Source

• An interactive playground with 3D driving simulations showing real AI decisions with visible sensor inputs. Source

Games

TypeSafe’s launch post shows Jev playing Doom at about 10 queries per second, at a cost of roughly $7 per hour.

People also made it play chess:

• Jev versus an OpenAI model. Source

• Two Jev instances playing against each other. Source

Other games:

• An NPC demo tested whether Jev could decide which characters should respond when a player speaks. The game engine sends the transcript and NPC descriptions, and Jev decides which NPC the player is addressing. Source

• Wikiracing, where Jev navigates from one Wikipedia page to another using only links on each page. Jev had to choose among hundreds or thousands of links per page. Source

Using Jev as an Agent Router for My Work

I wanted to give it a try too. Since Jev became available via OpenRouter today, I could skip the waiting list.

Currently, I’m automating my work at DataTalks.Club, AI Shipping Labs and I also have a lot of other operational work.

Because each piece of work is different and requires different tools, I need a different agent for each task:

• If a task is about DataTalks.Club course operations (cohorts, homeworks, projects, leaderboards, certificates) or DataTalks.Club events (workshops or podcast interviews), it should go to the DTC agent.

• If a task is about AI Shipping Labs (workshops, events, membership and payments, sprints, marketing pages), it should go to the AISL agent.

• Otherwise, it should go to the admin agent.

Routing with structured output

Previously, I’d use a simple model like gpt-4o-mini for routing with a structured output schema:

from enum import Enum
from pydantic import BaseModel, Field

class Destination(str, Enum):
    DATATALKS_CLUB = "datatalks_club"
    AI_SHIPPING_LABS = "ai_shipping_labs"
    ADMIN = "admin"

class RouteDecision(BaseModel):
    destination: Destination = Field(
        description=(
            "Which specialist should handle this request? Choose datatalks_club for "
            "DataTalks.Club course operations: cohorts, homeworks, learner projects "
            "and peer review, leaderboards, certificates, registrations, and meetups. "
            "Choose ai_shipping_labs for AI Shipping Labs operations: workshops and "
            "the membership tiers that gate them (main+ and above), events, plans and "
            "payments, shipping sprints, the book club, marketing pages, campaigns, "
            "and member CRM notes. Choose admin for personal task tracking in a local "
            "task list: capturing a task or reminder, listing tasks, marking one done."
        )
    )

The enum describes the directions to route the agent, so I first run a classifier to decide which agent to invoke, and then invoke that agent.

In code it can look like this:

from pydantic_ai import Agent
from site_agents.llm import openrouter_model

router = Agent(
    openrouter_model("openai/gpt-4o-mini"),
    instructions=GPT_ROUTER_INSTRUCTIONS,
    output_type=RouteDecision,
)

result = await router.run(request)
routing_decision = result.output

agents[routing_decision].run(request)

Routing with Jev

With Jev, the same routing decision is one Choice question:

ROUTER_INSTRUCTIONS = """
You are the dispatch layer for a small operations assistant.
Classify the user’s request into the single specialist that can do the work.
Do not answer the request and do not invent a fourth category.
""".strip()

ROUTING_CRITERIA = {
    "datatalks_club": (
        "DataTalks.Club course operations: cohorts and courses, homeworks and their "
        "questions, learner projects and peer review, leaderboards, certificates, "
        "graduates, course registrations, and community meetups."
    ),
    "ai_shipping_labs": (
        "AI Shipping Labs operations: workshops and the membership tiers that gate "
        "them (main+ and above), events, plans, payments and tier overrides, shipping "
        "sprints and their weekly checkpoints, the book club, marketing pages and "
        "site navigation, email campaigns, and member CRM notes."
    ),
    "admin": (
        "Personal task tracking in a local task list: capturing a task or reminder, "
        "listing open or completed tasks, and marking a task done. No website, course, "
        "workshop, or membership system is involved."
    ),
}

Now we prepare the payload and send a request:

payload = {
    "model": "~typesafe/jev-latest",
    "state": request,
    "questions": {
        "destination": {
            "type": "choice",
            "instructions": (
                f"{ROUTER_INSTRUCTIONS}\n\n"
                "Which specialist should handle this request? "
                "Choose exactly one option."
            ),
            "criteria": ROUTING_CRITERIA,
        }
    },
}

response = await client.post(
    "/api/alpha/decisions",
    headers={"Authorization": f"Bearer {api_key}"},
    json=payload,
)

The answer comes back in destination. Since the model can only return one of my three options, I can map it to Literal:

from typing import Literal

class JevChoiceAnswer(BaseModel):
    type: Literal["choice"]
    choice: Destination
    probabilities: dict[str, float]
    confidence: float

From there, dispatching is the same:

route = await route_request(request)
routing_decision = result.destination
agents[routing_decision].run(request)

Comparing the Two Routers

To compare Jev with gpt-4o-mini, I created 50 short, hand-labeled requests and marked the destination for each one. They cover the work each site actually has: cohorts, homeworks, projects and leaderboards for DataTalks.Club, workshops, membership tiers, sprints and marketing pages for AI Shipping Labs, and plain task capture for the admin list.

[
  {
    "id": "aisl-01",
    "request": "Create a workshop Tuesday 16:30 about JEV, main+ only.",
    "expected": "ai_shipping_labs"
  },
  {
    "id": "admin-02",
    "request": "Remind me tomorrow to submit the VAT receipts.",
    "expected": "admin"
  }
]

These are the results:

Both routers get every case right. It’s not a suprsize, though. But speed and price are different: Jev was 2.4 times faster and 3.4 times cheaper.

I ran the benchmark several times and the numbers barely moved - Jev’s median stayed between 0.32 and 0.34 seconds, gpt-4o-mini’s between 0.73 and 0.76 second. The cost was the same on every run.

I think it’s a very promising technology and I’ll most likely integrate it in my workflows.

Workshop on Tuesday

I want to do a more detailed workshop about Jev next Tuesday at 16:30 Berlin time on AI Shipping Labs. If you’re member, come join me!

Sign up for workshop

If you have a use case, bring it along too. See you there!

Tools

This week I bookmarked a few interesting tools that I want to try.

Rebuno’s execution dashboard shows durable agent steps, approvals, and execution status.

• rebuno: a library for adding durable step-level execution to agents: every tool and LLM call is recorded to a Postgres-backed event log, interrupted runs resume from the last recorded step, and YAML guardrails allow, deny, or hold any step for human approval. It works with LangChain, CrewAI, Pydantic AI, Mastra, and Vercel AI SDK.

• pizza-bot: an inbox for long-running AI agents. It puts work items that wait on your decision in Unread, but can continue to run. It’s something I wanted to build in CodeHive, but never managed.

• ordewell: a multi-agent task orchestrator that turns one goal into an ordered plan of coding-agent tasks, each with its own runner, model, thinking effort, and mode.

• panel: a research workspace where the agent works together with you. It shows all the files, PDFs, and notebooks in one dock. You interact with the agent in the same environment and don’t need back-and-forth switching.

The course is a practical and hands-on introduction to AI Engineering. After finishing the course, you’ll be able to create a project independently with tests, monitoring and a proper evaluation set. There are no slides in the course. Instead, we focus on coding a lot - also with help of AI assistants.

I selected the topics for the course to give you a solid foundation to work as an AI Engineer and be able to integrate AI into your product.

In this post, I’ll talk more about the course content, the projects from past cohorts, what participants say, and how to join.

In particular, I’ll cover:

• What is AI Engineering Buildcamp

• The running example, optional projects, and homework

• How the capstone works

• What people in the past cohorts built

• What participants say

• How to join Cohort 4

AI Engineering Buildcamp

In AI Engineering Buildcamp you learn AI engineering by building. We take one step at at a time, and at the end arrive at a fully-working system.

You’ll work on many projects during the course:

• The key running example project that evolves through the course - the documentation agent.

• Each week, there will be a new homework project. You will have to complete it independently to solidify the learning from the lessons.

• Optional example projects that show how the same ideas apply to other RAG and agent use cases

• Your capstone project, where you apply everything to a use case you choose

1. A Running Example Project

The running example through the course is the Documentation Agent. We index the Evidently documentation and build an AI agent on top of it.

This is a very practical application. Many companies have internal knowledge databases and need to be able to access them.

We start from RAG and add on top of it each week:

• Week 1 lays the foundation. It covers LLMs, the OpenAI API, RAG, and search. We index the docs and build a first RAG app to answer questions about them. From week 1, you already have a usable AI system.

• Week 2 helps you catch up on week 1 materials. It adds more examples to strengthen your understanding of RAG.

• Week 3 adds agentic capabilities. We turn the RAG app into an agent that can explore the docs database.

• Week 4 focuses on reliability. We add tests to verify that the agent works as intended. This includes classical unit tests and LLM-based judges.

• Week 5 focuses on monitoring. You see how the agent behaves in production. You collect the info you need to observe and debug it.

• Week 6 adds systematic evaluation. You learn to evaluate agents with a human in the loop. You create judges that mimic human evaluators and use synthetic data.

After six week, we’ll go through the same progression as any serious AI application: from simple RAG to a tool-using agent that we test, monitor and evaluate.

2. Optional Projects

If the main example is not enough, there are more optional projects that you can learn from.

For RAG, we’ll cover:

• FAQ Assistant: a chatbot for course FAQ data that uses RAG with boosting and filtering

• YouTube Transcript Summarizer: a system that extracts summaries and chapter structure from YouTube videos with structured output

• PDF Book Processor: a workflow for parsing a complex PDF, such as a book with math formulas and non-trivial layout. This kind of task often appears in take-home assignments and interviews.

We’ll have a few agent applications too:

• Web Search Agent: an agent that uses web search tools to find and filter and synthesize info from the internet

• YouTube Researcher: an agent that searches YouTube and fetches transcripts and produces structured research summaries

• Coding Agent: a functional coding agent that scaffolds Django applications

• Code Analysis Agent: an agent for analyzing and understanding codebases

• Deep Research Agent: a multi-stage research system that starts with broad search and expands into follow-up queries. It goes deeper into promising directions and fact-checks findings and generates a final article.

3. Homework Projects

The core weeks (weeks 1-6) end with a separate homework mini-project.

To solidify what you learned in the lessons, you’ll need to build another system yourself:

• Weeks 1 to 2: document processing with AI. You download books and extract PDF text and chunk docs and build a full RAG pipeline.

• Week 3: Wikipedia Agent. You implement search and page-fetching tools and build an agent with a framework of your choice.

• Week 4: testing through the DuckDB SQL Agent. You build an agent that translates natural language into SQL. You write pytest tests and add LLM judges and track costs.

• Week 5: Trivia Quizmaster Agent. You build an interactive trivia system and add monitoring it.

• Week 6: systematic evaluation through the Recipe Assistant Evaluation. You design 20 plus evaluation scenarios and run batch tests and detect hallucinations.

4. Capstone Project

On top of that, you work on your capstone during the course.

I want to help you to start with your capstone from day one and start building your project layer by layer.:

• Weeks 1 to 2: define the use case and build the first working version with RAG

• Week 3: make it agentic by adding tools. At this stage, your project moves beyond answering questions. It becomes a system that can make decisions.

• Week 4: make it reliable by adding testing

• Week 5: monitor it and start collecting logs from user interactions

• Week 6: evaluate the project based on collected interactions and user scenarios you generated

• Weeks 7 and 8: polish and extend and deploy the project

• Week 9: present the result and get peer feedback and continue developing it

My main focus for this cohort will be to make sure more people finish with a project. In the previous cohorts, some participants waited too long to decide what to build, and in the end they run out of time without submitting a project.

I want to address that directly. I will encourage to work on your projects from day 1.

If you already have an idea, I’ll share a framework and a set of instructions for your coding agent to implement it as quickly as possible.

If you don’t have an idea yet, I’ll share my design thinking approach to help you find something meaningful to solve.

I already used this approach in the last cohort, and one of the course graduates, Amar Agrawal, created a special app based on this approach: Project Ideation Tool.

Certificate

If you finish the project and do 3 peer reviews, you will get a certificate signed by me.

The certificate confirms that you shipped a capstone.

Past projects

I have already described what people from the past cohorts build.

Here are the articles:

• Cohort 1: 5 ideas for AI agents.

• Cohort 2: 9 Real-Life AI Projects from AI Engineering Buildcamp Graduates.

• Cohort 3: 13 AI Projects from AI Engineering Buildcamp.

I’m looking forward to seeing what you will build in Cohort 4!

What participants say

Here is what past participants say about the course.

Vishnu, Data Scientist at Humana, joined cohort 2.

Carmine, Senior Coach, Data Science and ML Engineer, joined cohort 3.

Lars, Data Scientist at NS, joined cohort 3.

You can find more reviews in the testimonial section of the course page.

Weekly Workload

The course lasts about nine weeks. It runs September 21 to November 22, 2026. It is designed for working professionals, but it still requires a meaningful commitment.

Most of the work is asynchronous. Each week features pre-recorded lessons and practical exercises and homework. Depending on your background and pace, this usually takes 3 to 10 hours per week. Project work takes 10 to 20 hours per week, distributed through the course.

There is one live session each week, mainly for office hours. This is when you can ask technical questions and discuss architecture decisions and receive feedback on your projects from me. I record these sessions and share them with everyone, along with a written summary.

This is not a passive course. It is intended for people who want to spend real time building and debugging, rather than just watching explanations and taking notes.

Scholarship

The scholarship form closed today. I received over 2,100 submissions. Selecting recipients was extremely difficult.

Today I contacted the people who were selected. If you do not see an email from me, you were not selected this time. I can not write to everyone individually.

If you applied and opted in for updates, I added you to this newsletter. That is why you see this email now. Thanks for all your applications.

When deciding, I focused on existing impact. I looked for people already doing something and helping their local community, and I prioritized regions where the course price is hard to afford.

Ways to Join

The standard price of the course is 1,799 USD.

1. You can use your company learning budget to cover the cost. I wrote a short guide with an email template you can use to reach out to your manager. Maven also has their own guide.

2. You can get a team discount. Teams of 2 to 9 get 20 percent off. Teams of 10 plus get 25 percent off.

3. I also provide full and partial scholarships for each cohort. For cohort 4, the form is already closed. See the Scholarship section above for details.

4. If none of the options above work for you, you can use the “SUBSTACK” code to get a 20% discount.

Enroll now

If you need a student discount or PPP pricing or have any questions about the joining options, contact me at alexey@datatalks.club, and I’ll help you.

Join the Course

AI Engineering Buildcamp is designed for hands-on learning. You will develop your own capstone project and complete mini-projects for homework and engage in optional practical projects. You will gain experience with RAG and agents and testing and observability and evaluation and deployment.

If you have been thinking about joining, registration is open until September 21.

At DataTalks.Club we’re running AI Dev Tools Zoomcamp and people are asking me where they can get access to coding agents for free or without paying a lot of money. So I figured I should take that post from X and turn it into an article.

While I do recommend getting a paid Codex or Claude subscription and use it for the course, there are some other providers that can get you a lot of tokens for less money or even for free.

In this post, I’ll share some of them. We’ll cover

• Free coding plans

• Cheap coding plans

• Current promotions

Let’s go!

Free coding plans

There are multiple platforms that currently offer free coding plans.

One of them is Tencent. They recently released Hy3 and now offer a free tier through its WorkBuddy and CodeBuddy assistants. It’s not available in Germany, so I couldn’t try it - but maybe it’ll work for you.

Another one is OpenCode Zen. They have a variety of models absolutely for free. One of these models is Muse Spark 1.3 from Meta.

The full name for this model is “Muse Spark 1.3 Contributor Free”. The “contributor” part here means that you’re “contributing” your data to Meta. That is, they can use your data to improve their model. That’s why it’s free - you pay for it with your data.

For me it’s not a concern. It’s quite a powerful model, it feels like Opus 5 level. I can have one or two heavy coding sessions with Muse 1.3 in OpenCode Zen absolutely for free.

Once I hit the limits, I can stay in OpenCode and switch the session to OpenCode Go - their coding plan.

Affordable coding plans

OpenCode Go costs $10 per month. For this $10 you get a lot. Specifically, you get the same Muse 1.3 Contributor, although without the “free” part.

Again, it’s a contributor model, so Meta is subsidizing 90% of the cost - at least this is what they say. And you can get a lot done with this model. Combine it with 1-2 free sessions from OpenCode Zen, and you’ll get 5+ hours of work per day with this plan.

And there are other models, like DeepSeek V4.1 Flash, which currently offers 75% discount (i.e. 4x more usage).

I used to recommend GitHub Copilot as the first coding plan. Not anymore - now it’s OpenCode Go.

Other cheap alternatives in the same $8-20 range:

• ChatGPT Plus ($20)

• Z.ai GLM Coding Plan Lite ($18)

• MiniMax Token Plan ($20)

• Command Code GOAT ($10)

Prices and quotas change fast, so check the current plan pages.

Cheap API access

So far we talked about coding plans. Instead, you can pay for API calls yourself and plug any API LLM platform to your coding agent.

If you do that directly with OpenAI or Anthropic and put $5 into your account, they will disappear before you can say “hello” to your coding agent.

But if you use DeepSeek’s V4.1 Flash during off-peak hours, you’ll be getting millions of tokens, which is enough to do multiple coding sessions.

Also, OpenRouter gives access to many free and cheap models. Any model ID ending in :free costs zero tokens. The rate limits are very strict though, but once you pay, the limits are lifted.

Current promotions

I really like Z.ai. I have used them since September last year through their API, and I’ve been using their coding plan since December.

Now they are regularly running promotions and giving away 300 million tokens for glm-5.3-flash. You have to use their Zcode harness, and the promotions are typically timed for weekends. It’s a lot of tokens - many people complain that these tokens expire before they can spend them.

Additionally, right now they are running a promo for all Z.ai coding plan subscribers. If you’re one of them, you get unlimited glm-5.3-flash tokens from 5pm to 3am CET.

Well, they aren’t completely “unlimited” - there are still rate-limits. But I don’t hit these limits unless I run 20+ agents in parallel.

That plus 300m tokens plus their quota reduction if you use Zcode - that combined gives you a lot of tokens.

One downside of Zcode is that it’s GUI only, no TUI. But I figured out how to connect it to Codex, so I can run Zcode from my terminal.

Z.ai is not the only company that runs promos. There are others:

• Tencent Cloud TokenHub gives every new account 1M free tokens per model, covering DeepSeek V4 Pro/Flash, GLM-5, MiniMax M3 and Kimi K2.6.

• Verdant currently offers GLM 5.3 Flash and DeepSeek V4 Flash for free within 5-hour and weekly limits.

• Cursor doubled included usage for its first-party models (Auto, Composer 2.5) on paid plans.

• Cognition (the company behind Devin) just released a new model SWE-2 and they made it unlimited in Devin paid plans.

Build more

With all these free models and promotions I no longer have a problem of having no tokens. My main problem is deciding what to build next.

I hope you’ll have the same problem. Have fun and enjoy building!

Tools

I haven’t included the section with new tools for quite some time. Now I want to fix it.

• engrim: a local-first SQLite memory engine that keeps one project-scoped episodic store across Antigravity, Claude Code, Cursor, Windsurf, and Codex. With their approach it’s very easy to switch models in the middle of a session, and you don’t need to send the whole session transcript when doing it.

• oto-dock: “The brains of your company, built on Claude Code & Codex, working on your Anthropic and OpenAI subscriptions.” It’s a self-hosted company OS where agents organized in departments connect to company tools, delegate to each other, and keep working unattended. I’m experimenting with automating as much of DataTalks.Club ops as I can, so this project looks promising.

The projects are real-life and include an exam questions generator, a chess coach, a RAG pipeline, job tools, and research assistants.

This is not the first post about the projects that AI Buildcamp graduates built. I wrote about the previous two cohorts earlier:

• First cohort: 5 ideas for AI agents

• Second cohort: 9 Real-Life AI Projects from AI Engineering Buildcamp Graduates

In this post I want to describe some of these projects.

Some of the course participants presented their projects live on Demo Day, so we’ll start with them.

Let’s start!

1) Exam Questions Generator by Salma Bouzid

Salma is a data analyst who took the course to get better at building an AI app. She’s leaving her job to work as a founder. Her project helps university professors generate exams from their own slide decks.

You upload a course PDF and pick how many questions you want. You also choose a mix of case studies, theory, and applied-concept questions. A professor can approve, edit, or reject each question, then export the exam. In the live demo she asked the model to tailor an HR-management exam to a business student in Berlin.

• The frontend is Next.js

• Supabase stores user actions plus API cost, PDF size, and regeneration prompts

• She started with Claude and later moved to OpenAI

• An LLM judge checks that a question is grounded in the course and not answerable without it

Salma’s LinkedIn

2) ATS Gap Analyser by Amar Agrawal

Amar’s ATS Gap Analyser is for people who apply to jobs and never find out what went wrong.

You paste a job URL and your CV. The app returns a match score out of 100, missing keywords, improvement suggestions, and a three-paragraph cover letter.

The app is live on Streamlit. It doesn’t store the CV or the job description, not even in the logs.

The agent has four tools.

• Extract job requirements

• Score the CV

• Suggest improvements

• Generate the cover letter

Suggestions retrieve from 50 ATS best-practice documents with minsearch.

• Eval used 50 manually written cases.

• Version 1 had 60% recall and 30% accuracy, and version 6 reached 88% recall.

• Amar kept accuracy lower on purpose: he would rather flag a missing skill than miss it.

Amar’s LinkedIn

3) Chess Coach Agent by Leo Cabibihan

Leo’s first project was a chess-coach chat, but it wasn’t useful. Chess Coach Agent starts from your real games instead. You import Lichess, Chess.com, or a PGN file. The app finds weak moments, explains them, and drills those positions.

Leo imported about 2,700 of his own Lichess games for the demo.

The stack includes FastAPI, React, Postgres with pgvector, and PydanticAI. It also uses Stockfish, BM25 lessons, Logfire, and it’s deployed on Render.

Practice uses spaced repetition at 1, 3, and 7 days, doubling up to 30. If there’s no API key, deterministic templates still run. CI uses a TestModel so tests stay offline.

Leo’s LinkedIn

4) Document Preparation Agent by Paulien Out and Alena Fojtik

Paulien and Alena build on-premise AI systems. They created Document Preparation Agent, which is RAG with extra metadata. This metadata is needed for filtering: their clients often want to add query qualifiers like “only the latest documents” or “only files written by person X”.

Because the customer data is private, the demo they used public arXiv papers, manuals, and EU documents. For v1 they focused only on PDFs only. There’s also no UI, only CLI.

Each manifest has author and document type, plus a summary, keywords, and numbered chunks.

Paulien’s LinkedIn and Alena’s LinkedIn

5) Research Radar by Hana Ben Ali

Hana is a structural engineer and a PhD student. She needs to stay of top of new research, but it’s very difficult: there are 100–300 new papers per day on arXiv. Scanning all the titles and abstracts manually takes hours, and relevant work is easy to miss.

Her Research Radar solves this problem. It fetches the new papers every morning, ranks them against her interests, and emails her the best matches.

The app also builds her interest profile based on the papers she already read, and the new recommendations are based on it.

It works in two stages:

1. Semantic search with cosine similarity and an author boost, keep top 15.

2. Re-rank with an LLM ranker to get top 5.

The top 5 go out by email through Resend.

Stars from the email write back to ground_truth_papers.csv, which is both the knowledge base and the eval set.

She started from 26 papers she had rated herself. After tuning a 0.6 semantic / 0.4 LLM blend, NDCG@5 went from 0.871 to 0.923. A Streamlit dashboard shows timestamps, average score, and ratings over time. The hardest part was making the ranker get better each morning.

Hana’s LinkedIn

The other submissions weren’t presented live. Here they are.

6) AI Diet Coach by Thet Su Win

AI Diet Coach is a meal planner for Southeast Asian users. Most Western diet apps ignore this group, that’s why Thet decided to create an app.

You set goals, restrictions, cuisines, and location. The agent builds a 7-day plan from 256 recipes: 201 from TheMealDB and 55 Asian dishes added by the author. It tracks meals and weight, swaps meals, and builds a shopping list.

It also looks up USDA nutrition data and suggests nearby restaurants through Google Maps. The app is live on Streamlit.

A judge went through four versions on 88 labeled answers, and accuracy moved from 46.6% to 73.9%.

Thet Su Win’s LinkedIn

7) AI Learning OS by Wesley Tan

Wesley wanted to solve the watch-later graveyard problem. Self-learners save dozens of videos and articles across YouTube playlists, browser bookmarks, and note-taking apps, but most of this material is never used.

They do finish some content, but it rarely becomes structured knowledge that’s possible to use later.

To solve that, Wesley created AI Learning OS.

It separates saved content from learned content. You queue YouTube videos, articles, markdown, and images. After a learning walkthrough, the system writes only what you actually studied into a knowledge base. Chat search cites that base and reports progress.

The agent has six tools

• Add to queue

• Browse queue

• Load for learning

• Consolidate to the knowledge base

• Search

• Get progress

Retrieval is hybrid BM25 plus dense vectors with reciprocal rank fusion. A 400-word chunk size with 5 results scored 15/15 on the tuning set. 600-word chunks with 7 results scored 12/15. An empty-knowledge-base test checks that the agent doesn’t invent content.

Wesley’s LinkedIn

8) Applied ML Teaching Copilot by Marco Teran

Marco’s Applied ML Teaching Copilot answers questions based on the Applied ML course materials. You can search, fetch a record by id, and get citations.

The stack is Streamlit, OpenAI, minsearch, PydanticAI, and Logfire. There’s a Streamlit demo.

The first judge scored 0.727 accuracy. The calibrated judge reached 1.0 on 33 labeled cases.

Marco’s LinkedIn

9) Datawarehouse Agent by Lars van Asseldonk

Lars’s Datawarehouse Agent is a natural-language-to-SQL assistant over Dutch Railways (NS) station-safety incidents. Non-technical staff ask a question and get a plain-language answer plus the SQL.

Lars used PydanticAI, OpenAI, DuckDB, and Streamlit. He also used Logfire, minsearch, and Plotly, and he recorded a demo video.

10) WorkerChronicle by Miki Foster

When applying for jobs, people often struggle to remember concrete examples from their past work and adapt them to each role. Miki built WorkerChronicle to help with this.

You use can use it when applying for jobs:

• Before applying, you import your resume, previous cover letters, and raw notes to build a library of projects and accomplishments.

• When you find a job, you give it the job description. The app finds relevant examples from your library, shows where evidence is missing, and creates an editable cover-letter outline.

• Before an interview, it uses the same material to prepare likely questions and remind you which examples to discuss.

Miki’s LinkedIn

11) Multi-Agent Software Development Platform by Nitesh Mishra

Nitesh’s Multi-Agent Software Development Platform turns a plain-English requirement into user stories, architecture, and backend code.

It also writes tests, reviews the generated code, and packages the result. Optionally, it can create a JIRA epic with stories and tasks, or open a draft GitHub pull request.

Each requirement goes through the same steps:

• PM agent creates user stories and acceptance criteria.

• Human confirmation approves the plan before code generation.

• Architect agent designs the system and creates a Mermaid diagram.

• Developer agent uses RAG to generate the backend code.

• QA agent writes the tests.

• Review agent uses RAG to find problems in the code.

• Artifact generation packages the code, tests, architecture, and README into a ZIP file.

Nitesh’s LinkedIn

12) LearnMate AI by Dianne Bronola

Dianne built LearnMate AI to help her learn better from YouTube videos.

Many of us have this problem: you watch a tutorial, understand it at the time, and forget most of it a week later. You can rewatch the video to refresh that knowledge, but it’s slow and ineffective. LearnMate helps with that problem.

You give it a video, it extracts concepts from it, connects them into a knowledge map. When the map is ready, it asks you questions about each concept, gives hints when you get stuck, and generates coding exercises if you need.

The stack includes FastAPI, React, PydanticAI, SQLite, ChromaDB, and OpenAI.

13) GapFinder by Katja Weber

Katja’s GapFinder is another tool to help with studying. You give it a link to a YouTube tutorial, it fetches the transcript, chunks it, and asks diagnostic questions.

Then it grades your answers against the video and reports:

• what you understood vs what you missed

• which sections you should rewatch

The stack is PydanticAI, OpenAI, Streamlit, Logfire, and minsearch.

Katja’s LinkedIn

New Cohort

I’m happy to see all these projects. At the same time, I want more participants to make it to the finish line and graduate with a completed project.

This is one of my main priorities for the next cohort. I’m actively researching what helps learners stay on track and finish with ambitious projects. I’ll apply what I learn to the course. You’ll see the results in the upcoming cohort.

The next cohort of the AI Engineering Bootcamp starts on September 21.

If you want to build a system like these, join us. Use code “SUBSTACK” to get 20% off.

AI Engineering Buildcamp

In this course, we use AI coding tools across the full software-development lifecycle.

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App

• Part 5: Coding Agent Building Blocks: Reusable Skills and Specialized Subagents

In this course we give you a tool-agnostic way to work with any coding agent and increate your productivity as a developer.

Watch the launch stream:

Start Here

Use these links to join the course and find the materials:

• Register for the course. Registration is free.

• Open the course platform to see deadlines, submit homework and projects.

• Star the course repository. It contains the module materials, recordings, homework, and final-project requirements.

• Read the course documentation for prerequisites, setup, logistics, and detailed guidance.

• Join the DataTalks.Club Slack for talking to your peer course participants and join the #course-ai-dev-tools-zoomcamp channel.

• Join the Telegram channel for announcements and deadline updates.

Course Curriculum

In this course, we use AI coding tools across the full software-development lifecycle.

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App

• Part 5: Coding Agent Building Blocks: Reusable Skills and Specialized Subagents

Module 1: AI-Native Developer Workflow

Giving a coding agent a vague request often produces a plausible application that doesn’t solve the problem you had in mind. We begin by improving the instructions and context we give it.

You turn a product idea into a specification and a backlog of small tasks. You create durable project context, then separate product management, implementation, and QA into focused sessions. You also learn how agent loops and multi-agent workflows can help with larger backlogs without removing your responsibility for review.

By the end of the module, you have a repeatable way to direct an agent and verify its work independently.

• Companion article

• Recording

• Module 1 materials

• Homework: Instructions · Submit — due 2026-09-07

Module 2: Build and Ship an AI-Assisted Full-Stack App

Next, you use that way of working to create a complete application. You write a product specification and build a frontend. Then you describe the API with OpenAPI, implement the backend, add SQLite persistence, and test the main behavior.

We use AI tools to produce the first version of each part, but we don’t accept generated code because it looks reasonable. We look at the code, run it, compare it with the specification, and write tests for the behavior we expect.

You finish with a tested full-stack application that runs locally and follows a documented OpenAPI specification.

• Companion article

• Recording

• Module 2 materials

• Homework: Instructions · Submit — due 2026-09-14

Module 3: Test, Containerize, and Deploy

A working local application is only the beginning. In this module, you test the seams between the frontend, backend, and database. You add integration tests, move from SQLite to Postgres, package the application with Docker, and run the checks automatically in CI.

Then you deploy the application to a public URL and connect deployment to the CI pipeline. When you merge a change that passes the required checks, the pipeline ships it.

You finish with a containerized application that other people can use and a documented way to test, deploy, and roll it back.

• Companion article

• Recording

• Module 3 materials

• Homework: Instructions · Submit — due 2026-09-21

Module 4: DevOps and Observability for AI-Built Apps

When the deployment command exits successfully, you know only that the command finished. You still need to check whether people can use the application.

You instrument an important request with OpenTelemetry and connect metrics, logs, and traces in Grafana. You write an alert for sustained user impact, collect a bounded evidence packet, and give a coding agent read-only access to that evidence. The agent can investigate and propose an action, while code outside the model enforces which actions are allowed.

You also run recurring security audits and record enough information to reconstruct an incident from the first alert to recovery.

• Companion article

• Recording

• Module 4 materials

• Homework: Instructions · Submit — due 2026-09-28

Module 5: Coding Agent Capabilities

In the final module, you look inside the coding-agent workflow and extend it around your project.

You work with project instructions, MCP, and reusable skills. You also explore hooks, specialized subagents, plugins, and small custom agents. We focus on each capability rather than one product’s name for it, so you can apply the ideas across modern coding agents.

You finish by packaging reusable instructions and a repeatable way to review changes in your application. You also include a specialist agent, an MCP tool or server, guardrails, and documented permissions.

• Companion article

• Recording

• Module 5 materials

• No homework

Office Hours + AI Shipping Labs Membership

As a part of the course, we will also have office hours. During these office hours you can ask questions about the course (and beyond).

There will be 4 sessions on Tuesdays at 18:30 Berlin time (Sep 08, Sep 15, Sep 22 and Sep 29).

Join Office Hours

Joining office hours requires a paid membership on AI Shipping Labs.

This course is completely free and there are no sponsors for this course, so I’m not earning anything by running it. If you want to support me, and also get more out of it, consider becoming an AI Shipping Labs member and joining office hours.

You will also get a personalized onboarding: we will discuss your current situation, your goals, and build a development plan for you.

Cohort Format

The lectures are pre-recorded, so you don’t need to attend a class at a particular time. In a live cohort, we move through the course together with shared deadlines, graded homework, and community support. You can also join the leaderboard, review peer projects, and work toward a certificate.

For each module:

1. Open the module folder in the course repository.

2. Watch the recording and follow the materials.

3. Complete the homework in your own public repository.

4. Submit your answers through the course platform before the deadline.

5. Ask questions and share what you learn in Slack.

Homework is optional for the certificate, but I recommend doing it. It gives you a smaller task for each module, helps you keep pace with the cohort, and prepares you for the final project. On-time homework and learning-in-public contributions also give you leaderboard points.

You can follow all the materials without joining the live cohort. If you study at your own pace, you can watch the same recordings, complete the homework for practice, and build a portfolio project. We offer graded submissions, peer review, and certificates during the live cohort.

For more information, read the guides to course work, homework, and joining a cohort after it starts.

Leaderboard

The leaderboard is optional: it doesn’t affect certification. It gives you a way to track your progress alongside the cohort and recognizes contributions beyond homework. You can earn points for submitting homework on time, learning in public, contributing to the FAQ, completing projects, and reviewing peer projects.

The screenshot below shows the AI Dev Tools Zoomcamp 2025 leaderboard. Your leaderboard profile is anonymous by default, but you can make your name, bio, and links public from your enrollment profile.

For more information, read the leaderboard documentation for the point categories, timing, and profile options.

Learn in Public

As you work through the course, share what you build and learn. A short post about a useful idea, a problem you solved, or a screenshot of your project reinforces the material and creates a public record of your progress. It can also bring feedback, new connections, and opportunities.

After publishing a post, submit its link through the course platform with your homework, project, or peer review. Each accepted link earns a leaderboard point.

For more information, read the learning-in-public guide for the submission process, post ideas, and examples.

FAQ and Questions

Start with the AI Dev Tools Zoomcamp FAQ when you run into a course-specific or technical problem. It collects answers from previous cohorts, so you may find the exact error or question already covered.

If the FAQ doesn’t answer your question, ask in the #course-ai-dev-tools-zoomcamp Slack channel. Include what you tried, the result you expected, the result you got, and enough context for someone else to reproduce the problem. When you find a useful answer that is missing from the FAQ, send it through the FAQ proposal form. Accepted FAQ contributions earn leaderboard points.

For more information, read the guide to asking questions for examples and troubleshooting guidance.

Final Project and Certificate

For the final project, you choose your own idea and build a complete application with AI assistance.

Your project should include:

• a frontend and backend

• an OpenAPI specification

• persistent storage

• automated tests

• containers and a public deployment

• clear instructions for running the application

You also document how you used AI. Show how you described tasks and gave the agent context. Explain how you reviewed generated code, tested the result, and handled permissions. In the later modules, you add operations, security, and agent-extension work around the application.

Homework isn’t required for the certificate. To earn the certificate, you need to submit a passing final project during the live cohort and complete the required peer reviews on time. Check the course platform for the exact project and review deadlines.

• Project 1: submit by 2026-10-19 and complete peer reviews by 2026-10-26.

• Project 2: submit by 2026-11-09 and complete peer reviews by 2026-11-16.

For more information, read the course-specific project requirements, the peer-review guide, and the certification guide.

Course Links

You don’t need to remember every link, but it helps to know where to go for each part of the course.

• Use the course platform for the schedule, deadlines, submission forms, scores, project, and certificate.

• Use the GitHub repository for module materials, recordings, code, homework, and project requirements.

• Use the course docs for setup and detailed guidance, and the Zoomcamp logistics docs for the shared rules.

• Use the course Slack channel for questions and discussion.

• Use Telegram for announcements.

• Use the YouTube playlist for recordings.

Start the Course

Open the course platform, star the course repository, and join Slack and Telegram. Then begin with Module 1.

You can still join after August 31. Check the platform for the deadlines that remain, and start working through the materials.

All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App

• Part 5: Coding Agent Building Blocks: Reusable Skills and Specialized Subagents (this article)

In this article, I want to cover the two most important aspects of working with coding assistants: skills and subagents.

• A skill describes how to perform a repeatable task.

• A subagent is an agent that runs in a separate context with clear task instructions.

In addition, I’ll cover my process for creating skills and show how to run multiple agents in parallel and turn your coding agent into a task orchestrator.

Getting these concepts is enough to work with coding agents productively, and you most likely won’t need anything else.

This article is loosely based on a live workshop I did for the course. In the workshop, I had to improvise, so the content doesn’t directly map to this article. But it’s still going to be useful, so check it out.

From Markdown Files to Skills and Subagents

In Part 1 of the course, AI-Native Development: Specifications, Loop and Graph Engineering, we touched on both skills and subagents, but I didn’t define them explicitly.

Specifically, we created a few documents:

• _docs/process.md and the other project documents contained reusable instructions (see here)

• _docs/team/pm.md, _docs/team/software-engineer.md, and _docs/team/qa-engineer.md described specialized roles for subagents (see here).

I wanted to keep the discussion high-level and not focus on the mechanics of any specific implementation.

We didn’t explicitly define them because we can tell a coding assistant:

Implement feature #101 following _docs/process.md

It will read the document and follow the steps there.

Or you can also specify the role of the agent at the beginning of a session:

You're a QA engineer. Your role is defined in `_docs/team/qa-engineer.md`. Review work for task #101.

Even better, you can ask it to start a subagent with this role:

Start a QA engineer subagent. The role is defined in `_docs/team/qa-engineer.md`. Review work for task #101.

It will work. The coding assistant can just go read the file and do the task.

But if you define these things as skills and subagents explicitly, you’ll be more effective in using them. In this article, I’ll show you how.

Let’s start with skills.

Building Block 1: Skills

A skill is a structured set of instructions for a repeatable procedure.

If you need to follow a specific sequence of actions for completing a task, you can document the steps in a skill. Then later you can ask the agent to use this skill to perform a task. It will read the file and follow the steps from there.

My Skills

I use skills a lot.

For example, one of my skills is for releasing a new version of a Python package. I maintain a lot of Python libraries that I need to regularly update and publish. I describe how I do it in My PyPI Release Pipeline for Python Libraries.

It’s a process with multiple steps:

• Run the tests

• Update the version in pyproject.toml

• Push the code to GitHub as a tag

• Verify that CI publishes the package

If I want to release a new version of minsearch, I don’t want to describe these steps each time I do it. Instead, I document this in a release skill and just say:

release a new version

The agent understands what I need, reads the skill, and follows the steps there.

The agent recognizes the release request and loads the relevant skill without an explicit skill command

Internally, a skill is a markdown file that looks like this:

---
name: release
description: Release a new version of the current project
---

Release

1. Inspect the project and determine its package registry.
2. Run the tests and other required checks.
3. Bump the version in the source of truth.
4. Commit the change and push a version tag.
5. Verify that the CI publishing job succeeds.

The main difference between a plain markdown file that we used previously and a skill is that the skill is automatically discoverable.

I can say:

Make a release. The process is described in _docs/release.md

And that will absolutely work. The agent will read the file and follow the steps there.

However, with skills I don’t need to say which file to use. The coding agent loads the list of skills into its context at startup. When I need to release a library, it can see that there’s a skill for that, so it goes ahead and uses it.

The agent can discover the right skill from the request, while an explicit command remains available when you want direct control

A skill is a markdown document, but in order to make it discoverable, we need to follow a convention:

• It must have a frontmatter section with a name and description

• It’s saved in .agents/skills/<NAME>/SKILL.md

And that’s it - the rest is up to you. You can also add other markdown documents or scripts together with the SKILL.md file in the same directory and describe how the agent should use them.

The Agent Skills specification describes this structure and the format in more detail.

If you want to better understand how skills work, the best way is to build a small coding agent with skills. In the AI Shipping Labs workshop Build a Coding Agent with Tools, Skills and PydanticAI I show you how. We first learn about agents and function calls, and then see how to implement a special function for loading skills.

Global Skills

There are some skills that all your projects need. But some are only needed for local work.

Examples of global skills:

• I use release for releasing a new version of a library, init-library to scaffold a new library or create-github-repo to create a GitHub repository

• I also fetch-youtube, fetch-loom, and fetch-zoom, to fetch recordings or transcripts for the agent to analyze.

• For creating diagrams, I have a diagram-creator skill.

I use these skills across multiple projects, so it’s better to make them global, so any agent on my computer can see and use them.

To create global skills, you need to put them in

• ~/.codex/skills for Codex

• ~/.claude/skills for Claude

• other coding assistants have a similar structure

I use multiple assistants, so I want all of them to use the same skills. For that, I maintain one canonical copy of the skills in my .agents repository, and then create symlinks from there to ~/.codex/skills, ~/.claude/skills and other places.

Project-Specific Skills

Most skills are not useful globally though - they are project-specific.

For example, in my projects:

• Course Management Platform has an incident-response skill. Every time there’s an alert, it knows how to investigate and fix the problem, similar to what we covered in Article 4.

• DataTalks.Club FAQ has skills for fetching course questions from Slack and turning recurring questions into FAQ entries.

They work in the same way as global skills, but you place them in your project root:

• ./.agents/skills/ for all the assistants except Claude

• ./.claude/skills/ for Claude Code

It’s annoying that Claude has its own format - just like with CLAUDE.md.

I solve this problem by creating a symlink from .claude/skills/ to .agents/skills/.

I hope Anthropic starts listening to its users and adopts the .agents convention too, so I can finally delete all these symlinks.

Creating Skills

You don’t need to create these skills manually. Coding assistants know the format, so you can ask them to create these skills.

However, I don’t just start a session and say “create a skill for releasing a new version of this library”. Usually, I do the task with the agent first and see where I need to correct it.

We work together until the task is done, and then at the end I say:

Save this as a skill. Cover the workflow we followed and keep all the corrections I made in mind, so the next agent can do the task correctly from the start.

This skill is specific to this project, so don't create a global skill.

If I don’t include the last line, Codex will create a global skill, so I always have to add it. I didn’t observe this behavior from other agents.

Improving an existing skill works in the same way. I trigger a skill, and if I see that something requires my intervention, I correct the agent. At the end of the session, I ask to update the process in the skill, so I don’t have to do it next time.

Building Block 2: Subagents

Most coding assistants allow you to start one session within another. We call the agents created this way “subagents”.

Subagents let one coding agent prompt another agent inside its own session

Session Context

The context for a session is all the information the agent has about the current task and all the actions it has taken.

It includes:

• AGENTS.md file

• Skills and tools definitions

• All the files it read

• All the code it read and wrote

• All the text you sent it and all the text it output

That’s a lot of information. As the session gets longer, the agent starts “forgetting” things from the beginning. This is called “context rot”.

That’s why I start a new focused session for each task. In the article From Idea to Production in 28 Prompts, I show the steps you need to take your idea to production and recommend starting a new session for each step.

Also, the current context will influence the future actions of the agent. If the agent implemented a feature, you shouldn’t ask the agent to test it in the same window. The implementation history will bias the agent, and it can skip some checks. Instead, it’s better to start a fresh session and ask the agent there to check the work.

Agentic Team

This is what we did in Part 1 of the course, AI-Native Development: Specifications, Loop and Graph Engineering. We defined multiple roles for our agentic team:

• Product manager (PM) - turns raw intake into specifications

• Software engineer (SWE) - implements the task based on the specifications

• QA engineer (QA) - validates that the task is implemented correctly

Each of these agents lives in a separate session. The context from SWE never gets into the QA session, so the QA engineer can test the system independently.

If I have an issue #101 that’s already groomed and I want to implement it, I can start a new session and say:

You're a software engineer agent. Your role is described in `_docs/team/swe.md`. Work on issue #101.

But instead, I can also launch it as a subagent:

Launch a subagent to implement #101. Follow the process.

The main session can keep working while a subagent handles a focused task in the background

It will create a new agent with an empty context (plus AGENTS.md) and automatically prompt it with something like this:

You're a software engineer. Your role is defined in `_docs/team/swe.md`.
Your task is to work on issue #101. Follow the process in `_docs/process.md`.

In some coding assistants, you’ll also be able to look inside the subagent sessions. In others, you’ll only know that the subagent is doing something, but you won’t know what exactly.

Defining Subagents

The coding assistant can read the agent definition from a markdown document (like _docs/team/swe.md) and launch a subagent.

Just like with skills, you can also create a reusable and discoverable subagent definition.

For example, for the software engineer, the definition could be:

---
name: software-engineer
description: Use this agent to execute a well-scoped coding assignment.
---

Implement the assigned task, add or update tests, run the project checks, and
report the files changed and any remaining risks. Do not expand the scope.

Or even this:

---
name: software-engineer
description: Use this agent to execute a well-scoped coding assignment.
---

Read your role definition from _docs/team/swe.md

You put these definitions in:

• .agents/agents/<NAME>.md for the majority of assistants

• .claude/agents/<NAME>.md for Claude

You can also define subagents globally, but I never needed it. My subagents are always project-specific.

Now you can ask:

Launch software engineer for #101

And it will start the software engineer agent.

A subagent can take on a specialized role, such as software engineering

Note that for some coding agents like Claude, you’ll have to restart the session to discover the agent you just defined.

Orchestration Session

When you start using subagents, your main session becomes the orchestrator.

If we have a process specified in process.md:

• PM grooms the issue.

• SWE implements it.

• QA checks it.

• If the issue passes the check, QA outputs PASS; otherwise, it outputs FAIL.

• If it outputs FAIL, SWE needs to reimplement it until QA accepts it.

This way, we create a graph.

The workflow from Part 1 is a graph: PM hands the issue to engineering, QA closes passing work, and failures loop back for reimplementation

This sequence needs to be enforced by something. Typically, we use the main coding session for that.

If we describe the process in our process.md document and link it in AGENTS.md, then we can ask:

Implement #101

It will automatically run #101 through the PM -> SWE -> QA sequence, each step will run in its own fresh context, and the orchestrator will make sure that on FAIL the task goes to SWE who will fix the problems.

Parallel Execution

In the first article, I only showed how to run this flow sequentially, and we can’t run it any other way: we always have to follow the PM -> SWE -> QA sequence.

But we can take care of multiple independent issues in parallel.

Our main agent is already the orchestrator, so it can run multiple subagents in parallel. But we need to make sure that the agents aren’t stepping on each other’s toes.

For that, we use git worktrees:

• For each new task, create a new git worktree

• The agents work independently in these worktrees

• You go through the entire PM -> SWE -> QA cycle in that worktree

• When the cycle is over and the task is done, the orchestrator merges it back into main (alternatively, you can have a conflict resolver agent)

• The worktree is deleted

The orchestrator’s role here becomes more than just launching subagents in order.

It also needs to:

• Keep track of the status of each task

• Schedule the agents

• Create and merge worktrees

• Prevent logical conflicts by picking tasks that don’t need to work on the same files

Agents work concurrently in isolated worktrees while the orchestrator controls review and merge order

To set it up, I put something like this in the process.md file:

Work in isolated git worktrees and run up to five agents in parallel.

You are the orchestrator. Your job is to:

- Split the backlog into independent issue tracks.
- Give each agent a separate worktree and clear file ownership.
- Keep the agent slots full while eligible work remains.
- Send completed work to a fresh reviewer.
- Return review findings to the correct implementer.
- Merge approved work into the main branch one issue at a time.
- Coordinate the work. Never implement the issues yourself.

This is the approach I use for the majority of my projects.

Summary

• Use a skill when you repeat the same sequence of actions.

• Use a subagent when a task needs a separate context or a specialized role.

• Use the main session as an orchestrator when several agents need to follow a process.

• Use git worktrees when independent tasks need to run in parallel.

If you want to learn more about using AI for development see the AI Dev Tools Zoomcamp course.

Across the series we build an application with AI coding agents from the idea to making it production-ready.

I showed how to do it in 3 stages:

• Stage 1: Build - coming up with an idea, creating frontend and backend, and running it locally

• Stage 2: Deploy - making the application testable and deployable, and setting up CI/CD

• Stage 3: Operate - adding dev/prod environments, collecting metrics, logs, and traces, and having AI agents react to incidents

In this article I’ll put everything together, so you can use it as a reference for developing your projects. I’ll keep it tool-agnostic: I don’t assume any specific coding agent, programming language, framework, cloud, or observability tool.

It doesn’t really matter which assistant you use. You can choose Codex, Claude Code, OpenCode, Grok Build, Cursor, OpenCode, or anything else you like.

I show a sequence of steps you can follow, one after another, to take your product from a vague idea to production.

Each step is a prompt that you can put directly into your coding assistant. In most cases, it helps to start a new session for each step: this way, only the important things are in your agent’s context so it can finish the work faster and better.

Let’s go!

Stage 1: Build

This is the first stage of our process.

In the build stage, we:

• Turn the raw idea into a concrete specification

• Create frontend and backend, and connect them

1. Specification

If you give a coding agent a vague idea, it’ll fill in the gaps with its own assumptions, and you’ll get an app you didn’t want.

It’s better to be very clear about what you want and capture this in a document called “specification”.

I usually use ChatGPT for that: I open it in dictation mode and do a brain dump with my idea, and then we iterate until it’s clear what it could look like.

You can start your discussion with your AI assistant using this prompt:

I want to build a tool for forming study groups.

Help me set the scope for this project precisely. I want to brainstorm with you and understand how the tool should work. Give me options.

Ask me one question at a time and keep your output short.

At the end, ask the assistant to summarize the discussion into a markdown document.

Create an empty folder for your project, and save the specs into _docs/specs.md.

Use this document to build the frontend.

2. Frontend

I build the frontend before the backend so I can test the idea quickly. It’s the least expensive way: you create a clickable interface and you test it with some mock data. If it looks the way you want, you can continue.

Put this into your coding assistant (or tools like Lovable, v0 or Bolt.new):

Create a system design interview application.

[Paste the ChatGPT-generated specification here.]

Centralize every backend call in one services layer, and create a mock implementation of it so the whole app runs without a real backend.

Add tests.

Save the frontend code in the frontend/ directory.

After you have iterated on the frontend and like it, it’s time to define how it will interact with the backend.

3. Manual test

Come up with a test you can use to validate that your application works. This should be a sequence of actions you perform to ensure the app works the way you want.

For an app that helps you find study partners, it could be something like:

1. Open the home page, sign up

2. Create a group for studying

3. Create an event there

4. Share the event

5. Sign up to the event from another browser session

6. Verify that the sign up is visible in the first session

You can use AI to come up with this scenario, but remember it - and maybe save it to _docs too.

4. AGENTS.md

AGENTS.md is loaded into the agent’s context every time you start a new session. You can add anything that you think is important for your app. For now let’s add one line there:

Make focused, regular commits, and explain your decisions in the commit messages so we can use them as logs

Later you can add more things here.

If you use Claude, create CLAUDE.md with a single line:

@AGENTS.md

5. OpenAPI schema

We previously asked the assistant to centralize all backend calls in one place. Now let’s use this to create the API specification. We will use this specification to define frontend-backend interaction:

Read the frontend's API client in frontend/

Create openapi.yaml at the repository root that defines every endpoint, method, path, request body, response body, and which endpoints need authentication.

Now we have the specification for our backend so we can build it.

6. Choose the backend stack

Before actually building it, you need to decide which technology you want to use for this application.

Ask the coding assistant to help:

Based on openapi.yaml, propose 2-3 backend stack options. Describe pros and cons for each. Recommend one.

You can challenge your AI assistant and ask it some questions, or just accept the option it suggests.

At the end, save the results to _docs/stack.md.

7. Backend

We have selected the technology, so let’s use it to build the backend.

Just like we created the frontend with mocked backend calls, we can create the backend with a mocked database. First, focus on making sure the frontend-backend integration works, and then connect a proper database.

Build backend in backend/ based on `_docs/stack.md` that implements the openapi.yaml spec.
Use an in-memory store and seed it with data so the frontend has something to show.
Add authentication with hashed passwords and bearer tokens for the endpoints that need it.
Write tests.

8. Makefile

I always have problems remembering how to start the backend, how to run the frontend, install the dependencies, and do other things. For that, I usually use Makefiles, so I can just type:

make backend

And the backend runs. Let’s define a Makefile:

Create a Makefile so I can easily run the application.

9. Connect frontend and backend

We created a backend but the frontend still serves the mock data. Let’s connect them:

Switch the frontend to use the real backend client.

Use the test scenario from before to make sure it works.

You may do a few iterations here. Test the application and ask the agent to fix the problems you discover until it works correctly.

10. Database

We don’t have a real database yet. Let’s add it. I usually use SQLite for local development, so I’ll use it here too.

Replace the in-memory store with a database. Use SQLite with ORM, so later we can easily switch to a different database. 

Instead of SQLite, your application may need something else, so you can chat with your coding assistant about it before you implement it.

After it’s done, use the test scenario again and iterate until it fully works.

Stage 2: Deploy

Our app works locally. Great.

Let’s deploy it and make it available for others to use:

• Containerize it

• Deploy it to the cloud

• Automate the deployment with CI/CD

11. Containerization

We need to put our application in a container, so we can later use any container management system for deploying it.

In our local environment, we have two processes: one for the frontend, one for the backend. We need to run them separately. In practice, you usually compile the frontend into static files and serve them with the backend.

So, instead of creating separate containers for the frontend and backend, we’ll create one that serves both:

Create a Dockerfile that builds the frontend with Node, then builds the backend. Backend should serve the frontend.

Build and run the image, then verify the flow with the test scenario and iterate until it works.

12. Production database

In the previous stage I used SQLite. It helps with keeping the local setup lightweight, but it’s not a database we can use in production. Typically we use something like Postgres.

You can ask your agent to select the best database for your case, but it’s very likely it’ll be Postgres.

So let’s add it:

Add Postgres support to the backend.

Use the test scenario to make sure the app works.

13. Docker Compose

To make it easier to manage the local environment with all the services, we use Docker Compose.

Let’s ask the agent to set it up:

Create docker-compose.yaml with two services: database and the app.

Run it with docker compose up and test it.

14. Integration

It’s possible that by now you already have tests. AI coding assistants are instructed to create them. But maybe in your project there are still none or too few.

Let’s make it explicit and ask the assistant to create them.

We also want to have a specific category of tests: integration tests. These tests are used to make sure the interaction between backend and the database is correct.

You can also ask the assistant to help you with finding the right test scenarios to include in the test suite:

Create integration tests that run against docker-compose.yaml. What scenarios should we test?

15. End-to-end test

Integration tests verify that the backend can connect to the database.

End-to-end tests verify the entire user-backend flow: from the browser to the frontend, from the frontend to the backend, and from the backend to the database.

So far, we have been doing these tests ourselves manually. Let’s now automate it.

Add an end-to-end test that runs against docker-compose.yaml.

Use Playwright to:

1. <Describe the first step>
2. <Describe the second step>
3. ...

Put the tests in the e2e/ folder in the repository root.

Now we don’t need to repeat this test manually every time we make a change.

16. Deployment stack

Now the application works locally and is properly tested. It’s time to deploy it. But where? There are many options.

Let’s choose what works best for us. Ask the coding agent:

Propose 2-3 ways to deploy the application. Compare cost, complexity and other factors.

<List any other constraints that you have>

It’s also helpful to use an infrastructure as code (IaC) tool to manage your cloud resources:

I want to use an infrastructure-as-code tool for managing my infra. What are my options?

It’s possible that for the deployment option you chose, there are no infrastructure-as-code options. In this case, just move on.

17. Deployment

Now that we have selected the stack, let’s deploy.

Ask the agent:

Deploy this application to <Selected option>.

If applicable, add:

use <IaC tool> for managing the infrastructure

18. CI/CD pipeline

We can deploy a few times manually, but we want to automate it. It’s done with CI/CD:

• CI is continuous integration - running tests every time you push to git

• CD is continuous deployment - automatically deploying when the tests pass

Create a CI/CD pipeline that:

- runs backend and frontend tests in parallel
- builds the Docker Compose stack and runs integration and end-to-end tests against it
- deploys the application
- validates that the deploy is successful by checking the health endpoint

You probably use GitHub for hosting code, so it’s likely that you’ll use GitHub Actions. If you don’t like it, chat with AI to select a different CI/CD tool.

We successfully deployed our application, but we’re not done yet. We need to make it more production-ready.

Stage 3: Operate

To make it more production-ready, we need to add a few things:

• Create dev/prod environments

• Capture metrics, logs, and traces

• Set up alerts

• Have an AI agent taking care of the alerts

Let’s do that.

19. Second environment

Now each push brings the updates directly to the deployment. But what if it has a bug that we missed?

To avoid that, we have two environments:

• Dev: the environment for testing. We deploy here on every push

• Prod: the environment that real users use. We promote a release from dev to prod only when we confirm that it works correctly

If we use infrastructure as code, it’s trivial to set up. If not, let the agent handle it.

Either way, it’s enough to just ask the agent to do it and it’ll figure the rest out:

Create a second, independent copy of our infrastructure. We will use the copy as production, and the existing infrastructure as a dev environment.

20. Manual promotion workflow

Now we automatically release the code to development on every push. Let’s set manual release promotion from dev to prod:

Create a manual GitHub Actions workflow that promotes the dev version to production.

21. Split build/deploy

It’s possible that in your CI/CD the build and deploy happen in one step (it happened to me when I was deploying to AWS EC2).

This is an anti-pattern. We want to build the image only once, first deploy it to development, and use it for production too if it works fine.

If for you these steps are not separate, we need to split them. Ask AI if you’re not sure.

Currently we build the image during the deploy stage.

Split it into two stages:

- Build: build the image and push it to a container registry
- Deploy: pull the image from the registry and serve it

The manual prod promotion CI/CD workflow pulls the currently deployed dev image to prod.

Tag each image using the YYYYMMDD-HHMMSS-shortsha pattern (e.g. "20260818-163457-83242da")

Depending on where you deploy your application, you may have to use different options for the container registry. Talk to AI if you’re not sure what to select and how to do it.

22. Observability and OpenTelemetry

Our application is deployed and serving real users in production. But we don’t know how reliable it is.

We need data for that. That’s called telemetry, and we will collect metrics, logs, and traces. The industry standard is OpenTelemetry, so you’ll probably use it.

Instrument the backend with OpenTelemetry.

Include in the telemetry:

- service name
- environment
- deployed version

To “instrument” the backend means to include the code for capturing the telemetry in our application. But we also need to save it somewhere.

23. Choose the observability backend

There are many options for storing the telemetry and displaying the dashboards. Some are open-source but you need to self-host them. Some are fully managed, but you’ll need to pay for using them. Some of the self-hosted options give a generous free tier.

Ask the agent to help us select:

Propose 2-3 options for storing the telemetry (metrics, logs, traces) via OTel and for displaying them on a dashboard.

<If you have any preferences or restrictions add them here>

24. OTel Collector stack

Now we’ve selected the stack. If you decide to go with open-source and self-hosting, you can also deploy it locally:

Add an OpenTelemetry Collector. Create "observability/" directory with Docker Compose for: <list selected technologies>. Keep this as a separate Compose project from the application stack.

Then deploy it:

Deploy the observability stack. It should be separate from the application stack. Connect both development and production to it.

If you decide to go with a managed service, connect it:

Add an OpenTelemetry Collector for <selected technology>. Help me configure it

25. Application metrics

We configured OTel. Let’s collect important metrics.

They are application-dependent and depend on what’s important for your product.

You can also ask the agent to come up with the most important metrics:

What are the most important product metrics I need to track in my application? Give me top 5

Start tracking them:

Track the following metrics with OTel

- <List them>

26. Dashboards

We’re tracking the metrics but not displaying them yet.

Let’s fix that:

Display the metrics we export in a dashboard. Make it possible to filter by environment and deployed version.

27. Alerts

We can’t babysit the metrics in a dashboard. Instead, we can configure an alert that will fire every time a metric goes too low or too high.

Add an actionable alert for <most important metric>. Include the service, environment, deployed version, owner, and dashboard URL in the alert.

It’s also useful to track 5xx error codes and for each error send an alert (or configure a threshold that makes sense for your application). Talk to your AI assistant to set it up.

28. On-call worker

Now when alerts fire, we get an email or a Slack message. But we still need to act on them.

We can have AI watch the alerts and fix the errors when they happen.

Let’s configure it.

Add an on-call-engineer/ directory with a script that connects to our alerts API.

When an alert fires, the script

- wakes up an on-call agent that runs in headless mode
- passes all the alert details

The on-call agent should be able to resolve the issue autonomously and commit the fix.

Your on-call engineer may have instructions that look like this:

You are the on-call engineer for this repository. An alert just fired.

Investigate the root cause. Read the code and reproduce the failure. If you find a real bug, make the smallest correction, run the backend tests, and commit the fix with a clear message.
If the alert is a false positive, explain why and do not change the code.

Stage 4: Going Further

In this article, I gave you 28 concrete steps to bring your idea to production. If you follow them, you’ll have a working application that can serve real users in less than a week.

But there are still many things you need to think about:

• Do you use a container manager? If not, consider using one.

• Do you use a managed database service? If not, consider using one.

• Regularly back up your database and test that the backups actually work.

• Learn about VPCs and see if you need one.

• When you start getting traffic, learn about scaling and load balancing.

Also, periodically ask Fable or GPT-5.6-Sol Max (preferably both) to audit your code for security vulnerabilities.

I’m only scratching the surface of a “real production system” in this article, but the stages and steps in this tutorial are more than enough to get started.

With this setup, you can use AI to build a product that runs reliably. The rest you can learn when you actually need it.

Have fun building!

And check our AI Dev Tools Zoomcamp if you want to learn more about using AI to increase your development productivity.

All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App (this article)

• Part 5: Coding Agent Building Blocks: Reusable Skills and Specialized Subagents

In part 2, we developed an application for conducting system design interviews. In part 3, we deployed it to a cloud environment and configured CI/CD.

In this article, we’ll make this deployment production-ready:

• Separate development and production environments

• Create a manual CI/CD trigger for promoting the app from development to production

• Use OpenTelemetry to collect metrics, traces, and logs and create

• Define application-specific metrics and create a dashboard that displays them

• Create an alert when the metrics change

• Use an AI agent as the on-call responder for the alerts

We will continue using AWS and CloudFormation, but the principles we show in this article are tool-agnostic and will work for any environment.

Recap

We started building the Interview Canvas project in Part 2: Build and Ship a Full-Stack App with AI Coding Assistants:

• We brainstormed with an AI assistant to come up with the specification

• Based on that we created a React-based frontend with no backend

• Next, we defined the frontend-backend OpenAPI schema

• Using the API contract we created the backend

• We added database support with SQLite and SQLAlchemy

The app was running locally, and then we deployed it in Part 3: Deploy a Full-Stack App with AI Coding Assistants:

• We containerized the application

• Then added Postgres support

• To make sure the system works reliably, we created integration and end-to-end tests

• We deployed it to AWS

• In the end, we set up CI/CD via GitHub Actions, so every push runs the tests and deploys the app to development

Now we take the application that’s already deployed and make it follow production best practices.

Dev and prod environments

When we push the code, we automatically deploy the changes, and they go live immediately. It’s okay when we are just starting to work on our project. But when we have real users, we want to be more careful and check that our changes didn’t introduce any regressions.

To avoid having this problem, we usually have two copies of the same environment:

• Dev (development): the environment we use internally for checking that everything works. Typically we run the latest version of our project there, and every time we push, the changes are automatically deployed there.

• Prod (production): this is what our users use. We don’t want to deploy every single change there automatically and we want to have more control over the process.

Because we previously defined our infrastructure as code, we can reuse most of it. We will most likely need to change a few things, like the size of the machine where the application is running, but the majority of the resources will stay the same.

Let’s ask our coding assistant to create a copy of the existing environment and call it “production”:

Create a second, independent copy of our infrastructure. We will use the copy as production, and the existing infrastructure as a dev environment.

Once we have the prod environment, let’s update our CI/CD. We will only deploy to dev on every push. For production, we will have a manual action that will take whatever development has and promote (apply) it to production.

Create a manual GitHub Actions workflow that promotes the dev version to production.

Now we have two environments.

Container repository

In the pipeline we have so far, the Docker image is built during the deploy stage. I deploy to EC2 by executing the build script on the machine and then running the image in Docker.

It’s an anti-pattern.

There are multiple problems with this approach:

1. The deploy stage is actually two things: build and deploy. If build fails, there should be no deploy, so it’s better to have these steps separately.

2. When we promote the dev version to production, we have to build again. During this time, many things could have changed, so the build will not be identical to dev, and it can cause problems.

So we split the deploy step into two separate steps:

• Build the image and upload it to a container registry

• Pull this image from the registry during the deploy

When promoting to prod, we simply pull the same image to prod.

For AWS, we can use Amazon ECR as the registry. You can also push your images to Docker Hub or another container registry if you’re running outside of AWS and your cloud doesn’t have a special service for that.

Let’s implement that:

Currently we build the image during the deploy stage.

Split it into two stages:

- Build: build the image and push it to a container registry (ECR)
- Deploy: pull the image from the registry and serve it

The manual prod promotion CI/CD workflow pulls the currently deployed dev image to prod.

Tag each image using the YYYYMMDD-HHMMSS-shortsha pattern (e.g. "20260818-163457-83242da")

Now the build step produces a tagged image, and the deploy step deploys it to dev. We can test the dev application, and when we later promote it to production, we will be certain that it’s exactly the same image.

Observability

Having two environments helps us avoid accidentally pushing buggy code to production. But accidents will still happen, and we need to make sure we detect them and react as fast as possible.

For that, we need to have observability. “Observability” means collecting information about the application so we can understand its behavior. With it, when something breaks, we can quickly find the problem.

We achieve observability by adding monitoring to our applications. At the minimum, we need to collect basic performance metrics like CPU and memory utilization and requests per second (RPS).

If we see that CPU and memory utilization are growing and RPS is dropping, something may be off.

This information alone is not enough. It doesn’t explain what happened inside a request and why it’s causing errors or degraded performance. For that, we need to collect more.

OpenTelemetry

OpenTelemetry (often abbreviated as “OTel”) is the industry standard for telemetry.

Telemetry is all the information that the application produces:

• Metrics - requests per second, response latency, and number of errors

• Logs - timestamped records of individual events, like an error message or a failed database query

• Traces - all the steps (called “spans”) that a single request makes through the entire application

OTLP is the protocol that applications use to send this telemetry.

For many popular libraries, we can start capturing telemetry with just a few lines of code. This process is called “instrumenting” - adding telemetry collection to an application. Auto-instrumentation can instrument libraries like FastAPI without changing their code.

Let’s do it. Ask the coding agent:

Instrument the backend with OpenTelemetry.

Include in the telemetry:

- service name
- environment
- deployed version

The application now produces telemetry and exports it over OTLP. We still need to decide where to send and store it, and how to view it. Let’s do that now.

OTel Collectors

We export telemetry over OTLP, but it’s not saved anywhere. For that, we define an OpenTelemetry Collector between the application and the storage systems.

There are many services you can use for that. Ask your AI assistant and select the option that works best for your application.

In our case, I’ll use:

• Prometheus for metrics

• Loki for logs

• Tempo for traces

• Grafana for dashboards

I had never used Loki or Tempo before, but when I told my coding agent that I wanted to use Prometheus and Grafana, it suggested including Loki and Tempo too.

So let’s implement it:

Add an OpenTelemetry Collector.

Create "observability/" directory with Docker Compose for:

- OpenTelemetry Collector
- Prometheus
- Loki
- Tempo
- Grafana

Keep this as a separate Compose project from the application stack

Now we have the infrastructure for observability, so we’re ready to build a dashboard.

Metrics

Metrics are numerical measurements collected over time. They can show a current value, such as the number of active participants, or count events during an interval, such as rooms created or errors in the last five minutes.

For System Design Canvas, useful measurements may include:

• Total number of interview rooms

• Number of active rooms and number of active participants

• Number of elements on the canvas across the application

• Change propagation delay: when I add an element to the canvas, how long does it take for you to see it

• Number of errors

Let’s start collecting some of them:

Track these application metrics:

- interview rooms created
- active interview participants
- canvas elements created
- failures in component creation

Include the environment and deployed version

Now we have the data and can display it in a dashboard:

Add a Grafana panel with these metrics. Make it possible to filter by environment and deployed version.

Run it locally and test it by opening our application, creating a room, adding a canvas element, and checking that it appears in Grafana.

Deploy the observability stack

If you use the same stack as me (Prometheus, Grafana and others), we need to deploy and manage them ourselves.

If you use a managed OTel-compatible service such as CloudWatch, Grafana Cloud, Datadog, or Sentry, you can skip this step.

So if you’re following my steps, let’s deploy our stack:

Deploy the observability stack. It should be separate from the application stack.
Connect both development and production to it.

After it finishes, you can open Grafana and perform the same test that we did locally.

Note: here we deploy Grafana and the other services openly on the Internet, so everyone can access them. In practice, you put sensitive resources like observability services and databases on private networks, restrict network access, and require authentication so only authorized users can access them.

Alerting

We have the dashboards, but we can’t watch them all the time to see if some metrics drop or grow too large. If this happens, we need a system that triggers an alert saying that something happened.

Add an actionable alert for repeated canvas component-creation failures.

Use a threshold and duration that represent real user impact. Include the
service, environment, deployed version, owner, and dashboard URL in the alert.

On-Call Engineer

Now that we have alerts, we need to react to them. In companies, there’s usually an on-call engineer. That’s the engineer who receives the alert when something happens. They need to figure out what’s happening and find a quick solution to stop the problem.

When a metric crosses its threshold for long enough, the alert fires and the on-call engineer receives it.

Our AI assistant can be the on-call engineer, and if something happens, it can try to fix it. In real scenarios, AI on-call agents also need to understand if the issue is serious enough to escalate the alert to a human on-call engineer.

In our case, we won’t do it. We will implement a system that checks the observability system for alerts, and if something is happening, an agent session will start and try to fix the problem.

Add an on-call-engineer/ directory with a script that polls the observability
alert API every minute.

When an alert fires, pass the alert details to a headless coding agent.

As a result, we have a worker that polls for alerts and starts an agent session when one fires.

In my case, the agent had this prompt:

You are the on-call engineer for this repository. An alert just fired.

Investigate the root cause. Read the code and reproduce the failure.
If you find a real bug, make the smallest correction, run the backend tests,
and commit the fix with a clear message.

If the alert is a false positive, explain why and do not change the code.

This is a small proof-of-concept script. In reality, you will probably have a system that looks like this:

• An alert is triggered and sent via SNS or a similar service.

• A Lambda reacts to that alert and starts a container job.

• The job launches Codex or Claude in headless mode.

• Once the session is over, the logs are saved and the compute is terminated.

Let’s test our idea by introducing a bug.

Introducing a bug

We want to test that this system works, so let’s start a coding agent and ask it to add a bug or find an existing one.

Introduce a realistic bug in canvas component creation.

For some requests, creating a component should fail even though the existing tests pass. Keep the failure reproducible so we can test that the bug causes an alert and the on-call response.

Here, the goal is to debug the process and make sure our on-call engineer actually wakes up and solves the problem.

After a few iterations, we have an on-call engineer that wakes up when there’s an alert and tries to fix the problem.

Clean up

After you finish everything for the module, I recommend cleaning up everything. We created these things for learning, so we shouldn’t forget to clean them up. If you don’t, expect a bigger bill at the end of the month.

Ask your agent to clean the infra:

Delete the CloudFormation stacks.

Check yourself that everything is indeed gone. You may want to double-check everything in the AWS console or ask another agent to scan the running resources in your account.

Next steps after this module

We started with an idea, and now our setup is much closer to production. But here I could only briefly touch on the most important concepts.

There are many more things that you should consider for a real production application:

• Here I deploy to EC2 by running a shell script on the instance. It’s okay for a proof of concept but very problematic for production. Use a container management system like ECS or an alternative.

• Sometimes you will need to roll back a change you promoted to prod. Ask your agent to make it easy to do.

• Use a managed database service.

• Regularly back up your database. It’s best if your backups live outside of your infrastructure-as-code stack and you have multiple independent copies.

• Test that you can actually use these backups.

• Put internal services and resources on private subnets inside a VPC, and restrict network access to them.

• If your application needs to handle a lot of traffic, learn about scaling and load balancing. Container management systems make this easier to manage.

• Ask Fable or GPT-5.6-Sol Max (better both) to audit your code for security vulnerabilities. Do it multiple times.

Next in the series

With this article, we finish developing our end-to-end application. We took it from a raw idea and turned it into a system that’s much closer to production.

In the last article, we will look into coding agent capabilities:

• MCP

• skills

• commands

• plugins

• specialized agents

You can find the course materials and the next cohort in AI Dev Tools Zoomcamp. It’s free.

It also appears in interview questions. In interviews, you’ll definitely hear something like “how do you evaluate a RAG pipeline?” or “how do you measure agent performance?”. If you don’t include evals in your home assignment, it’s a red flag.

Why is it such an important skill? Because it’s hard, and it matters.

It’s easy to build an agent these days. I show how to do it in my “AI Hero: 7-Day AI Agents Crash-Course”. You need to get an OpenAI key, take any agentic framework, define the instructions and the tools, and that’s it. You have an agent.

But this agent will break in so many ways. It will:

• Give a wrong answer

• Confidently hallucinate a plausible answer

• Finish without giving any answer

• Get into a loop and call the same tool over and over again

• Exhaust the context

• Call a tool with invalid arguments

That’s just the tip of the iceberg. To make sure this agent works reliably, you need to test and evaluate it.

Also, without evaluations, you’re blind. You don’t know how the agent is performing. You also can’t really change anything: every change you introduce may break the system in many ways, including the areas where you least expect it.

Agent evaluation is not simple. It takes a lot of time to understand the problem, get input from real users, and gather the right data. It’s not something you can just delegate to Claude and forget about it.

That’s what makes evals such an important part of AI Engineering.

In this article, I will tell you about my approach to evaluations:

• Start manually by “vibe-checking” the system, but also collect logs as early as possible

• Create a tool for labelling the logs and put together a gold standard dataset

• Create a judge that’s aligned with our judgement

• Break the agent like a QA engineer

• Get more data by generating it synthetically

• Start collecting data from real users

• Monitor your system with online evaluation

• Always refine your gold standard dataset

Let’s get into details.

Collecting the Gold Standard Dataset

You have an agent. What’s next?

There’s nothing wrong with “vibe-checking” it: you poke it, ask questions you want it to answer, and look at the results. If they aren’t good, you figure out how to fix it.

But already at this step you can be a bit more organized and start logging what the agent is doing. When you ask a question and see the answer, make sure they are saved somewhere. You do a few vibe-checking sessions, and you already have 10-15 logged records.

Now you can systematically look at these logs and classify each record into

• “good”: the answer is what you expect

• “bad”: the answer is not what you wanted to see

To help me do it, I usually vibe-code a small labelling tool.

This gives you a “version 0” of the gold standard dataset - the dataset you will use for evaluating your system (sometimes it’s also called the “ground truth” dataset).

Don’t delegate this step to coding assistants. It’s only a few examples, so you can go through them relatively quickly. But this way, you will get a lot of insights into how your agent is behaving, what’s working and what isn’t.

Judge alignment

Once you manually labelled 10-15 examples, it’s time to start automating it.

I use coding assistants for that:

• Ask the assistant to analyze the logs and classify the examples into good or bad.

• See if the results match yours. If they don’t, ask the agent to explain its decision - maybe you missed something, or the agent didn’t have enough context.

• Iterate until you both agree.

• At the end of the session, ask the agent to create a judge.md file that describes the classification rules.

• Read this file carefully to see that the rules are generic and not specific to the examples you have in the dataset.

We just created a judge - a system that evaluates our agent.

Now we need to test that using judge.md alone is sufficient:

• Start a new session

• Ask to start a subagent which reads judge.md and classifies the records in your dataset

• If something is off, ask the agent to change judge.md

• Iterate until you both align on what’s good and bad

This process is called “alignment” - we align the judge with what we think is good or bad.

It’s the same as training a binary classifier that predicts your decisions. But instead of weights, you have a judge.md file. (And it’s also helpful to have a train-test split here too! But later, when you have more data.)

At this point, you also get a metric - the fraction of good examples. Now if you change something in your agent, you can re-run the judge and make sure this metric doesn’t go down.

Breaking the agent

We have the initial version of our gold standard dataset. But it’s not enough. Now we want to find cases that break our system.

QA engineers have been breaking software for many decades. For coming up with cases that should derail our agent, we can borrow some concepts from their frameworks.

1. Equivalence partitions

The first step is equivalence partitioning. You divide the input space into groups where inputs within each group should produce similar behavior.

If we have a RAG agent that answers questions about our docs, it could be:

• Questions about topics in the documentation,

• Questions about relevant topics but not in the documentation,

• Questions about irrelevant topics,

• Ambiguous questions,

• Questions in different languages

Once you come up with a list of groups, come up with 2-3 questions from each group. You can ask AI to help you with the entire process.

2. Boundary testing

The next concept is boundary testing.

We already have some questions from the previous step, but we can be even more aggressive in our testing.

After defining the partitions, QA engineers would test the boundaries of each partition. We can do the same.

For a RAG agent, come up with queries that:

• Exactly match a document that we have in our database (easy case),

• Are semantically related but use different words (typical case),

• Are semantically related to multiple documents (ambiguous case),

• Are completely off-topic (boundary case).

Further alignment

This way you can easily add 30-40 more questions to your gold standard dataset.

Let’s take all the data we have, and run it against our agent. Then use the judge to classify the output.

We already have a tool that we created earlier for manual evaluation. At this point, I adjust this tool to also support checking the output of the judge. For each record, I just decide whether I “agree” or “disagree” with the judge. I find it more convenient than labelling the new logs as “good” or “bad”.

If we have many “disagree” cases, tune the judge following the same process as before. We will never have 100% alignment, so we don’t need to be too strict about it, but the judge should be correct most of the time.

Using the Judge to improve your agent

Now we have a fully automated process that:

1. Runs the agent against our data

2. Runs the judge against the output of step 1

3. Outputs a metric - the fraction of “good” results

Because it’s automated, we can use it to improve our agent. You can ask the coding assistant to do it.

I recommend following the implementer-tester pattern for that:

• Define the implementer subagent that modifies the agent

• Define the judge subagent that uses judge.md to evaluate the output

• Iterate until your “good” ratio is acceptable

Synthetic data

So far we focused on the quality of our gold standard dataset. In this section, I want to talk about the quantity.

Quantity is also important because agents are non-deterministic. Even if the agent works properly on our small dataset, it will break when we get more traffic - just because of probability and statistics. When you expose your agent to a bigger number of inputs, things will break: with more repetitions, you increase the chances of something going wrong. That’s why you want quantity here.

We can generate a lot more data using AI. Open your AI assistant, ask it to analyze your code, describe to it your target users, and then tell it to generate questions that these users are likely to ask.

For a RAG system, you can:

• Take a document from your knowledge base

• Ask AI to come up with 5 questions based on this document

• Do it for all documents

You will end up with a lot of questions. You can keep all of them, or take a sample.

Then follow the same process: run your agent against this data and then use the judge to evaluate the results.

This way you will probably run into many different problems:

• The agent will loop without stopping and exhaust the context

• It won’t follow the instructions and do something that it shouldn’t

• A tool will break and your agent won’t finish the task

And many others. You don’t know in advance what you’ll get, which is why you want to expose the agent to a wide variety of inputs.

Once you identify the scenarios where your agent breaks or doesn’t perform well, add them to your gold standard dataset.

Then you can fix the agent, and re-run the whole evaluation set to make sure you don’t introduce any regressions.

Start testing on real users as soon as possible

We have evaluated our agent on many cases, so we’re confident that it’s working okay. But we always need more data.

Now it’s time to put it in front of real users:

• Invite people for user testing sessions to check your system

• If you plan to integrate it into an existing product, include it in a small subset of the traffic

• If it’s a small personal project, ask your friends and family to test it.

• If it’s a course project, ask your classmates to play with it.

• Share it on social media.

Make sure you’re collecting the logs (be direct with the user about what you’re collecting) and use this for adding new cases into the gold standard dataset.

Add +1/-1 buttons. This way, the users can give you explicit feedback, and if somebody dislikes the answer, you should inspect it.

Online evaluation

We have a system for offline evaluations: we run the agent against our dataset and we use the judge to score the output to see how well it behaves.

It’s okay, but it’s not scalable. Instead, we want to run it continuously: every time somebody uses our agent, we react in real time - save the logs, evaluate them, and display the results on the dashboard.

We can’t really continue using our coding agent for that, so we’ll need to create a stand-alone judge.

It will give you a real-time view of the agent’s quality: the performance on live traffic, and new cases that you can add to the gold standard dataset as they appear.

Periodically inspect bad results, and if you see something new, add these cases to your gold standard dataset. Also sample the good results to make sure they are actually good.

For inspecting this live traffic I also often use coding assistants. I take a sample of the data, save it locally and ask them to go through these cases with me.

Multiple judges

We started with a good/bad evaluator, but I didn’t really explain what it means. And it anyways depends on your problem.

Over time, we will need to have multiple judges, each evaluating the agent from a different angle:

• Task completion: the agent actually solved the user’s problem

• Relevance/Correctness: the answer is relevant/correct

• Groundedness: the claims in the answer are supported by the documents in the knowledge base

• Completeness: the answer includes all the required information

• Following instructions: the agent follows the instructions correctly

• Trajectory optimality: the sequence of tool calls the agent used to complete the task is an optimal one

There are other criteria too that you can consider for your application.

But I usually recommend starting with one judge and then adding others as your system matures.

The agent evaluation checklist

As a summary for this article, I wanted to give you a checklist that you can use next time you work on a new agent.

Gold standard dataset:

- [ ] Vibe-check the agent and log 10-15 interactions
- [ ] Label results as "good" or "bad" yourself
- [ ] Define equivalence partitions and test cases within each
- [ ] Include boundary cases (exact match, paraphrase, multiple matching documents, off-topic)
- [ ] Generate synthetic questions

The judge:

- [ ] judge.md describes the classification rules in generic terms, not specific to your examples
- [ ] The judge runs as a subagent that reads only judge.md
- [ ] Check the judge's answers with "agree" or "disagree"
- [ ] Use the fraction of "good" results as your metric
- [ ] Add more judges as your system matures

Improving the agent:

- [ ] The evaluation pipeline is automated: run the agent, run the judge, get the metric
- [ ] Each time you make a change, run the full evaluation set to catch regressions

Online evaluation:

- [ ] Put the agent in front of real users
- [ ] Collect user feedback
- [ ] Score online traffic with a stand-alone judge, show results on a dashboard
- [ ] Inspect bad results periodically, add new failing cases to the eval dataset

If you want to learn more about building AI Agents, I teach that in my course AI Engineering Buildcamp: From RAG to Agents. In the course, I also cover evaluation in a lot of detail with real code, real tools, and real agents.

The next cohort starts September 21. You can use code “SUBSTACK” to get 20% off. See you there!

All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants (this article)

• Part 4: DevOps and Observability for an AI-Built App

• Part 5: Coding Agent Building Blocks: Reusable Skills and Specialized Subagents

In this article, I want to take a web application and make it accessible for everyone on the internet.

We will:

• Take the application that’s already working locally

• Containerize the application

• Switch from SQLite to Postgres

• Add integration tests

• Deploy to AWS

• Set up CI/CD via GitHub Actions

It’s based on the second half of the full-day workshop I did for AI Shipping Labs: Build and Deploy a Full-Stack App with AI Coding Assistants.

Recap

In the previous article, we started building an application for system-design interviews. An interviewer creates a session and shares a link with a candidate. When the candidate updates something on the canvas, the interviewer sees the updates in real time.

• First, we created the frontend only with React

• Then we created an OpenAPI specification for defining the frontend-backend API

• Next, we created the backend from this specification

• We added database support with SQLite and SQLAlchemy

You can find the code in the interview-canvas-share repository.

Containerization

When we run the application locally, we need to execute two commands: one for the frontend and one for the backend.

Let’s run them in two separate terminals:

# first terminal
cd frontend
npm run dev

# second terminal
make run

So, we have two services, and we need to deploy them to production. We may think that we need two containers: one for the frontend and one for the backend.

During development, we run the frontend as a separate service because we use Vite. Vite watches the frontend code and refreshes the page when we make changes. It’s very convenient because we can see our changes immediately.

In production, the frontend code doesn’t change while the application is running. We build it once and get a set of static HTML, CSS, and JavaScript files.

This means we don’t need a separate container for the frontend: the backend can serve these files. So, we need only one container.

Ask the coding assistant to create it:

Create a Dockerfile that builds the frontend with Node, then builds a Python image with the backend and the frontend static files.

Backend should serve the frontend.

We get a two-stage Docker build:

• First, we use a Node.js image to compile the frontend

• Then we build the backend and copy only the frontend files without the Node.js dependencies

You can see the Dockerfile here.

Build and run the image from the repository root:

docker build -t sdip:latest .

docker run --rm -p 8000:8000 \
  -v sdip-data:/data \
  -e SDIP_DATABASE_URL=sqlite:////data/sdip.db \
  --name sdip sdip:latest

Here we specify a named Docker volume sdip-data that will keep the SQLite database between container runs.

Open the application at localhost:8000 and test it:

• Create an interview session

• Open the join link in a different browser window

• Move an element in the candidate window

• Check that the interviewer sees the change

We’ll repeat this test again. I’ll refer to it as the two-session test.

Switch from SQLite to Postgres

SQLite is very convenient for local development. It keeps the data in a single file and doesn’t need a separate database server.

But for production, we typically use Postgres or a similar database.

When we set up the foundation in the previous article, we asked the coding agent to use SQLAlchemy.

I did this on purpose because I knew that later I’d switch to Postgres.

Start Postgres locally:

docker run -d \
  --name interview-canvas-db \
  -e POSTGRES_USER=sdip \
  -e POSTGRES_PASSWORD=sdip \
  -e POSTGRES_DB=sdip \
  -p 5432:5432 \
  -v interview-canvas-pgdata:/var/lib/postgresql/data \
  postgres:16-alpine

Now we can ask the assistant to use it:

Add Postgres support to the backend.

Run the backend against Postgres from the repository root:

export SDIP_DATABASE_URL=postgresql://sdip:sdip@localhost:5432/sdip
make run

When it’s done, repeat the two-session test.

Docker Compose

Previously, I started a Postgres container with a separate command. But now let’s put all the services our application needs inside one Docker Compose file.

With this file, we can run our entire application with a single command docker compose up.

Ask the assistant to implement it:

Create docker-compose.yaml with two services: Postgres and the app.

The file defines the database and adds a health check, so our application waits until Postgres is ready to accept connections.

Start it:

docker compose up --build

In our case, it runs the application at localhost:8100.

Integration and end-to-end tests

The AI assistant might have created some backend tests.

If it didn’t, ask it to create integration tests:

Create integration tests that run against docker-compose.yaml. What scenarios should we test?

We added two things that can potentially break, so let’s test them too. We’ll verify that:

• The frontend compiles correctly

• The backend can communicate with Postgres

Ask the AI assistant to implement a test:

Add an end-to-end test that runs against docker-compose.yaml.

Use Playwright to:

1. Log in as the interviewer (session 1).
2. Create an interview session.
3. Share the join link.
4. Join from a separate client as the candidate (session 2).
5. Change the canvas as the candidate (session 2).
6. Verify that the interviewer sees the change (session 1).

Put the tests in the e2e/ folder in the repository root.

After it finishes, we can run the tests with a single make command:

make e2e

Deploy to AWS

We’re now certain that the application works well. We can deploy it.

Our application runs in a container and only needs Postgres, so we have a lot of options for deploying it. We can use Render, Railway, Fly.io, or any other managed container system.

Last year we deployed to Render, but this year I want to deploy to AWS. You don’t have to use AWS, and instead you can ask the coding assistant to recommend an environment for your application.

Ask your assistant to deploy it:

Deploy this application to AWS. Use AWS CloudFormation.

For that to work, you need to have an AWS user. I typically create a temporary user with admin permissions and watch every step of what the coding agents are doing.

You can see what I got here. It runs the app, Postgres, and Caddy (adds HTTPS and WSS support for our app) on one EC2 instance.

It’s fine for a proof-of-concept, but using managed database services (such as RDS) is better. We will not do it here.

CI/CD with GitHub Actions

We used a user with the admin permissions to deploy the application. It’s okay for the first deployment, but only when we carefully watch it. The next step is to configure CI/CD and remove that access.

• Continuous integration (CI) means that every time we make a change and push it to GitHub, we automatically run all the tests to make sure we didn’t break anything.

• Continuous deployment (CD) is about deploying this change automatically.

In GitHub, we use GitHub Actions for that.

Let’s configure it. Every time we make a push to main, we want to:

• run frontend and backend tests

• build the containers

• run the integration tests

• run the end-to-end tests

• if all the tests pass, deploy the new version

For the last step, the runner (the process that will deploy the application) will need to be able to access our AWS infrastructure. We will use OpenID Connect (OIDC) for this: the runner will assume a role with the necessary permissions and update the application.

Create the role and the workflow:

Create a CI/CD pipeline that:

- runs backend and frontend tests in parallel
- builds the Docker Compose stack and runs integration and end-to-end tests against it
- deploys to AWS using a GitHub OIDC role
- validates that the deploy is successful by checking the health endpoint

The finished workflow uses a restricted AWS role for deployment.

Change something in the application, commit, and push to see it go live.

Clean up

When we’re done, we need to delete all the resources created by CloudFormation:

aws cloudformation delete-stack --stack-name sdip
aws cloudformation wait stack-delete-complete --stack-name sdip

The next step after deployment

Let’s recap what we have done so far:

• We created the frontend application with React

• Then we defined the frontend-backend API with OpenAPI specs

• Based on the specs, we created the backend

• Next, we added database support with SQLite and SQLAlchemy to the backend

• To make it easier to deploy the application, we put both frontend and backend inside one container. The backend serves the frontend.

• To go to production, we replaced SQLite with Postgres.

• Next, we simplified running everything locally with Docker Compose

• Once everything was in Compose, we created an end-to-end test

• We took the application that worked locally and deployed it to AWS using CloudFormation

• Finally, we created a CI/CD deployment pipeline to deploy every change automatically.

But we still need to do a lot more for the app to run reliably:

• Dev and prod environments

• Observability: logs, metrics, alerts

• Using AI as the first responder when the application stops working

We’ll cover it in the next lesson.

In this article I take that algorithm and update it for AI Engineering. The updates are minimal and you can use this algorithm for any other engineering job.

The Job Search Algorithm

The algorithm is straightforward:

1. Network: Connect with people

2. Apply: Apply for jobs even if you don’t qualify

3. Build: Create projects and share them online

4. Interview: Record what you were asked

5. Reflect: Review and improve

6. Repeat

This is a while loop. Continue networking, applying, building, and interviewing until you receive an offer you like.

Step 1: Network

Networking helps you understand the market, find opportunities, and learn what companies are actually looking for.

Use LinkedIn and meetups for that.

Talk to people working in your field. Talk to people who work as AI engineers. Ask about the projects they work on, and the technologies they use. And ask if they are hiring and ask for a referral.

If you are a junior, look for other juniors who have recently got a job. Ask what helped them get hired, which companies they targeted, what the interview process involved, and what they would do differently.

A conversation doesn’t have to end with a referral. By talking to people you discover companies that hire, understand which skills to develop, improve your search strategy, and meet interesting people.

Step 2: Apply

Start applying for jobs right now. Many people postpone it because “they are not ready". You are ready. Apply even if you don’t qualify.

Don’t reject yourself because you are unsure if you’re qualified. Apply and let the hiring team evaluate your application. If you apply, you have a chance to move forward. If you don’t, you never will.

When you look at the job requirements, don’t treat them all as mandatory. The job description always describes the ideal candidate and often these candidates don’t exist.

There are, of course, cases where you shouldn’t apply. If you’re a junior, but the position is senior, then don’t apply. If you match only 20% of the requirements, it’s also a bad sign. If you’re in doubt, apply.

If you apply for relevant roles but don’t hear back, review your CV. Make sure it shows the experience, skills, and projects that are relevant to the positions you apply.

Recently I published a workshop at AI Shipping Labs about tailoring your CV for AI engineering roles that you may find helpful (heads up: it requires a paid subscription).

Step 3: Build

You learn by doing. When you apply, note which skills are required for the positions you apply for, and build projects using these technologies.

Always timebox these projects. Set the time limit to 10-20 hours and then move on to the next project. At this stage, your goal is to have an awareness of different technologies and an understanding when to use them. You can go deeper when you’re already working.

I wrote a full guide on building projects with six steps you can follow:

A complete project should include working code in a public GitHub repository and a clear README. The README is the most important document of your project, so make sure you put enough effort into polishing it before you add a project into your portfolio.

A few weeks ago I wrote an article about that:

Don’t build these projects silently. Share what you build and what you learned online.

Here’s a suggested sequence of posts for LinkedIn and/or X:

1. Describe the project idea and the technologies

2. For each milestone, post a screenshot or a code snippet with what you built

3. When the project is ready, announce it and publish the link

4. Optionally, write a blog post

And yes, there’s another article I wrote about it:

Step 4: Interview

Once your CV gets past the screening stage, you get invited to an interview.

Don’t treat interviews as exams. You’re not in school. It’s a two-way process: they evaluate you, but you also evaluate them.

Interviewing is a also skill. Your first interview may not be great, but it improves with practice. Each subsequent interview helps you identify gaps in your preparation and see what you should do differently next time.

Some interviews will lead to rejections. It’s normal and may happen for many reasons outside of your control: the company chose someone with more suitable experience, hired an internal candidate, changed the role, or froze hiring. Don’t take the rejections personally, learn from them and move on.

The hiring process varies between companies, but typically you will have some of these interviews in the pipeline:

1. Recruiter interview: a short conversation about your background, motivation, availability, salary expectations, and basic fit

2. Theoretical interview: questions about LLMs, RAG, agents

3. Coding interview: coding, Python, SQL, code review, or practical engineering tasks

4. Project deep dive: a deeper discussion of your experience and past projects

5. System design: defining the architecture of a project

6. Behavioral interview: examples of how you dealt with different situations at work (and outside)

7. Take-home assignment: a small project you complete at home

You probably won’t get all of them: some stages may be bundled in one round, and other stages may be absent.

Now let’s go through each step of the hiring pipeline. In this article, I mostly used the materials from AI Engineering Field Guide. You will find a lot more relevant information there if you’re preparing for AI Engineering interviews.

⭐ Star the Field Guide repo

1. Recruiter interview

The recruiter is the first person you speak with during the interview process.

It’s not a technical interview. They will tell you about the company, the role and ask some basic questions.

You should be ready to answer:

• “What do you know about our company?” Make sure you do some basic research. If you can’t answer it, the interview process is likely to stop there (it happened to me!).

• “What are your salary expectations?” Research the market and be ready to provide a realistic range. You can dodge this question and delay it to other stages, but I don’t think it’s helpful.

• “Tell me about yourself”. Prepare a 3-minutes story. It will be useful at later stages too.

Use this interview to make sure you understand what the interview process looks like and what the expectations are at each stage of the process. Ask for tips how to prepare for these stages.

2. Theoretical interview

There will be one or multiple rounds where the company checks if you understand how LLM applications work. They may appear in a dedicated technical interview or as part of the project deep dive or AI system design sessions.

For AI Engineering, you will be asked about:

• LLM behavior, prompting, context windows, and structured outputs

• RAG, retrieval, reranking, and common ways RAG fails

• Agents, tool use, tool design

• Evaluation and debugging

• Monitoring, cost, latency, and scalability

Fine-tuning, model training, and transformer internals can appear too, but this only happens for roles that require these skills.

You can see theoretical questions in the “Theory” part of AI Engineering Field Guide interview preparation section.

3. Technical interview

Coding rounds generally fall into two categories: implementation and algorithm rounds.

Traditionally, for implementation rounds, you’re given a task like “implement a web crawler”. You’re not expected to deliver a fully working solution, but rather demonstrate your approach to coding. You may need to design (in the “object-oriented design” sense of the word) the interaction between different components, define proper abstractions, and then implement only some parts of it.

However, more and more often companies give a vague task and ask you to use AI to implement it. You are expected to produce a reasonably complete working solution. The interviewer evaluates how you interact with the coding agent, how you review the code, test it, correct errors, and refine the implementation.

The algorithm rounds usually involve easy and medium Leetcode-style problems. Hard problems are rare. There is already a lot of information on the Internet on how to prepare for them, so I won’t spend time on it. Joining Leetcode and solving some problems there definitely helps.

You can see technical questions in the “Coding” part of AI Engineering Field Guide interview preparation section.

4. Project deep dive

Project deep dive interviews assess your level of seniority and check if your CV matches the reality.

At my previous jobs I regularly conducted these interviews.

I start by asking the candidate to introduce themselves and then ask what they were doing recently. Our discussion naturally goes into projects they worked on and then I start going deeper with questions:

• What problem does this project solve?

• How does it work?

• Which part did you build?

• Who made the technical decisions? How?

• What went wrong?

• What would you change if you were implementing this project today?

This is by no means an exhaustive list, but it should give you some ideas of what to expect.

I recently had a workshop about it at AI Shipping Labs. Based on it, I wrote an article: AI Engineering Hiring Manager Interview (it’s free).

You can see deep-dive questions in the “Project Deep-Dive” part of AI Engineering Field Guide interview preparation section.

5. System design

If you’re applying for more senior roles, you’ll have a system design round. They are very uncommon for junior roles.

There are two types of system design interviews: “classical” system design and AI system design.

AI engineers are software engineers who specialize in AI, so they are expected to design software systems too. That’s why one of the interview rounds may be classical system design.

In these interviews you will need to design a system related to what the company and the prospective team do.

A food delivery company may ask you to design an order-processing or delivery-tracking system. When I interviewed at Yandex, they asked me to design search. When I interviewed with Meta, they asked me to design a geo-aware recommender system. When I worked at OLX and we were hiring people to join the recommender platform, we were asking to design a recommender system for an online marketplace.

But for AI engineers there’s also an AI system design round. Common tasks include designing a document Q&A system, a coding assistant, an AI search engine, or an agent. The specific system will depend on what the team is doing.

For these interviews, clarify the requirements, ask which data is available and what’s the expected scale. After that, sketch the main components of the system.

You can see AI system design questions in the “AI System Design” part of AI Engineering Field Guide interview preparation section.

6. Behavioral interview

The behavioral interview checks how you collaborate, make decisions, take responsibility, and handle setbacks. “Tell me about a time you needed to help a colleague”. There’s nothing specific in these interviews for AI engineers.

The best way to prepare for these interviews is to research the company’s values. You can ask the recruiter about them too. For each value, prepare examples of how you demonstrated them. As a general reference, I recommend Amazon’s Leadership Principles.

You can see behavioral questions in the “Behavioral” part of AI Engineering Field Guide interview preparation section.

7. Take-home assignment

In take-home assignments you need to build a small AI application, submit the code, a README, and a working demo. It’s also typical to have a defense session where you need to present the solution and answer questions about it.

Before starting, clarify the requirements, ask if you can use AI tools, and what the evaluation criteria are.

Before finishing, make sure you have tests, an evaluation set, and a good README that describes the system well.

Take-home assignments are time-consuming. They are a good addition to your project portfolio, but if you interview with multiple companies, and all of them give you homework, it can add up.

If you’re short on time, it’s okay to decline them. When you do, make sure you have projects in your portfolio that demonstrate the qualities they want to assess. Then you can say “I unfortunately don’t have time to work on it, but I have similar projects on GitHub and can walk you through one”. Don’t expect it to work with everyone, but some companies may agree to this.

When getting a home task, watch out for warning signs like these:

• You get an automated take-home assignment right after applying, even before speaking to anyone.

• The company states the task takes two hours, but the requirements suggest it could take several days.

• The company won’t clarify how your submission will be assessed or utilized.

In these cases you may spend time working on the test and never hear back from the company after submitting your project.

You can see examples of tasks in the “Home Assignments” part of AI Engineering Field Guide interview preparation section.

Step 5: Reflect (after the interview)

After each interview, write down all the questions you were asked and tasks you were given. What was difficult? Which answers could have been better?

You can also talk to AI, describe in detail how the interview went, and ask for feedback.

Write that all down in your interview journal.

This will help you identify areas where you can improve, and give better answers in the next interviews.

If you’re rejected, ask the company for feedback. Many companies will decline: there’s a non-zero chance that you may sue them and use this feedback against them, so they won’t risk it. But some will share it - most likely over a phone call. Also, don’t take it personally. Rejections are part of the process.

Continue networking, applying for more jobs, keep building and sharing what you learn.

If you get an offer - congratulations! But don’t rush into accepting it. Evaluate if the package is good and if you will like working there.

If you’re interviewing with other companies, finish the process with them too. It’s okay to be explicit about it with the company that made the offer. If they are pushing you to accept it as soon as possible, this is a red flag, so consider this carefully before agreeing. You got one offer, so you’ll definitely get others.

Good luck and have fun!

Share Alexey On Data

What I’ve been working on this week

1) Building and deploying a website with ChatGPT Work

This week, I experimented with building a small website using ChatGPT Work.

I’m really impressed. I sent it photos of a music festival brochure (it was Brutal Assault) and asked it to turn them into a website. It extracted the schedule, built a festival planner, and deployed it to a public URL that I could share with others.

I didn’t even need to run Codex. Everything happened from my phone from the ChatGPT app. It also has built-in authentication (via ChatGPT) and a built-in database.

2) FAQ Assistant in Practice

Last week I wrote an article about the FAQ assistant that we use at DataTalks.Club. Then this week I did a webinar about it where I gave a more detailed overview of the system.

3) Building a full-stack application with AI

I hosted the second workshop for AI Dev Tools Zoomcamp. In the first, we talked about specification-driven development, loop and graph engineering. In the second, I showed how to build an end-to-end application with frontend, backend, and a database. As an example, we built an application for system design interviews.

I’m hosting the notes on Substack, so you can read them here:

There will be more workshops and events for AI Dev Tools Zoomcamp:

• August 10: Test, Containerize, and Deploy an AI-Assisted App

• August 18: DevOps and Observability for AI-Built Apps

• August 24: Pre-Course Live Q&A

• August 31: Course launch

I will publish the materials from all these workshops as Substack articles too.

4) Inference Engineering - AI Shipping Labs book club

We are also starting a book club at AI Shipping Labs.

Our first book is Inference Engineering by Philip Kiely. It covers model architecture, hardware, serving software, optimization, cost, reliability, and running inference systems in production. The digital edition is available for free.

The kickoff is on August 10 (Monday) at 5:00 PM CET. We will review the book, discuss the reading cadence, and decide together how the sessions and discussions should work.

Join the kickoff

5) Stock Markets Analytics Zoomcamp

In DataTalks.Club, we’re preparing the next cohort of Stock Markets Analytics Zoomcamp. It starts on August 24.

The course provides a gentle introduction to stock market trading, Python programming in Google Colab, analytics, and data visualization.

Register for the course

Edited by Valeriia Kuka

In the first article, I wrote about turning an idea into a specification. The specification is only the beginning. In this article, we will create an end-to-end application from scratch. We will cover both frontend and backend, and add a database.

All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants (this article)

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App

• Part 5: Coding Agent Building Blocks: Reusable Skills and Specialized Subagents

We will create an application for system-design interviews.

As an interviewer, you create a session and share the link with the interviewee. They can create diagrams in the app, and you see the changes in real time.

Both frontends connect to the same room through WebSockets, so the changes appear in both sessions simultaneously. The FastAPI backend handles those events and saves them to SQLite.

You can find the code in the interview-canvas-share repository.

It’s based on my full-day workshop I did for AI Shipping Labs: Build and Deploy a Full-Stack App with AI Coding Assistants.

Overview

Like in the previous article, we start with an idea. As this is only an idea, we turn it into a specification using ChatGPT.

From there, we start building:

1. Generate a frontend that uses mocked backend calls.

2. Establish backend-frontend contract by creating OpenAPI specifications from the frontend service layer.

3. Implement the FastAPI backend, connect it to the frontend, and test the application.

4. Add persistence with SQLite and SQLAlchemy.

At each stage we get something concrete that we can test:

• First we define the specification and make sure it reflects what we want to build.

• After that, we use the specs to create a frontend prototype that we can interact with. We mock backend calls so we can test the idea.

• Then we connect the frontend to the backend. We start with an in-memory store to make sure the frontend-backend connection works.

• Finally, we add a database so the application keeps its state after a restart.

At the end, we have a working local application that is ready for deployment.

Start with a specification

Before building the frontend, we need to describe the application precisely. If we don’t do it, we will get something that works but we don’t need.

For our application, we need to specify:

• who creates an interview session

• how a candidate joins it

• which components they can place on the canvas

• how both people see changes in real time

This is called “Specification-Driven Development”. We don’t spend a lot of time here because it was the focus of the previous article AI-Native Development: Specifications, Loop and Graph Engineering.

For creating the specification, I always use ChatGPT in dictation mode. Give the assistant as much information as possible at this stage.

You can see the result here.

Frontend First

Now we have the specification, but it’s only text. There are multiple options of what you can do next:

• Focus on the database layer, define the entities, and work all the way up through backend to frontend

• Alternatively, you can start with specifying OpenAPI and define how backend and frontend interact and build both independently from there

• Or you can focus on the frontend first and then build the rest

All these approaches make sense and have their pros and cons.

For simple projects that you’re building alone, I recommend starting with frontend. You can quickly judge if you’re moving in the right direction, and if it solves your problem or not.

Treat it as a way to test your idea and your specification, and do it before you build the rest of the application.

For this step I like using Lovable, which generates a React app from one prompt. Claude Design, Replit, v0, and Bolt are similar tools that you can use instead.

Alternatively, you can start with coding assistants directly and ask them to create a React app (or whatever technology you want).

I use Lovable because it creates really nice designs. Also, I’m not a frontend engineer and I don’t know much about the frontend world. Lovable makes technology choices for me that I know will make sense, while a coding agent can select something randomly.

Now open any tool of your choice and give it this prompt:

Create a system design interview application. 

[Paste the ChatGPT-generated specification here.]

Centralize every backend call in one services layer, and create a mock
implementation of it so the whole app runs without a real backend.

Add tests.

This part is quite important:

Centralize every backend call in one services layer, and create a mock implementation of it so the whole app runs without a real backend.

Without it, the agent can do something arbitrary, but here we explicitly say that we want a mock service. Later, it will become the single point of integration of our frontend with backend. And because we mock it, it will work from the beginning.

Lovable creates a React app in TypeScript. We can interact with it in the Lovable interface.

Next, save it to GitHub. If you use Lovable:

• Select the plus icon in the bottom-left corner

• Connect your GitHub account.

• Lovable creates a private repository. I usually change it to public.

Move the Frontend into the Project

Next, we can clone the repository locally.

For the rest of the project, I want this setup:

/backend     # backend application and its tests
/docs        # supporting documentation
/frontend    # frontend application
AGENTS.md    # instructions for coding agents
openapi.yaml # API agreement

So let’s create these folders and move all the frontend stuff to “frontend”, and the specification we created to “/docs/spec.md”.

After we re-arranged the files, commit the changes.

At this point, you should also be able to run the application locally and test that things work the way you want. If they don’t, use a coding agent to fix it.

To run a project you exported from Lovable:

cd frontend
npm i
npm run dev

AGENTS.md

We already discussed the importance of AGENTS.md in the first article.

Let’s create one for this project too. Place it in the repository root:

for backend, use uv for dependency management. a few useful commands:

uv sync
uv add <PACKAGE-NAME>
uv run python <PYTHON-FILE>

regularly commit code to git 

This is only the starting point and it will change as your project grows.

OpenAPI Specifications

When creating frontend, we asked the AI assistant to put everything in a centralized service layer. Later we will replace it with actual calls to backend.

But now we should define the specification - the agreement between frontend and backend. We will use OpenAPI for that.

This specification gives explicit information about the endpoints, paths, request bodies, response bodies, and authentication rules.

Let’s create it:

Read the frontend's API client in frontend/

Create openapi.yaml at the repository root.

Specify the backend this frontend expects: every endpoint, method, path, request body, response body, and which endpoints need authentication.

You can skip this step. But I wouldn’t recommend it.

It takes a few minutes, but has many benefits. The backend gets a precise target instead of being inferred from frontend code. Not only we save tokens this way, but also get a clear picture of what exactly the backend needs.

You can see the result in openapi.yaml.

The Backend

Now we have the OpenAPI specs and we can use this file to create the backend. We will use Python and FastAPI for that. But you can choose any technology you want.

When I start with frontend, I use a mocked backend and then replace it with a real one. In the same way, for FastAPI backends, I start with a mocked database, and then later change it.

I do that because I want to make sure the frontend-backend connection works, and until it’s smooth, I don’t worry about the persistence.

Let’s ask the coding assistant to implement it:

Build a FastAPI backend in backend/ that implements the openapi.yaml spec.

Use an in-memory store and seed it with data so the frontend has something to show. Add authentication with hashed passwords and bearer tokens for the endpoints that need it.

Split the code into modules - routers, models, store, auth

Write tests

After 5-10 minutes, you will have the backend ready.

Makefile

When it’s ready, you can run the application. Normally, for FastAPI, the command is something like that:

cd backend
uv run uvicorn backend.main:app --reload --port 8091

I’ll use port 8091 but you can replace it with any other port you like.

But for me it’s always hard to remember these commands, so I ask the coding assistant to create a Makefile:

Create a Makefile so I can easily run it.

Then running it is as simple as

make run

When it’s running, open http://localhost:8091/docs in the browser. You will see the OpenAPI specification of the implemented backend.

We can compare it with the actual specs. From this point we no longer need the original openapi.yaml - the one that’s generated by FastAPI is enough.

Connecting Frontend and Backend

The backend is runnable now, so we can connect it to the frontend:

Switch the frontend to use the real backend client. 

It most likely won’t work from the first try. You will hist CORS errors and probably some others.

It’s also time to test the application thoroughly. Open two browser windows:

• Create a session in one and get the join link

• Use the join link in the second window

• Interact with the system in the second window and see the changes being propagated to the first one.

If something doesn’t work, ask the agent to fix it.

Database

At this point we have a working application with frontend and backend. They can connect to each other.

But because we don’t have a real database yet, when the backend is restarted, all the data is lost.

Let’s fix it:

Replace the in-memory store with a database. Use SQLite and SQLAlchemy.
Use an environment variable to configure which DB the server should connect to.
Make it database-agnostic - later we will add support for other databases (e.g. Postgres).

There are a few important parts in this prompt:

• First, I ask it to use SQLite. It’s a very nice and lightweight database for testing the application locally.

• At some point, you’ll want to switch to a different database for production, e.g. Postgres. Using SQLAlchemy makes this process very simple.

• But we also mention that explicitly so the agent doesn’t use any SQLite specific features in the first version.

After 5-10 minutes you will have a fully working end-to-end application that’s running locally. Now you can test it: create multiple sessions and see that changes in one are reflected in another. Stop your backend server, start again, and check that all the changes are still there.

If something doesn’t work, ask the coding assistant to fix it.

Ready for Deployment

Now we have a working application:

• we turned an idea into a specification

• created frontend

• defined the API contract

• implemented the backend from the contract

• connected them

• added SQLite for persistence

But this application is only working locally. Next, we need to deploy it. That’s something we will do in the next article. We will take what we developed here and turn it into a deployed service.

We will:

• containerize the frontend and backend

• add integration tests for the full application flow

• set up CI to run the checks automatically

• run Postgres with Docker Compose

• add database migrations

• deploy the application to a public environment

• set up CD to deploy changes after the checks pass

Step-by-Step Approach

In this workshop, we created a working application using the step-by-step approach. At the end of each step, we had something functioning:

• Clear speciation

• Working frontend prototype with mock backend

• Properly integrated backend with mock database

• Persistence with SQLite

For each step, we started from a new session and drove the agent to produce a working version.

But we did it one prompt at a time. After the first version is working, it’s time to introduce the development process. We talked about it in the first article. This process will help you application continue working properly as it grows bigger.

Next in the Series

In the next modules we continue working with this app:

• Deployment: deploy the application to the cloud and set up CI/CD

• DevOps and Observability: add observability to the application and use AI as the first line of support when the application breaks

You can find the entire course in the course repository. It’s free.

⭐ Star the course repo

This article is based on the the workshop I did for AI Shipping Labs: Build and Deploy a Full-Stack App with AI Coding Assistants. In that workshop I created the snake game, deployed it with AWS and configured CI/CD to make sure tests run on every push. Check it out!

There are two components:

• FAQ curation: building the FAQ dataset from student contributions, Slack discussions, and YouTube transcripts. It lives in DataTalksClub/faq.

• FAQ retrieval: the Slack bot that answers students’ questions using this dataset. It lives in DataTalksClub/faq-assistant and runs on AWS Lambda.

In this article, I want to describe these components and the entire project in more detail:

• the data sources and how they become the FAQ dataset

• the architecture and how data flows through it

• the decisions I made and why I made them

• how I evaluate each part of the system

ZoomcampQABot

Yes, I already wrote about the FAQ assistant in From Google Docs to an Automated FAQ System for DataTalks.Club Courses.

In that article, I described the ZoomcampQABot bot developed by Alex Litvinov. When it occasionally breaks down, I have to pull in Alex and ask him to fix the issue. Sometimes it would be handling an expired key, or sometimes it’s fixing a small bug.

Two months ago his OpenAI account ran out of money, so the bot stopped working. I had to ping Alex again. I always felt bad that he uses his own money to pay for the project, so I offered to take it over and run it in the DataTalks.Club infra. Typically he’d decline it, but this time he agreed.

This is the setup ZoomcampQABot uses:

• OpenAI for generating the answers

• Fly.io for hosting

• HuggingFace for embeddings

• Milvus for vector search, running on Zilliz Cloud in production, with four collections and query engines

• Cohere for reranking

• Upstash Redis as an embeddings cache

• LangSmith for feedback logging

The data comes from multiple sources:

• the FAQ dataset

• Slack history

• Course GitHub repository data

• YouTube transcripts

Each course has a separate ingest entrypoint on its own cron job in GitHub Actions. YouTube videos are pulled in manually.

This setup makes a lot of sense, but I couldn’t port it easily to the DataTalks.Club infra. I wanted to use it in combination with the Au-Tomator Slack Bot that I described in Building and Maintaining a Slack Moderation Bot for an 88k-Member Community.

The Au-Tomator bot runs on Serverless - on AWS Lambda. It’s been running for years now and I never had to pay for it: it was always under the free tier for AWS Lambda usage. And when I have to pay for it, I expect it to be minimal. So I wanted to run the FAQ bot on Serverless too.

So I had to redesign the Slack bot. I’ll describe the result later in the article, but now I’ll cover the main data source: the FAQ dataset. This dataset influenced many architectural decisions in the assistant, so I want to talk about it first.

Part 1: The FAQ Dataset

The FAQ dataset is open for everyone. We host the website via GitHub pages and the data is in the DataTalksClub/faq repository.

Previously it lived in a bunch of Google Docs. It was convenient, but this approach had a few problems:

• It was frequently vandalized, so I had to manually roll the documents back.

• I wanted to automate the curation with AI, but using AI in Google Docs is not trivial.

Eventually I moved the data to a bunch of markdown files. I described the migration process in From Google Docs to an Automated FAQ System for DataTalks.Club Courses.

Now with AI assistants I spend a lot less time maintaining this dataset, and I can keep it clean and up-to-date.

There are multiple sources of questions for the FAQ dataset:

• The questions that people contributed via GitHub

• Discussions in Slack

• Q&A videos on YouTube

FAQ Automation: User-Contributed FAQ Records

Anyone can contribute to the FAQ dataset:

• You submit an issue, specifying the question, the course and your answer.

• A GitHub Actions workflow indexes the entire dataset with minsearch.

• It then searches twice - on the question alone, and on the question and answer together - and combines the two results with reciprocal rank fusion.

• It sends the results to OpenAI, which returns a structured decision: NEW, UPDATE, DUPLICATE or WRONG_COURSE.

• For NEW or UPDATE, it opens a pull request.

• For DUPLICATE or WRONG_COURSE, it closes the issue.

FAQ Automation Evaluations

When the first version of the script was created, we didn’t have any evaluation set. I mostly relied on my gut feeling. But as more people started using it, the number of incorrect decisions increased. I needed to improve it. Naturally, I didn’t want to fix it blindly and just hope for the best: I needed a proper evaluation framework.

For creating it, I used those incorrect decisions. I analyzed the issues where I needed to manually adjust the result, and focused on the cases that were sufficiently different from each other.

Not all decisions made by the script are equal. If the agent makes a mistake in a NEW or UPDATE decision, it’s easy to correct. I use AI assistants to help me with that.

But if the decision is DUPLICATE or WRONG_COURSE, the PR will never be created, and the issue will be closed. In this case, I will not even have a chance to review them. Thus, a mistake in this case is way more expensive. Which means, I had to give these cases more importance in the eval set to make sure that these cases don’t happen often.

There’s also a problem with collecting the evals data from historical decisions. The FAQ data is constantly changing, so we have to use a “leave-one-out” setup.

When the script decides that something is NEW and we later use it in the evals, it’s no longer NEW: the item was already merged into the dataset. So if we run it with the same issue again, it will say DUPLICATE instead of NEW.

In order to properly test the NEW cases, we therefore need to remove the record from our dataset. So if we have 200 records, and the record D is the one we added, we remove the D and test this case against the remaining 199 records.

When I analyzed all my past corrections, the most common error turned out to be incorrect section placement. For example, instead of placing a record about projects in the “Project” section, it would place it in “General”. My eval set also tests for that.

Right now I have 61 cases:

• 38 expecting NEW

• 10 expecting DUPLICATE

• 7 expecting WRONG_COURSE

• 4 not expecting WRONG_COURSE

• 2 expecting UPDATE

In addition to testing the whole flow, I have a retrieval-only evaluation set. I use this part to test that duplicate detection works correctly. This part is less interesting, so I’ll skip it. You can read more about it in the project’s README.

Bulk Reviewing

I usually let the PRs accumulate and process them every two weeks.

For that, I use AI assistants. I have a custom skill that lets me go through each PR and merge it as is, or fix it if it’s not correct.

If I come across an interesting error, I ask the assistant to add it to our evals set. I don’t try to fix it immediately. I wait till we have a few more cases, then I ask the assistant to see if there are simple ways to fix them.

If some cases can’t be fixed easily, I don’t sweat over it. I just let it sit in the evals with a FAIL status. I don’t want the system prompt to grow too large and handle all the corner cases. But I do want to know that these corner cases exist.

Other Sources: Slack and YouTube

Our Slack is very active. Course participants are asking questions and helping each other. In many cases, these discussions are worth saving in the FAQ dataset.

For that I regularly go through all the Slack threads. I ask my AI assistant to go through each thread, and if something is useful, it’s saved as a new record.

Also, for each course I run a few live YouTube sessions, for example:

• pre-course Q&A session

• course launch streams

• occasional office hours

I get the transcript and use AI assistant to extract potential Q&A candidates. If there’s something new, it also goes to the dataset.

Part 1 Overview

The main focus of the first part is the dataset curation.

I review all the PRs that our FAQ automation creates in batches, and use AI to turn Slack discussions and YouTube videos into focused FAQ records.

Now let’s go back to the retrieval side. This FAQ dataset is used as the main data source for the Slack bot.

Part 2: Slack Bot

When you mention @Au-Tomator (my bot) or @ZoomcampQABot from Alex, both would perform RAG:

• Use search to fetch the candidate FAQ records

• Pass them to OpenAI

• Return the answer from the LLM and post it in the thread as a reply

ZoomcampQABot has a lot of moving parts though. Most of them are not straightforward to deploy to the serverless environment. So I decided to simplify it.

The first thing I dropped was indexing Slack threads and YouTube videos. I already extract this information and put it directly to the FAQ dataset, so this part is not needed anymore.

Second, I removed the vector database. I know that it improves retrieval, but at the cost of having to maintain more infra. I want to keep the setup very lean. If the bot is sometimes not right, it’s not a huge problem: I and other community members can simply correct it in Slack.

Zero-Dependency Serverless Deployment

My to-go search library for text search is minsearch. I wrote about it in Minsearch: The Small Search Library Behind My RAG Workshops and Courses.

However, deploying minsearch to AWS Lambda isn’t trivial. When I created this library, my focus was on teaching the concepts, not on deploying it to serverless. Internally, it users Scikit-Learn and Pandas, and these libraries in turn pull in numpy and scipy. Normally, everyone working with data already has these libraries in their standard setup. But for Lambda, they are too heavy and take well over the 50 MB limit.

So I decided to replace minsearch with a zero-dependency search engine written entirely in pure-Python. I called it zerosearch. It has only one optional dependency - stemlite, which is a stemmer I use across all my small search libraries (zerosearch, minsearch and SQLiteSearch).

Another challenge was Pydantic. We typically use for structured output, and it works fine in usual environments, but it’s not trivial to get it right for Lambda. It has a compiled Rust core, and you need to make sure the binaries that you send package into your Lambda deployment are compatible. Plus, loading the binaries core and building the model schemas adds to the cold start time.

So I removed it along with requests and did all the calls to OpenAI using only the standard Python library.

Keeping the Index Fresh

Then I started thinking about what to do with updating the index. If a record in the FAQ changes, the search needs to reflect it. Alex solved that by pulling in data using a daily cron job. But I wanted to do this as soon as the data in FAQ changes.

Almost all my projects have a CI/CD workflow: when I push to main, the code change is propagated to a live environment. I usually use it for code changes, but for this project I did the same for data changes too.

When a push happens, I re-build the whole index, and push it together with the source code in Lambda. It’s quite small: it’s 8 MB total.

Once it’s deployed, the lambda loads the local updated index, and can serve the fresh data.

FAQ Assistant Retrieval Evaluation

I replaced vector search with text search, and used zerosearch to make it fast and work well in Serverless. But it would still lose to vector search in terms of retrieval quality.

How can I make sure I squeeze the absolute best from text search? For that I needed to have a proper evaluation set.

I built the ground truth dataset from real Slack threads:

• Get a Slack dump with all the course channels

• Take all the Slack threads (9,900 of them)

• Filter them down to keep actual questions

At the end, I had a sample of 130 records:

• 60 for Data Engineering Zoomcamp

• 40 Stock Markets Analytics

• 30 AI Dev Tools

For each of the questions, I found and marked the correct documents in the index. Then I evaluated the hit-rate and MRR at k=1,3,5.

Once I had this dataset, I could start experimenting with search optimization.

I tried multiple options:

• Taking raw question as is

• Keyword expansion

• Different variants of query rewriting

The best option is to take a Slack message and turn it into a bunch of keywords while preserving exact error messages, tool names, commands and filenames.

For example, it turns this Slack message:

Does anyone know why minsearch fails when I run uv run python index.py? I get KeyError: 'course' after renaming documents.json to faq.json.

into these keywords:

minsearch FAQ indexing course field "KeyError: 'course'" "uv run python index.py" index.py documents.json faq.json

For query rewriting I use gpt-4o-mini because it’s the fastest and cheapest model, but for the actual generation I rely on gpt-5.4-mini.

FAQ Assistant Generation Evaluation

In the previous part I explained how I evaluate search. I also evaluate generation.

For the evaluation dataset, I collect feedback from Slack. When a bot answers a question, and somebody corrects it or add something extra, it means that the bot couldn’t answer the question properly. So we can include this question in the evals.

So far it’s not large and has a few cases like:

• the answer from the bot is incomplete

• the answer is incorrect

• the search doesn’t find anything

Most of the time the way to fix these problems is not tuning the prompt, but fixing the FAQ dataset itself. I check why a record wasn’t retrieved, or why the answer didn’t contain the correct information, and then see how I should update the FAQ records to make it work next time.

Then I’d fix the record, re-run evaluation, and re-deploy the bot.

Deploying with Au-Tomator

I use Au-Tomator to help me with Slack management. So I wanted to use the existing setup, not create a new Slack bot.

I already have two Lambdas for Au-Tomator:

• The router: routes and filters Slack events. It also makes sure we acknowledge the request from Slack quickly within 3 seconds (a requirement from Slack) and forward the request to the next lambda.

• The automator: the bot itself, it handles all the reactions.

And I added the third one:

• The FAQ assistant: rewrites the question, searches the index, and generates the answer

Now when you mention the bot, it first goes to Au-Tomator, and then Au-Tomator sends it to the FAQ assistant. The assistant gets in the question, and send back the answers. Posting the response to Slack is handled by Au-Tomator.

As a bonus, now with this setup, I can trigger the FAQ assistant with :faq: reaction. Previously this reaction would only post a message to the Slack thread saying “Go check the FAQ”.

Now it actually checks the FAQ and generates the answer.

Other Retrieval Sources

I don’t only search the FAQ dataset. I also index the DataTalks.Club documentation and the Markdown files from each course repository.

I combine all three sources into one search index and package it with the FAQ assistant Lambda. In Slack, either a mention or a :faq: reaction triggers Au-Tomator. It sends the question to the assistant and posts the answer back to the thread.

The Full Picture

In this article, we discussed the two parts of the FAQ system:

• The FAQ dataset, where we primarily talked about the dataset curation

• The FAQ assistant, which is a Slack bot deployed via AWS Lambda

The FAQ assistant reads data from the FAQ dataset and helps students find answers to their questions faster.

This is how the entire system looks like:

• I turn GitHub issues, useful Slack discussions, and live-session transcripts into reviewed FAQ records.

• I combine the FAQ, docs website, and course-repository Markdown into a zerosearch index. The index is rebuilt every time we push to the FAQ repo.

• A mention or :faq: reaction triggers Au-Tomator that sends the question to the FAQ assistant and publishes the answer in Slack.

• I use wrong and missing answers as evaluation cases and FAQ updates, so the next index contains the correction.

Curation over Complexity

This year, I re-recorded the entire LLM Zoomcamp course from scratch. There, we use the FAQ dataset as the main running example.

When I introduce the FAQ dataset, and we do the first RAG example, I mention that the example looks very simple. But the reason it looks so simple is because the complex work has already happened: the dataset was already cleaned and prepared. In reality, in your AI projects, most of the time will go into data cleaning and data curation.

In this article, I tried to explain how exactly I did it.

We can always have a better model, use a vector database, or a faster machine, but none of that will help if your data is missing, outdated or unclear.

When the assistant gives a bad answer, it’s not because of the tech stack, but because of the data. So the most useful fix to make it better is to improve the dataset.

In the FAQ case, I collect the feedback from real users, use it to improve the underlying dataset and thus improve the system itself:

1. Students contribute issues to the FAQ dataset.

2. Useful discussions in Slack become curated FAQ records.

3. The updated dataset is indexed and deployed with the Lambda.

4. Au-Tomator uses it to answer future questions.

5. Incorrect or incomplete answers become evaluation cases and help improve the system.

Because of this loop, I’m comfortable with a simpler architecture that I described in this article.

Since January 2026 I scrape AI Engineering jobs monthly for AI Engineering Field Guide. So far we have 4,894 descriptions, and among them, the number of FDE-related postings increased from 28 in January to 108 in July.

According to this dataset, the AI Engineering job market doubled in these 6 months. But the number of FDE listings grew 4 times! That is, it’s growing at twice the overall AI Engineering market rate.

In this article, I analyze 113 FDE postings to understand what this work involves. Since our dataset focuses on AI Engineering roles, it’s limited to FDEs working in AI.

We discuss

• how companies define the role

• which skills they expect

• how FDE differs from applied AI engineering, solutions engineering, and consulting

At the end of the article, we also include a quick self-assessment checklist to help you determine whether the FDE role is for you.

Forward-Deployed Engineer

Palantir created the forward-deployed engineering model in 2006. They had a general platform, but making it useful required adapting it to each customer’s data, workflows, and infrastructure.

To solve it, they placed their engineers in their customer’s teams. FDEs identified their problem, configured Palantir’s platforms, built integrations and custom components. They also supported product development: when the same problem appeared across deployments, Palantir would integrate this as a feature to the main platform.

Palantir’s FDE model applies directly to AI in enterprises today. General-purpose models must be adapted to each customer’s data, systems, and workflows before they can deliver value in production.

There’s a “deployment gap” - the gap between a prototype and a working customer system. It exists because many enterprises don’t have the engineering capacity or product knowledge to close it themselves. FDEs fill that gap.

A forward-deployed engineer is a customer-facing software engineer who turns an ambiguous business problem into a working production system. They work alongside the customer to define the problem, design the solution, adapt and integrate the company’s product, build any missing software, deploy and debug the system, and feed recurring customer needs back into the core product.

FDE’s responsibilities: our analysis

We collected a dataset of 113 FDE jobs descriptions from January to July, and then analyzed the expected responsibilities.

They include:

• Direct client engagement (90% of postings)

• Building and deploying production system (87% of postings)

• System, API, and data integration (62% of postings)

• Project scoping, discovery, and requirements gathering (51% of postings)

• System testing, evaluation, and monitoring (41% of postings)

• Channeling customer feedback back into the core product roadmap (39% of postings)

• Developing prototypes, demos, and proofs of concept (25% of postings)

• Travel or client on-site presence (10% of postings)

If we compare it with the rest of the dataset, only 21% of the other AI roles are expected to interact with the clients.

Here’s how we can describe the FDE’s main responsibilities

1) Understand the customer’s problem

FDEs usually begin with a desired business outcome. They work with users, domain experts, and engineering teams to understand the existing workflow, examine the available data, and identify technical constraints. They then translate what they learn into deployment requirements and success criteria.

2) Build the production AI application

Once the problem is clear, the FDE writes the software to solve it, handling conventional production engineering tasks like authentication, permissions, testing, error handling, deployment, logging, and observability.

3) Connect the application to the customer’s systems

FDEs also work on integration, and this need is present in 62% of job postings. They build connectors, data pipelines, transform data, configure access controls, integrate APIs, and deploy applications in the customer’s cloud.

4) Deploy, evaluate, and debug the system

Production deployment expands the FDE’s role beyond implementation. The engineer must define evaluation criteria for realistic conditions and use logs, traces, metrics, and user feedback to identify failures post-release. FDE continue to be involved in the project after deployment.

5) Bring field findings back to the product

Some deployment issues are unique to individual customers, while others highlight common needs. FDEs identify these recurring problems and relay customer feedback to the product team, with 39% of postings referencing this contribution. This process connects customer delivery and product development, allowing the FDE to adapt the product for one customer and identify reusable capabilities for others.

Skills companies ask for

It’s natural that with these responsibilities FDEs are expected to have a broad mix of skills.

Software engineering

The technology stack varies from job to job, but all of the FDEs are expected to deploy production software.

Python is the most frequently mentioned programming language, appearing in 89% of job postings. Other commonly referenced languages and frameworks include:

• TypeScript: 29%

• SQL: 23%

• React: 14%

Applied AI

FDEs are expected to build applications with existing models, not train them. The most frequently mentioned gen AI technologies are:

• Prompt engineering: 56%

• RAG: 50%

• LangChain: 32%

• AI agents: 30%

• Vector databases: 23%

• Agentic workflows: 19%

Cloud and deployment

In the job descriptions, the three major cloud providers appear at similar rates:

• AWS: 40%

• GCP: 36%

• Azure: 32%

Companies also seek candidates with experience in the following areas:

• Docker: 35%

• Kubernetes: 31%

• CI/CD: 27%

Customer-facing work

The technical requirements represent only part of the role. With 92% of FDE postings identified as customer-facing, engineers must also navigate technical discovery and tackle evolving requirements.

FDEs need to be able to:

• Understand existing business workflows

• Translate user needs into technical requirements

• Communicate constraints and tradeoffs to stakeholders

• Collaborate with domain specialists and customer engineering teams

• Adapt implementations based on new information

• Differentiate customer-specific integration from reusable product gaps

How the role differs from related titles

FDEs have a broad set of responsibilities, so it’s natural that the role overlaps with many others.

The main difference is the type of code FDEs write, how long they work with one customer, and the involvement in production deployment.

An FDE builds and deploys systems for specific customers, writes production code, and stays involved until the deployment works.

• Consultant: Works within a defined engagement and usually leaves after delivering recommendations or an implementation. FDE: Stays through production deployment, owns more of the outcome, and feeds customer needs back into the product.

• Solutions or sales engineer: Focuses on demos, technical validation, and supporting multiple deals before handing off implementation. FDE: Writes production code, owns deployment, and works more deeply with fewer customers.

• AI engineer: Builds production AI systems, often with more focus on the underlying product or platform. FDE: Places greater emphasis on customer integration, deployment, and close collaboration.

Who is a good fit

FDEs come from different technical backgrounds:

• AI engineer: Strong in LLM applications, RAG, agents, evaluation, and production AI. Gap: Customer discovery, enterprise integration, and ownership of customer-specific deployments.

• Software or product engineer: Strong in APIs, backend or full-stack development, testing, and shipping software. Gap: Applied AI, evaluation, data pipelines, and customer discovery.

• Data engineer: Strong in pipelines, integrations, distributed systems, cloud platforms, and production operations. Gap: Full-stack development, LLM systems, evaluation, and product discovery.

• ML engineer: Strong in model deployment, monitoring, evaluation, and production ML. Gap: Broader product engineering, customer-specific integrations, and direct customer work.

• Platform, DevOps, or SRE engineer: Strong in infrastructure, CI/CD, security, reliability, observability, and debugging. Gap: Application development, applied AI, customer discovery, and workflow analysis.

• Solutions engineer or technical consultant: Strong in customer discovery, solution design, technical communication, and varied customer environments. Gap: Production software development, long-term ownership, and applied AI evaluation.

• Data scientist: Strong in experimentation, model evaluation, analysis, domain knowledge, and stakeholder communication. Gap: Maintainable application development, systems integration, infrastructure, and deployment.

For our dataset and set of responsibilities, AI engineering is the closest technical background because the tools and systems often overlap. (But our data is probably biased because in our scrapes we focus on AI Engineering roles.)

The strongest fit is someone who can combine three capabilities:

• Build and operate production software across application code, data, integrations, and infrastructure.

• Work directly with customers to turn unclear problems into technical requirements and measurable outcomes.

• Remain responsible through deployment, evaluation, and production debugging.

Is this role for you?

Forward-deployed engineering may suit you if you want to be a hands-on engineer while working closely with customers and owning projects through production.

The role may be less suitable if you prefer narrow technical ownership, stable specifications, limited customer interaction, or handing projects off before production.

Before applying, ask yourself:

• Do I want regular customer interaction to be part of my engineering work?

• Can I make progress when requirements are incomplete or changing?

• Am I comfortable working across several parts of the stack?

• Do I want to own deployment and production results, not only implementation?

• Can I explain technical constraints to non-specialists?

• Am I willing to work within the customer’s existing systems and infrastructure?

• Do I enjoy turning recurring customer problems into reusable product improvements?

Conclusion

FDE is still an inconsistent job title, but we can see a clear focus on customers and production.

Compared to traditional AI engineering, FDEs place a stronger, more consistent emphasis on customer discovery, systems integration, and deployment responsibilities. For engineers who want that combination, forward-deployed engineering provides a path to remain technical while owning a broader part of the production outcome.

Edited by Valeriia Kuka.

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering (this article)

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App

• Part 5: TBA

I start with AI-native developer workflows. We will see how to turn a raw idea into a specification, give coding agents the right context, and verify what they build.

Coding agents now write code faster than I can read it.

When we give an agent a task, it can quickly implement it. But if a task is vague, the agent fills the gaps with its own assumptions. A weak agent that misunderstands us writes fifty lines of broken code. A strong agent that misunderstands us creates eight files, wires them together, and adds tests that pass. The code works, but it isn’t what we needed.

We no longer spend most of our time typing. We spend it saying precisely what we want and checking what came back.

In this article, I show how to make the request specific so the agents don’t need to guess. Then we decompose the request into tasks and assign them to a product manager, a software engineer, and a QA engineer. Finally, we implement all the tasks in a backlog through a loop.

We cover topics like:

• Spec-driven development

• Context engineering

• Loop engineering

• Graph engineering

We’ll use a deliberately vague project idea: a tool for weekly feedback for projects. It doesn’t say who gives the feedback, who receives it, or what “projects” means.

You can see the final result in the retroloop repository.

Specs before code

We need to understand what we want to build before the agent produces the first line of code. We have to think it through in detail and give explicit instructions. If we do that, the agent will produce something close to what we want.

We call this “spec-driven development”. We start with the specification, make sure it aligns with our vision, and only then write the code from it.

In our example, “a tool for weekly feedback for projects”, many things aren’t clear.

• Who are the users for this tool?

• What problem will they solve?

• How are they going to use it?

If we don’t specify these things and give the idea directly to a coding agent, it’ll fill the gaps.

When I asked Claude Code to implement this project, the only prompt I gave was “a tool for weekly feedback for projects”. It came up with weekly-feedback, a command-line tool for tracking weekly project status. It records wins, issues, blockers, and next steps. It also created documentation and covered the app with 62 tests, all of which passed.

It all works perfectly, but that’s far from what I needed. I needed a web tool for a team retrospective that captures feedback from teams in the form of “Start/Stop/Continue”. It was my fault for not telling Claude that.

Start in a chat assistant

Instead of giving a prompt directly to the coding assistant, I start in a chat application and talk the idea through. I use ChatGPT in dictation mode for this.

I begin with the same vague idea:

I want to build a tool for weekly feedback for projects.

Help me set the scope for this project precisely. I want to brainstorm with you
and understand how the tool should work. Give me options.

Ask me one question at a time and keep your output short.

This way, we can use AI as our brainstorming partner and find out precisely what we want:

• Who contributes feedback? All team members.

• What format do they use? Start/Stop/Continue.

• Is the feedback anonymous? Names appear by default, but contributors can choose to remain anonymous.

• What can people see before the reveal? Only their own cards.

• How does the facilitator reveal the feedback? All cards appear at the same time.

• What happens next? The team clusters the cards, then each person casts three votes for the topics to discuss and can give multiple votes to one topic.

• What does the team record? Decisions and action items from the discussion.

• Can the facilitator add a recording? They can upload audio, video, or a transcript after the meeting, but built-in recording isn’t part of the first version.

When we finish, I ask for a file with all the specifications:

Save everything to a markdown file that I can download.

Download the file and save it as plan.md.

Bootstrapping a project

Create a project from this specification:

mkdir project-name
cd project-name

git init

Copy the plan.md file:

mkdir -p _docs
mv ~/Downloads/plan.md _docs/plan.md
git add _docs/plan.md
git commit -m “Add project plan”

You can find the plan.md file from this project in the Retroloop repository.

I try to commit as often as possible, after every meaningful decision. With those commits, we can review what the agent changed. If something isn’t working well, we can easily return to the last good state.

Choose the stack and architecture

During the brainstorming session, we didn’t choose the tech stack.

Ask the coding agent to come up with several options:

Read _docs/plan.md. Propose multiple options for the tech stack and
explain each option.

Don't write code yet.

It proposes multiple options and explains the tradeoffs of each one.

I choose Django because I know it well enough to review. In your case, you can select any technology you’re comfortable with. It’s also okay not to have a preference and to let the agent select what it thinks will work best.

Turn the decisions into a backlog

Now that we’ve settled on the tech stack, we can ask the agent to decompose the specifications into a backlog with tasks:

Create a backlog with tasks in _docs/tasks.md.

Each task should be small enough to finish in one session, and
independent enough that I could hand it to someone who has not read
the others.

Use this template for each task:

## <number>. <title>
Goal: <one line>
Description: <two or three sentences on what the task involves>

The first task should be setting up an empty project with a passing test.

Don't write code yet.

It created these tasks.md.

Review the tasks and ask the agent to merge tasks that are too small or split tasks that don’t fit into one session. We want to create an MVP - the first version of the app. If something is out of scope for your vision of the MVP, remove it.

When we’re happy with the tasks, move them to a task tracker. I use GitHub issues for that.

Ask the agent to do it:

Create a public GitHub repo for this project.
Move each task from _docs/tasks.md into a GitHub issue.

For that to work, we need the gh CLI tool authenticated and the repo connected to the GitHub remote.

From this point on, GitHub issues are the canonical tasks and the only active backlog. We no longer need _docs/tasks.md.

Context engineering

The repository has a backlog now. When we start a new session, however, the agent doesn’t know which task we mean. It must figure that out every time.

These details go in AGENTS.md, which coding agents like Codex or OpenCode read when they start a new session.

Claude Code reads CLAUDE.md, while I use multiple coding assistants and want my workflow to be tool-agnostic.

That’s why I also create CLAUDE.md with a single line:

@AGENTS.md

It tells Claude to read AGENTS.md.

This is called context engineering. With prompt engineering, we control one message in one session. With context engineering, we control what agents know when they start a new session and what information they can find while they work. We include useful facts and working rules they would otherwise have to rediscover.

AGENTS.md

To make this context available in every new session, create AGENTS.md:

Commands

- `uv sync` - install dependencies
- `uv run pytest` - the whole suite
- `uv run pytest tests/test_home.py` - one test file

Rules

- Dependencies are added in `pyproject.toml`. Do not add one without
  asking

The other documents

In addition to AGENTS.md, I usually have a few other Markdown documents in my projects.

The main one is process.md, which I use to describe how work is organized. It could live inside AGENTS.md, but I keep it separate.

Create _docs/process.md:

- Tasks are GitHub issues, one at a time
- Read the acceptance criteria before starting and before closing
- Commit regularly

As I continue working on a project, I may create other documents, such as:

• testing-guidelines.md for testing

• design-system.md so the UI doesn’t drift every session

• api.md, which describes what the API should look like

I keep them together in _docs/ and link them from AGENTS.md:

Documents

- `_docs/process.md` - how work is organized
- Before writing tests, read `_docs/testing-guidelines.md`
- For anything touching the UI, read `_docs/design-system.md`

The agent reads AGENTS.md at the start of every session, so it knows where to find the process, testing, and design rules if it needs them.

This way, it will load the design system only for a UI task and the testing guidelines only for a testing task. By loading each document only when it’s relevant, we keep AGENTS.md short while we continue adding written context to the project.

These documents are living documents, and I update them often. If I need to correct an agent during a coding session, I can ask it to modify the documents. Next time, it knows what I need, so I don’t have to correct it again.

You can use a prompt like this:

Based on the corrections I made, find the relevant documents and update them.
Commit the current work before changing the documents.

Bootstrap the first task

With AGENTS.md and process.md in place, we can start a new session and ask the agent to implement the first task:

Implement task 1.

For this project, the agent creates the Django app, dependencies, and a passing test.

Grooming: the product manager agent

We have a backlog of tasks, but they’re not precise enough.

We discussed this problem already: if the task isn’t specific, the agent will fill in the gaps during implementation. We risk spending time and tokens on something we don’t need.

Instead, we should ask the AI assistant to fill these gaps before writing any code. Then we review the specification, correct it, and give it to the coding agent to implement.

This process is called “grooming”: we groom a task to make it more specific. Then an engineer can implement it without asking a single question.

In real teams, product managers usually do this work. Here we’ll create a team of agents, and the first role we’ll define will be a PM.

Create a document:

_docs/team/
  pm.md

Inside, write the description for the product manager agent:

You’re a Product Manager

You groom a task before anyone implements it.

- Read the issue as written
- Rewrite it using the template in `_docs/task-template.md`
- Make the acceptance criteria checkable - someone should be able to
  point at the screen and say yes or no
- Think about the edge cases the person who filed it did not consider
- Do not write any code

Definition of done:

- The issue has all four sections filled in
- Every acceptance criterion can be checked by looking at the result
- Everything moved out of scope links to a follow-up issue
- An engineer who has never spoken to you could implement it from the
  issue and the documents it links

If something does not belong in this task, do not silently drop it.
File a follow-up issue and list it under out of scope with a link to
that issue, so it is clear what was moved and where it went.

A groomed task has four sections:

1. Goal - one or two sentences on what should be true afterwards.

2. Acceptance criteria - checkable statements.

3. Out of scope - what this change must not do.

4. Constraints - files it should stay inside, libraries it should or shouldn’t use, prior decisions it has to follow.

We save the issue template as _docs/task-template.md:

## Goal

One or two sentences on what should be true when this is done.

## Acceptance criteria

- [ ] A statement you can check by looking at the result
- [ ] One line per case, including the awkward ones

## Out of scope

- Something that does not belong in this task, moved to #TASK-NUMBER

## Constraints

- Files this should stay inside
- Libraries to use
- Guidelines to follow

We’ll need to groom every task, so we’ll add it to process.md:

Roles

- PM - grooms a task before anyone implements it, follows _docs/team/pm.md

We can now start a new session and ask the agent to groom an issue:

Groom issue #4

After it finishes, review the result.

We can catch a misunderstanding most cheaply while grooming: the issue is a paragraph, and correcting it costs one sentence. If we catch the same misunderstanding after implementation, we need a rewrite.

Loop engineering

After grooming one issue, we can ask the agent to groom the rest:

Groom all GitHub issues. Process one issue at a time.

This will mostly work, but the agent may eventually stop. It might say, “I’ve groomed issues 1, 2, and 3. Do you want me to proceed?”

The answer is almost always “yes”, but the agent has stopped and is waiting for us to say that explicitly. In many cases, I want the agent to continue automatically.

To do it, we can give the agent a goal:

/goal groom all issues

The /goal command will prompt the agent to continue, so we won’t need to do it manually. Instead, we delegate that responsibility to the harness: the system around an agent, such as Claude Code or Codex. When the agent stops, the harness checks whether the condition has been met. If it hasn’t, the harness resumes the work.

This approach is called “loop engineering”. It’s similar to a while loop: we repeat the work until a condition is met.

With loop engineering, the system runs a coding agent repeatedly instead of having us drive it manually, prompt by prompt.

There are multiple “engineering” levels when we work with coding agents:

• Prompt engineering - what we say when we interact with the agent

• Context engineering - what the agent knows before it starts and what it can get during the session

• Loop engineering - when it stops working

• Graph engineering - who does what when there’s more than one agent (we’ll discuss it later)

In June 2026, Addy Osmani published the Loop Engineering essay that gave it a name, and Peter Steinberger compressed the idea into one tweet:

A loop needs a stop condition: a checkable statement that tells the harness when to stop.

For /goal groom all issues, the stop condition is “all issues are groomed”. After each agent run, the harness checks whether that condition is true and resumes the agent if it isn’t.

The stop condition must be something the model can evaluate. “All issues are groomed”, “all tests pass”, and “no file is over 200 lines” are checkable, but “make the code better” isn’t. If the stop condition isn’t checkable, the agent can stop too early or run forever.

Claude Code and Codex provide the /goal loop by default. If your harness doesn’t provide it, you can implement it yourself using stop hooks.

Implementation: the software engineer agent

After grooming the issue, we can give it to a software engineer, the agent who will write the code.

Define the second role:

_docs/team/
  software-engineer.md

Put this definition inside:

You’re a Software Engineer

You implement one groomed task at a time.

- Read the issue and implement what it describes
- Implement against the acceptance criteria, do not change them
- Stay inside the files and constraints the issue names
- Write tests for what you built
- Do not close the issue
- Commit regularly

Definition of done:

- Every acceptance criterion in the issue is implemented
- Tests are written for the new behaviour, and the whole suite passes
- The work is committed
- The issue is still open, with a comment saying what you did

If an acceptance criterion is wrong, impossible, or contradicts
another one, create a comment on the issue about it.

Add one more line to process.md:

Roles

- PM - grooms a task before anyone implements it, follows _docs/team/pm.md
- Engineer - implements one groomed task, follows _docs/team/software-engineer.md

Then ask the agent to implement a task in a fresh session:

Implement issue #2

The engineer stops when the code is written and its own tests pass. It’s still too early to say that the task is properly implemented, so we need to test it.

Testing: the QA engineer agent

When the same agent writes and judges the code, it’s grading its own homework.

If we ask, “Is this correct?” we’ll get a definite “yes”, but the agent might have missed many edge cases.

In the real world, we ask other people to validate our work. We have code reviews, and many teams have designated QA engineers whose focus is making sure the code is reliable.

That’s why we’ll also add a QA engineer to our team.

Add the third role:

_docs/team/
  qa-engineer.md

The description:

You’re a QA Engineer

You check finished work against the issue that specified it.

- Read the acceptance criteria from the issue
- Check each one against what the code actually does
- Run the tests, and say which ones you ran
- Look for the cases the criteria describe but the tests do not cover
- Do not fix anything you find. Report it by creating a comment

Your output is a verdict: PASS or FAIL. It is FAIL if a single
acceptance criterion fails. Post it as a comment on the issue:

## QA: FAIL

- [x] A visitor can create an account with a username and password - PASS
- [ ] A duplicate username shows a visible error - FAIL
      Submitted an existing username and received an unhandled error

Tests: `uv run pytest`, 18 passed, 0 failed

Definition of done:

- The comment starts with PASS or FAIL
- Every acceptance criterion has a verdict against it
- Every FAIL says what you did and what happened
- The test command and its result are included
- Nothing in the code was changed

Ignore what the implementation says it does. Only the acceptance
criteria and the running code count.

Note: you may need to adjust this role for your project if you don’t use uv and pytest.

And the last line in process.md:

Roles

- PM - grooms a task before anyone implements it, follows _docs/team/pm.md
- Engineer - implements one groomed task, follows _docs/team/software-engineer.md
- QA - checks the result against the acceptance criteria, follows _docs/team/qa-engineer.md

Then, in a new session, ask:

Test issue #2

If we get a PASS, that’s great. If we get a FAIL, that’s also useful: we caught a regression. So we start a new engineer session, use the QA comment as input, and ask the engineer to fix it.

We iterate until QA says PASS.

Graph engineering

We have three roles:

• Product manager grooms an issue

• Software engineer implements it

• QA engineer tests it, outputs PASS or FAIL

If QA says FAIL, the engineer will need to reimplement it. Otherwise, the task is done.

We can visualize this process as a graph:

If we have an orchestrator that launches these agents automatically instead of us doing it manually, we get “graph engineering”. We define a graph with specialized agents as nodes and describe how the work moves from one to another.

The earliest explicit use of the term I found in the article from Josh C. Simmons (July 4, 2026): The Graph Engineering phase.

Peter Steinberger sparked a wider discussion in a July 18 post by asking whether the conversation had shifted from loops to graphs. A few hours later, Hamel Husain used “graph engineering” in an article headline.

Hamel put just a GIF saying “stop it” in the article, but people liked the term and started using it for multi-agent orchestration.

I wouldn’t call it a new idea, though, and loop engineering is definitely not dead. Even with graphs, we still need a loop to drive the work and orchestrate it across multiple agents with different roles.

The orchestrator

To implement it, we need an orchestrator.

Describe it in process.md:

Orchestrator

The main session is the orchestrator. It launches the PM, the engineer
and QA as subagents. It does not groom, implement or test itself.

Lifecycle

1. Pick the next open issue from the backlog
2. PM grooms it
3. Engineer implements it
4. QA verifies it
5. On FAIL, back to step 3 with the QA comment as input
6. On PASS, close the issue
7. Repeat until the backlog is empty

Rules

- Do not skip step 2
- The engineer does not close the issue
- QA does not fix the code, only outputs PASS or FAIL
- The orchestrator closes the issue only after QA outputs PASS

We can now start a fresh session and launch the loop:

/goal work through the backlog

The agent reads AGENTS.md, finds process.md, follows the lifecycle, and dispatches the agents according to the documents in _docs/team/.

Because we combine it with a goal, the orchestrator runs until the condition holds.

You can see the result in retroloop.

Note: this approach takes significantly more time and tokens than a direct loop with only a software engineer. In many cases, you don’t need this complexity. Often, a simple prompt or loop is enough.

I’ve followed this approach on many projects, including the AI Shipping Labs website.

I describe it in I Built an AI Agent Team for Software Development. Here, I distill that article into a tutorial.

Next in the series

We have now finished the first module. It’s theoretical, but the next modules are very practical.

We will implement an application from the specification to making it production-ready in 3 steps:

• Build: Build and Ship a Full-Stack App with AI Coding Assistants

• Deploy: Deploy a Full-Stack App with AI Coding Assistants

• Operate: DevOps and Observability for an AI-Built App

Think of it as the project’s landing page. It should help someone quickly understand what you built, why it matters, and whether they should explore it further. This applies to pet projects, course submissions, and portfolio projects you create to support a job application.

At my previous job, I was heavily involved in hiring. I conducted the first technical screening after the recruiter call, which meant reviewing candidates’ profiles and projects. I would open a repository, scan the README, and decide within about ten seconds whether to continue. Most of the time, I wouldn’t continue. If the README didn’t tell me what the project was about or how to run it, I wouldn’t spend time trying to figure that out myself.

I now see the same problem in the courses I run. Each cohort produces hundreds of student projects, and all of my courses use peer review, in which participants assess one another’s work against a set of criteria.

Reviewers need to understand the problem, inspect the implementation, and determine which criteria the project satisfies. A README that makes this difficult can cost the author course points. I have seen projects lose points because important work, such as evaluation results, was buried several notebooks deep and never mentioned in the README.

In this article, I'll explain how your README should look like, which sections it should contain, and who you should write for, so your project gets the attention it deserves.

It's an extended version of the material I prepared earlier this year to the students of my AI Engineering Buildcamp course. Right now the participants of LLM Zoomcamp will find it especially useful, but the same principles apply to the other Zoomcamps and to most engineering projects.

Throughout the article, I will use one running example: Fitness Assistant, a RAG application that helps users find exercises and suitable alternatives. I built it two years ago as a reference project for LLM Zoomcamp. The repository was already in reasonable shape, but I refined its README further while preparing this article.

Start with the reader

Before you write a README, think about the person who is going to read it.

I suggest writing for three audiences at once: peer reviewers, hiring teams, and your future self.

Peer reviewers

The first audience is someone assessing your project against a list of criteria.

In the Zoomcamps I run, participants review one another’s projects and decide how many points each submission earns. They need to find evidence for the problem description, evaluation, monitoring, reproducibility, and other criteria.

Even when your project is not part of a course, it helps to imagine that it will be reviewed this way.

An imaginary peer reviewer gives you a useful standard. Assume that someone will inspect the repository carefully and ask whether you have covered the important parts of the project. Your README should help them find the evidence without searching through every file and notebook.

Hiring teams

The second audience is the hiring team.

A recruiter, hiring manager, or engineer may not spend as much time on the initial review. They first want to understand what the project is and whether it is relevant to the role. Only then will they decide whether to inspect the implementation, evaluation, tests, or architecture in more detail.

This is why optimizing for a peer reviewer is still useful. The README should be easy to scan, but it should also contain enough detail for a technical reader who decides to go deeper.

Your future self

The third audience is your future self.

After several months, you may return to the repository and no longer remember how the project works, how to run it, why you made certain decisions, or what you planned to improve.

A good README should give you enough context to understand the project without reconstructing it from the code and commit history.

Write for all three

These audiences are different, but their needs are compatible.

The hiring team needs a clear overview and a reason to continue. The peer reviewer needs evidence and detail. Your future self needs enough context to understand and continue the work.

The strongest README supports all three: it is easy to scan at first, detailed enough to inspect, and complete enough to return to later.

What does the reader need to understand?

Most readers approach a project with four broad questions:

1. What is this project?

2. Does it work well?

3. Can I run and reproduce it?

4. How was it built?

Let’s start with the first question.

Part 1: What is this project?

Readers need to understand what you built.

The beginning of the README should answer three questions:

• Who is the project for?

• What problem does it solve?

• What did you build to solve it?

Use the title and short description to provide the immediate answer. Then explain the problem in more detail and show how the project works.

1. Title and description

Start with a clear project title and one or two sentences that describe the user, the problem, and the solution.

Avoid opening with a list of technologies. This tells the reader which tools you used, but not what the application does or why anyone would use it.

For Fitness Assistant, I use this description:

In two sentences, the reader learns:

• The user: fitness beginners without regular access to a trainer

• The problem: choosing exercises and finding alternatives can be difficult

• The solution: a conversational AI assistant that provides guidance

The exact formula will vary by project, but a useful starting point is:

You can add a second sentence to explain the main obstacle or why the project matters.

For example:

Keep the description concrete. Phrases such as “AI-powered platform,” “innovative solution,” and “intelligent system” add little unless you explain what the system actually does.

If you struggle to write the description, generate several alternatives and compare them. You can ask an AI assistant to analyze the repository or to describe the project based on your notes. However, you still need to choose the version that accurately represents the project and makes sense to your audience.

2. Problem

The short description introduces the problem, and the problem section explains it.

Here’s a simple structure that works well:

1. What is the user trying to do?

2. What makes that difficult?

3. Why are the existing options insufficient?

4. Who experiences this problem?

For my Fitness Assistant project, the problem is described like that:

• Beginners often struggle to choose appropriate exercises

• Gyms can feel intimidating when someone does not know how to use the equipment

• An exercise may be unsuitable because of an injury, missing equipment, or personal preference

• Personal trainers can provide guidance, but they are not always available or affordable

Keep this section proportional to the project. You don’t need a long market analysis for a small course project. A few specific paragraphs are usually enough to establish why the application exists and who benefits from it.

3. Demo

After you explain the project, show it working.

A reader should not need to install dependencies, create API keys, and start several services just to see what the application does. Many readers will decide whether to continue before they run anything locally.

You can demonstrate the project with:

• Live application

• Short video

• GIF

• Screenshots

• Sample input and output

Use the format that best represents the project. A screenshot may be enough for a dashboard or static interface. A short video is usually more useful when the application involves several steps, dynamic output, or interaction with an AI assistant.

For an AI application, the demo might show:

1. User entering a realistic request

2. System retrieving or processing the relevant information

3. Application producing an answer

4. User providing feedback or asking a follow-up question

Keep the main demonstration focused. Show the core workflow first rather than spending several minutes explaining the repository structure or installation process.

You can cover evaluation results, monitoring dashboards, and architecture later in the video, but a reader should see the application working within the first few moments.

If the project is deployed, place the live link near the top of the README. A reader can then try the application without setting it up locally. Keep the video or screenshots as well because the hosted version may later become unavailable or require credentials.

For Fitness Assistant, I recorded a video that shows the conversation flow and the main application features.

Part 2: Does it work well?

For an AI engineering project, readers need three kinds of evidence:

• Evaluation: How well does the AI system perform?

• Testing: Does the software behave as expected?

• Monitoring: What happens when the application processes real requests?

1. Evaluation

Without evaluation, readers can see that the application generates answers, but they cannot judge how consistently or accurately it performs.

Describe the evaluation dataset and the kinds of cases it includes. The dataset should reflect the situations in which you expect the application to operate. A system evaluated only on simple, carefully written examples may perform poorly when users phrase the same requests differently.

Also explain what you changed after establishing the baseline. The result is more useful when the reader can follow the progression from the initial approach to the final one.

For my Fitness Assistant, the README reports both retrieval and end-to-end RAG evaluation. After adding field boosting, retrieval reached a 94% hit rate and 90% mean reciprocal rank. The final RAG flow produced answers classified as relevant in 83% of cases when using gpt-4o-mini. The README also includes the parameters that produced the best retrieval results and links to the evaluation notebooks and datasets.

Keep the summary in the README, but store the detailed code and results in a separate, clearly named location. Document how to rerun the evaluation so the reader can verify the results or test a different approach.