All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App (this article)

• Part 5: TBA

In part 2, we developed an end-to-end application for conducting system design interviews. In part 3, we deployed it to AWS.

In this part, we will take the application we deployed previously, and make it more production-ready:

• Define dev and prod environments

• Introduce observability: logs, metrics, alerts

• Use AI as the first responder when the application stops working

This article is a draft and will be updated after the workshop

Overview

Right now, every push to main goes straight to the one environment we have, and nothing tells us the app is unhealthy until we notice by hand. We fix that in three steps: separate dev from prod so a bad change never reaches users directly, add the telemetry and alerting that would have caught last part’s failure on its own, and give an agent a narrow, auditable way to respond.

Dev and prod environments

So far, a push to main deploys straight to the only environment we have. That’s fine for a demo, but it means every change - tested or not - reaches the same instance our users hit.

We want a place to deploy to first, and a deliberate, separate step to promote a change from there to production.

We already have infrastructure as code for our deployment from the previous part, and it already parameterizes what differs between environments, such as the domain and the instance size. That means we don’t design anything new - we ask for a second, independent copy of what we already have, and treat what we already have as dev from now on.

Create a second, independent copy of our deployment infrastructure for a
production environment.

It must be able to run alongside the existing one with its own database and compute. The existing becomes the dev environment.

Once prod exists, split the deploy step in CI/CD so each environment gets its own path:

Split our CI/CD deploy step into two:

Deploy to dev: on every push to main, after tests pass, deploy to the dev
environment automatically (we already do it today)

Deploy to prod: a manual workflow that promotes the version running in dev to prod.

The key property is that prod never builds from source on its own - it redeploys the exact commit that already proved itself in dev. Promotion is a deliberate, logged action, not a side effect of pushing code.

Instrumenting

Two environments buy us a safe place to promote from, but not visibility. A bad deploy can still sit in prod for hours before anyone notices, because nothing yet tells us the app is unhealthy. And even once we notice, “error rate went up” isn’t enough - we need to get from that to one failed request, its log, and the commit that served it. A dashboard that only shows CPU and memory can look perfectly fine while the one thing users care about is silently failing. That’s what we fix next.

We use OpenTelemetry for this: a vendor-neutral format for traces, metrics, and logs. A metric tells us the canvas-broadcast error rate went up; a trace shows one failed request’s path through the backend; a log holds the exception. Tagging all three with the same service name, environment, and deployed commit is what lets us walk from a spike on a chart to the one request, and the one log line, behind it.

We ask the coding assistant to instrument one important operation first:

Instrument the FastAPI backend with OpenTelemetry.

Export traces and metrics with OTLP (no collector yet). Include service name, environment and deployed git commit

The app sends OTLP to an OpenTelemetry Collector, which exports it to whichever backends we pick.

Exporting straight from the app to each backend instead would mean an application change every time a backend changes, and a slow backend could stall the app’s own exporters. The collector is one stable boundary that handles batching and retries outside the request path.

Add OTLP collector.

Create the observability/ directory with Docker Compose for an OpenTelemetry
Collector, Prometheus, Loki, Tempo, and Grafana.

Build the dashboard

The collector is running, but nothing draws it yet. We ask for one panel - the one the rest of this article leans on:

Add one Grafana panel: canvas-broadcast error rate, with exemplars enabled so
a point links to its trace.

Look at it locally first

Before shipping anywhere, we run the stack next to the app on our own machine and generate one real trace, so we know the prompt above actually worked before going looking for it in dev or prod:

docker compose -f observability/docker-compose.yaml up -d

Then we hit the app a few times so there’s something to see, and open Grafana at

http://localhost:3000

- no tunnel needed here, since nothing on a laptop is public in the first place. Once the panel we just asked for shows real numbers, it’s safe to ship.

Run it in dev and prod, and reach it privately

A demo dashboard on a laptop can’t tell us about a real failure, so this stack needs to run next to the app in every environment. We deploy it the same way as everything else - push to main, dev picks it up automatically, then a promotion ships it to prod:

Deploy the observability/ stack alongside the app, in every environment.

Follow one failed request

Let’s break the app:

Introduce a bug that makes canvas broadcasts fail for a fraction of calls.

Same pipeline as before: main, then dev, then a promotion to prod. Then we use the app normally until one edit fails.

With that one failure in hand, we open the dashboard and answer these, in order, by moving between panels rather than guessing:

1. Did users receive errors? - the canvas-broadcast error-rate panel moved.

2. Which operation failed? - that’s the panel that moved; the metric is already labeled by operation.

3. Can we open one representative request? - click an exemplar on that panel to jump straight to one trace.

4. Which log belongs to that request? - the trace carries a trace ID; look it up in Loki.

5. Which deployment served it? - the trace and the log both carry the deployed commit as an attribute.

A dashboard that can’t answer these - even a good-looking one - hasn’t instrumented the incident we actually care about.

Create one actionable alert

We don’t import a large alert pack - just one alert for a sustained increase in failed canvas broadcasts, requiring both:

• enough requests failed to affect users

• the failure lasted long enough that one brief blip doesn’t page anyone

This follows the Prometheus alerting guidance: keep alerts few, tolerate small blips, and page on symptoms that need action. CPU, memory and connection counts still belong on the dashboard - they help explain an incident, they just don’t need to wake anyone up on their own.

We ask the assistant for the rule and its test together:

Create a Prometheus alert for a sustained canvas-broadcast error rate.

Wake the on-call engineer

Everything from here lives in one folder in the app repo, on-call-engineer/. Rather than stand up something to receive a webhook, we poll: a script checks Prometheus’s alert API on an interval and starts the on-call agent the moment our alert shows up firing.

Add an on-call-engineer/ folder with a script that polls Prometheus's
/api/v1/alerts every minute.

For this proof of concept, that script runs from a cron job on our own machine. Nothing needs to be reachable from the internet for this to work - the script reaches out to Prometheus, not the other way around.

A cron job on a laptop only works while someone remembers to leave it running, though. The production version of the same idea is a scheduled serverless function - an AWS Lambda on an EventBridge schedule, a Cloud Run job on Cloud Scheduler, or your platform’s equivalent - firing the identical poll script every minute. Nothing runs between checks: the platform starts the function, it polls, and if the alert isn’t firing it exits and gets torn down immediately. That’s a stronger version of “provisioned only when needed” than standing up a box on alert, since even the polling itself costs nothing while the app is healthy. We don’t build that here, but the poll script doesn’t change to get there - only what’s calling it on a timer.

Let it investigate and fix

The agent isn’t handed a pre-built evidence bundle - it collects its own, with read access to exactly the systems it needs: Prometheus, Loki, Tempo, and git log. And it’s allowed to do more than look: edit code in a fresh checkout, run the test suite, and push a fix if it has one.

You are the on-call agent for one production alert: {alert.json}

You have read access to Prometheus, Loki, Tempo, and git log. Investigate
first - write your evidence and reasoning to incidents/<timestamp>/, and
don't call something the root cause unless the evidence rules out the main
alternatives.

If you find a fix you're confident in, make it on a fresh branch, run the
test suite, and push only if it passes. If you're not confident, or nothing
you tried passes, don't push - write an escalation instead (see below).

It also appears in interview questions. In interviews, you’ll definitely hear something like “how do you evaluate a RAG pipeline?” or “how do you measure agent performance?”. If you don’t include evals in your home assignment, it’s a red flag.

Why is it such an important skill? Because it’s hard, and it matters.

It’s easy to build an agent these days. I show how to do it in my “AI Hero: 7-Day AI Agents Crash-Course”. You need to get an OpenAI key, take any agentic framework, define the instructions and the tools, and that’s it. You have an agent.

But this agent will break in so many ways. It will:

• Give a wrong answer

• Confidently hallucinate a plausible answer

• Finish without giving any answer

• Get into a loop and call the same tool over and over again

• Exhaust the context

• Call a tool with invalid arguments

That’s just the tip of the iceberg. To make sure this agent works reliably, you need to test and evaluate it.

Also, without evaluations, you’re blind. You don’t know how the agent is performing. You also can’t really change anything: every change you introduce may break the system in many ways, including the areas where you least expect it.

Agent evaluation is not simple. It takes a lot of time to understand the problem, get input from real users, and gather the right data. It’s not something you can just delegate to Claude and forget about it.

That’s what makes evals such an important part of AI Engineering.

In this article, I will tell you about my approach to evaluations:

• Start manually by “vibe-checking” the system, but also collect logs as early as possible

• Create a tool for labelling the logs and put together a gold standard dataset

• Create a judge that’s aligned with our judgement

• Break the agent like a QA engineer

• Get more data by generating it synthetically

• Start collecting data from real users

• Monitor your system with online evaluation

• Always refine your gold standard dataset

Let’s get into details.

Collecting the Gold Standard Dataset

You have an agent. What’s next?

There’s nothing wrong with “vibe-checking” it: you poke it, ask questions you want it to answer, and look at the results. If they aren’t good, you figure out how to fix it.

But already at this step you can be a bit more organized and start logging what the agent is doing. When you ask a question and see the answer, make sure they are saved somewhere. You do a few vibe-checking sessions, and you already have 10-15 logged records.

Now you can systematically look at these logs and classify each record into

• “good”: the answer is what you expect

• “bad”: the answer is not what you wanted to see

To help me do it, I usually vibe-code a small labelling tool.

This gives you a “version 0” of the gold standard dataset - the dataset you will use for evaluating your system (sometimes it’s also called the “ground truth” dataset).

Don’t delegate this step to coding assistants. It’s only a few examples, so you can go through them relatively quickly. But this way, you will get a lot of insights into how your agent is behaving, what’s working and what isn’t.

Judge alignment

Once you manually labelled 10-15 examples, it’s time to start automating it.

I use coding assistants for that:

• Ask the assistant to analyze the logs and classify the examples into good or bad.

• See if the results match yours. If they don’t, ask the agent to explain its decision - maybe you missed something, or the agent didn’t have enough context.

• Iterate until you both agree.

• At the end of the session, ask the agent to create a judge.md file that describes the classification rules.

• Read this file carefully to see that the rules are generic and not specific to the examples you have in the dataset.

We just created a judge - a system that evaluates our agent.

Now we need to test that using judge.md alone is sufficient:

• Start a new session

• Ask to start a subagent which reads judge.md and classifies the records in your dataset

• If something is off, ask the agent to change judge.md

• Iterate until you both align on what’s good and bad

This process is called “alignment” - we align the judge with what we think is good or bad.

It’s the same as training a binary classifier that predicts your decisions. But instead of weights, you have a judge.md file. (And it’s also helpful to have a train-test split here too! But later, when you have more data.)

At this point, you also get a metric - the fraction of good examples. Now if you change something in your agent, you can re-run the judge and make sure this metric doesn’t go down.

Breaking the agent

We have the initial version of our gold standard dataset. But it’s not enough. Now we want to find cases that break our system.

QA engineers have been breaking software for many decades. For coming up with cases that should derail our agent, we can borrow some concepts from their frameworks.

1. Equivalence partitions

The first step is equivalence partitioning. You divide the input space into groups where inputs within each group should produce similar behavior.

If we have a RAG agent that answers questions about our docs, it could be:

• Questions about topics in the documentation,

• Questions about relevant topics but not in the documentation,

• Questions about irrelevant topics,

• Ambiguous questions,

• Questions in different languages

Once you come up with a list of groups, come up with 2-3 questions from each group. You can ask AI to help you with the entire process.

2. Boundary testing

The next concept is boundary testing.

We already have some questions from the previous step, but we can be even more aggressive in our testing.

After defining the partitions, QA engineers would test the boundaries of each partition. We can do the same.

For a RAG agent, come up with queries that:

• Exactly match a document that we have in our database (easy case),

• Are semantically related but use different words (typical case),

• Are semantically related to multiple documents (ambiguous case),

• Are completely off-topic (boundary case).

Further alignment

This way you can easily add 30-40 more questions to your gold standard dataset.

Let’s take all the data we have, and run it against our agent. Then use the judge to classify the output.

We already have a tool that we created earlier for manual evaluation. At this point, I adjust this tool to also support checking the output of the judge. For each record, I just decide whether I “agree” or “disagree” with the judge. I find it more convenient than labelling the new logs as “good” or “bad”.

If we have many “disagree” cases, tune the judge following the same process as before. We will never have 100% alignment, so we don’t need to be too strict about it, but the judge should be correct most of the time.

Using the Judge to improve your agent

Now we have a fully automated process that:

1. Runs the agent against our data

2. Runs the judge against the output of step 1

3. Outputs a metric - the fraction of “good” results

Because it’s automated, we can use it to improve our agent. You can ask the coding assistant to do it.

I recommend following the implementer-tester pattern for that:

• Define the implementer subagent that modifies the agent

• Define the judge subagent that uses judge.md to evaluate the output

• Iterate until your “good” ratio is acceptable

Synthetic data

So far we focused on the quality of our gold standard dataset. In this section, I want to talk about the quantity.

Quantity is also important because agents are non-deterministic. Even if the agent works properly on our small dataset, it will break when we get more traffic - just because of probability and statistics. When you expose your agent to a bigger number of inputs, things will break: with more repetitions, you increase the chances of something going wrong. That’s why you want quantity here.

We can generate a lot more data using AI. Open your AI assistant, ask it to analyze your code, describe to it your target users, and then tell it to generate questions that these users are likely to ask.

For a RAG system, you can:

• Take a document from your knowledge base

• Ask AI to come up with 5 questions based on this document

• Do it for all documents

You will end up with a lot of questions. You can keep all of them, or take a sample.

Then follow the same process: run your agent against this data and then use the judge to evaluate the results.

This way you will probably run into many different problems:

• The agent will loop without stopping and exhaust the context

• It won’t follow the instructions and do something that it shouldn’t

• A tool will break and your agent won’t finish the task

And many others. You don’t know in advance what you’ll get, which is why you want to expose the agent to a wide variety of inputs.

Once you identify the scenarios where your agent breaks or doesn’t perform well, add them to your gold standard dataset.

Then you can fix the agent, and re-run the whole evaluation set to make sure you don’t introduce any regressions.

Start testing on real users as soon as possible

We have evaluated our agent on many cases, so we’re confident that it’s working okay. But we always need more data.

Now it’s time to put it in front of real users:

• Invite people for user testing sessions to check your system

• If you plan to integrate it into an existing product, include it in a small subset of the traffic

• If it’s a small personal project, ask your friends and family to test it.

• If it’s a course project, ask your classmates to play with it.

• Share it on social media.

Make sure you’re collecting the logs (be direct with the user about what you’re collecting) and use this for adding new cases into the gold standard dataset.

Add +1/-1 buttons. This way, the users can give you explicit feedback, and if somebody dislikes the answer, you should inspect it.

Online evaluation

We have a system for offline evaluations: we run the agent against our dataset and we use the judge to score the output to see how well it behaves.

It’s okay, but it’s not scalable. Instead, we want to run it continuously: every time somebody uses our agent, we react in real time - save the logs, evaluate them, and display the results on the dashboard.

We can’t really continue using our coding agent for that, so we’ll need to create a stand-alone judge.

It will give you a real-time view of the agent’s quality: the performance on live traffic, and new cases that you can add to the gold standard dataset as they appear.

Periodically inspect bad results, and if you see something new, add these cases to your gold standard dataset. Also sample the good results to make sure they are actually good.

For inspecting this live traffic I also often use coding assistants. I take a sample of the data, save it locally and ask them to go through these cases with me.

Multiple judges

We started with a good/bad evaluator, but I didn’t really explain what it means. And it anyways depends on your problem.

Over time, we will need to have multiple judges, each evaluating the agent from a different angle:

• Task completion: the agent actually solved the user’s problem

• Relevance/Correctness: the answer is relevant/correct

• Groundedness: the claims in the answer are supported by the documents in the knowledge base

• Completeness: the answer includes all the required information

• Following instructions: the agent follows the instructions correctly

• Trajectory optimality: the sequence of tool calls the agent used to complete the task is an optimal one

There are other criteria too that you can consider for your application.

But I usually recommend starting with one judge and then adding others as your system matures.

The agent evaluation checklist

As a summary for this article, I wanted to give you a checklist that you can use next time you work on a new agent.

Gold standard dataset:

- [ ] Vibe-check the agent and log 10-15 interactions
- [ ] Label results as "good" or "bad" yourself
- [ ] Define equivalence partitions and test cases within each
- [ ] Include boundary cases (exact match, paraphrase, multiple matching documents, off-topic)
- [ ] Generate synthetic questions

The judge:

- [ ] judge.md describes the classification rules in generic terms, not specific to your examples
- [ ] The judge runs as a subagent that reads only judge.md
- [ ] Check the judge's answers with "agree" or "disagree"
- [ ] Use the fraction of "good" results as your metric
- [ ] Add more judges as your system matures

Improving the agent:

- [ ] The evaluation pipeline is automated: run the agent, run the judge, get the metric
- [ ] Each time you make a change, run the full evaluation set to catch regressions

Online evaluation:

- [ ] Put the agent in front of real users
- [ ] Collect user feedback
- [ ] Score online traffic with a stand-alone judge, show results on a dashboard
- [ ] Inspect bad results periodically, add new failing cases to the eval dataset

If you want to learn more about building AI Agents, I teach that in my course AI Engineering Buildcamp: From RAG to Agents. In the course, I also cover evaluation in a lot of detail with real code, real tools, and real agents.

The next cohort starts September 21. You can use code “SUBSTACK” to get 20% off. See you there!

All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants (this article)

• Part 4 and 5: TBA

In this article, I want to take a web application and make it accessible for everyone on the internet.

We will:

• Take the application that’s already working locally

• Containerize the application

• Switch from SQLite to Postgres

• Add integration tests

• Deploy to AWS

• Set up CI/CD via GitHub Actions

It’s based on the second half of the full-day workshop I did for AI Shipping Labs: Build and Deploy a Full-Stack App with AI Coding Assistants.

Recap

In the previous article, we started building an application for system-design interviews. An interviewer creates a session and shares a link with a candidate. When the candidate updates something on the canvas, the interviewer sees the updates in real time.

• First, we created the frontend only with React

• Then we created an OpenAPI specification for defining the frontend-backend API

• Next, we created the backend from this specification

• We added database support with SQLite and SQLAlchemy

You can find the code in the interview-canvas-share repository.

Containerization

When we run the application locally, we need to execute two commands: one for the frontend and one for the backend.

Let’s run them in two separate terminals:

# first terminal
cd frontend
npm run dev

# second terminal
make run

So, we have two services, and we need to deploy them to production. We may think that we need two containers: one for the frontend and one for the backend.

During development, we run the frontend as a separate service because we use Vite. Vite watches the frontend code and refreshes the page when we make changes. It’s very convenient because we can see our changes immediately.

In production, the frontend code doesn’t change while the application is running. We build it once and get a set of static HTML, CSS, and JavaScript files.

This means we don’t need a separate container for the frontend: the backend can serve these files. So, we need only one container.

Ask the coding assistant to create it:

Create a Dockerfile that builds the frontend with Node, then builds a Python image with the backend and the frontend static files.

Backend should serve the frontend.

We get a two-stage Docker build:

• First, we use a Node.js image to compile the frontend

• Then we build the backend and copy only the frontend files without the Node.js dependencies

You can see the Dockerfile here.

Build and run the image from the repository root:

docker build -t sdip:latest .

docker run --rm -p 8000:8000 \
  -v sdip-data:/data \
  -e SDIP_DATABASE_URL=sqlite:////data/sdip.db \
  --name sdip sdip:latest

Here we specify a named Docker volume sdip-data that will keep the SQLite database between container runs.

Open the application at localhost:8000 and test it:

• Create an interview session

• Open the join link in a different browser window

• Move an element in the candidate window

• Check that the interviewer sees the change

We’ll repeat this test again. I’ll refer to it as the two-session test.

Switch from SQLite to Postgres

SQLite is very convenient for local development. It keeps the data in a single file and doesn’t need a separate database server.

But for production, we typically use Postgres or a similar database.

When we set up the foundation in the previous article, we asked the coding agent to use SQLAlchemy.

I did this on purpose because I knew that later I’d switch to Postgres.

Start Postgres locally:

docker run -d \
  --name interview-canvas-db \
  -e POSTGRES_USER=sdip \
  -e POSTGRES_PASSWORD=sdip \
  -e POSTGRES_DB=sdip \
  -p 5432:5432 \
  -v interview-canvas-pgdata:/var/lib/postgresql/data \
  postgres:16-alpine

Now we can ask the assistant to use it:

Add Postgres support to the backend.

Run the backend against Postgres from the repository root:

export SDIP_DATABASE_URL=postgresql://sdip:sdip@localhost:5432/sdip
make run

When it’s done, repeat the two-session test.

Docker Compose

Previously, I started a Postgres container with a separate command. But now let’s put all the services our application needs inside one Docker Compose file.

With this file, we can run our entire application with a single command docker compose up.

Ask the assistant to implement it:

Create docker-compose.yaml with two services: Postgres and the app.

The file defines the database and adds a health check, so our application waits until Postgres is ready to accept connections.

Start it:

docker compose up --build

In our case, it runs the application at localhost:8100.

Integration and end-to-end tests

The AI assistant might have created some backend tests.

If it didn’t, ask it to create integration tests:

Create integration tests that run against docker-compose.yaml. What scenarios should we test?

We added two things that can potentially break, so let’s test them too. We’ll verify that:

• The frontend compiles correctly

• The backend can communicate with Postgres

Ask the AI assistant to implement a test:

Add an end-to-end test that runs against docker-compose.yaml.

Use Playwright to:

1. Log in as the interviewer (session 1).
2. Create an interview session.
3. Share the join link.
4. Join from a separate client as the candidate (session 2).
5. Change the canvas as the candidate (session 2).
6. Verify that the interviewer sees the change (session 1).

Put the tests in the e2e/ folder in the repository root.

After it finishes, we can run the tests with a single make command:

make e2e

Deploy to AWS

We’re now certain that the application works well. We can deploy it.

Our application runs in a container and only needs Postgres, so we have a lot of options for deploying it. We can use Render, Railway, Fly.io, or any other managed container system.

Last year we deployed to Render, but this year I want to deploy to AWS. You don’t have to use AWS, and instead you can ask the coding assistant to recommend an environment for your application.

Ask your assistant to deploy it:

Deploy this application to AWS. Use AWS CloudFormation.

For that to work, you need to have an AWS user. I typically create a temporary user with admin permissions and watch every step of what the coding agents are doing.

You can see what I got here. It runs the app, Postgres, and Caddy (adds HTTPS and WSS support for our app) on one EC2 instance.

It’s fine for a proof-of-concept, but using managed database services (such as RDS) is better. We will not do it here.

CI/CD with GitHub Actions

We used a user with the admin permissions to deploy the application. It’s okay for the first deployment, but only when we carefully watch it. The next step is to configure CI/CD and remove that access.

• Continuous integration (CI) means that every time we make a change and push it to GitHub, we automatically run all the tests to make sure we didn’t break anything.

• Continuous deployment (CD) is about deploying this change automatically.

In GitHub, we use GitHub Actions for that.

Let’s configure it. Every time we make a push to main, we want to:

• run frontend and backend tests

• build the containers

• run the integration tests

• run the end-to-end tests

• if all the tests pass, deploy the new version

For the last step, the runner (the process that will deploy the application) will need to be able to access our AWS infrastructure. We will use OpenID Connect (OIDC) for this: the runner will assume a role with the necessary permissions and update the application.

Create the role and the workflow:

Create a CI/CD pipeline that:

- runs backend and frontend tests in parallel
- builds the Docker Compose stack and runs integration and end-to-end tests against it
- deploys to AWS using a GitHub OIDC role
- validates that the deploy is successful by checking the health endpoint

The finished workflow uses a restricted AWS role for deployment.

Change something in the application, commit, and push to see it go live.

Clean up

When we’re done, we need to delete all the resources created by CloudFormation:

aws cloudformation delete-stack --stack-name sdip
aws cloudformation wait stack-delete-complete --stack-name sdip

The next step after deployment

Let’s recap what we have done so far:

• We created the frontend application with React

• Then we defined the frontend-backend API with OpenAPI specs

• Based on the specs, we created the backend

• Next, we added database support with SQLite and SQLAlchemy to the backend

• To make it easier to deploy the application, we put both frontend and backend inside one container. The backend serves the frontend.

• To go to production, we replaced SQLite with Postgres.

• Next, we simplified running everything locally with Docker Compose

• Once everything was in Compose, we created an end-to-end test

• We took the application that worked locally and deployed it to AWS using CloudFormation

• Finally, we created a CI/CD deployment pipeline to deploy every change automatically.

But we still need to do a lot more for the app to run reliably:

• Dev and prod environments

• Observability: logs, metrics, alerts

• Using AI as the first responder when the application stops working

We’ll cover it in the next lesson.

In this article I take that algorithm and update it for AI Engineering. The updates are minimal and you can use this algorithm for any other engineering job.

The Job Search Algorithm

The algorithm is straightforward:

1. Network: Connect with people

2. Apply: Apply for jobs even if you don’t qualify

3. Build: Create projects and share them online

4. Interview: Record what you were asked

5. Reflect: Review and improve

6. Repeat

This is a while loop. Continue networking, applying, building, and interviewing until you receive an offer you like.

Step 1: Network

Networking helps you understand the market, find opportunities, and learn what companies are actually looking for.

Use LinkedIn and meetups for that.

Talk to people working in your field. Talk to people who work as AI engineers. Ask about the projects they work on, and the technologies they use. And ask if they are hiring and ask for a referral.

If you are a junior, look for other juniors who have recently got a job. Ask what helped them get hired, which companies they targeted, what the interview process involved, and what they would do differently.

A conversation doesn’t have to end with a referral. By talking to people you discover companies that hire, understand which skills to develop, improve your search strategy, and meet interesting people.

Step 2: Apply

Start applying for jobs right now. Many people postpone it because “they are not ready". You are ready. Apply even if you don’t qualify.

Don’t reject yourself because you are unsure if you’re qualified. Apply and let the hiring team evaluate your application. If you apply, you have a chance to move forward. If you don’t, you never will.

When you look at the job requirements, don’t treat them all as mandatory. The job description always describes the ideal candidate and often these candidates don’t exist.

There are, of course, cases where you shouldn’t apply. If you’re a junior, but the position is senior, then don’t apply. If you match only 20% of the requirements, it’s also a bad sign. If you’re in doubt, apply.

If you apply for relevant roles but don’t hear back, review your CV. Make sure it shows the experience, skills, and projects that are relevant to the positions you apply.

Recently I published a workshop at AI Shipping Labs about tailoring your CV for AI engineering roles that you may find helpful (heads up: it requires a paid subscription).

Step 3: Build

You learn by doing. When you apply, note which skills are required for the positions you apply for, and build projects using these technologies.

Always timebox these projects. Set the time limit to 10-20 hours and then move on to the next project. At this stage, your goal is to have an awareness of different technologies and an understanding when to use them. You can go deeper when you’re already working.

I wrote a full guide on building projects with six steps you can follow:

A complete project should include working code in a public GitHub repository and a clear README. The README is the most important document of your project, so make sure you put enough effort into polishing it before you add a project into your portfolio.

A few weeks ago I wrote an article about that:

Don’t build these projects silently. Share what you build and what you learned online.

Here’s a suggested sequence of posts for LinkedIn and/or X:

1. Describe the project idea and the technologies

2. For each milestone, post a screenshot or a code snippet with what you built

3. When the project is ready, announce it and publish the link

4. Optionally, write a blog post

And yes, there’s another article I wrote about it:

Step 4: Interview

Once your CV gets past the screening stage, you get invited to an interview.

Don’t treat interviews as exams. You’re not in school. It’s a two-way process: they evaluate you, but you also evaluate them.

Interviewing is a also skill. Your first interview may not be great, but it improves with practice. Each subsequent interview helps you identify gaps in your preparation and see what you should do differently next time.

Some interviews will lead to rejections. It’s normal and may happen for many reasons outside of your control: the company chose someone with more suitable experience, hired an internal candidate, changed the role, or froze hiring. Don’t take the rejections personally, learn from them and move on.

The hiring process varies between companies, but typically you will have some of these interviews in the pipeline:

1. Recruiter interview: a short conversation about your background, motivation, availability, salary expectations, and basic fit

2. Theoretical interview: questions about LLMs, RAG, agents

3. Coding interview: coding, Python, SQL, code review, or practical engineering tasks

4. Project deep dive: a deeper discussion of your experience and past projects

5. System design: defining the architecture of a project

6. Behavioral interview: examples of how you dealt with different situations at work (and outside)

7. Take-home assignment: a small project you complete at home

You probably won’t get all of them: some stages may be bundled in one round, and other stages may be absent.

Now let’s go through each step of the hiring pipeline. In this article, I mostly used the materials from AI Engineering Field Guide. You will find a lot more relevant information there if you’re preparing for AI Engineering interviews.

⭐ Star the Field Guide repo

1. Recruiter interview

The recruiter is the first person you speak with during the interview process.

It’s not a technical interview. They will tell you about the company, the role and ask some basic questions.

You should be ready to answer:

• “What do you know about our company?” Make sure you do some basic research. If you can’t answer it, the interview process is likely to stop there (it happened to me!).

• “What are your salary expectations?” Research the market and be ready to provide a realistic range. You can dodge this question and delay it to other stages, but I don’t think it’s helpful.

• “Tell me about yourself”. Prepare a 3-minutes story. It will be useful at later stages too.

Use this interview to make sure you understand what the interview process looks like and what the expectations are at each stage of the process. Ask for tips how to prepare for these stages.

2. Theoretical interview

There will be one or multiple rounds where the company checks if you understand how LLM applications work. They may appear in a dedicated technical interview or as part of the project deep dive or AI system design sessions.

For AI Engineering, you will be asked about:

• LLM behavior, prompting, context windows, and structured outputs

• RAG, retrieval, reranking, and common ways RAG fails

• Agents, tool use, tool design

• Evaluation and debugging

• Monitoring, cost, latency, and scalability

Fine-tuning, model training, and transformer internals can appear too, but this only happens for roles that require these skills.

You can see theoretical questions in the “Theory” part of AI Engineering Field Guide interview preparation section.

3. Technical interview

Coding rounds generally fall into two categories: implementation and algorithm rounds.

Traditionally, for implementation rounds, you’re given a task like “implement a web crawler”. You’re not expected to deliver a fully working solution, but rather demonstrate your approach to coding. You may need to design (in the “object-oriented design” sense of the word) the interaction between different components, define proper abstractions, and then implement only some parts of it.

However, more and more often companies give a vague task and ask you to use AI to implement it. You are expected to produce a reasonably complete working solution. The interviewer evaluates how you interact with the coding agent, how you review the code, test it, correct errors, and refine the implementation.

The algorithm rounds usually involve easy and medium Leetcode-style problems. Hard problems are rare. There is already a lot of information on the Internet on how to prepare for them, so I won’t spend time on it. Joining Leetcode and solving some problems there definitely helps.

You can see technical questions in the “Coding” part of AI Engineering Field Guide interview preparation section.

4. Project deep dive

Project deep dive interviews assess your level of seniority and check if your CV matches the reality.

At my previous jobs I regularly conducted these interviews.

I start by asking the candidate to introduce themselves and then ask what they were doing recently. Our discussion naturally goes into projects they worked on and then I start going deeper with questions:

• What problem does this project solve?

• How does it work?

• Which part did you build?

• Who made the technical decisions? How?

• What went wrong?

• What would you change if you were implementing this project today?

This is by no means an exhaustive list, but it should give you some ideas of what to expect.

I recently had a workshop about it at AI Shipping Labs. Based on it, I wrote an article: AI Engineering Hiring Manager Interview (it’s free).

You can see deep-dive questions in the “Project Deep-Dive” part of AI Engineering Field Guide interview preparation section.

5. System design

If you’re applying for more senior roles, you’ll have a system design round. They are very uncommon for junior roles.

There are two types of system design interviews: “classical” system design and AI system design.

AI engineers are software engineers who specialize in AI, so they are expected to design software systems too. That’s why one of the interview rounds may be classical system design.

In these interviews you will need to design a system related to what the company and the prospective team do.

A food delivery company may ask you to design an order-processing or delivery-tracking system. When I interviewed at Yandex, they asked me to design search. When I interviewed with Meta, they asked me to design a geo-aware recommender system. When I worked at OLX and we were hiring people to join the recommender platform, we were asking to design a recommender system for an online marketplace.

But for AI engineers there’s also an AI system design round. Common tasks include designing a document Q&A system, a coding assistant, an AI search engine, or an agent. The specific system will depend on what the team is doing.

For these interviews, clarify the requirements, ask which data is available and what’s the expected scale. After that, sketch the main components of the system.

You can see AI system design questions in the “AI System Design” part of AI Engineering Field Guide interview preparation section.

6. Behavioral interview

The behavioral interview checks how you collaborate, make decisions, take responsibility, and handle setbacks. “Tell me about a time you needed to help a colleague”. There’s nothing specific in these interviews for AI engineers.

The best way to prepare for these interviews is to research the company’s values. You can ask the recruiter about them too. For each value, prepare examples of how you demonstrated them. As a general reference, I recommend Amazon’s Leadership Principles.

You can see behavioral questions in the “Behavioral” part of AI Engineering Field Guide interview preparation section.

7. Take-home assignment

In take-home assignments you need to build a small AI application, submit the code, a README, and a working demo. It’s also typical to have a defense session where you need to present the solution and answer questions about it.

Before starting, clarify the requirements, ask if you can use AI tools, and what the evaluation criteria are.

Before finishing, make sure you have tests, an evaluation set, and a good README that describes the system well.

Take-home assignments are time-consuming. They are a good addition to your project portfolio, but if you interview with multiple companies, and all of them give you homework, it can add up.

If you’re short on time, it’s okay to decline them. When you do, make sure you have projects in your portfolio that demonstrate the qualities they want to assess. Then you can say “I unfortunately don’t have time to work on it, but I have similar projects on GitHub and can walk you through one”. Don’t expect it to work with everyone, but some companies may agree to this.

When getting a home task, watch out for warning signs like these:

• You get an automated take-home assignment right after applying, even before speaking to anyone.

• The company states the task takes two hours, but the requirements suggest it could take several days.

• The company won’t clarify how your submission will be assessed or utilized.

In these cases you may spend time working on the test and never hear back from the company after submitting your project.

You can see examples of tasks in the “Home Assignments” part of AI Engineering Field Guide interview preparation section.

Step 5: Reflect (after the interview)

After each interview, write down all the questions you were asked and tasks you were given. What was difficult? Which answers could have been better?

You can also talk to AI, describe in detail how the interview went, and ask for feedback.

Write that all down in your interview journal.

This will help you identify areas where you can improve, and give better answers in the next interviews.

If you’re rejected, ask the company for feedback. Many companies will decline: there’s a non-zero chance that you may sue them and use this feedback against them, so they won’t risk it. But some will share it - most likely over a phone call. Also, don’t take it personally. Rejections are part of the process.

Continue networking, applying for more jobs, keep building and sharing what you learn.

If you get an offer - congratulations! But don’t rush into accepting it. Evaluate if the package is good and if you will like working there.

If you’re interviewing with other companies, finish the process with them too. It’s okay to be explicit about it with the company that made the offer. If they are pushing you to accept it as soon as possible, this is a red flag, so consider this carefully before agreeing. You got one offer, so you’ll definitely get others.

Good luck and have fun!

Share Alexey On Data

What I’ve been working on this week

1) Building and deploying a website with ChatGPT Work

This week, I experimented with building a small website using ChatGPT Work.

I’m really impressed. I sent it photos of a music festival brochure (it was Brutal Assault) and asked it to turn them into a website. It extracted the schedule, built a festival planner, and deployed it to a public URL that I could share with others.

I didn’t even need to run Codex. Everything happened from my phone from the ChatGPT app. It also has built-in authentication (via ChatGPT) and a built-in database.

2) FAQ Assistant in Practice

Last week I wrote an article about the FAQ assistant that we use at DataTalks.Club. Then this week I did a webinar about it where I gave a more detailed overview of the system.

3) Building a full-stack application with AI

I hosted the second workshop for AI Dev Tools Zoomcamp. In the first, we talked about specification-driven development, loop and graph engineering. In the second, I showed how to build an end-to-end application with frontend, backend, and a database. As an example, we built an application for system design interviews.

I’m hosting the notes on Substack, so you can read them here:

There will be more workshops and events for AI Dev Tools Zoomcamp:

• August 10: Test, Containerize, and Deploy an AI-Assisted App

• August 18: DevOps and Observability for AI-Built Apps

• August 24: Pre-Course Live Q&A

• August 31: Course launch

I will publish the materials from all these workshops as Substack articles too.

4) Inference Engineering - AI Shipping Labs book club

We are also starting a book club at AI Shipping Labs.

Our first book is Inference Engineering by Philip Kiely. It covers model architecture, hardware, serving software, optimization, cost, reliability, and running inference systems in production. The digital edition is available for free.

The kickoff is on August 10 (Monday) at 5:00 PM CET. We will review the book, discuss the reading cadence, and decide together how the sessions and discussions should work.

Join the kickoff

5) Stock Markets Analytics Zoomcamp

In DataTalks.Club, we’re preparing the next cohort of Stock Markets Analytics Zoomcamp. It starts on August 24.

The course provides a gentle introduction to stock market trading, Python programming in Google Colab, analytics, and data visualization.

Register for the course

Edited by Valeriia Kuka

In the first article, I wrote about turning an idea into a specification. The specification is only the beginning. In this article, we will create an end-to-end application from scratch. We will cover both frontend and backend, and add a database.

All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants (this article)

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4 and 5: TBA

We will create an application for system-design interviews.

As an interviewer, you create a session and share the link with the interviewee. They can create diagrams in the app, and you see the changes in real time.

Both frontends connect to the same room through WebSockets, so the changes appear in both sessions simultaneously. The FastAPI backend handles those events and saves them to SQLite.

You can find the code in the interview-canvas-share repository.

It’s based on my full-day workshop I did for AI Shipping Labs: Build and Deploy a Full-Stack App with AI Coding Assistants.

Overview

Like in the previous article, we start with an idea. As this is only an idea, we turn it into a specification using ChatGPT.

From there, we start building:

1. Generate a frontend that uses mocked backend calls.

2. Establish backend-frontend contract by creating OpenAPI specifications from the frontend service layer.

3. Implement the FastAPI backend, connect it to the frontend, and test the application.

4. Add persistence with SQLite and SQLAlchemy.

At each stage we get something concrete that we can test:

• First we define the specification and make sure it reflects what we want to build.

• After that, we use the specs to create a frontend prototype that we can interact with. We mock backend calls so we can test the idea.

• Then we connect the frontend to the backend. We start with an in-memory store to make sure the frontend-backend connection works.

• Finally, we add a database so the application keeps its state after a restart.

At the end, we have a working local application that is ready for deployment.

Start with a specification

Before building the frontend, we need to describe the application precisely. If we don’t do it, we will get something that works but we don’t need.

For our application, we need to specify:

• who creates an interview session

• how a candidate joins it

• which components they can place on the canvas

• how both people see changes in real time

This is called “Specification-Driven Development”. We don’t spend a lot of time here because it was the focus of the previous article AI-Native Development: Specifications, Loop and Graph Engineering.

For creating the specification, I always use ChatGPT in dictation mode. Give the assistant as much information as possible at this stage.

You can see the result here.

Frontend First

Now we have the specification, but it’s only text. There are multiple options of what you can do next:

• Focus on the database layer, define the entities, and work all the way up through backend to frontend

• Alternatively, you can start with specifying OpenAPI and define how backend and frontend interact and build both independently from there

• Or you can focus on the frontend first and then build the rest

All these approaches make sense and have their pros and cons.

For simple projects that you’re building alone, I recommend starting with frontend. You can quickly judge if you’re moving in the right direction, and if it solves your problem or not.

Treat it as a way to test your idea and your specification, and do it before you build the rest of the application.

For this step I like using Lovable, which generates a React app from one prompt. Claude Design, Replit, v0, and Bolt are similar tools that you can use instead.

Alternatively, you can start with coding assistants directly and ask them to create a React app (or whatever technology you want).

I use Lovable because it creates really nice designs. Also, I’m not a frontend engineer and I don’t know much about the frontend world. Lovable makes technology choices for me that I know will make sense, while a coding agent can select something randomly.

Now open any tool of your choice and give it this prompt:

Create a system design interview application. 

[Paste the ChatGPT-generated specification here.]

Centralize every backend call in one services layer, and create a mock
implementation of it so the whole app runs without a real backend.

Add tests.

This part is quite important:

Centralize every backend call in one services layer, and create a mock implementation of it so the whole app runs without a real backend.

Without it, the agent can do something arbitrary, but here we explicitly say that we want a mock service. Later, it will become the single point of integration of our frontend with backend. And because we mock it, it will work from the beginning.

Lovable creates a React app in TypeScript. We can interact with it in the Lovable interface.

Next, save it to GitHub. If you use Lovable:

• Select the plus icon in the bottom-left corner

• Connect your GitHub account.

• Lovable creates a private repository. I usually change it to public.

Move the Frontend into the Project

Next, we can clone the repository locally.

For the rest of the project, I want this setup:

/backend     # backend application and its tests
/docs        # supporting documentation
/frontend    # frontend application
AGENTS.md    # instructions for coding agents
openapi.yaml # API agreement

So let’s create these folders and move all the frontend stuff to “frontend”, and the specification we created to “/docs/spec.md”.

After we re-arranged the files, commit the changes.

At this point, you should also be able to run the application locally and test that things work the way you want. If they don’t, use a coding agent to fix it.

To run a project you exported from Lovable:

cd frontend
npm i
npm run dev

AGENTS.md

We already discussed the importance of AGENTS.md in the first article.

Let’s create one for this project too. Place it in the repository root:

for backend, use uv for dependency management. a few useful commands:

uv sync
uv add <PACKAGE-NAME>
uv run python <PYTHON-FILE>

regularly commit code to git 

This is only the starting point and it will change as your project grows.

OpenAPI Specifications

When creating frontend, we asked the AI assistant to put everything in a centralized service layer. Later we will replace it with actual calls to backend.

But now we should define the specification - the agreement between frontend and backend. We will use OpenAPI for that.

This specification gives explicit information about the endpoints, paths, request bodies, response bodies, and authentication rules.

Let’s create it:

Read the frontend's API client in frontend/

Create openapi.yaml at the repository root.

Specify the backend this frontend expects: every endpoint, method, path, request body, response body, and which endpoints need authentication.

You can skip this step. But I wouldn’t recommend it.

It takes a few minutes, but has many benefits. The backend gets a precise target instead of being inferred from frontend code. Not only we save tokens this way, but also get a clear picture of what exactly the backend needs.

You can see the result in openapi.yaml.

The Backend

Now we have the OpenAPI specs and we can use this file to create the backend. We will use Python and FastAPI for that. But you can choose any technology you want.

When I start with frontend, I use a mocked backend and then replace it with a real one. In the same way, for FastAPI backends, I start with a mocked database, and then later change it.

I do that because I want to make sure the frontend-backend connection works, and until it’s smooth, I don’t worry about the persistence.

Let’s ask the coding assistant to implement it:

Build a FastAPI backend in backend/ that implements the openapi.yaml spec.

Use an in-memory store and seed it with data so the frontend has something to show. Add authentication with hashed passwords and bearer tokens for the endpoints that need it.

Split the code into modules - routers, models, store, auth

Write tests

After 5-10 minutes, you will have the backend ready.

Makefile

When it’s ready, you can run the application. Normally, for FastAPI, the command is something like that:

cd backend
uv run uvicorn backend.main:app --reload --port 8091

I’ll use port 8091 but you can replace it with any other port you like.

But for me it’s always hard to remember these commands, so I ask the coding assistant to create a Makefile:

Create a Makefile so I can easily run it.

Then running it is as simple as

make run

When it’s running, open http://localhost:8091/docs in the browser. You will see the OpenAPI specification of the implemented backend.

We can compare it with the actual specs. From this point we no longer need the original openapi.yaml - the one that’s generated by FastAPI is enough.

Connecting Frontend and Backend

The backend is runnable now, so we can connect it to the frontend:

Switch the frontend to use the real backend client. 

It most likely won’t work from the first try. You will hist CORS errors and probably some others.

It’s also time to test the application thoroughly. Open two browser windows:

• Create a session in one and get the join link

• Use the join link in the second window

• Interact with the system in the second window and see the changes being propagated to the first one.

If something doesn’t work, ask the agent to fix it.

Database

At this point we have a working application with frontend and backend. They can connect to each other.

But because we don’t have a real database yet, when the backend is restarted, all the data is lost.

Let’s fix it:

Replace the in-memory store with a database. Use SQLite and SQLAlchemy.
Use an environment variable to configure which DB the server should connect to.
Make it database-agnostic - later we will add support for other databases (e.g. Postgres).

There are a few important parts in this prompt:

• First, I ask it to use SQLite. It’s a very nice and lightweight database for testing the application locally.

• At some point, you’ll want to switch to a different database for production, e.g. Postgres. Using SQLAlchemy makes this process very simple.

• But we also mention that explicitly so the agent doesn’t use any SQLite specific features in the first version.

After 5-10 minutes you will have a fully working end-to-end application that’s running locally. Now you can test it: create multiple sessions and see that changes in one are reflected in another. Stop your backend server, start again, and check that all the changes are still there.

If something doesn’t work, ask the coding assistant to fix it.

Ready for Deployment

Now we have a working application:

• we turned an idea into a specification

• created frontend

• defined the API contract

• implemented the backend from the contract

• connected them

• added SQLite for persistence

But this application is only working locally. Next, we need to deploy it. That’s something we will do in the next article. We will take what we developed here and turn it into a deployed service.

We will:

• containerize the frontend and backend

• add integration tests for the full application flow

• set up CI to run the checks automatically

• run Postgres with Docker Compose

• add database migrations

• deploy the application to a public environment

• set up CD to deploy changes after the checks pass

Step-by-Step Approach

In this workshop, we created a working application using the step-by-step approach. At the end of each step, we had something functioning:

• Clear speciation

• Working frontend prototype with mock backend

• Properly integrated backend with mock database

• Persistence with SQLite

For each step, we started from a new session and drove the agent to produce a working version.

But we did it one prompt at a time. After the first version is working, it’s time to introduce the development process. We talked about it in the first article. This process will help you application continue working properly as it grows bigger.

Next in the Series

The remaining modules will build on the same app:

• Deployment: we will deploy the application to the cloud and set up CI/CD

• DevOps and Observability: we will add observability to the application and use AI as the first line of support when the application breaks

You can find the entire course in the course repository. It’s free.

⭐ Star the course repo

This article is based on the the workshop I did for AI Shipping Labs: Build and Deploy a Full-Stack App with AI Coding Assistants. In that workshop I created the snake game, deployed it with AWS and configured CI/CD to make sure tests run on every push. Check it out!

There are two components:

• FAQ curation: building the FAQ dataset from student contributions, Slack discussions, and YouTube transcripts. It lives in DataTalksClub/faq.

• FAQ retrieval: the Slack bot that answers students’ questions using this dataset. It lives in DataTalksClub/faq-assistant and runs on AWS Lambda.

In this article, I want to describe these components and the entire project in more detail:

• the data sources and how they become the FAQ dataset

• the architecture and how data flows through it

• the decisions I made and why I made them

• how I evaluate each part of the system

ZoomcampQABot

Yes, I already wrote about the FAQ assistant in From Google Docs to an Automated FAQ System for DataTalks.Club Courses.

In that article, I described the ZoomcampQABot bot developed by Alex Litvinov. When it occasionally breaks down, I have to pull in Alex and ask him to fix the issue. Sometimes it would be handling an expired key, or sometimes it’s fixing a small bug.

Two months ago his OpenAI account ran out of money, so the bot stopped working. I had to ping Alex again. I always felt bad that he uses his own money to pay for the project, so I offered to take it over and run it in the DataTalks.Club infra. Typically he’d decline it, but this time he agreed.

This is the setup ZoomcampQABot uses:

• OpenAI for generating the answers

• Fly.io for hosting

• HuggingFace for embeddings

• Milvus for vector search, running on Zilliz Cloud in production, with four collections and query engines

• Cohere for reranking

• Upstash Redis as an embeddings cache

• LangSmith for feedback logging

The data comes from multiple sources:

• the FAQ dataset

• Slack history

• Course GitHub repository data

• YouTube transcripts

Each course has a separate ingest entrypoint on its own cron job in GitHub Actions. YouTube videos are pulled in manually.

This setup makes a lot of sense, but I couldn’t port it easily to the DataTalks.Club infra. I wanted to use it in combination with the Au-Tomator Slack Bot that I described in Building and Maintaining a Slack Moderation Bot for an 88k-Member Community.

The Au-Tomator bot runs on Serverless - on AWS Lambda. It’s been running for years now and I never had to pay for it: it was always under the free tier for AWS Lambda usage. And when I have to pay for it, I expect it to be minimal. So I wanted to run the FAQ bot on Serverless too.

So I had to redesign the Slack bot. I’ll describe the result later in the article, but now I’ll cover the main data source: the FAQ dataset. This dataset influenced many architectural decisions in the assistant, so I want to talk about it first.

Part 1: The FAQ Dataset

The FAQ dataset is open for everyone. We host the website via GitHub pages and the data is in the DataTalksClub/faq repository.

Previously it lived in a bunch of Google Docs. It was convenient, but this approach had a few problems:

• It was frequently vandalized, so I had to manually roll the documents back.

• I wanted to automate the curation with AI, but using AI in Google Docs is not trivial.

Eventually I moved the data to a bunch of markdown files. I described the migration process in From Google Docs to an Automated FAQ System for DataTalks.Club Courses.

Now with AI assistants I spend a lot less time maintaining this dataset, and I can keep it clean and up-to-date.

There are multiple sources of questions for the FAQ dataset:

• The questions that people contributed via GitHub

• Discussions in Slack

• Q&A videos on YouTube

FAQ Automation: User-Contributed FAQ Records

Anyone can contribute to the FAQ dataset:

• You submit an issue, specifying the question, the course and your answer.

• A GitHub Actions workflow indexes the entire dataset with minsearch.

• It then searches twice - on the question alone, and on the question and answer together - and combines the two results with reciprocal rank fusion.

• It sends the results to OpenAI, which returns a structured decision: NEW, UPDATE, DUPLICATE or WRONG_COURSE.

• For NEW or UPDATE, it opens a pull request.

• For DUPLICATE or WRONG_COURSE, it closes the issue.

FAQ Automation Evaluations

When the first version of the script was created, we didn’t have any evaluation set. I mostly relied on my gut feeling. But as more people started using it, the number of incorrect decisions increased. I needed to improve it. Naturally, I didn’t want to fix it blindly and just hope for the best: I needed a proper evaluation framework.

For creating it, I used those incorrect decisions. I analyzed the issues where I needed to manually adjust the result, and focused on the cases that were sufficiently different from each other.

Not all decisions made by the script are equal. If the agent makes a mistake in a NEW or UPDATE decision, it’s easy to correct. I use AI assistants to help me with that.

But if the decision is DUPLICATE or WRONG_COURSE, the PR will never be created, and the issue will be closed. In this case, I will not even have a chance to review them. Thus, a mistake in this case is way more expensive. Which means, I had to give these cases more importance in the eval set to make sure that these cases don’t happen often.

There’s also a problem with collecting the evals data from historical decisions. The FAQ data is constantly changing, so we have to use a “leave-one-out” setup.

When the script decides that something is NEW and we later use it in the evals, it’s no longer NEW: the item was already merged into the dataset. So if we run it with the same issue again, it will say DUPLICATE instead of NEW.

In order to properly test the NEW cases, we therefore need to remove the record from our dataset. So if we have 200 records, and the record D is the one we added, we remove the D and test this case against the remaining 199 records.

When I analyzed all my past corrections, the most common error turned out to be incorrect section placement. For example, instead of placing a record about projects in the “Project” section, it would place it in “General”. My eval set also tests for that.

Right now I have 61 cases:

• 38 expecting NEW

• 10 expecting DUPLICATE

• 7 expecting WRONG_COURSE

• 4 not expecting WRONG_COURSE

• 2 expecting UPDATE

In addition to testing the whole flow, I have a retrieval-only evaluation set. I use this part to test that duplicate detection works correctly. This part is less interesting, so I’ll skip it. You can read more about it in the project’s README.

Bulk Reviewing

I usually let the PRs accumulate and process them every two weeks.

For that, I use AI assistants. I have a custom skill that lets me go through each PR and merge it as is, or fix it if it’s not correct.

If I come across an interesting error, I ask the assistant to add it to our evals set. I don’t try to fix it immediately. I wait till we have a few more cases, then I ask the assistant to see if there are simple ways to fix them.

If some cases can’t be fixed easily, I don’t sweat over it. I just let it sit in the evals with a FAIL status. I don’t want the system prompt to grow too large and handle all the corner cases. But I do want to know that these corner cases exist.

Other Sources: Slack and YouTube

Our Slack is very active. Course participants are asking questions and helping each other. In many cases, these discussions are worth saving in the FAQ dataset.

For that I regularly go through all the Slack threads. I ask my AI assistant to go through each thread, and if something is useful, it’s saved as a new record.

Also, for each course I run a few live YouTube sessions, for example:

• pre-course Q&A session

• course launch streams

• occasional office hours

I get the transcript and use AI assistant to extract potential Q&A candidates. If there’s something new, it also goes to the dataset.

Part 1 Overview

The main focus of the first part is the dataset curation.

I review all the PRs that our FAQ automation creates in batches, and use AI to turn Slack discussions and YouTube videos into focused FAQ records.

Now let’s go back to the retrieval side. This FAQ dataset is used as the main data source for the Slack bot.

Part 2: Slack Bot

When you mention @Au-Tomator (my bot) or @ZoomcampQABot from Alex, both would perform RAG:

• Use search to fetch the candidate FAQ records

• Pass them to OpenAI

• Return the answer from the LLM and post it in the thread as a reply

ZoomcampQABot has a lot of moving parts though. Most of them are not straightforward to deploy to the serverless environment. So I decided to simplify it.

The first thing I dropped was indexing Slack threads and YouTube videos. I already extract this information and put it directly to the FAQ dataset, so this part is not needed anymore.

Second, I removed the vector database. I know that it improves retrieval, but at the cost of having to maintain more infra. I want to keep the setup very lean. If the bot is sometimes not right, it’s not a huge problem: I and other community members can simply correct it in Slack.

Zero-Dependency Serverless Deployment

My to-go search library for text search is minsearch. I wrote about it in Minsearch: The Small Search Library Behind My RAG Workshops and Courses.

However, deploying minsearch to AWS Lambda isn’t trivial. When I created this library, my focus was on teaching the concepts, not on deploying it to serverless. Internally, it users Scikit-Learn and Pandas, and these libraries in turn pull in numpy and scipy. Normally, everyone working with data already has these libraries in their standard setup. But for Lambda, they are too heavy and take well over the 50 MB limit.

So I decided to replace minsearch with a zero-dependency search engine written entirely in pure-Python. I called it zerosearch. It has only one optional dependency - stemlite, which is a stemmer I use across all my small search libraries (zerosearch, minsearch and SQLiteSearch).

Another challenge was Pydantic. We typically use for structured output, and it works fine in usual environments, but it’s not trivial to get it right for Lambda. It has a compiled Rust core, and you need to make sure the binaries that you send package into your Lambda deployment are compatible. Plus, loading the binaries core and building the model schemas adds to the cold start time.

So I removed it along with requests and did all the calls to OpenAI using only the standard Python library.

Keeping the Index Fresh

Then I started thinking about what to do with updating the index. If a record in the FAQ changes, the search needs to reflect it. Alex solved that by pulling in data using a daily cron job. But I wanted to do this as soon as the data in FAQ changes.

Almost all my projects have a CI/CD workflow: when I push to main, the code change is propagated to a live environment. I usually use it for code changes, but for this project I did the same for data changes too.

When a push happens, I re-build the whole index, and push it together with the source code in Lambda. It’s quite small: it’s 8 MB total.

Once it’s deployed, the lambda loads the local updated index, and can serve the fresh data.

FAQ Assistant Retrieval Evaluation

I replaced vector search with text search, and used zerosearch to make it fast and work well in Serverless. But it would still lose to vector search in terms of retrieval quality.

How can I make sure I squeeze the absolute best from text search? For that I needed to have a proper evaluation set.

I built the ground truth dataset from real Slack threads:

• Get a Slack dump with all the course channels

• Take all the Slack threads (9,900 of them)

• Filter them down to keep actual questions

At the end, I had a sample of 130 records:

• 60 for Data Engineering Zoomcamp

• 40 Stock Markets Analytics

• 30 AI Dev Tools

For each of the questions, I found and marked the correct documents in the index. Then I evaluated the hit-rate and MRR at k=1,3,5.

Once I had this dataset, I could start experimenting with search optimization.

I tried multiple options:

• Taking raw question as is

• Keyword expansion

• Different variants of query rewriting

The best option is to take a Slack message and turn it into a bunch of keywords while preserving exact error messages, tool names, commands and filenames.

For example, it turns this Slack message:

Does anyone know why minsearch fails when I run uv run python index.py? I get KeyError: 'course' after renaming documents.json to faq.json.

into these keywords:

minsearch FAQ indexing course field "KeyError: 'course'" "uv run python index.py" index.py documents.json faq.json

For query rewriting I use gpt-4o-mini because it’s the fastest and cheapest model, but for the actual generation I rely on gpt-5.4-mini.

FAQ Assistant Generation Evaluation

In the previous part I explained how I evaluate search. I also evaluate generation.

For the evaluation dataset, I collect feedback from Slack. When a bot answers a question, and somebody corrects it or add something extra, it means that the bot couldn’t answer the question properly. So we can include this question in the evals.

So far it’s not large and has a few cases like:

• the answer from the bot is incomplete

• the answer is incorrect

• the search doesn’t find anything

Most of the time the way to fix these problems is not tuning the prompt, but fixing the FAQ dataset itself. I check why a record wasn’t retrieved, or why the answer didn’t contain the correct information, and then see how I should update the FAQ records to make it work next time.

Then I’d fix the record, re-run evaluation, and re-deploy the bot.

Deploying with Au-Tomator

I use Au-Tomator to help me with Slack management. So I wanted to use the existing setup, not create a new Slack bot.

I already have two Lambdas for Au-Tomator:

• The router: routes and filters Slack events. It also makes sure we acknowledge the request from Slack quickly within 3 seconds (a requirement from Slack) and forward the request to the next lambda.

• The automator: the bot itself, it handles all the reactions.

And I added the third one:

• The FAQ assistant: rewrites the question, searches the index, and generates the answer

Now when you mention the bot, it first goes to Au-Tomator, and then Au-Tomator sends it to the FAQ assistant. The assistant gets in the question, and send back the answers. Posting the response to Slack is handled by Au-Tomator.

As a bonus, now with this setup, I can trigger the FAQ assistant with :faq: reaction. Previously this reaction would only post a message to the Slack thread saying “Go check the FAQ”.

Now it actually checks the FAQ and generates the answer.

Other Retrieval Sources

I don’t only search the FAQ dataset. I also index the DataTalks.Club documentation and the Markdown files from each course repository.

I combine all three sources into one search index and package it with the FAQ assistant Lambda. In Slack, either a mention or a :faq: reaction triggers Au-Tomator. It sends the question to the assistant and posts the answer back to the thread.

The Full Picture

In this article, we discussed the two parts of the FAQ system:

• The FAQ dataset, where we primarily talked about the dataset curation

• The FAQ assistant, which is a Slack bot deployed via AWS Lambda

The FAQ assistant reads data from the FAQ dataset and helps students find answers to their questions faster.

This is how the entire system looks like:

• I turn GitHub issues, useful Slack discussions, and live-session transcripts into reviewed FAQ records.

• I combine the FAQ, docs website, and course-repository Markdown into a zerosearch index. The index is rebuilt every time we push to the FAQ repo.

• A mention or :faq: reaction triggers Au-Tomator that sends the question to the FAQ assistant and publishes the answer in Slack.

• I use wrong and missing answers as evaluation cases and FAQ updates, so the next index contains the correction.

Curation over Complexity

This year, I re-recorded the entire LLM Zoomcamp course from scratch. There, we use the FAQ dataset as the main running example.

When I introduce the FAQ dataset, and we do the first RAG example, I mention that the example looks very simple. But the reason it looks so simple is because the complex work has already happened: the dataset was already cleaned and prepared. In reality, in your AI projects, most of the time will go into data cleaning and data curation.

In this article, I tried to explain how exactly I did it.

We can always have a better model, use a vector database, or a faster machine, but none of that will help if your data is missing, outdated or unclear.

When the assistant gives a bad answer, it’s not because of the tech stack, but because of the data. So the most useful fix to make it better is to improve the dataset.

In the FAQ case, I collect the feedback from real users, use it to improve the underlying dataset and thus improve the system itself:

1. Students contribute issues to the FAQ dataset.

2. Useful discussions in Slack become curated FAQ records.

3. The updated dataset is indexed and deployed with the Lambda.

4. Au-Tomator uses it to answer future questions.

5. Incorrect or incomplete answers become evaluation cases and help improve the system.

Because of this loop, I’m comfortable with a simpler architecture that I described in this article.

Since January 2026 I scrape AI Engineering jobs monthly for AI Engineering Field Guide. So far we have 4,894 descriptions, and among them, the number of FDE-related postings increased from 28 in January to 108 in July.

According to this dataset, the AI Engineering job market doubled in these 6 months. But the number of FDE listings grew 4 times! That is, it’s growing at twice the overall AI Engineering market rate.

In this article, I analyze 113 FDE postings to understand what this work involves. Since our dataset focuses on AI Engineering roles, it’s limited to FDEs working in AI.

We discuss

• how companies define the role

• which skills they expect

• how FDE differs from applied AI engineering, solutions engineering, and consulting

At the end of the article, we also include a quick self-assessment checklist to help you determine whether the FDE role is for you.

Forward-Deployed Engineer

Palantir created the forward-deployed engineering model in 2006. They had a general platform, but making it useful required adapting it to each customer’s data, workflows, and infrastructure.

To solve it, they placed their engineers in their customer’s teams. FDEs identified their problem, configured Palantir’s platforms, built integrations and custom components. They also supported product development: when the same problem appeared across deployments, Palantir would integrate this as a feature to the main platform.

Palantir’s FDE model applies directly to AI in enterprises today. General-purpose models must be adapted to each customer’s data, systems, and workflows before they can deliver value in production.

There’s a “deployment gap” - the gap between a prototype and a working customer system. It exists because many enterprises don’t have the engineering capacity or product knowledge to close it themselves. FDEs fill that gap.

A forward-deployed engineer is a customer-facing software engineer who turns an ambiguous business problem into a working production system. They work alongside the customer to define the problem, design the solution, adapt and integrate the company’s product, build any missing software, deploy and debug the system, and feed recurring customer needs back into the core product.

FDE’s responsibilities: our analysis

We collected a dataset of 113 FDE jobs descriptions from January to July, and then analyzed the expected responsibilities.

They include:

• Direct client engagement (90% of postings)

• Building and deploying production system (87% of postings)

• System, API, and data integration (62% of postings)

• Project scoping, discovery, and requirements gathering (51% of postings)

• System testing, evaluation, and monitoring (41% of postings)

• Channeling customer feedback back into the core product roadmap (39% of postings)

• Developing prototypes, demos, and proofs of concept (25% of postings)

• Travel or client on-site presence (10% of postings)

If we compare it with the rest of the dataset, only 21% of the other AI roles are expected to interact with the clients.

Here’s how we can describe the FDE’s main responsibilities

1) Understand the customer’s problem

FDEs usually begin with a desired business outcome. They work with users, domain experts, and engineering teams to understand the existing workflow, examine the available data, and identify technical constraints. They then translate what they learn into deployment requirements and success criteria.

2) Build the production AI application

Once the problem is clear, the FDE writes the software to solve it, handling conventional production engineering tasks like authentication, permissions, testing, error handling, deployment, logging, and observability.

3) Connect the application to the customer’s systems

FDEs also work on integration, and this need is present in 62% of job postings. They build connectors, data pipelines, transform data, configure access controls, integrate APIs, and deploy applications in the customer’s cloud.

4) Deploy, evaluate, and debug the system

Production deployment expands the FDE’s role beyond implementation. The engineer must define evaluation criteria for realistic conditions and use logs, traces, metrics, and user feedback to identify failures post-release. FDE continue to be involved in the project after deployment.

5) Bring field findings back to the product

Some deployment issues are unique to individual customers, while others highlight common needs. FDEs identify these recurring problems and relay customer feedback to the product team, with 39% of postings referencing this contribution. This process connects customer delivery and product development, allowing the FDE to adapt the product for one customer and identify reusable capabilities for others.

Skills companies ask for

It’s natural that with these responsibilities FDEs are expected to have a broad mix of skills.

Software engineering

The technology stack varies from job to job, but all of the FDEs are expected to deploy production software.

Python is the most frequently mentioned programming language, appearing in 89% of job postings. Other commonly referenced languages and frameworks include:

• TypeScript: 29%

• SQL: 23%

• React: 14%

Applied AI

FDEs are expected to build applications with existing models, not train them. The most frequently mentioned gen AI technologies are:

• Prompt engineering: 56%

• RAG: 50%

• LangChain: 32%

• AI agents: 30%

• Vector databases: 23%

• Agentic workflows: 19%

Cloud and deployment

In the job descriptions, the three major cloud providers appear at similar rates:

• AWS: 40%

• GCP: 36%

• Azure: 32%

Companies also seek candidates with experience in the following areas:

• Docker: 35%

• Kubernetes: 31%

• CI/CD: 27%

Customer-facing work

The technical requirements represent only part of the role. With 92% of FDE postings identified as customer-facing, engineers must also navigate technical discovery and tackle evolving requirements.

FDEs need to be able to:

• Understand existing business workflows

• Translate user needs into technical requirements

• Communicate constraints and tradeoffs to stakeholders

• Collaborate with domain specialists and customer engineering teams

• Adapt implementations based on new information

• Differentiate customer-specific integration from reusable product gaps

How the role differs from related titles

FDEs have a broad set of responsibilities, so it’s natural that the role overlaps with many others.

The main difference is the type of code FDEs write, how long they work with one customer, and the involvement in production deployment.

An FDE builds and deploys systems for specific customers, writes production code, and stays involved until the deployment works.

• Consultant: Works within a defined engagement and usually leaves after delivering recommendations or an implementation. FDE: Stays through production deployment, owns more of the outcome, and feeds customer needs back into the product.

• Solutions or sales engineer: Focuses on demos, technical validation, and supporting multiple deals before handing off implementation. FDE: Writes production code, owns deployment, and works more deeply with fewer customers.

• AI engineer: Builds production AI systems, often with more focus on the underlying product or platform. FDE: Places greater emphasis on customer integration, deployment, and close collaboration.

Who is a good fit

FDEs come from different technical backgrounds:

• AI engineer: Strong in LLM applications, RAG, agents, evaluation, and production AI. Gap: Customer discovery, enterprise integration, and ownership of customer-specific deployments.

• Software or product engineer: Strong in APIs, backend or full-stack development, testing, and shipping software. Gap: Applied AI, evaluation, data pipelines, and customer discovery.

• Data engineer: Strong in pipelines, integrations, distributed systems, cloud platforms, and production operations. Gap: Full-stack development, LLM systems, evaluation, and product discovery.

• ML engineer: Strong in model deployment, monitoring, evaluation, and production ML. Gap: Broader product engineering, customer-specific integrations, and direct customer work.

• Platform, DevOps, or SRE engineer: Strong in infrastructure, CI/CD, security, reliability, observability, and debugging. Gap: Application development, applied AI, customer discovery, and workflow analysis.

• Solutions engineer or technical consultant: Strong in customer discovery, solution design, technical communication, and varied customer environments. Gap: Production software development, long-term ownership, and applied AI evaluation.

• Data scientist: Strong in experimentation, model evaluation, analysis, domain knowledge, and stakeholder communication. Gap: Maintainable application development, systems integration, infrastructure, and deployment.

For our dataset and set of responsibilities, AI engineering is the closest technical background because the tools and systems often overlap. (But our data is probably biased because in our scrapes we focus on AI Engineering roles.)

The strongest fit is someone who can combine three capabilities:

• Build and operate production software across application code, data, integrations, and infrastructure.

• Work directly with customers to turn unclear problems into technical requirements and measurable outcomes.

• Remain responsible through deployment, evaluation, and production debugging.

Is this role for you?

Forward-deployed engineering may suit you if you want to be a hands-on engineer while working closely with customers and owning projects through production.

The role may be less suitable if you prefer narrow technical ownership, stable specifications, limited customer interaction, or handing projects off before production.

Before applying, ask yourself:

• Do I want regular customer interaction to be part of my engineering work?

• Can I make progress when requirements are incomplete or changing?

• Am I comfortable working across several parts of the stack?

• Do I want to own deployment and production results, not only implementation?

• Can I explain technical constraints to non-specialists?

• Am I willing to work within the customer’s existing systems and infrastructure?

• Do I enjoy turning recurring customer problems into reusable product improvements?

Conclusion

FDE is still an inconsistent job title, but we can see a clear focus on customers and production.

Compared to traditional AI engineering, FDEs place a stronger, more consistent emphasis on customer discovery, systems integration, and deployment responsibilities. For engineers who want that combination, forward-deployed engineering provides a path to remain technical while owning a broader part of the production outcome.

Edited by Valeriia Kuka.

I start with AI-native developer workflows: how to turn an idea into a specification, give coding agents the right context, and verify what they build.

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering (this article)

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4 and 5: TBA

Coding agents now write code faster than I can read it.

When we give an agent a task, it can quickly implement it. But if a task is vague, the agent fills the gaps with its own assumptions. A weak agent that misunderstands us writes fifty lines of broken code. A strong agent that misunderstands us creates eight files, wires them together, and adds tests that pass. The code works, but it isn’t what we needed.

We no longer spend most of our time typing. We spend it saying precisely what we want and checking what came back.

In this article, I show how to make the request specific so the agents don’t need to guess. Then we decompose the request into tasks and assign them to a product manager, a software engineer, and a QA engineer. Finally, we implement all the tasks in a backlog through a loop.

We cover topics like:

• Spec-driven development

• Context engineering

• Loop engineering

• Graph engineering

We’ll use a deliberately vague project idea: a tool for weekly feedback for projects. It doesn’t say who gives the feedback, who receives it, or what “projects” means.

You can see the final result in the retroloop repository.

Specs before code

We need to understand what we want to build before the agent produces the first line of code. We have to think it through in detail and give explicit instructions. If we do that, the agent will produce something close to what we want.

We call this “spec-driven development”. We start with the specification, make sure it aligns with our vision, and only then write the code from it.

In our example, “a tool for weekly feedback for projects”, many things aren’t clear.

• Who are the users for this tool?

• What problem will they solve?

• How are they going to use it?

If we don’t specify these things and give the idea directly to a coding agent, it’ll fill the gaps.

When I asked Claude Code to implement this project, the only prompt I gave was “a tool for weekly feedback for projects”. It came up with weekly-feedback, a command-line tool for tracking weekly project status. It records wins, issues, blockers, and next steps. It also created documentation and covered the app with 62 tests, all of which passed.

It all works perfectly, but that’s far from what I needed. I needed a web tool for a team retrospective that captures feedback from teams in the form of “Start/Stop/Continue”. It was my fault for not telling Claude that.

Start in a chat assistant

Instead of giving a prompt directly to the coding assistant, I start in a chat application and talk the idea through. I use ChatGPT in dictation mode for this.

I begin with the same vague idea:

I want to build a tool for weekly feedback for projects.

Help me set the scope for this project precisely. I want to brainstorm with you
and understand how the tool should work. Give me options.

Ask me one question at a time and keep your output short.

This way, we can use AI as our brainstorming partner and find out precisely what we want:

• Who contributes feedback? All team members.

• What format do they use? Start/Stop/Continue.

• Is the feedback anonymous? Names appear by default, but contributors can choose to remain anonymous.

• What can people see before the reveal? Only their own cards.

• How does the facilitator reveal the feedback? All cards appear at the same time.

• What happens next? The team clusters the cards, then each person casts three votes for the topics to discuss and can give multiple votes to one topic.

• What does the team record? Decisions and action items from the discussion.

• Can the facilitator add a recording? They can upload audio, video, or a transcript after the meeting, but built-in recording isn’t part of the first version.

When we finish, I ask for a file with all the specifications:

Save everything to a markdown file that I can download.

Download the file and save it as plan.md.

Bootstrapping a project

Create a project from this specification:

mkdir project-name
cd project-name

git init

Copy the plan.md file:

mkdir -p _docs
mv ~/Downloads/plan.md _docs/plan.md
git add _docs/plan.md
git commit -m “Add project plan”

You can find the plan.md file from this project in the Retroloop repository.

I try to commit as often as possible, after every meaningful decision. With those commits, we can review what the agent changed. If something isn’t working well, we can easily return to the last good state.

Choose the stack and architecture

During the brainstorming session, we didn’t choose the tech stack.

Ask the coding agent to come up with several options:

Read _docs/plan.md. Propose multiple options for the tech stack and
explain each option.

Don't write code yet.

It proposes multiple options and explains the tradeoffs of each one.

I choose Django because I know it well enough to review. In your case, you can select any technology you’re comfortable with. It’s also okay not to have a preference and to let the agent select what it thinks will work best.

Turn the decisions into a backlog

Now that we’ve settled on the tech stack, we can ask the agent to decompose the specifications into a backlog with tasks:

Create a backlog with tasks in _docs/tasks.md.

Each task should be small enough to finish in one session, and
independent enough that I could hand it to someone who has not read
the others.

Use this template for each task:

## <number>. <title>
Goal: <one line>
Description: <two or three sentences on what the task involves>

The first task should be setting up an empty project with a passing test.

Don't write code yet.

It created these tasks.md.

Review the tasks and ask the agent to merge tasks that are too small or split tasks that don’t fit into one session. We want to create an MVP - the first version of the app. If something is out of scope for your vision of the MVP, remove it.

When we’re happy with the tasks, move them to a task tracker. I use GitHub issues for that.

Ask the agent to do it:

Create a public GitHub repo for this project.
Move each task from _docs/tasks.md into a GitHub issue.

For that to work, we need the gh CLI tool authenticated and the repo connected to the GitHub remote.

From this point on, GitHub issues are the canonical tasks and the only active backlog. We no longer need _docs/tasks.md.

Context engineering

The repository has a backlog now. When we start a new session, however, the agent doesn’t know which task we mean. It must figure that out every time.

These details go in AGENTS.md, which coding agents like Codex or OpenCode read when they start a new session.

Claude Code reads CLAUDE.md, while I use multiple coding assistants and want my workflow to be tool-agnostic.

That’s why I also create CLAUDE.md with a single line:

@AGENTS.md

It tells Claude to read AGENTS.md.

This is called context engineering. With prompt engineering, we control one message in one session. With context engineering, we control what agents know when they start a new session and what information they can find while they work. We include useful facts and working rules they would otherwise have to rediscover.

AGENTS.md

To make this context available in every new session, create AGENTS.md:

Commands

- `uv sync` - install dependencies
- `uv run pytest` - the whole suite
- `uv run pytest tests/test_home.py` - one test file

Rules

- Dependencies are added in `pyproject.toml`. Do not add one without
  asking

The other documents

In addition to AGENTS.md, I usually have a few other Markdown documents in my projects.

The main one is process.md, which I use to describe how work is organized. It could live inside AGENTS.md, but I keep it separate.

Create _docs/process.md:

- Tasks are GitHub issues, one at a time
- Read the acceptance criteria before starting and before closing
- Commit regularly

As I continue working on a project, I may create other documents, such as:

• testing-guidelines.md for testing

• design-system.md so the UI doesn’t drift every session

• api.md, which describes what the API should look like

I keep them together in _docs/ and link them from AGENTS.md:

Documents

- `_docs/process.md` - how work is organized
- Before writing tests, read `_docs/testing-guidelines.md`
- For anything touching the UI, read `_docs/design-system.md`

The agent reads AGENTS.md at the start of every session, so it knows where to find the process, testing, and design rules if it needs them.

This way, it will load the design system only for a UI task and the testing guidelines only for a testing task. By loading each document only when it’s relevant, we keep AGENTS.md short while we continue adding written context to the project.

These documents are living documents, and I update them often. If I need to correct an agent during a coding session, I can ask it to modify the documents. Next time, it knows what I need, so I don’t have to correct it again.

You can use a prompt like this:

Based on the corrections I made, find the relevant documents and update them.
Commit the current work before changing the documents.

Bootstrap the first task

With AGENTS.md and process.md in place, we can start a new session and ask the agent to implement the first task:

Implement task 1.

For this project, the agent creates the Django app, dependencies, and a passing test.

Grooming: the product manager agent

We have a backlog of tasks, but they’re not precise enough.

We discussed this problem already: if the task isn’t specific, the agent will fill in the gaps during implementation. We risk spending time and tokens on something we don’t need.

Instead, we should ask the AI assistant to fill these gaps before writing any code. Then we review the specification, correct it, and give it to the coding agent to implement.

This process is called “grooming”: we groom a task to make it more specific. Then an engineer can implement it without asking a single question.

In real teams, product managers usually do this work. Here we’ll create a team of agents, and the first role we’ll define will be a PM.

Create a document:

_docs/team/
  pm.md

Inside, write the description for the product manager agent:

You’re a Product Manager

You groom a task before anyone implements it.

- Read the issue as written
- Rewrite it using the template in `_docs/task-template.md`
- Make the acceptance criteria checkable - someone should be able to
  point at the screen and say yes or no
- Think about the edge cases the person who filed it did not consider
- Do not write any code

Definition of done:

- The issue has all four sections filled in
- Every acceptance criterion can be checked by looking at the result
- Everything moved out of scope links to a follow-up issue
- An engineer who has never spoken to you could implement it from the
  issue and the documents it links

If something does not belong in this task, do not silently drop it.
File a follow-up issue and list it under out of scope with a link to
that issue, so it is clear what was moved and where it went.

A groomed task has four sections:

1. Goal - one or two sentences on what should be true afterwards.

2. Acceptance criteria - checkable statements.

3. Out of scope - what this change must not do.

4. Constraints - files it should stay inside, libraries it should or shouldn’t use, prior decisions it has to follow.

We save the issue template as _docs/task-template.md:

## Goal

One or two sentences on what should be true when this is done.

## Acceptance criteria

- [ ] A statement you can check by looking at the result
- [ ] One line per case, including the awkward ones

## Out of scope

- Something that does not belong in this task, moved to #TASK-NUMBER

## Constraints

- Files this should stay inside
- Libraries to use
- Guidelines to follow

We’ll need to groom every task, so we’ll add it to process.md:

Roles

- PM - grooms a task before anyone implements it, follows _docs/team/pm.md

We can now start a new session and ask the agent to groom an issue:

Groom issue #4

After it finishes, review the result.

We can catch a misunderstanding most cheaply while grooming: the issue is a paragraph, and correcting it costs one sentence. If we catch the same misunderstanding after implementation, we need a rewrite.

Loop engineering

After grooming one issue, we can ask the agent to groom the rest:

Groom all GitHub issues. Process one issue at a time.

This will mostly work, but the agent may eventually stop. It might say, “I’ve groomed issues 1, 2, and 3. Do you want me to proceed?”

The answer is almost always “yes”, but the agent has stopped and is waiting for us to say that explicitly. In many cases, I want the agent to continue automatically.

To do it, we can give the agent a goal:

/goal groom all issues

The /goal command will prompt the agent to continue, so we won’t need to do it manually. Instead, we delegate that responsibility to the harness: the system around an agent, such as Claude Code or Codex. When the agent stops, the harness checks whether the condition has been met. If it hasn’t, the harness resumes the work.

This approach is called “loop engineering”. It’s similar to a while loop: we repeat the work until a condition is met.

With loop engineering, the system runs a coding agent repeatedly instead of having us drive it manually, prompt by prompt.

There are multiple “engineering” levels when we work with coding agents:

• Prompt engineering - what we say when we interact with the agent

• Context engineering - what the agent knows before it starts and what it can get during the session

• Loop engineering - when it stops working

• Graph engineering - who does what when there’s more than one agent (we’ll discuss it later)

In June 2026, Addy Osmani published the Loop Engineering essay that gave it a name, and Peter Steinberger compressed the idea into one tweet:

A loop needs a stop condition: a checkable statement that tells the harness when to stop.

For /goal groom all issues, the stop condition is “all issues are groomed”. After each agent run, the harness checks whether that condition is true and resumes the agent if it isn’t.

The stop condition must be something the model can evaluate. “All issues are groomed”, “all tests pass”, and “no file is over 200 lines” are checkable, but “make the code better” isn’t. If the stop condition isn’t checkable, the agent can stop too early or run forever.

Claude Code and Codex provide the /goal loop by default. If your harness doesn’t provide it, you can implement it yourself using stop hooks.

Implementation: the software engineer agent

After grooming the issue, we can give it to a software engineer, the agent who will write the code.

Define the second role:

_docs/team/
  software-engineer.md

Put this definition inside:

You’re a Software Engineer

You implement one groomed task at a time.

- Read the issue and implement what it describes
- Implement against the acceptance criteria, do not change them
- Stay inside the files and constraints the issue names
- Write tests for what you built
- Do not close the issue
- Commit regularly

Definition of done:

- Every acceptance criterion in the issue is implemented
- Tests are written for the new behaviour, and the whole suite passes
- The work is committed
- The issue is still open, with a comment saying what you did

If an acceptance criterion is wrong, impossible, or contradicts
another one, create a comment on the issue about it.

Add one more line to process.md:

Roles

- PM - grooms a task before anyone implements it, follows _docs/team/pm.md
- Engineer - implements one groomed task, follows _docs/team/software-engineer.md

Then ask the agent to implement a task in a fresh session:

Implement issue #2

The engineer stops when the code is written and its own tests pass. It’s still too early to say that the task is properly implemented, so we need to test it.

Testing: the QA engineer agent

When the same agent writes and judges the code, it’s grading its own homework.

If we ask, “Is this correct?” we’ll get a definite “yes”, but the agent might have missed many edge cases.

In the real world, we ask other people to validate our work. We have code reviews, and many teams have designated QA engineers whose focus is making sure the code is reliable.

That’s why we’ll also add a QA engineer to our team.

Add the third role:

_docs/team/
  qa-engineer.md

The description:

You’re a QA Engineer

You check finished work against the issue that specified it.

- Read the acceptance criteria from the issue
- Check each one against what the code actually does
- Run the tests, and say which ones you ran
- Look for the cases the criteria describe but the tests do not cover
- Do not fix anything you find. Report it by creating a comment

Your output is a verdict: PASS or FAIL. It is FAIL if a single
acceptance criterion fails. Post it as a comment on the issue:

## QA: FAIL

- [x] A visitor can create an account with a username and password - PASS
- [ ] A duplicate username shows a visible error - FAIL
      Submitted an existing username and received an unhandled error

Tests: `uv run pytest`, 18 passed, 0 failed

Definition of done:

- The comment starts with PASS or FAIL
- Every acceptance criterion has a verdict against it
- Every FAIL says what you did and what happened
- The test command and its result are included
- Nothing in the code was changed

Ignore what the implementation says it does. Only the acceptance
criteria and the running code count.

Note: you may need to adjust this role for your project if you don’t use uv and pytest.

And the last line in process.md:

Roles

- PM - grooms a task before anyone implements it, follows _docs/team/pm.md
- Engineer - implements one groomed task, follows _docs/team/software-engineer.md
- QA - checks the result against the acceptance criteria, follows _docs/team/qa-engineer.md

Then, in a new session, ask:

Test issue #2

If we get a PASS, that’s great. If we get a FAIL, that’s also useful: we caught a regression. So we start a new engineer session, use the QA comment as input, and ask the engineer to fix it.

We iterate until QA says PASS.

Graph engineering

We have three roles:

• Product manager grooms an issue

• Software engineer implements it

• QA engineer tests it, outputs PASS or FAIL

If QA says FAIL, the engineer will need to reimplement it. Otherwise, the task is done.

We can visualize this process as a graph:

If we have an orchestrator that launches these agents automatically instead of us doing it manually, we get “graph engineering”. We define a graph with specialized agents as nodes and describe how the work moves from one to another.

The earliest explicit use of the term I found in the article from Josh C. Simmons (July 4, 2026): The Graph Engineering phase.

Peter Steinberger sparked a wider discussion in a July 18 post by asking whether the conversation had shifted from loops to graphs. A few hours later, Hamel Husain used “graph engineering” in an article headline.

Hamel put just a GIF saying “stop it” in the article, but people liked the term and started using it for multi-agent orchestration.

I wouldn’t call it a new idea, though, and loop engineering is definitely not dead. Even with graphs, we still need a loop to drive the work and orchestrate it across multiple agents with different roles.

The orchestrator

To implement it, we need an orchestrator.

Describe it in process.md:

Orchestrator

The main session is the orchestrator. It launches the PM, the engineer
and QA as subagents. It does not groom, implement or test itself.

Lifecycle

1. Pick the next open issue from the backlog
2. PM grooms it
3. Engineer implements it
4. QA verifies it
5. On FAIL, back to step 3 with the QA comment as input
6. On PASS, close the issue
7. Repeat until the backlog is empty

Rules

- Do not skip step 2
- The engineer does not close the issue
- QA does not fix the code, only outputs PASS or FAIL
- The orchestrator closes the issue only after QA outputs PASS

We can now start a fresh session and launch the loop:

/goal work through the backlog

The agent reads AGENTS.md, finds process.md, follows the lifecycle, and dispatches the agents according to the documents in _docs/team/.

Because we combine it with a goal, the orchestrator runs until the condition holds.

You can see the result in retroloop.

Note: this approach takes significantly more time and tokens than a direct loop with only a software engineer. In many cases, you don’t need this complexity. Often, a simple prompt or loop is enough.

I’ve followed this approach on many projects, including the AI Shipping Labs website.

I describe it in I Built an AI Agent Team for Software Development. Here, I distill that article into a tutorial.

Next in the series

We have now finished the first module.

In later modules, we build on the same workflow:

• Building and shipping a full-stack app end to end: spec to frontend, backend, tests, Docker, deployment and CI/CD

• Coding agent capabilities: MCP, skills, plugins, hooks and custom agents

• AI for security, audit and DevOps

• Taking a project of your own from an empty folder to something running

Think of it as the project’s landing page. It should help someone quickly understand what you built, why it matters, and whether they should explore it further. This applies to pet projects, course submissions, and portfolio projects you create to support a job application.

At my previous job, I was heavily involved in hiring. I conducted the first technical screening after the recruiter call, which meant reviewing candidates’ profiles and projects. I would open a repository, scan the README, and decide within about ten seconds whether to continue. Most of the time, I wouldn’t continue. If the README didn’t tell me what the project was about or how to run it, I wouldn’t spend time trying to figure that out myself.

I now see the same problem in the courses I run. Each cohort produces hundreds of student projects, and all of my courses use peer review, in which participants assess one another’s work against a set of criteria.

Reviewers need to understand the problem, inspect the implementation, and determine which criteria the project satisfies. A README that makes this difficult can cost the author course points. I have seen projects lose points because important work, such as evaluation results, was buried several notebooks deep and never mentioned in the README.

In this article, I'll explain how your README should look like, which sections it should contain, and who you should write for, so your project gets the attention it deserves.

It's an extended version of the material I prepared earlier this year to the students of my AI Engineering Buildcamp course. Right now the participants of LLM Zoomcamp will find it especially useful, but the same principles apply to the other Zoomcamps and to most engineering projects.

Throughout the article, I will use one running example: Fitness Assistant, a RAG application that helps users find exercises and suitable alternatives. I built it two years ago as a reference project for LLM Zoomcamp. The repository was already in reasonable shape, but I refined its README further while preparing this article.

Start with the reader

Before you write a README, think about the person who is going to read it.

I suggest writing for three audiences at once: peer reviewers, hiring teams, and your future self.

Peer reviewers

The first audience is someone assessing your project against a list of criteria.

In the Zoomcamps I run, participants review one another’s projects and decide how many points each submission earns. They need to find evidence for the problem description, evaluation, monitoring, reproducibility, and other criteria.

Even when your project is not part of a course, it helps to imagine that it will be reviewed this way.

An imaginary peer reviewer gives you a useful standard. Assume that someone will inspect the repository carefully and ask whether you have covered the important parts of the project. Your README should help them find the evidence without searching through every file and notebook.

Hiring teams

The second audience is the hiring team.

A recruiter, hiring manager, or engineer may not spend as much time on the initial review. They first want to understand what the project is and whether it is relevant to the role. Only then will they decide whether to inspect the implementation, evaluation, tests, or architecture in more detail.

This is why optimizing for a peer reviewer is still useful. The README should be easy to scan, but it should also contain enough detail for a technical reader who decides to go deeper.

Your future self

The third audience is your future self.

After several months, you may return to the repository and no longer remember how the project works, how to run it, why you made certain decisions, or what you planned to improve.

A good README should give you enough context to understand the project without reconstructing it from the code and commit history.

Write for all three

These audiences are different, but their needs are compatible.

The hiring team needs a clear overview and a reason to continue. The peer reviewer needs evidence and detail. Your future self needs enough context to understand and continue the work.

The strongest README supports all three: it is easy to scan at first, detailed enough to inspect, and complete enough to return to later.

What does the reader need to understand?

Most readers approach a project with four broad questions:

1. What is this project?

2. Does it work well?

3. Can I run and reproduce it?

4. How was it built?

Let’s start with the first question.

Part 1: What is this project?

Readers need to understand what you built.

The beginning of the README should answer three questions:

• Who is the project for?

• What problem does it solve?

• What did you build to solve it?

Use the title and short description to provide the immediate answer. Then explain the problem in more detail and show how the project works.

1. Title and description

Start with a clear project title and one or two sentences that describe the user, the problem, and the solution.

Avoid opening with a list of technologies. This tells the reader which tools you used, but not what the application does or why anyone would use it.

For Fitness Assistant, I use this description:

In two sentences, the reader learns:

• The user: fitness beginners without regular access to a trainer

• The problem: choosing exercises and finding alternatives can be difficult

• The solution: a conversational AI assistant that provides guidance

The exact formula will vary by project, but a useful starting point is:

You can add a second sentence to explain the main obstacle or why the project matters.

For example:

Keep the description concrete. Phrases such as “AI-powered platform,” “innovative solution,” and “intelligent system” add little unless you explain what the system actually does.

If you struggle to write the description, generate several alternatives and compare them. You can ask an AI assistant to analyze the repository or to describe the project based on your notes. However, you still need to choose the version that accurately represents the project and makes sense to your audience.

2. Problem

The short description introduces the problem, and the problem section explains it.

Here’s a simple structure that works well:

1. What is the user trying to do?

2. What makes that difficult?

3. Why are the existing options insufficient?

4. Who experiences this problem?

For my Fitness Assistant project, the problem is described like that:

• Beginners often struggle to choose appropriate exercises

• Gyms can feel intimidating when someone does not know how to use the equipment

• An exercise may be unsuitable because of an injury, missing equipment, or personal preference

• Personal trainers can provide guidance, but they are not always available or affordable

Keep this section proportional to the project. You don’t need a long market analysis for a small course project. A few specific paragraphs are usually enough to establish why the application exists and who benefits from it.

3. Demo

After you explain the project, show it working.

A reader should not need to install dependencies, create API keys, and start several services just to see what the application does. Many readers will decide whether to continue before they run anything locally.

You can demonstrate the project with:

• Live application

• Short video

• GIF

• Screenshots

• Sample input and output

Use the format that best represents the project. A screenshot may be enough for a dashboard or static interface. A short video is usually more useful when the application involves several steps, dynamic output, or interaction with an AI assistant.

For an AI application, the demo might show:

1. User entering a realistic request

2. System retrieving or processing the relevant information

3. Application producing an answer

4. User providing feedback or asking a follow-up question

Keep the main demonstration focused. Show the core workflow first rather than spending several minutes explaining the repository structure or installation process.

You can cover evaluation results, monitoring dashboards, and architecture later in the video, but a reader should see the application working within the first few moments.

If the project is deployed, place the live link near the top of the README. A reader can then try the application without setting it up locally. Keep the video or screenshots as well because the hosted version may later become unavailable or require credentials.

For Fitness Assistant, I recorded a video that shows the conversation flow and the main application features.

Part 2: Does it work well?

For an AI engineering project, readers need three kinds of evidence:

• Evaluation: How well does the AI system perform?

• Testing: Does the software behave as expected?

• Monitoring: What happens when the application processes real requests?

1. Evaluation

Without evaluation, readers can see that the application generates answers, but they cannot judge how consistently or accurately it performs.

Describe the evaluation dataset and the kinds of cases it includes. The dataset should reflect the situations in which you expect the application to operate. A system evaluated only on simple, carefully written examples may perform poorly when users phrase the same requests differently.

Also explain what you changed after establishing the baseline. The result is more useful when the reader can follow the progression from the initial approach to the final one.

For my Fitness Assistant, the README reports both retrieval and end-to-end RAG evaluation. After adding field boosting, retrieval reached a 94% hit rate and 90% mean reciprocal rank. The final RAG flow produced answers classified as relevant in 83% of cases when using gpt-4o-mini. The README also includes the parameters that produced the best retrieval results and links to the evaluation notebooks and datasets.

Keep the summary in the README, but store the detailed code and results in a separate, clearly named location. Document how to rerun the evaluation so the reader can verify the results or test a different approach.

2. Testing

Explain which tests the project includes and how to run them with one command (if possible).

When the project requires additional services or configuration, state that before the command. A reader should not have to discover that the tests require a database, an API key, or a running container from the resulting error messages.

If the project does not have automated tests, say so. My Fitness Assistant currently has no automated tests. I list this as a limitation.

This tells the reader that I understand what is missing and prevents the README from suggesting that the project is more complete than it is. Honesty is more useful than vague claims when the repository contains no evidence to support them.

3. Monitoring

Evaluation tells you how the system performed on a prepared dataset. Monitoring helps you understand what happens after people begin using it.

In the monitoring section, describe the metrics your system captures and where it stores them. Explain how a reader can view this data, which specific events or metrics you prioritize, and how these observations enable you to diagnose and resolve issues. If you have a dashboard, include a screenshot and explain what its panels show.

For Fitness Assistant, the application stores conversations in PostgreSQL and displays monitoring data in Grafana. The README includes a screenshot and explains the dashboard panels, which track:

• 5 most recent conversations

• Positive and negative user feedback

• OpenAI API cost

• Token usage

• Model used

• Response time

Part 3: Can I run and reproduce it?

After readers understand what the project does and how well it performs, they may want to run it themselves.

1. Quickstart

The quickstart should provide the shortest reliable path from a clean machine to a running application.

Include prerequisites, the repository clone command, dependency installation, environment configuration, required services, and the command that starts the application.

For example:

git clone https://github.com/username/project-name.git
cd project-name
uv sync
cp .env.example .env
docker compose up

It will also help to mention some core tools and technologies you used:

• Python version

• Package manager, such as uv, Poetry, or pip

• Docker and Docker Compose

• Node.js version, if the project includes a frontend

• Database or system dependencies that do not run in containers

If the project supports several setup methods, show the recommended one first.

In addition to quickstart, also include:

• Local development: detailed instructions for running services separately

• Troubleshooting: solutions to common setup problems

2. Data and configuration

Many projects fail to run because the code is available, but the required data or configuration is not.

Document every external input the reader needs. Name the source, file, destination, and command when possible:

python scripts/download_data.py

For configuration, list the required environment variables and explain what each one controls.

Provide an example file such as .env.example:

OPENAI_API_KEY=
POSTGRES_HOST=postgres
POSTGRES_DB=fitness
POSTGRES_USER=fitness
POSTGRES_PASSWORD=

Also, clearly state which settings are required and which are optional ones. A reader should know whether the application can start without a monitoring service, cloud account, or third-party API.

For Fitness Assistant, the README contains a description of a dataset that I used for creating the assistant.

3. Deployment

If the project is deployed, place the live URL near the top of the README, immediately after the title and description.

The deployment section itself should explain where the application is hosted, which services it uses, how deployment works, which environment variables or secrets are required, whether deployment happens manually or through CI/CD, how someone else could deploy their own version

Depending on the project, you might deploy it with Streamlit Community Cloud, Render, Railway, a managed container platform, or a virtual machine running Docker.

For example:

The API runs on Render, PostgreSQL is hosted on Neon, and the frontend is deployed on Vercel. A GitHub Actions workflow deploys the API after tests pass on the main branch.

If deployment requires several services, include a short description of how they connect. Link to longer deployment instructions when the process would make the main README too long.

Fitness Assistant does not currently have a public deployment. The README states this and provides two alternatives

1. Start the complete system with Docker Compose

2. Run the application and supporting services locally

Part 4: How was it built?

Once readers understand what the project does, how well it performs, and how to run it, they may want to inspect the implementation in more detail.

1. Architecture

Use the architecture section to show how data and requests move through the system.

You can create the diagram with Mermaid, an image, or a simple text-based flow. The format matters less than whether the reader can understand the system without first reading the code.

For Fitness Assistant, the architecture diagram shows how the main components interact:

1. A user sends a request to the Flask application.

2. The application uses minsearch to retrieve relevant exercises.

3. It sends the request and retrieved context to OpenAI.

4. It returns the generated answer to the user.

5. It stores the conversation and usage information in PostgreSQL.

6. Grafana reads the stored data and displays it in a monitoring dashboard.

It also helps to add a short explanation below the diagram. Diagrams show relationships well, but they do not always explain why a component exists or what responsibility it has.

2. Project structure

The project structure section helps readers find the code behind the claims you make elsewhere in the README.

Include a simplified file tree with one-line descriptions:

Do not paste the complete output of tree if the repository contains dozens of generated, configuration, or unimportant files. Show the paths that help the reader understand and navigate the project.

If you use notebooks, explain what each notebook contains. Names such as notebook1.ipynb or final_v2.ipynb give the reader little information. Prefer descriptive filenames, and state which notebook contains the reported results.

3. Decisions and trade-offs

The technologies you chose matter less than the reasoning behind them.

Describe the decisions that materially shaped the project, such as model selection, retrieval method, database choice, framework choice, etc.

A useful format is:

For example:

Or:

4. CI/CD

The CI/CD section explains which checks and deployment steps run automatically.

At a minimum, document what triggers the workflow, which checks run, what happens when a check fails, and whether successful changes are deployed.

For example:

AI evaluations can be slower and more expensive than conventional tests, so you may not want to run the full evaluation suite on every commit. You could run a smaller set on pull requests and the complete evaluation before a release.

If deployment is automated, describe the sequence. Link to the workflow files so readers can inspect the implementation.

If the project has no CI/CD, do not create a section that implies otherwise. You can mention its absence under limitations instead.

Nice to have: What is the project scope and its limitations?

A good README also helps readers understand where it stops.

Describing these boundaries shows that you can assess your own work realistically and distinguish between what you implemented, tested, and only considered.

1. Limitations

Use the limitations section to explain what the project does not currently handle well.

Be specific. Name the boundary and its consequence:

Or:

For Fitness Assistant, the limitations include the absence of automated tests and a public deployment. The dataset is also relatively small and synthetic, so the application cannot provide complete or medically reliable exercise advice.

Do not hide a known weakness behind vague wording. At the same time, do not turn the section into an apology. State the boundary plainly and explain its practical effect.

2. Future work

Future work should follow from the limitations or from the evidence you gathered while building and evaluating the project.

Keep the list realistic and prioritized. Explain what you would change and why:

3. Self-evaluation

For a course project, you may also include a self-evaluation against the project rubric.

Go through each criterion, state the score you believe the project earns, and point to the evidence that supports it.

This helps the reviewer navigate the repository and shows that you assessed the project against the same criteria they will use.

Do not award points without evidence or treat the self-evaluation as the final score. The reviewer still needs to verify your claims and make an independent judgment.

A self-evaluation can also reveal gaps before submission. If you cannot point to evidence for a criterion, either improve the project or adjust the score you claim.

You can use an AI assistant to perform an initial review of the repository against the rubric. Ask it to identify the evidence for each criterion and flag anything it cannot verify. Then check every conclusion yourself because the assistant may overlook files, misunderstand the implementation, or give credit for features that are only mentioned but not implemented.

Common mistakes

I see the same README problems across course projects in almost every cohort:

• Missing setup instructions. If I cannot run the project from a clean clone, it loses points on reproducibility.

• No evaluation. An AI application without evaluation is only a demo.

• No demo. If readers have to run the code before they can see what it does, many will not.

• Important work is buried in notebooks. Link to the evaluation, prompts, experiments, and results from the README.

• A stale README. The commands, screenshots, metrics, and deployment links should match the project’s current version.

• A wall of text. Use clear sections, screenshots, diagrams, commands, and concrete results to make the README easy to scan.

• Planned work is described as implemented. Use the present tense only for features that already exist. Put unfinished ideas under limitations or future work.

• A README written as an afterthought. A list of technologies and a few setup commands is not enough. Explain the problem, show the project, and provide evidence that it works.

Using AI to write and review the README

A coding agent can inspect your repository and help draft the README, but do not publish the first version it generates.

Start by asking it to identify the main application flow, setup commands, configuration, tests, evaluation, monitoring, and deployment files. Then use that information to draft one section at a time for a specific audience.

For example:

AI can also review the finished README against the repository or a course rubric. Ask it to flag unsupported claims, missing files, incorrect commands, undocumented environment variables, and criteria for which it cannot find evidence.

Do not force every section from this article into the README. Include only what is relevant to the project, and order the sections according to what your reader needs most.

Finally, verify everything yourself. Coding agents can invent commands, misunderstand the architecture, describe planned features as implemented, or report outdated results. Run every command and check every number, path, link, and claim before publishing.

Good project examples

Here are a few strong READMEs from AI Engineering Buildcamp and LLM Zoomcamp participants:

• AI Diet Coach: Clear problem statement, live demo, screenshots, architecture diagram, and monitoring dashboards.

• Research Radar: Strong one-line description, clear evaluation results, parameter tuning, deployment, CI, and 67 tests.

• CineRAG: Separate retrieval and answer-quality evaluations, with datasets, metrics, results, and instructions.

• Applied ML Teaching Copilot: Detailed evaluation process, 60 test scenarios, clear test separation, and self-evaluation against the rubric.

Other good examples:

• Chess Coach Agent

• Meal Map

• Datawarehouse Agent

• AA Bot

More projects are available on the AI Engineering Buildcamp Cohort 3 and LLM Zoomcamp 2025 project pages.

What I’ve Been Working on This Week

AI Dev Tools Zoomcamp 2026 starts on August 31

Registration is open for the next cohort of AI Dev Tools Zoomcamp.

This is a free hands-on course about using AI developer tools in real software development workflows.

I’m preparing the 4 new workshops to update all of the course content:

• ​AI-Native Developer Workflow: Using AI Tools Without Losing Control

• ​Build and Ship an AI-Assisted Full-Stack App

• ​Coding Agent Capabilities: MCP, Skills, Plugins, and Custom Agents

• ​Open-Source AI Tools for Security, Audit, and DevOps

Resource: Turn AI agent traces into useful data

Coding agents such as Claude Code, Codex, and Copilot produce detailed traces about sessions, model usage, tool calls, token consumption, and cost. In this hands-on LLM Zoomcamp workshop, Alona shows how to turn those raw logs into structured data and dashboards.

You’ll learn how to:

• Ingest local coding-agent traces and logs from an API

• Build data pipelines with dlt and load the results into DuckDB

• Explore usage, models, projects, and token consumption in Marimo dashboards

• Deploy pipelines and reports so they can run regularly and be shared with a team

The workshop is part of LLM Zoomcamp, but you can follow it independently.

Watch the workshop

Edited by Valeriia Kuka

I used to do everything manually, but in 2025, I started using coding agents and creating more libraries.

Eventually, I automated the whole process. In this article, I’ll tell you how I do it and walk you through the entire workflow:

• Starting a library and choosing a name

• Publishing the first version

• Releasing through CI

• Automating the process with agents

1. Start with an Idea

I build something to solve a specific problem that I’m facing. Then I realize I may want to reuse it, either as a command-line utility or as a small library for other projects.

Before I create a repo, I do a brief validation step. I try to understand what I’m building and whether it is worth packaging as a library.

I usually start brainstorming and researching with ChatGPT. I explain the problem, explore what the library should do, and check whether similar tools already exist. If they do, I look at how they solve the problem and whether they fit my needs.

This also gives me a clear brief I can use if I decide to move forward with the library.

I described this process before when I wrote about building SQLiteSearch.

2. Choose a Name

Once I have a clear brief, I choose a name. This is harder than it sounds. I need a name that fits the project, is short enough to use, and is still available on PyPI.

I usually start a new session with an agent and paste the exported conversation describing the problem I created in the previous step. In this prompt, I also explain that I want to turn the description into a library, and ask the agent to help me brainstorm names. I also ask it to check whether each name is available.

The availability check is pretty simple, because PyPI exposes package metadata at this URL:

https://pypi.org/pypi/{name}/json

If the request returns 404, the name is available. If it returns 200, someone has already taken it.

Then I iterate. If a name is taken, I ask for more options. If a name is available but does not feel right, I keep brainstorming.

Heru is one example. The idea was “one tool to rule them all” (all the agents), so I wanted a short name connected to The Lord of the Rings, maybe something Elvish. I brainstormed with the agent, checked availability along the way, and eventually landed on Heru.

Quse was similar. I wanted a short name that still meant something. After several iterations, I landed on Quse, short for “quota use.”

I usually spend around 10 minutes on this step. By the end, I have a name that fits the project, is available on PyPI, and works for both the GitHub repo and the package.

I wrote more about how Heru and Quse got their names in Six Projects That Didn’t Make It.

3. Create the First Version

Once I choose the name, I create a GitHub repository and publish its first version.

I usually make the repo public for two reasons:

• I contribute a lot to open source, and I want others to see the code, use it, and maybe contribute to it too

• Public repos get a larger GitHub Actions quota than private repos

At this stage, I don’t need a complete library. I create a skeleton: a minimal working version that I can publish.

I also decide the project structure up front. Across my projects, I keep this structure fairly consistent. I use Hatch, with hatchling as the build backend, and uv for tooling.

For a minimal library, I use a layout like this:

library_name/
├── library_name/__init__.py
├── library_name/__version__.py        # __version__ = “0.0.1”
├── library_name/cli.py                # only if the package installs a CLI
├── tests/__init__.py
├── pyproject.toml
├── Makefile
├── README.md
├── .gitignore
├── .python-version
└── uv.lock

I put the package metadata in pyproject.toml:

[build-system]
requires = [”hatchling”]
build-backend = “hatchling.build”

[project]
name = “<library_name>”
dynamic = [”version”]
description = “<package-description>”
readme = “README.md”
license = {text = “WTFPL”}
authors = [{name = “Alexey Grigorev”, email = “alexey@datatalks.club”}]
requires-python = “>=3.12”
dependencies = [”<dependencies>”]

[tool.hatch.build.targets.wheel]
packages = [”<library_name>”]

[tool.hatch.version]
path = “<library_name>/__version__.py”

If the library has a command-line tool, I add an entry point so users can run the command after installation:

[project.scripts]
stylint = “<library_name>.cli:main”

That is the whole skeleton: a package directory, a version file, a place for tests, and a pyproject.toml file with the build configuration.

4. Publish the Project

Once the first version is ready, I can publish it.

For manual publishing, I need two accounts: one on Test PyPI and one on PyPI. I use Test PyPI to verify that the package builds correctly and installs as expected before publishing it to the real index.

I also need API tokens for both accounts. I generate a token in each account and put both tokens in ~/.pypirc in my home folder. I keep separate token-authenticated entries for pypi and testpypi:

[pypi]
username = __token__
password = pypi-...

[testpypi]
username = __token__
password = pypi-...

Hatch uses these credentials.

With the tokens configured, the manual publishing flow is simple:

uv run hatch build
uv run hatch publish --repo test
uv run hatch publish

I build the package, publish it to Test PyPI, check that everything looks right, and then publish it to PyPI.

This is the manual flow. I documented it in the minsearch README so I would not have to remember the commands every time.

5. Set Up CI/CD

I can publish manually, but I prefer to release through CI. I want releases to work the same way on any computer I use. It should not matter whether I have the right tokens on my machine or whether my local environment is configured correctly.

Instead of running hatch publish on my own machine, I let GitHub Actions publish the package when I push a version tag.

Before I release, I bump the version. For a simple bug fix, I bump the patch version. For a more substantial change, or for a breaking change, I bump the minor version. I rarely update the major version.

I put the publishing workflow in .github/workflows/publish.yml. GitHub Actions run it whenever I push a tag that starts with v. The workflow does three things:

• builds the package with uv build

• checks that the built package version matches the tag

• publishes the package with the official pypa/gh-action-pypi-publish action

For authentication, I store the PyPI token as a GitHub secret called PYPI_API_TOKEN. This is the same token I keep in ~/.pypirc, but GitHub stores it as a repo secret. At release time, I do not need the token on my machine.

So the release process is:

1. Bump the version in __version__.py

2. Commit the change

3. Create a v<version> tag

4. Push the tag

In code, it looks like that:

git tag “$VERSION”
git push origin “v$VERSION”

When I push the tag, GitHub Actions builds the package and uploads it to PyPI.

This approach is not specific to Python. I can use the same flow for other ecosystems: push a tag through GitHub, then let CI run whatever publishing logic the project needs. That can mean publishing to PyPI, another registry, building binaries, or uploading artifacts elsewhere. The only requirement is that CI has access to the necessary credentials.

rustkyll is a good example. It is a static site generator written in Rust. When I push a v* tag, its workflow cross-compiles binaries for six platforms, attaches the binaries to a GitHub Release, and publishes binary-bundled wheels to Test PyPI and then to PyPI. The publishing logic is different, but the trigger is the same: push a version tag.

6. Automate Publishing with Skills

Once I have the manual and CI flows, I can stop repeating the setup by hand. I use a few agent skills that cover the whole pipeline:

• init-library

• setup-pypi-ci

• release

I use init-library to scaffold a new Python package. I give it a name, a short description, the dependencies, and whether the package needs a CLI.

The skill creates the standard layout:

• the package directory,

• __init__.py,

• __version__.py seeded at 0.0.1,

• a tests directory,

• Pyproject.toml,

• Makefile, and

• README.md.

It also sets up the project with uv. Once I have the repo on GitHub, I move to the next skill.

I use setup-pypi-ci to set up tag-triggered CI publishing. The skill copies the publish.yml workflow, adds the make release target, reads the PyPI token from ~/.pypirc, and saves it in GitHub as the PYPI_API_TOKEN secret. After that, I can release the package by bumping the version and pushing a tag.

I use release to make a release. By default, it bumps the patch version, commits the change, pushes the tag, watches the GitHub Actions run until it finishes, and writes GitHub release notes from the git log. The skill is registry-agnostic: I can use it for PyPI, crates.io, npm, or anything else that publishes from CI when I push a v* tag.

I keep these skills in my .agents repo, which works as my AI assistant dotfiles. It gives me one source of truth for the same skills across Claude Code, Codex, and OpenCode. So I can use the same release process with every agent I use.

This workflow helps me maintain more and more libraries and create new ones quickly. What used to be a manual process is now a set of reusable skills.

What I’ve Been Working on Recently

1) Ingesting Agent Traces with dltHub

This week, we ran a hands-on workshop with dltHub on ingesting agent traces.

When you build AI agents, you need to understand what they are doing. You need to see tool calls, intermediate steps, token usage, outcomes, and failures rather than leaving them hidden in logs.

In the workshop, we used dltHub Pro to build a pipeline that captures agent traces, normalizes them into a consistent schema, and transforms nested, variable-length trace data into tables we can query and analyze.

Then we used this data to build reports. These reports help us understand how agents behave, where they fail, and what we can improve. The goal was to stop flying blind and create a reporting layer that makes agent debugging more practical.

2) Updating the Agent Guardrails Workshop

I also updated the workshop on building an agent with guardrails.

In this workshop, we build a Data Engineering Zoomcamp FAQ agent. We start with a simple RAG baseline, turn it into an agent with one search tool, and then add guardrails around it.

We add input checks for topic and intent, output checks for unsafe promises or policy violations, and multiple guardrails with clear responsibilities. We also run guardrail checks concurrently so they do not add too much latency.

We write the examples in Python with Pydantic, minsearch, and the plain OpenAI Python SDK. I wanted the approach to stay framework-neutral, so we implement the agent loop and the guardrails directly instead of relying on a specific agent framework.

We take inspiration from the OpenAI Agents SDK guardrails, but we implement the guardrails ourselves. If your agent framework does not provide input and output guardrails, you can still use the same approach.

See the workshop

3) Adapting a Resume with AI

Another workshop was about adapting a resume with AI.

We built a pipeline that starts with a general resume for an engineering role. Once we have that base version, we adapt it to a specific domain, for example, education. At this stage, we make the experience most relevant to that domain more visible.

Then we take one more step: we adapt the domain-specific resume to a concrete company. Instead of rewriting the resume randomly for every application, we move through three levels:

• General resume for the role

• Resume adapted to the domain

• Resume adapted to a specific company in that domain

That was the main idea of the workshop: use AI not just to rewrite a resume, but to create a structured process for positioning your experience.

See the workshop

Interesting Tools

• Webwright is a lightweight, terminal-based browser agent framework from Microsoft that gives an LLM a CLI to spawn browser sessions and complete web tasks. Instead of the step-by-step Playwright MCP loop, it has the model write a re-runnable Python Playwright script end-to-end, so the agent’s browsing history becomes a single code file you can rerun, adapt, and debug. It’s a CLI alternative to Playwright MCP for browser testing and automation, built on just httpx, pydantic, playwright, and typer, with OpenAI, Anthropic, and OpenRouter backends.

Edited by Valeriia Kuka

In this newsletter, I want to share a repeatable process for selecting your next portfolio project. We focus on the AI Engineering roles, but it will also work for any role in engineering or data.

Why Building a Portfolio Project in the First Place

First of all, an active portfolio that you usually store on your GitHub profile raises your chances of getting hired, because most hiring managers look at it. Additionally, it helps others notice your work, leading to collaboration, freelance offers, and conversations you wouldn’t have otherwise. You can think of it as your landing page that presents you and your skills without requiring you to talk about yourself.

Types of Portfolio Projects

I divide all the projects into several categories, and I think this categorization may also help you better understand what could become your portfolio project.

1) Role-targeted projects

Role-targeted projects are the projects that you build to get hired for a specific role. You pick a domain, research the companies in it, and build something that speaks to their work.

In this post, we’ll focus on role-targeted projects, and the framework I describe here gives you a way to find ideas for them. But it’s also helpful to know about other project types I collected below.

2) Personal projects

These are the kinds of projects that solve the problems that you yourself regularly face and want to build a solution for them.

Most personal projects might not be directly related to the field you’re applying to. And, individually, they aren’t strong portfolio pieces. But together they show that you notice problems and build tools to solve them.

I have many such projects. When something in my work or life is annoying, and I see a fix, I build it. For example, I have a tool called ssh-autoforward that helps me manage my remote servers. It scans the remote ports and forwards a port the moment it opens

3) Take-home assignments

When a company gives you a take-home assignment as part of an interview, the code you produce is yours. That’s why take-home assignments can become portfolio projects, too.

This is especially fair when the company doesn’t pay you for the interview. Publish it, say it was part of an interview process, and show how you code.

4) Hackathon, community, and open-source projects

Any project that you created in public or contributed to can also become your portfolio project. It can be your hackathon contribution, community work, or contributions to open-source projects.

The strongest signal for hiring managers is that other people saw, reviewed, or used your work’s product, meaning it serves others, not just you.

Now that we’ve covered why you should have a portfolio and what to fill it with, let’s move to the framework itself and cover each step separately.

The Project-Selection Framework

There are two broad job search strategies that you can use.

Spray-and-Pray Strategy

The first one is more chaotic, less controllable, and less efficient. It’s called spray-and-pray. You apply to every job you can find and hope someone responds. You don’t have any selection criteria or filters to select job offerings you want to apply for. You spread your effort, and that strategy doesn’t guarantee you meaningful results.

Portfolio Strategy with Domain-Based Selection

The second option is more focused. You think about which domains and companies you want to work for and use it as a North Star that guides you through the skills you need to pick up and projects to build. It doesn’t mean you stop applying to other jobs. You still can, but you have a direction that you’re putting your efforts into. And it also means your project starts to have a narrower scope, and you can tune it later and add details.

And here are the 6 steps that describe the project selection process:

1. Pick a domain where you want to work.

2. Find the companies in that domain that are (were) hiring for the role you’re applying to.

3. Analyze their job descriptions and engineering blog posts.

4. Extract the problems they solve.

5. Find problems that are common across several companies in the domain.

6. Pick one problem and design a project around it, choosing a technology stack that matches what those companies use.

How to Run this Framework

You can run this framework on your own, but I’d recommend using an AI assistant to speed up your research.

Here’s what you need if you want to use AI:

1. An AI assistant, like ChatGPT, Gemini, Grok, or a coding agent, like Codex or Claude Code. A coding agent can create and move files, so you keep the research organized as you go. With a chat assistant, you’ll need to manage the files yourself.

2. Time to read about the companies that you’ll collect. The agent can do the first pass quickly, but you still need to go through the sites, job descriptions, and blogs by hand so you understand the problems, not just the list.

Now let’s go through the framework step by step.

Subscribe now

Step 1: Start from the domain

Start by choosing a domain where you might want to work.

This is what makes the project-selection process different from the spray-and-pray strategy. Instead of applying to every role you can find, you choose a domain first and use it to guide your research, skill selection, and project ideas.

It is also better than targeting only one company. One opening can close before you finish the project. A single company may also have a very specific problem that does not transfer well to other interviews. But when several companies in the same domain address the same problem, a single portfolio project can become relevant to many conversations.

The domain does not have to be one you already know well. Existing domain knowledge helps, especially if you are switching careers, but interest is enough to start.

Step 2: Find companies in the domain

Once you have a domain, list 5 to 10 companies that hire for the role you want to be hired for or work on problems where this role is relevant.

Current and past openings are useful, but they are not the only source. A company can still help with your research if it has engineering blogs, case studies, product pages, or publicly available technical material. At this stage, you are trying to understand what companies in the domain build and what problems appear across the market.

This is where you can use your AI assistant to combine your job-market data with live online search. That makes the method easier for someone else to repeat.

The request looks like this:

Step 3: Analyze job descriptions and tech blogs

After you have a list of companies, the next step is to understand what they work on.

Use two sources: job descriptions and engineering blogs.

Job descriptions show what a company expects an engineer to do. They usually include responsibilities, team context, and required skills. From them, you can infer what the company is building and what kind of work it expects you to handle.

Engineering blogs show what teams have already built and what problems they had to solve. In education, for example, a blog post about an AI conversation partner can reveal problems around dialogue quality, learner feedback, grounding, evaluation, or latency. That kind of detail is useful because it gives you material for portfolio project ideas.

Ask the agent to find relevant job descriptions and engineering blog posts for each company on the list, and to split them into two files:

When you go through the job descriptions, do not fixate on the list of technologies. Use the tech stack as a clue, not as the project idea.

Focus on what the company does and what problem it solves with those technologies.

Once you have the job descriptions and blog posts, extract the problems from them.

Step 4: Extract problems

Next, turn the source material into a list of problems. But don’t jump to solutions yet.

At this stage, you only want to understand that there are people facing this problem, what input they have, and what they want to get as a result of interacting with your solution.

Point the agent at the job descriptions and engineering blogs it produced, and ask it to extract the problems. Don’t forget to ask it to specify sources for each problem it finds:

Step 5: Find problems shared across companies

A single-company problem is narrow. A problem that several companies care about is usually a better portfolio target than one that appears only once.

Ask the agent to group the per-company problems into themes that appear across more than one company:

Step 6: Pick a problem and choose technologies

Now you have a list of shared problems. The next step is to pick one and turn it into a project.

Create project candidates

Start by generating several possible projects from one theme. This gives you options before you commit.

You can use this prompt:

Pick one project and choose technologies that match the target companies

Pick one project from your list and choose technologies that target companies already use, so the project reads as domain-relevant when you discuss it in an interview.

Ask the agent for the stack:

The agent suggests technologies based on the data it found, but the final decision is yours.

Don’t try to learn too many new tools in one project. Pick a small set, build the project, and make it usable.

Once you have the project and stack, you are ready to build.

Where to go from here

You now have the full framework. From here, you can start building.

But here are a few more things that deserve extra attention, and I think it’s important to note them to you.

Add evaluations and monitoring

For an AI engineering portfolio or most data professionals, evaluations matter more than any other feature.

A project that generates output but never checks whether the output is correct is still incomplete.

Use an AI assistant to help you build faster, but be deliberate about these parts:

1. An evaluation harness that scores the output against the criteria that matter for the problem

2. Logging or monitoring, so you can inspect what the system did

3. Tests, so the core pieces stay correct as you change the project

4. A README that a hiring manager can skim quickly

Make the README scannable

Most hiring managers won’t read your code. They will skim the project page first and decide whether to look for more information. Optimize for that.

The README should explain, in a minute or less:

1. What you built

2. Who it is for

3. What input it takes

4. What output it produces

5. How you evaluate whether it works

6. Why the technology choices make sense

Ask friends or colleagues to read the README and tell you what they understood. If they cannot explain the project back to you, rewrite it.

The code still matters. If someone opens a random file, it should look like the work you did and put effort into. But the README is where most people will decide whether the project is worth a closer look.

Make the project usable

A project that runs and produces results is stronger than one that is technically complete but never used. Get it to a state where you can feed it real input, run it end-to-end, and get useful output back.

Keep building in the same direction

One project is a start. You can build more projects by using your initial shortlist of project candidates.

When working on a new project, slightly vary the focus or stack while still aiming at the same domain.

For example, one project can use OpenAI, the next one – Anthropic. Another can use a knowledge graph approach based on a case study. Over time, you build a small portfolio that shows the range of your skills and the tools you can work with, based on real market data rather than random choices.

By the time you interview, you will have projects to talk about. When the interviewer asks what you built, you can explain the domain, problem, input, output, evaluation approach, and why you chose the stack.

Build a habit

Landed your new job? Congratulations! However, don’t stop building. Actively look for new project ideas and consistently build a habit. This will keep your skills sharp and your approach hands-on. Over time, starting new projects will become increasingly straightforward as you refine your process and build upon your existing foundation.

This article is based on a workshop I recently did at AI Shipping Labs. Check our website if you want to see more details (free with signup).

Check the Workshop

What I’ve Been Working on Recently

1) Podwiki: A Wiki from Five Years of Podcasts

I’ve been working on something I call Podwiki: a Wikipedia-style resource built from all the podcasts I’ve recorded over the last five years.

I gave the task to Codex and used the goal feature. It worked on the project for about two days and processed a lot of material. It also burned through the limits quickly, so I had to reset them several times. The result is still in progress, but it already looks useful: a large, structured knowledge base from years of podcast conversations.

2) AI Shipping Labs: New Sprint and Small Improvements

We also launched a new sprint in AI Shipping Labs.

To prepare for it, I worked on a set of small product improvements. Nothing huge individually, but the kind of polishing work that makes the experience smoother for members once a new sprint starts.

3) AI Dev Tools Zoomcamp New Cohort

We opened registration for the next cohort of AI Dev Tools Zoomcamp. The live cohort starts on August 31, 2026.

The course is still free, and the materials are also available in self-paced mode. But the live cohort offers the full structure: graded homework, a leaderboard, peer review, and a certificate. The self-paced version is better if you just want to go through the materials on your own, but it does not include scoring, peer review, leaderboard, or certificate.

One small but important improvement: registration is now hosted directly on the course management platform rather than in Airtable.

Register here

4) Trying Fable for Codebase Audits

I’ve also been trying Fable while I have access to it.

One use case I’m exploring is codebase audits. I used it to run several parallel audit agents over a codebase, and the results were interesting. Two auditors finished with 15 findings between them, including issues such as an all-or-nothing usage-panel parser, an unbounded transcript read that could run out of memory, and a corrupt settings file that could cause the app to crash-loop at launch. Three more auditors were still running at that point.

This is the kind of task where parallel agents can be useful: not writing the whole system, but inspecting different parts of an existing codebase and surfacing problems that are easy to miss.

5) New Course Concept: Product Shipping Zoomcamp

I also spent time thinking through a new course concept: Product Shipping Zoomcamp.

The idea is an end-to-end product development course in which students take an idea from concept to a deployed, improved, measured, publicly launched, and peer-reviewed product. The key design decision is that community and build-in-public start in Module 1, not at the end. Students should not disappear for weeks, build something alone, and only show it when it is finished. They should get feedback, share progress, and improve the product throughout the course.

If this kind of course sounds interesting, leave a comment with what you would want to see in it.

Leave a comment

Resource: Ship Production Code 2x Faster With AI

If you want to go deeper into AI-assisted software development, we recently published a DataTalksClub workshop with Mihail Eric on shipping production code faster with AI.

The workshop goes beyond basic vibe coding and focuses on a more structured development process. Mihail walks through his RAPID framework for using AI in real codebases: onboarding to legacy systems, analyzing the architecture, constraining agents with sufficient context, preventing scope creep with PRD templates, and coordinating multiple agents without losing control over code quality.

Edited by Valeriia Kuka

Some of them work, but I don’t need them anymore. Others don’t work at all. Some turned out to be more complex than I expected, or I couldn’t build them the way I imagined. I abandoned many of them at different stages.

My GitHub profile shows 195 repositories, and that’s only my personal account. I also experiment with projects in the DataTalks.Club and AI Shipping Labs organizations on GitHub. With coding agents, I write a lot more code than ever before, because I can try ideas faster. Naturally, I also create more projects that don’t go anywhere.

But I don’t think this is a problem.

These projects may not reach their final form, and I may not use them afterward, but building them still teaches me something. Some help me understand how to work with coding assistants. Others help me think through a vague problem in practice.

Often, I start with a problem that is not clear yet. I try one solution, build part of it, and then realize it does not quite fit. So I adjust the idea, remove some parts, try a different approach, or abandon the project entirely. The project itself may not survive, but the problem becomes clearer.

That’s why I don’t think abandoned projects are necessarily failures or a waste of time. Often, they are part of the learning process.

In this post, I’ll share:

• A few projects I started and later abandoned

• Why some of them were still useful

• Which ideas survived as smaller tools

• What I learned from having all this dead weight in my repositories

Code Explainer

In October 2025, I built a small Streamlit app to help understand other people’s repositories. I pasted in a GitHub URL, waited for it to download the main branch, and asked questions about the code.

Behind the scenes, an agent could list directories, read files, and grep through the codebase. The interface showed what it was looking at, which files it had analyzed, and how much the request cost. I could also ask follow-up questions without starting over.

Then I started using Claude Code and realized I didn’t need Code Explainer at all. I can just ask Claude Code, Codex, or another coding agent to explain any project. They can clone the repository, inspect the files, and answer questions directly.

So now it’s much simpler to ask a coding assistant than to maintain a separate tool for this.

The Fitness Tracker

Around December 2025, when I first got a Claude Code subscription, I started building my own fitness tracker. I think everyone who tracks workouts eventually wants to build one.

It was my first project where I used Claude Code more intentionally. At that point I was trying to understand how it was different from other assistants I used previously and how much of a real project they could handle.

I already had my own fitness process. I had tried several apps and trackers, but none of them matched the way I train. Since I understood the domain and had a clear idea of what I wanted, I decided to build my own.

At first, I asked Claude Code to create both the frontend and the backend, and I did not give it many constraints. I told it to choose whatever stack it wanted. I did not like the result: it didn’t work and I didn’t understand why. So I asked it to rebuild the backend in Django and use a different frontend stack.

I wanted to define workout presets for different days, start a workout from a preset, and log the details the way I actually train: warm-ups, bodyweight exercises, drop sets, weights, and reps. I also wanted the tracker to remember the last weights I used and save unfinished workouts on the server, so I could refresh the page or switch devices without losing the session.

After that, I kept adding more features: food and meal logging, reusable meal templates, bodyweight and sleep tracking, and a metabolism view that connected these inputs. By that point, the project had a lot of custom logic, and none of it worked properly.

In the end, I gave up on it. It was my first proper Claude Code project and I built it for myself, around a domain I understood well. It did not become something I use every day, but it helped me understand how Claude Code behaves on a real project: where it helps, where it needs more guidance, and how quickly a personal tool can grow once I start adding every feature I want.

That made it easier to try other projects afterward.

The Metabolism Simulator

In January 2026, I started experimenting with Claude Code more seriously. I wanted to see whether I could make it keep working for a long time without coming back every few minutes to press “continue”.

I tried the Ralph Wiggum plugin, which prompts Claude to continue whenever it stops.

The first project I tested with this setup was a metabolism simulator. I was interested in the topic, but I was also using the project as an experiment. What happens if I let Claude Code work on the same app for many hours with a continuation loop?

Since I do sports, I’m genuinely interested in how metabolism works. What happens when I go for a run without breakfast? What happens when I eat and then go to the gym? How should I eat after a workout? What changes if I want to cut or gain muscle?

I wanted to build a simulator tailored specifically to me. It would show what happens when a person eats, trains, or sleeps: glucose, hormones, energy, and muscle recovery. Ideally, I wanted to connect it to my fitness tracker and use my own diet and training data. It would be like a personal nutrition coach that already knows my routines.

There was a computer game in the 90s called Komputerschik, or “the Computer Guy”. You had a computer and could buy more RAM or upgrade other parts. I wanted something like that, but for a human body. I wanted to change inputs and see cause and effect.

I asked Claude to plan the app first. Then I told it to use a client-server architecture and write frontend and backend unit tests, integration tests, and end-to-end Playwright tests. After that, I turned on the loop and let it work.

After three hours, Claude had built the interface, but the buttons did not work. I asked it to test the app through Playwright and fix the broken flows. After 20 hours, it had added many more features, but food and exercise logging still did not work.

It turned out that instead of fixing the API problem, it just decided to display demo data, making the app look functional while the backend was broken. I had to tell it not to hide a broken backend behind a fallback.

I let Claude run for a few more days. By the end, it had created a well-designed application with a simulation loop, a dashboard, food and exercise data, hormone charts, and many other features. Not everything worked, but it was clear that Claude had produced a lot of code.

The loop can produce something that looks good, but it isn’t reliable. Claude can get sloppy and delete a test instead of fixing it. An agent can keep working forever, but it still needs steering. You need clear requirements, grooming, tests, review, and somebody to accept the work.

You need a development process.

That’s why I started working on my agent team in February 2026. I still use that process now. The metabolism simulator failed as a metabolism app, but it succeeded as an experiment. It showed me what breaks when I let an agent run without enough structure, and it pushed me toward the process I use now.

For this concrete app, I don’t think I’ll come back to it. If I do anything in this direction, I’ll start from scratch and think more deliberately about what exactly I want to model.

CodeHive

I already wrote about my agent team for software development. The team had five agents: an orchestrator, a project manager, a software engineer, a QA engineer, and an on-call engineer.

I tested it on five projects, but kept seeing the same problems. Claude Code could pause in the middle of a task and ask, “Shall we proceed?” when there are a lot of unfinished tasks. Or it said the work was done, even though it wasn’t.

I also couldn’t see what a subagent was doing. It could spend an hour on a task, and I had no way to tell whether it was making progress or was stuck. Sometimes the orchestrator ignored the setup entirely and sent a task straight to the SWE without PM grooming or QA verification.

The methodology lived in markdown files, so the agent could ignore any part of it. I wanted the application to enforce the pipeline instead: PM always grooms the task first, then SWE implements it, QA verifies it, and PM accepts it. The roles, acceptance criteria, and definition of done are be part of the tool, not another prompt.

At that time, I also started hitting Claude Code’s usage limits, which made me realize that relying on a single provider was a bad idea.

To deal with all of that, I started working on CodeHive. I wanted it to provide:

• Enforced PM, SWE, QA, and acceptance pipeline

• Multiple agent backends, including Claude Code, Codex, GitHub Copilot, and Z.ai

• A non-blocking task pool

• Subagent status and intervention controls

• Automatic GitHub issue intake

• Access from a phone, computer, browser, and terminal

CodeHive got too ambitious. Around that time, I discovered Termius and realized I didn’t need a complex multi-client system just to work from my phone. Opening Termius on my phone was enough, so I no longer needed the web interface.

CodeHive wasn’t a complete waste. I worked out much of the agent-team approach there. But turning it into a tool I would actually use would take a very long time, so I stopped working on it.

Litehive

Litehive came out of CodeHive. After CodeHive grew too large, I pulled out the two parts I actually needed: the enforced pipeline and the ability to switch between agent engines.

Litehive was the local-first command-line version. No interface. No web app. Each project had a small .litehive folder containing its configuration and context, while the task queue, reports, agent sessions, and runtime state were stored in SQLite outside the repository.

The hard-coded pipeline from CodeHive became Litehive’s main job. It moved each task through grooming, implementation, testing, acceptance, and a Git commit. I could queue, pause, resume, or requeue work, inspect its logs and journal, and recover a task when a runner crashed or an agent got stuck.

Each task ran in its own git worktree. When it passed the stages, Litehive committed the work, merged it into the main workspace, and pushed it.

The multiple-engine requirement also came from CodeHive. Litehive could run Codex, Claude Code, GitHub Copilot, Gemini, and OpenCode, and I could switch a task to another engine when I was approaching a limit. The skills and context were written in a format any of these agents could read.

I spent more than a month on this project and burned a huge pile of tokens. I rewrote it several times, almost from scratch, but it never became stable.

I hadn’t thought the architecture through properly, so the agents filled in the gaps. The project got complicated, and each rewrite removed some checks and edge cases from the previous version.

The state machine was the one part I really liked, because it made the system deterministic. But with the looser orchestrator I use now, I can launch many sessions at once and let it merge the results. That is much harder with a fixed state machine.

I don’t think I’ll come back to Litehive. It’s simpler for me now to start an agent and let it follow the process I have tuned for Codex and Claude Code.

Looking back, CodeHive and Litehive were part of what helped me improve my approach to managing the agent team.

At first, I thought I needed a full system around a team of agents: roles, queues, status tracking, approvals, multiple clients, and a hard pipeline. Then CodeHive became too large. I extracted the core into Litehive. Then Litehive showed me that even the smaller version was still too rigid and too complicated for the way I actually work.

In the end, I arrived at a simpler setup: I talk to the agents directly and use a process that I have tuned through practice. That turned out to be more flexible than wrapping everything in a fixed application.

But I don’t think I would have reached that conclusion without building CodeHive and Litehive first. They helped me understand which parts of the process mattered and which parts were just extra machinery.

What came out of Litehive

Litehive cost me a lot of time and tokens, but two parts of it were useful enough to extract into separate projects.

Heru

Litehive had to run coding agents without opening their interactive chat interfaces. I wanted it to start Codex, Claude Code, GitHub Copilot, Gemini, or OpenCode from the command line, pass in a task, capture the output, and resume the same session later if needed.

It’s not that simple: all the engines have different flags, output formats, session IDs, and rules for resuming a previous run.

At first, I put all of that provider-specific logic inside Litehive. The same codebase handled the task queue and also knew how to start each CLI, which flags to pass, how to parse its output, where to find the session ID, and how to resume a previous run.

Eventually I pulled the provider-specific part into a separate tool called Heru. I kept the task queue and orchestration logic in Litehive and moved the provider-specific execution logic into Heru. Litehive decided which task should run next and which agent should handle it, and heru translated that decision into the right command for the selected provider.

I named it “Heru” because I wanted one tool to “rule them all”. “Heru” means “Lord” in one of the Elvish languages.

I used it with Claude Code and Codex, but also often with GitHub Copilot when it was cheaper. Now Copilot’s pricing makes it less useful, so I stopped relying on it. But new coding agents keep appearing, and I can easily add them to Heru.

Quse

I work on a lot of projects in parallel, so I often run out of tokens. I use Codex, Claude Code and other providers, so when I hit limits in one tool, I switch to another.

But when I’m working on a long task that I start in Claude Code, and get close to the quota, it can get interrupted at the worst possible moment. I wanted Litehive to manage that situation and switch to another provider automatically. When Claude reached 95% of its available usage, Litehive would send new tasks to Codex or Copilot instead. And when Claude is back to 0%, it can start receiving tasks again.

To make that work, I needed a unified way to check usage quotas across providers. Each provider reports usage differently, so I built a small tool called Quse to handle that logic. The name means “Quota Use”.

Quse normalizes quota checks for Codex, Claude Code, Copilot, and Z.ai. When I run it from the command line, it shows how much usage I have left for each provider and when the limit resets.

This is probably the project from this whole list that I use the most right now.

I also integrated Quse into PocketShell, the phone shell app that I actively use now. When I approach limits, my phone sends me push notification, so I can switch to another provider.

Quse is the small part of Litehive that became a daily tool. It does not run agents on its own, but it gives me the information I need to decide whether to keep using the current provider or move the next task elsewhere.

The Mermaid diagram tool

merm, my Mermaid diagram tool, was another project where I tested the agent-team process.

I built it because I wanted to render Mermaid diagrams on a server without installing Node and a headless browser just to produce an SVG. Merm is a pure Python renderer: it parses Mermaid markup, lays out the diagram, and writes SVG or PNG files.

It has a command-line interface, so I can render a .mmd file or pipe Mermaid text into it. I started with flowcharts, then expanded the renderer to support sequence, class, state, ER, Gantt, pie, mindmap, and git graph diagrams.

I polished this project to a good state. I did not spend much of my own time on it, though it probably used a lot of tokens. More importantly, I used the project to test and refine the agent-team process on a real codebase.

The tool works, but I do not have a strong need for it right now. Most of the time I need to render diagrams in a browser, so I simply use the mermaid.js library.

Maybe I will come back to Merm later if I build a more dedicated workflow for diagrams. For now, this project is not exactly abandoned. It works. I just do not need it enough to use it regularly.

Cleaning up the Dead Weight

Abandoning some projects along the way is a normal part of the experimentation process. Not everything you build has to become useful,.

A lot of projects die. Some just sit on GitHub and never get touched again. And that’s fine.

The valuable part is what you learn from doing the project, or the process you developed while building it. Sometimes it’s a smaller tool that splits off from it. Sometimes it’s simply the realization that you should not solve this problem in this particular way.

For me, if one out of ten projects actually sticks and becomes part of my real workflow, I would already consider that a success.

The important thing is to keep experimenting, notice when something is not working, stop when it is no longer useful, and move on with a better understanding than you had before.

And while preparing this article, I decided to clean up some of my repositories and delete those I no longer needed. There are probably still a lot left that I don’t need, and I’ll likely delete those later too.

What I’ve Been Working on Recently

1) Updated the AI Engineering Field Guide

I continued working on the AI Engineering Field Guide, a data-driven resource about AI engineering roles, skills, interviews, and career paths.

This week, I refreshed the job data behind the guide and updated the repo. It now draws on 4,894 real job descriptions, interview experiences, and practitioner stories. The guide covers role analysis, skills, responsibilities, use cases, interview preparation, company-by-company interview processes, learning paths, and portfolio strategy.

2) Vercel FAQ Agent workshop

I presented the Vercel FAQ Agent workshop at AI Shipping Labs.

We take the FAQ agent from earlier workshops and ship it on Vercel. Now a Next.js app serves the UI, a serverless function runs the agent loop, and the chat model is reached through the Vercel AI Gateway.

The workshop compares four ways to build the same agent on Vercel: AI SDK with MiniSearch, Eve with durable sessions, Eve with Upstash Vector, and a Python FastAPI backend for teams that prefer Python.

I was really impressed with eve and how easy it is to deploy agents with Vercel. But there’s one downside: it’s all in TypeScript.

3) Full-Stack Vibe Coding workshop

I also worked on the Full-Stack Vibe Coding workshop. It was an offline workshop, but I made the materials available for free on the AI Shipping Labs website.

The workshop builds Snake as a full-stack web app end-to-end, with a coding assistant doing most of the work. We start with a React frontend generated in Lovable, write an OpenAPI spec that the frontend and backend agree on, then use a coding assistant to build a FastAPI backend from that spec.

After that, we add a real database, connect the frontend to the backend, run tests, containerize the app with Docker Compose, deploy it to AWS with infrastructure as code, and add a GitHub Actions CI/CD pipeline.

The point is to show how far you can get with a coding assistant, even if you do not have deep frontend, backend, or DevOps experience, as long as you can guide the work and review what the assistant produces.

Interesting Tools

• Omnigent is an open-source meta-harness from Databricks for coordinating coding agents. Instead of relying on one agent, you can run several agents in one session, for example, one agent writes code and another reviews it. Omnigent supports Claude Code, Codex, Pi, the OpenAI Agents SDK, and YAML-defined agents. It also moves some controls out of prompts and into the harness itself: spend caps, approval gates, sandboxed tool execution, and an egress proxy that can inject secrets without exposing them to the agent. The project is released under Apache 2.0.

• LiteParse is a fast open-source document parser from the LlamaIndex team. It runs locally, so you can parse documents without cloud dependencies or proprietary LLM features. It supports PDF, DOCX, XLSX, PPTX, and images; uses PDFium for spatial text parsing with bounding boxes; and can run OCR through bundled Tesseract or an HTTP OCR server. The Rust core has bindings for Python, Node.js/TypeScript, the browser through WASM, and a CLI, so it should fit into most local document-processing pipelines.

This happened when Exasol, a database company, asked me to run a workshop for them. They released a new version of their database, Exasol Personal. It’s normally a paid service, but this edition runs in your own AWS account. You need an account and a few permissions, then you can create a cluster and use it from your laptop.

For me, it was very easy to set it up. But then I started thinking about how to make it scale to 50-60 workshop participants, who will most likely be unprepared.

So I needed to find a way for the participants to provision resources in my AWS account without giving them my AWS keys.

The solution I found turned out to be useful not only for workshops, but also for coding agents. I don’t want to give my agents permanent access to my main AWS account. Instead, I want to restrict them to a sandbox environment and decide when and for how long they can have access.

The workshop problem

I needed to find a way to give 50 or 60 people in a room access to provision AWS resources from their own machines, with nothing to install and no keys to copy. You sit down, open the workshop environment, and it works.

I didn’t want to give the participants my AWS key. First, it’s a very bad idea for security reasons. Second, how do I even do it?

Type it on the screen, send it by email, drop it in a repo for a few minutes? Once a key is out, anyone who has it can provision whatever they want until I disable it. If somebody starts mining Bitcoin on my account, that’s my problem.

I also didn’t want to use my main AWS account. I’m not a security expert, and with 50+ people I don’t know (and don’t necessarily trust), things may go wrong.

And finally, I wanted to run in GitHub Codespaces. It’s very convenient, and I use it for all my workshops. You click a button and get a remote machine with everything installed and the same environment as everyone else. I didn’t know if it’d be possible for this workshop, but I wanted to find an equally convenient option.

EC2 instance profiles

The first thing I thought about was EC2 instance profiles.

You attach a role to an instance, and when you start it, it gets access to all the resources you configured. You don’t need to worry about managing keys: EC2 does it for you.

But to create instances with the profiles, I’d need to distribute my key to the participants again, who would then use it to provision the instances.

So I wanted something that’s like EC2 instance profiles, but it would work without having to share my keys, and ideally outside of AWS – on GitHub Codespaces.

I needed to reproduce the way instance profiles work. The flow in EC2 looks like this:

1. You launch an EC2 instance with a role attached

2. EC2 calls STS and assumes that role

3. STS returns temporary credentials

4. Those credentials become available through the instance metadata service

5. The AWS CLI, boto3, and other SDKs know where to find them

6. When the credentials expire, AWS refreshes them automatically

If I could do something similar, but for Codespaces, it’d solve my problem: when the workshop is over, I just deactivate the profile, and my key is never used.

What I built: a credential endpoint I host

I started looking for alternatives and found this environment variable:

AWS_CONTAINER_CREDENTIALS_FULL_URI

It tells the AWS SDK where to fetch credentials. It is typically used in container environments, but it also works outside them. So I can point it to an HTTP URL and run a small service behind it. The service needs to create temporary credentials, and when they expire, the AWS SDK automatically asks for a refresh.

In my case, I created a Lambda that assumes an AWS role and returns temporary credentials in the format the SDK expects.

It’s the same flow as instance profiles, but I provide the endpoint instead of AWS doing it via the metadata service.

It moves the mechanism off EC2 and lets it run anywhere the SDK runs, including a Codespace.

Wiring it into Codespaces

I didn’t want this anywhere near my personal account. So I created a separate AWS sandbox account, put the Lambda there, and built a repository template for the workshop. The template carried a dev container, so when participants forked the repo and opened it in Codespaces, they got the right tools and the credential URL already configured. They opened a Codespace, ran the workshop commands, and the SDK pulled credentials from the Lambda. They never had to know the Lambda existed.

Second, the repo was public, so anyone with the URL could find the credential endpoint, and I didn’t want the Lambda handing out credentials to the whole internet. So I added a secret check and shared the secret offline during the workshop. I wrote it on the screen. I think it was something funny, like bananas.

You can check the code for this on GitHub:

1. aws-workshop-credentials for the credential-vending Lambda.

2. exasol-workshop-starter for the dev-container template.

The same problem with coding agents

The same pattern helps with coding agents.

Maybe you use Claude Code or Codex too, and sometimes they need AWS. I was running them on my laptop, where I also kept admin AWS credentials. And I am guilty of running them in skip-permissions mode, where the agent doesn’t ask before reading a file, running a command, or changing something. Approving every small step is annoying, so I told it to do whatever it wanted.

What could go wrong?

Plenty. One of the agents dropped my production database. I wrote about that here: How I Dropped Our Production Database and Now Pay 10% More for AWS. When you delete an RDS instance, the backups go with it. I thought I had daily backups to fall back on. I didn’t. I opened support requests, couldn’t recover anything, and recovery only became possible after I upgraded to business support.

This post isn’t about that incident. But that incident is the reason I now think about how agents get AWS access at all. The rule I took from it: an agent should never have a path to production.

My first step was to move agents to a remote sandbox server. If an agent decides to delete something, it can’t do that because it lacks access. The server itself is disposable. I can easily recreate it, so if an agent breaks it, that’s fine.

Second, real deployments go through CI/CD. When I need to apply a Terraform change, I let the agents write the files and push them to GitHub, then I apply the change from my laptop. On my laptop for infra work, I don’t use skip-permissions mode, so I stay in control of anything that touches real infrastructure.

But sometimes I want my agents to experiment with infra on AWS. It happens when I need to work out how to deploy a new service, which resources it needs, and which permissions it requires. For this, I created a separate sandbox AWS account. When agents need access to it, I run a script that writes a credential file into the project folder the agent is working in, and the agent gets access for about an hour. The tool is aws-sandbox-cli.

The result is that agents have enough room to explore and break things, but only within the sandbox, and never on a path that reaches production.

Turning access on from my phone

But sometimes I’m not at my laptop, I’m on my phone, and I still want agents to work. I travel often, and now and then I want to let an agent figure out some inference work while I’m away from my desk.

I don’t have admin AWS credentials on my phone, and I don’t want them there. Reaching my laptop instead isn’t an option either. When I travel, it’s in my backpack, and when I’m out for the day, it’s at home behind a router, so SSHing into it isn’t trivial. And even when I can access the sandbox, I don’t want agents hitting AWS at will. They might provision ten GPU instances, and then I have a very expensive problem.

So I reused the credential URL. The remote sandbox already points its credential URI at a Lambda, so I built a phone app that toggles that Lambda on and off. When the toggle is on, the Lambda hands out credentials. When it’s off, the sandbox asks and fails. The access is temporary, and I can hold it for an hour. When I need an agent to run an experiment, I open the app, flip the toggle on, and flip it off when I’m done. The tool is phone-aws-gate. I connect to the sandbox from the phone with Termius, an SSH client for phones.

For example, when I connect to the remote server and run:

aws sts get-caller-identity

It fails because there is no AWS access. I flip the toggle on my phone; it asks for my fingerprint and grants access for 15 minutes, with a timer running. Now get-caller-identity works. I flip the toggle off, and the access is gone.

Where I use it now

It started as a workshop problem: how do I let 50 or 60 people use AWS without handing out long-lived keys? The answer was a credential endpoint I control. The same endpoint now does two more jobs. It gives coding agents temporary access to a sandbox account and sits behind a toggle on my phone, so I can turn that access on and off from anywhere.

One mechanism covered all three because the underlying need was the same each time: temporary, revocable AWS access with no long-lived keys sitting on a machine.

I run the credential endpoint now, so its security is mine to get right, and the access is only as locked down as I make it. The workshop ran behind a word I wrote on a screen. That was a deliberate trade for a setup I could throw away the same day, not a pattern I’d reuse for anything that had to last.

The rule I follow now is the one the dropped database taught me. Before anything gets AWS access, whether it’s a room full of strangers, an agent, or a script, I ask the same questions. Is the access scoped to a sandbox? Is it temporary? Can I revoke it? And can it reach production? If the answer to that last one is yes, I stop and fix it first.

What I’ve been working on this week

Sprint prep with Codex

I spent a lot of the week getting AI Shipping Labs ready for the next sprint. Codex ran for over two days on platform features, and I gave it a lot of feedback along the way. Most of it was fixes and polish of existing features, making what already exists look and work better. It kept going until everything on my list was done.

Cloudflare Workers workshop

On Wednesday, we ran an AI Shipping Labs workshop on Cloudflare Workers, and it was more interesting than I expected. The idea was to take the FAQ agent from our earlier deployment workshops and move its whole online path onto Cloudflare.

The question we wanted to answer was whether Cloudflare could replace the usual stack for a small FAQ agent: a FastAPI server, a separate vector database, and a paid always-on host.

It mostly can. Cloudflare runs the full online path, with one caveat: ingestion still runs locally as an operator command rather than as a public endpoint, so you load the FAQ into Vectorize yourself when you need to.

We built that path end-to-end, creating credentials and a Vectorize index, ingesting the FAQ, running the Worker locally against the real Cloudflare services, then deploying it and cleaning it up. We also tried a Python Worker rewrite. Workers are still cheap, and for small pet projects, they're close to ideal.

A lighter FAQ assistant, and zerosearch

After that workshop, I tried to port the DataTalks.Club FAQ assistant to Cloudflare Workers and make it as small as possible. The Python layer on Workers doesn’t support extra libraries, so I rewrote MinSearch as a zero-dependency, pure-Python library called zerosearch.

A shared email service for the course management platform

I started wiring email into the course management platform, which wasn’t possible before. It uses Amazon SES to send mail tailored to our setup, starting with the course management platform.

Emails already work in AI Shipping Labs, which I finished a few weeks ago, but I want to pull the logic into a separate service, DataMailer, so all our platforms can share it. My current goal is to get the course management platform emails working first, then move the AI Shipping Labs emails over, and maybe the weekly emails after that. If it ends up replacing MailChimp, great. If not, that’s fine, and we keep MailChimp. But that’s the direction.

Interesting Tools

• Omnigent is an open-source meta-harness from Databricks. It sits above the coding agents you already use (Claude Code, Codex, Pi, the OpenAI Agents SDK, or your own YAML-defined agents) and makes them work together as one system. You can run several agents in a single session, for example, one vendor’s agent reviewing another’s code. Policies like spend caps and approval gates are enforced at the harness layer instead of through prompts, and tools run inside an OS sandbox with an egress proxy that injects secrets the agent never sees. Sessions follow you across the terminal, web, mobile, and a macOS app, and you can share a live one by URL so teammates can watch or co-drive. It’s released under Apache 2.0.

• Ponytail is an AI agent skill that pushes coding agents to write the least code that does the job, what its author calls “the laziest senior dev in the room,” replacing fifty lines with one. Before writing anything, the agent works down a ladder: does this need to exist, does the standard library or a native platform feature already cover it, can it be done in one line? It builds only the minimum that works, but never cuts validation, error handling, security, or accessibility. It installs as a plugin for Claude Code, Codex, and a dozen other agents. On agentic benchmarks, it cut code by around 54% while staying cheaper and faster than the unassisted baseline.

Resources

Ivan Brigida ran a workshop on building a Stock Research Agent. He had clearly prepared a lot, and I liked how he presented it.

Edited by Valeriia Kuka

A lot of the time, that is completely fine. For a simple website, one-shot is more than enough, and it’s way better than what I could ever design myself.

But I don’t want my websites to look too generic. There are thousands of web pages now that all look the same. In writing, we have markers like “delve” or excessive bold formatting that give away AI-generated text. In design, there are similar elements you notice immediately.

One-shot designs also get hard to use. They tend to grow overly complex, with elements not always where I’d expect them. Each new page brings elements that don’t match the rest. Eventually, the whole thing becomes messy.

I am not a designer, and I never liked front-end or design work. But I build a lot of user-facing tools now, both web and mobile. I want them to look nice and clean, with elements placed logically.

One-Shot AI Designs

When you look at AI-generated designs, the first thing you notice is the feature grid: cards with an icon, a heading, a short description, and a colored accent. I find it very generic and often boring. And these left-border accents give away AI designs immediately.

When you ask AI to implement something more complicated than a landing page, it tends to cram elements together. For admin panel interfaces where you want to perform some actions, AI will put a lot of buttons together to the right from the title. I always have to correct it and ask to put the buttons under the title.

Also, coding agents like columns. Agents split forms from metadata panels, or put metrics, tables, and settings panels at the same visual level. The page becomes quite busy quickly.

I can see these patterns repeated across different projects with both Codex and Claude producing similar designs.

My Own Designs

While you can notice AI-generated patterns, my own work is by no means better. I’m not a designer and always wanted to stay as far away from front-end work as possible.

But I still needed to create web pages, like the main DataTalks.Club website that I created back in 2020. I used the Bootstrap framework because it was something I was already familiar with, so it was easier for me to build something on top of that.

It has a headline, a signup form, a few illustrations, and a lot of empty space. It did the job, but you can clearly see that I’m not a front-end engineer.

Another example is the Course Management Platform. This is the platform we use at DataTalks.Club to run courses. For each course we have homework and project submission forms, a leaderboard and a few other useful things.

Previously I had to manage everything via Google Forms and Google Spreadsheets that were stitched together with a bunch of Python scripts. I started working on automating all that in 2023, and again chose Bootstrap as the CSS framework. I ran the first cohort of the Data Engineering Zoomcamp with it in 2024 and it has been saving so much of my time since then.

For a long time that was fine. The platform did its job, and styling was never the priority for me.

Migrating the Course Platform to Tailwind in Codex

In November 2024, one of the course participants filed an issue to migrate the course platform to Tailwind.

I had no time for it, and I knew nothing about Tailwind, so I didn’t implement it. There were attempts to redesign the website from other contributors, but I didn’t like the outcome, so none of that work eventually was merged.

But in 2026, I had a Codex subscription and in a few days my token usage would reset. When it happens, I want to make sure I use my subscription as much as possible, otherwise these tokens would be wasted.

I use these tokens to clear the backlog in many of the open source projects I’ve been working on for the last few years.

One of these projects was the Course Management Platform and I finally reached the Tailwind issue. I had wanted to redesign the platform since the day I created it. It was functional and was already saving a lot of time but the look could definitely be improved. I thought that the day to do it finally had come.

So I dropped the issue into the Codex session and asked it to create a redesign. I don’t have any screenshots from that day, but the output was terrible. I didn’t like the one-shotted design at all. It screamed “AI-generated” at me and looked like millions of other websites on the internet.

Using GPT Image to Generate Interface Mockups

Around that time, GPT Image 2 came out. People on X were posting AI-generated screenshots with the caption “this is not a screenshot”.

I thought that if GPT Image is so good at generating screenshots, maybe it’s also good at generating interfaces.

I took a few screenshots of the Course Management Platform and asked ChatGPT to redesign it. With a few iterations I got a result that I liked. It wasn’t the final result, but the direction was definitely good.

Once the desktop version was ready, I also asked to generate the mobile ones.

Then I gave the results to Codex and asked it to build the pages in that style. The result was far from identical, but directionally it was good.

After that, I needed to iterate on the pages separately. I moved things around a bit until I got the result I personally wanted.

Using GitHub Primer as the Design System

The first pages were ready, and I started redesigning other screens.

I immediately saw that the style drifted from the original design mockups:

• The buttons and other UI elements were placed in unexpected places

• The CSS styles for these UI elements were not consistent across the pages

• Also, the gaps between elements were different. Even the width of the pages wasn’t consistent

I needed to come up with a specification of how exactly the elements should look like, where they should be placed, the gaps between them, the colors and the styles. I needed a design reference that the agents could use when implementing the changes or adding new elements.

I started talking with Codex and told it to find a good reference that actually fits our system, so we can base all the design decisions on it.

Codex suggested GitHub Primer and I liked it a lot, so we documented the result in design-system.md.

The Algorithm

Just giving the mockups and the design system wasn’t enough, so I still needed to check all the pages and see things that I didn’t like. I then asked to move them around until I liked the positions and styling.

There are still a lot of things that I want to improve, especially in the internal part of the website for the instructors. But overall I’m quite satisfied with the outcome. You can check the platform at https://courses.datatalks.club/.

So the algorithm I arrived at was:

1. Describe to ChatGPT what kind of site I want and what should be on the page

2. Ask it to generate a mockup image

3. Iterate on the image until I like the direction

4. Give the image to the coding agent and ask it to implement the layout

5. Check the results, note the things you don’t like, fix a few of them

6. Ask the agent to pick up a reference design system (or a combination of multiple) and create the design system document

7. Add new pages to your project, make sure the agents follow the design system, and when you don’t like what they create design-wise, ask them to document that in the system

This approach also worked well for other websites that I work on, like AI Shipping Labs.

Android Applications

It also works well for other user-facing applications, not only web.

Right now I am working on an app that lets me manage agents from my phone. I call it Pocket Shell. The first version already works, but the interface is still rough.

I took these screenshots and asked ChatGPT to redesign it. It took a few iterations until I arrived at the result I liked (directionally).

I used the mockups to set the direction, then polished the details in code. There’s still a lot of work in the app, but I like the current screens much more.

This workflow helps me a lot with design, even though I still don’t think I’m good at it. Of course, I still rely entirely on coding agents to do the implementation, but at least now the designs that come out of the process look less generic.

What I’ve Been Working On Recently

1. LLM Zoomcamp Launch

We launched LLM Zoomcamp this Monday and 754 people joined the launch session.

This year I reworked most of the content and we also added a new module about orchestration.

The course is up-to-date and if you’re looking to start with AI Engineering, you can do it now. Join us for the next 10 weeks and let’s learn together!

Join LLM Zoomcamp

Also, at the beginning of the stream I asked people how they discovered the course and a lot of people said that AI recommended it. That’s really cool! I’m happy that ChatGPT and Claude are making the right recommendations when it comes to courses.

Also, many AI Shipping Labs community members are doing the course too, and I’ll be hosting office hours there.

2. Vector Search Workshop at AI Shipping Labs

This week I run another workshop at AI Shipping Labs. During our first ever workshop in April I wanted to deploy an agent end-to-end: create a FastAPI backend, a frontend and deploy it to the cloud. My original plan was also to include vector search, but we didn’t have time back then. This week we finally did it.

As the service for hosting the database, we chose Turso and used SQLiteSearch on top of it to do vector search. You can check the code here.

During the workshop, we also discovered CloudFlare Vectorize - a fully serverless vector database. I want to learn how to use it and cover in the next workshops.

My Talk “Managing AWS Access for Humans and AI Agents”

On Tuesday I gave a talk at Berlin AWS Group meetup, where I shared my approach for providing AWS Access in low-trust environments like public workshops or for your AI agents.

I plan to write an article based on this talk, but you can already check the video.

Subscribe now

Tools

• LiteParse is a fast, open-source document parser from the LlamaIndex team that runs entirely on your machine with no cloud dependencies or proprietary LLM features. It does spatial text parsing with bounding boxes via PDFium, handles PDF, DOCX, XLSX, PPTX, and images, and offers selective OCR through bundled Tesseract or any HTTP OCR server. The Rust core ships with bindings for Python, Node.js/TypeScript, the browser (WASM), and a CLI, so you can plug it into a local document pipeline from whatever stack you use.

Edited by Valeriia Kuka

At first, this puzzled me. The answer sounded obvious: install it and start using it. Besides, there’s already plenty of material. Anthropic offers a free Claude Code 101 course and comprehensive Claude Code docs, and OpenAI has a Codex quickstart.

But that’s not what people are really asking. The real question is bigger - It’s not “how do I start with Claude Code” but multiple questions in one:

• How do I configure Claude Code with the correct model and permissions?

• Which slash commands should I learn?

• Do I need skills, subagents, plugins, and MCP servers?

• Should I run it locally, in Docker, or remotely?

• Can I use it from my phone?

• What about all the tricks people post on X every day, like the Ralph loop or teams of agents?

• Can someone show me exactly which buttons to press, and in which order?

I understand where these questions come from. Open X and you see an endless stream of people sharing what they built and which tricks they used. I add to that stream too, with this newsletter.

This creates FOMO. You start thinking you need to figure out everything before you can start. But you don’t, and in this newsletter I’ll explain why.

Here’s how I’d get started with a coding agent if I were starting from scratch:

1. Choose an assistant

2. Use it on a real task

3. Try automation

4. Document repeated work

5. Turn useful documents into skills

6. Use subagents when the context becomes a problem

Work through these steps in order. Don’t start with the advanced setup or you risk getting even more confused.

Subscribe now

Step 1: Choose an Assistant

There are many AI coding assistants now:

• Claude Code

• Codex

• OpenCode

• GitHub Copilot

• Antigravity

• Cline

I used to recommend GitHub Copilot because, for $10 per month, you could get quite far, but that’s no longer how I think about it. Copilot’s billing is now token-based, which makes it expensive to use.

Right now, the best value for many people is ChatGPT Plus, which costs $20/month, because Codex is included with ChatGPT plans.

Claude Code is also included with Claude Pro ($20/month) and Max ($100/month or $200/month). But the $20/month plan limits can disappear quickly, and it won’t get you far. If you use Claude Code a lot, you’d probably need to switch to the $100/month plan.

The tools differ, and I can’t say one is clearly better for everyone. At the beginning, pick the one that fits your budget.

Don’t spend too much time comparing them. You’ll learn more from one finished task than from reading another thread about which agent is best.

Step 2: Use The Assistant On One Real Task

After you pick an agent, use it on something small enough to check end-to-end.

For example:

• Write a Python or bash script

• Solve homework from a course

• Build a mini-project, like the snake game we did in AI Dev Tools Zoomcamp

• Fix a small problem in a project you already use

You don’t need MCPs, skills, subagents, plugins, or custom prompts yet. At this stage, you only need to check two things: can the agent solve your problem, and how do you like to work with it?

Decide How Much Permission To Give It

When you start, the agent asks for permission for every action. It gets annoying fast.

You have two options:

• Build an allow-list of approved actions: Every time the agent requests a command, you approve it and tell it not to ask for that action again. Over time, the useful commands become automatic.

• Run the agent in skip-permissions mode, also called YOLO mode: The agent can run any command executable from your terminal. You have to understand the risk. If the agent decides to delete something, it can.

I run most of my projects in YOLO mode. Sometimes speed matters more than control. I want to hand the agent a task for the night and walk away without having to approve every single step.

For serious tasks, I use the allow-list approach. If I’m touching production infrastructure, Terraform, cloud permissions, billing, or anything where a mistake is expensive, I don’t use YOLO. Well, I tried once, and it taught me a good lesson.

There’s no universal rule. It depends on how much control you want and how much the result matters.

Step 3: Use Agents For Automation

I’d been using coding agents for a long time, and they were already useful, but I realized how valuable they are only when I started using them for automation.

For example, right now, I’m preparing videos for LLM Zoomcamp. I need to download videos from YouTube, cut them into separate chunks, and upload them back as individual videos. I also want each video to have time codes, so I download the subtitles and turn them into chapters.

I delegate most of this to Claude: I only need to

• Check that the cuts are clean

• Bulk-upload the videos, because the support for that in YouTube API is limited

• Hit Save so each video goes from draft to published, because there’s no API for that either

The rest is on Claude: it sees the uploaded videos, pulls their IDs, updates the title and description, adds the timecodes, and puts them into the playlist in the right order.

Creating this kind of automation is pretty simple. I just had to describe what I needed, and Claude found the video links, downloaded them, fetched the transcripts, cut the videos with ffmpeg, and walked me through setting up YouTube API access. After that, I checked the cuts and uploaded the videos.

Overall, if a service has an API or a command-line tool, an AI agent can figure out how to use it from the terminal.

Over the last year, I automated many things this way:

• Creating homework submission forms for courses.datatalks.club

• Creating GitHub repositories

• Publishing new versions of the libraries I maintain

Step 4: Document Repeated Processes

Once I automate something and know I’ll need it again, I document it.

Usually, I work with the agent until the task is done, and then I ask it to write down the process in a markdown file. The LLM Zoomcamp docs for cutting YouTube videos live here.

Next time, I can point the agent at the doc, and when I needed to add more videos to the LLM Zoomcamp playlist, that’s exactly what I did.

Without a doc, the agent starts from scratch and burns time and tokens figuring out the same thing again.

For YouTube, this may not matter much, since there’s plenty of information online and the agent already knows a lot. But for less common workflows, it doesn’t have the context.

For example, I use agents to publish homework to courses.datatalks.club. To do that, the agent needs to know:

• The production URL

• Where the API key lives

• What is available in the API

• The exact payload shape

• What to do when things go wrong

Without a doc, I need to explain this every time.

With a doc, I can just say “use this process” and let the agent continue from there.

Step 5: Turn Useful Documents Into Skills

Once a document becomes part of your regular process, turn it into a skill.

A skill is a markdown file saved in a specific folder with a specific name.

For Claude Code skills, the folder is:

.claude/skills/<skill-name>/SKILL.md

For Codex skills, the folder is:

.codex/skills/<skill-name>/SKILL.md

The file needs YAML frontmatter with a name and a description. That’s the only required part.

Here’s an example from my course-management-agent repo:

---
name: course-content
description: Manage courses, homeworks, and projects via REST API
---
# Course Content API
## Overview
This skill provides commands to manage courses, homeworks, and projects via the REST API. Supports list, create, update state/dates/description, and guarded delete.

Once the file is in the right place, the agent picks it up automatically. With a skill in place, I can write a request as usual: “Here is the homework, create a form for it.” The agent recognizes the skill, loads it, and does the rest.

Without the skill, I need to point it at the docs every time.

You don’t have to write skills by hand. Ask the agent to document the process and turn it into a skill, and Claude Code and Codex both have built-in help for creating skills, so they can put the file in the right folder and format.

If you’re wondering whether something should become a skill, don’t overthink it. Document what you do at the end of each coding session, and if you come back to that document repeatedly, turn it into a skill.

Step 6: Use Subagents When Context Gets Too Large

As you work with an agent, every question and answer accumulates in the context. Eventually, the context fills up and the quality drops.

I first noticed how useful subagents are while setting up my Telegram writing assistant.

It works like this:

• I send it content: voice notes, text messages, links, videos

• I run /process from Telegram

• Claude turns the pile into articles, summaries, and saved notes

The /process command is a skill that describes how to turn the input into writing material.

Now imagine I send this:

• Ten voice notes that should become two or three articles

• A tweet I liked and want to save

• A three-hour YouTube video I want to summarize

If I run /process and the main agent handles everything in the same context, the long YouTube transcript pollutes the rest of the work: the article draft gets worse, the summary isn’t great either, and after that, there’s no clean context to continue from.

One way out is to process the simple items first and handle the YouTube video in a separate session later.

But there’s an alternative: the main agent starts a subagent with specific instructions: summarize this transcript. The subagent runs in its own context, saves the result, and reports back. The main agent only sees the task, the status, and the final summary. The huge transcript never enters the main context.

That’s the main reason I use subagents.

If the task is small and the context is clean, I keep it in the main session. If the task is large, noisy, or research-heavy, I send it to a subagent.

Each subagent works on a specific task:

• Summarize this long transcript and save the notes

• Inspect this large repository and report the architecture

• Review these logs and extract the failure pattern

• Search these files and return the exact places that need changes

Other Tips

Keep Project Context Files

In my own projects, I usually create a CLAUDE.md and an AGENTS.md.

Claude Code reads CLAUDE.md. Codex and OpenCode read AGENTS.md.

These files explain the project to the agent:

• What the project does

• How to run tests

• Which commands are safe

• Where important files live

• What conventions to follow

• What not to touch

The agent reads this file when it starts. That saves me from explaining the project from scratch each time and spares the agent from crawling through the whole repository to figure out how things work.

Because I reset sessions often, these files matter a lot.

Reset The Session Often

The subcommand I use most often is the one that resets the session. It’s called reset, new or clear, depending on the agent.

Every new task starts with a clean context. The agent reads the most important instructions from AGENTS.md and starts working on the task

Use Goals For Long-Running Work

Another command I use actively right now is goal.

You give the agent a completion condition in plain language, for example:

/goal work through the backlog

Then the agent keeps starting new turns until the condition is met.

It’s the productized version of the Ralph loop: a loop that keeps feeding work to a the agent, until a certain condition is met.

Codex shipped it first. Claude Code added a near-identical version shortly after, with a separate small evaluator model that checks whether the condition is satisfied after each turn and, if not, sends guidance and starts another turn. It pairs well with skip-permissions mode, so each turn runs without approval prompts.

Beyond reset and goal, I don’t use many slash commands.

My Setup Lives In Dotfiles

To understand what others are doing, I recommend that you first take all the steps that I described in the article. Complex setups can be confusing, especially when you’re just starting to work with agents.

Here, I’d like to share my own setup, which is available in a public repository: github.com/alexeygrigorev/.claude. Despite the name, it bootstraps and configures all three agents I use from one shared clone. Every machine and every assistant gets the same setup.

Here’s what’s inside this repo and the role of each file:

• An installer script clones the repo

• A configure script wires everything into the right home-directory locations

• The configure step takes a target: claude, codex, opencode, or all

• A config/ folder holds per-assistant settings

• A skills/ folder holds the shared skills

• A shared .bashrc defines the aliases I use daily

The skills are symlinked into each assistant, so Claude Code, Codex, and OpenCode can share them.

Some examples from that repo:

• create-github-repo

• fetch-youtube

• fetch-loom

• fetch-google-recorder

• init-library

• jina-reader

• openai-transcribe

• regular-ping

• release

• setup-pypi-ci

• stylint

The .bashrc also defines short aliases. I use c for Claude, cc for continue-session, csp for Claude with skip permissions, cy for Codex in bypass mode, and oc for OpenCode.

The repo also has guardrails. Claude settings register a PreToolUse hook that blocks dangerous commands unless I confirm them. For example, it blocks rm -rf /, force pushes, dropping a database, and terraform apply.

This setup isn’t required on day one. I built it after I’d faced the same tasks enough times to get annoyed. Don’t overcomplicate things, install any agent and just start experimenting with it for solving your tasks. Along the way, you’ll figure out the rest and which agent capabilities you really need.

Subscribe now

What I’ve Been Working On This Week

Preparing For The Upcoming Workshop On Running Durable Agents In Production

I’ve been preparing for the upcoming workshop on Running Durable Agents in Production.

Agent demos often work well until something interrupts them: the process crashes, a deploy happens, a tool call fails, or an approval never comes back. In production, this means the workflow needs a way to recover, continue, and show what happened.

That’s what Nicholas Lotz will cover in the workshop: how to save state outside the agent process, resume after failures, retry tool calls, pause safely for approvals, inspect execution traces, and deploy multi-agent apps to real infrastructure.

Register here

Monitoring LLM Applications Workshop

I also worked on a workshop about monitoring LLM applications.

In the workshop, we build a Streamlit chat app, store conversations in PostgreSQL, capture LLM call metrics, add user feedback, use an LLM judge, and put everything on dashboards. We also go through Grafana, synthetic data, Docker Compose, and a few next steps, like OpenTelemetry and alerting.

Freestyle Workshop At AI Shipping Labs

For AI Shipping Labs, I did a freestyle coding workshop on hosting open-source LLMs with vLLM on rented RunPod GPUs.

We compared Cloud Run and Lambda pricing, picked GPUs, calculated VRAM for DeepSeek models, fixed out-of-memory errors with quantization, set up SSH, used uv, kept things alive with tmux, protected endpoints with API keys, and packaged everything into a Docker image.

LLM Zoomcamp 2026 Pre-Course Live Q&A

I also hosted the LLM Zoomcamp 2026 pre-course Q&A.

The live cohort starts on June 8, so this was a chance to answer questions before the course begins. We talked about what we’ll build, who the course is for, prerequisites, homework, the leaderboard, certificates, capstone projects, and how to get the most out of the DataTalks.Club community.

LLM Zoomcamp is free and open-source. Basic Python and command-line experience are enough to get started.

Tools

• PaperBanana: an agentic framework that automates the creation of publication-ready methodology diagrams and statistical plots directly from paper text, references, or even rough sketches. By orchestrating specialized agents for retrieval, planning, rendering, and self-critique, it reduces the manual bottleneck of academic illustration while maintaining scientific accuracy and visual consistency. It is built specifically for researchers who want high-quality figures without spending hours in drawing tools.

• Dexter: an autonomous financial research agent that thinks, plans, and learns as it works. It performs analysis using task planning, self-reflection, and real-time market data. Think Claude Code, but built specifically for financial research. It features intelligent task planning, autonomous execution, self-validation, real-time access to financial data, and safety features such as loop detection and step limits.

Interesting Resources

• Claude Code and Large-Context Reasoning: materials from a hands-on O’Reilly Live Learning course by Tim Warner that teaches how to build production-ready AI-assisted development workflows with Claude Code. It covers large-context reasoning, MCP-based persistent memory, agents, and custom skills, with practical examples for code review, automation, and CI/CD.

• awesome-slash: a curated GitHub list of tools, patterns, and projects built around slash-command interfaces. It’s a practical reference for anyone designing command-driven workflows, bots, or developer tools that rely on concise, action-oriented commands instead of complex UIs.

Edited by Valeriia Kuka

Search, or retrieval, is one of the most important parts of RAG. I needed a way to teach it without asking participants to install Docker or Elasticsearch.

So I built minsearch: a small in-process Python search library. It started as the smallest thing I needed to teach retrieval in a notebook, then grew as the course examples changed.

In this post, I will share:

• Why was Elasticsearch too much for this setup

• How the first version worked

• How it became a PyPI package

• Why I added an appendable index and vector search

• How I used Claude to make it faster

• When minsearch is the right tool

Subscribe now

Why Elasticsearch Was Too Much

In the first run of LLM Zoomcamp, I needed to show participants how to index a small dataset, submit a query, and retrieve relevant documents. Usually, it was a few thousand documents, sometimes fewer. Most of the examples were run in notebooks, sometimes on Google Colab, using open-source LLMs on a GPU. And working with notebooks was my main motivation for creating minsearch as a lightweight alternative to more complex search engines.

For a normal production system, I would probably reach for Elasticsearch. It is powerful, and I knew it well. But for a workshop notebook, it was too much. It requires a server, Docker, configuration, and operational details that weren’t the point of the lesson.

In the course or workshop setup, everything should run within a single notebook. I looked for a small Python library that could do a good-enough lexical search within the same Python process as the notebook, but I didn’t find anything that fit.

At that time, I had been doing text processing and search for quite some time, so building a small in-process search library myself wasn’t hard. Even back then, when coding agents weren’t as good as they are now, I could describe what I wanted to a chat assistant, get code back, and ask for a few fixes.