DevOps and Observability for an AI-Built App

Part 4 of AI Dev Tools Zoomcamp

This is the forth article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.

All articles in the series:

• Part 1: AI-Native Development: Specifications, Loop and Graph Engineering

• Part 2: Build and Ship a Full-Stack App with AI Coding Assistants

• Part 3: Deploy a Full-Stack App with AI Coding Assistants

• Part 4: DevOps and Observability for an AI-Built App (this article)

• Part 5: TBA

In part 2, we developed an end-to-end application for conducting system design interviews. In part 3, we deployed it to AWS.

In this part, we will take the application we deployed previously, and make it more production-ready:

• Define dev and prod environments

• Introduce observability: logs, metrics, alerts

• Use AI as the first responder when the application stops working

This article is a draft and will be updated after the workshop

Overview

Right now, every push to main goes straight to the one environment we have, and nothing tells us the app is unhealthy until we notice by hand. We fix that in three steps: separate dev from prod so a bad change never reaches users directly, add the telemetry and alerting that would have caught last part’s failure on its own, and give an agent a narrow, auditable way to respond.

Dev and prod environments

So far, a push to main deploys straight to the only environment we have. That’s fine for a demo, but it means every change - tested or not - reaches the same instance our users hit.

We want a place to deploy to first, and a deliberate, separate step to promote a change from there to production.

We already have infrastructure as code for our deployment from the previous part, and it already parameterizes what differs between environments, such as the domain and the instance size. That means we don’t design anything new - we ask for a second, independent copy of what we already have, and treat what we already have as dev from now on.

Create a second, independent copy of our deployment infrastructure for a
production environment.

It must be able to run alongside the existing one with its own database and compute. The existing becomes the dev environment.

Once prod exists, split the deploy step in CI/CD so each environment gets its own path:

Split our CI/CD deploy step into two:

Deploy to dev: on every push to main, after tests pass, deploy to the dev
environment automatically (we already do it today)

Deploy to prod: a manual workflow that promotes the version running in dev to prod.

The key property is that prod never builds from source on its own - it redeploys the exact commit that already proved itself in dev. Promotion is a deliberate, logged action, not a side effect of pushing code.

Instrumenting

Two environments buy us a safe place to promote from, but not visibility. A bad deploy can still sit in prod for hours before anyone notices, because nothing yet tells us the app is unhealthy. And even once we notice, “error rate went up” isn’t enough - we need to get from that to one failed request, its log, and the commit that served it. A dashboard that only shows CPU and memory can look perfectly fine while the one thing users care about is silently failing. That’s what we fix next.

We use OpenTelemetry for this: a vendor-neutral format for traces, metrics, and logs. A metric tells us the canvas-broadcast error rate went up; a trace shows one failed request’s path through the backend; a log holds the exception. Tagging all three with the same service name, environment, and deployed commit is what lets us walk from a spike on a chart to the one request, and the one log line, behind it.

We ask the coding assistant to instrument one important operation first:

Instrument the FastAPI backend with OpenTelemetry.

Export traces and metrics with OTLP (no collector yet). Include service name, environment and deployed git commit

The app sends OTLP to an OpenTelemetry Collector, which exports it to whichever backends we pick.

Exporting straight from the app to each backend instead would mean an application change every time a backend changes, and a slow backend could stall the app’s own exporters. The collector is one stable boundary that handles batching and retries outside the request path.

Add OTLP collector.

Create the observability/ directory with Docker Compose for an OpenTelemetry
Collector, Prometheus, Loki, Tempo, and Grafana.

Build the dashboard

The collector is running, but nothing draws it yet. We ask for one panel - the one the rest of this article leans on:

Add one Grafana panel: canvas-broadcast error rate, with exemplars enabled so
a point links to its trace.

Look at it locally first

Before shipping anywhere, we run the stack next to the app on our own machine and generate one real trace, so we know the prompt above actually worked before going looking for it in dev or prod:

docker compose -f observability/docker-compose.yaml up -d

Then we hit the app a few times so there’s something to see, and open Grafana at

http://localhost:3000

- no tunnel needed here, since nothing on a laptop is public in the first place. Once the panel we just asked for shows real numbers, it’s safe to ship.

Run it in dev and prod, and reach it privately

A demo dashboard on a laptop can’t tell us about a real failure, so this stack needs to run next to the app in every environment. We deploy it the same way as everything else - push to main, dev picks it up automatically, then a promotion ships it to prod:

Deploy the observability/ stack alongside the app, in every environment.

Follow one failed request

Let’s break the app:

Introduce a bug that makes canvas broadcasts fail for a fraction of calls.

Same pipeline as before: main, then dev, then a promotion to prod. Then we use the app normally until one edit fails.

With that one failure in hand, we open the dashboard and answer these, in order, by moving between panels rather than guessing:

1. Did users receive errors? - the canvas-broadcast error-rate panel moved.

2. Which operation failed? - that’s the panel that moved; the metric is already labeled by operation.

3. Can we open one representative request? - click an exemplar on that panel to jump straight to one trace.

4. Which log belongs to that request? - the trace carries a trace ID; look it up in Loki.

5. Which deployment served it? - the trace and the log both carry the deployed commit as an attribute.

A dashboard that can’t answer these - even a good-looking one - hasn’t instrumented the incident we actually care about.

Create one actionable alert

We don’t import a large alert pack - just one alert for a sustained increase in failed canvas broadcasts, requiring both:

• enough requests failed to affect users

• the failure lasted long enough that one brief blip doesn’t page anyone

This follows the Prometheus alerting guidance: keep alerts few, tolerate small blips, and page on symptoms that need action. CPU, memory and connection counts still belong on the dashboard - they help explain an incident, they just don’t need to wake anyone up on their own.

We ask the assistant for the rule and its test together:

Create a Prometheus alert for a sustained canvas-broadcast error rate.

Wake the on-call engineer

Everything from here lives in one folder in the app repo, on-call-engineer/. Rather than stand up something to receive a webhook, we poll: a script checks Prometheus’s alert API on an interval and starts the on-call agent the moment our alert shows up firing.

Add an on-call-engineer/ folder with a script that polls Prometheus's
/api/v1/alerts every minute.

For this proof of concept, that script runs from a cron job on our own machine. Nothing needs to be reachable from the internet for this to work - the script reaches out to Prometheus, not the other way around.

A cron job on a laptop only works while someone remembers to leave it running, though. The production version of the same idea is a scheduled serverless function - an AWS Lambda on an EventBridge schedule, a Cloud Run job on Cloud Scheduler, or your platform’s equivalent - firing the identical poll script every minute. Nothing runs between checks: the platform starts the function, it polls, and if the alert isn’t firing it exits and gets torn down immediately. That’s a stronger version of “provisioned only when needed” than standing up a box on alert, since even the polling itself costs nothing while the app is healthy. We don’t build that here, but the poll script doesn’t change to get there - only what’s calling it on a timer.

Let it investigate and fix

The agent isn’t handed a pre-built evidence bundle - it collects its own, with read access to exactly the systems it needs: Prometheus, Loki, Tempo, and git log. And it’s allowed to do more than look: edit code in a fresh checkout, run the test suite, and push a fix if it has one.

You are the on-call agent for one production alert: {alert.json}

You have read access to Prometheus, Loki, Tempo, and git log. Investigate
first - write your evidence and reasoning to incidents/<timestamp>/, and
don't call something the root cause unless the evidence rules out the main
alternatives.

If you find a fix you're confident in, make it on a fresh branch, run the
test suite, and push only if it passes. If you're not confident, or nothing
you tried passes, don't push - write an escalation instead (see below).