<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Alexey On Data]]></title><description><![CDATA[Practical writing on building, testing, and operating AI systems, workflows, and automation, based on my projects and experiments. Subscribe to get ideas you can reuse in your own work.]]></description><link>https://aishippingblog.com</link><image><url>https://substackcdn.com/image/fetch/$s_!otON!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png</url><title>Alexey On Data</title><link>https://aishippingblog.com</link></image><generator>Substack</generator><lastBuildDate>Thu, 19 Nov 2026 15:34:43 GMT</lastBuildDate><atom:link href="https://aishippingblog.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Alexey Grigorev]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[alexeyondata@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[alexeyondata@substack.com]]></itunes:email><itunes:name><![CDATA[Alexey Grigorev]]></itunes:name></itunes:owner><itunes:author><![CDATA[Alexey Grigorev]]></itunes:author><googleplay:owner><![CDATA[alexeyondata@substack.com]]></googleplay:owner><googleplay:email><![CDATA[alexeyondata@substack.com]]></googleplay:email><googleplay:author><![CDATA[Alexey Grigorev]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[A Free Course on Generative Media Starts on Monday]]></title><description><![CDATA[New Zoomcamp: Production AI Agents for Generative Media]]></description><link>https://aishippingblog.com/p/a-free-course-on-generative-media</link><guid isPermaLink="false">https://aishippingblog.com/p/a-free-course-on-generative-media</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 13 Nov 2026 22:44:03 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Z9Eg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Z9Eg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 424w, https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 848w, https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 1272w, https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Production AI Agents for Generative Media: a free, open source, project-based course with DataTalks.Club and Nebius. An agent plans, writes and checks image, voice and video assets on serverless GPUs, from 1 to 1,000+ records, with a cost per asset&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Production AI Agents for Generative Media: a free, open source, project-based course with DataTalks.Club and Nebius. An agent plans, writes and checks image, voice and video assets on serverless GPUs, from 1 to 1,000+ records, with a cost per asset" title="Production AI Agents for Generative Media: a free, open source, project-based course with DataTalks.Club and Nebius. An agent plans, writes and checks image, voice and video assets on serverless GPUs, from 1 to 1,000+ records, with a cost per asset" srcset="https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 424w, https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 848w, https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 1272w, https://substackcdn.com/image/fetch/$s_!Z9Eg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef5cbb76-1d98-4b7b-93cc-7dc30cc679a0_2560x1280.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>On Monday, we start <a href="https://github.com/DataTalksClub/production-ai-agents-for-media-generation">Production AI Agents for Generative Media</a>. It&#8217;s a new free course at DataTalks.Club, and we run it together with Nebius.</p><p>In the course, we generate images, voice, and video with open-weight models. We need GPUs to run these models, and Nebius gives course participants GPU credits through the <a href="https://dev.nebius.com/builders">Nebius AI Builder Program</a>.</p><p>Generating one image is easy. But when we need a thousand images or videos, we quickly run into problems.</p><p>In the course, we deal with these problems:</p><ul><li><p>GPUs are expensive, and we pay for every hour</p></li><li><p>GPU jobs fail halfway, and we don&#8217;t want to pay for the same work twice</p></li><li><p>Videos are large, so sending them back in an API response is slow</p></li><li><p>Agents aren&#8217;t reliable, and the quality changes from run to run</p></li></ul><p>We build an agent that takes a short brief and produces an image, a voice track, or a video. It checks the result, and if the result isn&#8217;t good enough, it tries again. Then we scale it from one record to thousands. </p><p>In this post, I&#8217;ll cover:</p><ul><li><p>What we build in the course</p></li><li><p>The content of each module</p></li><li><p>Our capstone project at the end of the course</p></li><li><p>How to apply and how the GPU credits work</p></li></ul><p>We also talked about the course and answered many questions in the recent Q&amp;A stream. You can watch it here:</p><div id="youtube2-iACtrkiXndw" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;iACtrkiXndw&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/iACtrkiXndw?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><h2><strong>The Media Factory</strong></h2><p>In the course, across all four modules, we build a media factory. We give it a record from our data, and it makes a personalized card or ad for it.</p><ul><li><p>First, the agent plans what the asset should look like and writes the text for it</p></li><li><p>Then it calls GPU models to make the image and the voice or video</p></li><li><p>If the result doesn&#8217;t look good enough, the agent throws it away and tries again</p></li></ul><p>We start with one record and then scale to thousands.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-DV8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-DV8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!-DV8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!-DV8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!-DV8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-DV8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The Personalized Media Factory: a brief goes to an agent, GPU models generate image, voice or video, a check sends assets that aren't good enough back to revise and regenerate, and the factory scales from 1 to 1,000+ records with a cost per asset&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The Personalized Media Factory: a brief goes to an agent, GPU models generate image, voice or video, a check sends assets that aren't good enough back to revise and regenerate, and the factory scales from 1 to 1,000+ records with a cost per asset" title="The Personalized Media Factory: a brief goes to an agent, GPU models generate image, voice or video, a check sends assets that aren't good enough back to revise and regenerate, and the factory scales from 1 to 1,000+ records with a cost per asset" srcset="https://substackcdn.com/image/fetch/$s_!-DV8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!-DV8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!-DV8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!-DV8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3481d0-bac0-4c75-a6eb-4b93768a242e_3360x1800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The media factory that we build during the course.</figcaption></figure></div><p></p><h2><strong>Course Structure</strong></h2><p>We start the course on Monday with a launch session, then go through four modules, and finish with the capstone project.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!YODM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YODM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!YODM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!YODM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!YODM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YODM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/caf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Course structure: Module 0 Welcome, Module 1 Make it work, Module 2 Make it intelligent, Module 3 Make it scale, Module 4 Make it production-ready, then the capstone&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Course structure: Module 0 Welcome, Module 1 Make it work, Module 2 Make it intelligent, Module 3 Make it scale, Module 4 Make it production-ready, then the capstone" title="Course structure: Module 0 Welcome, Module 1 Make it work, Module 2 Make it intelligent, Module 3 Make it scale, Module 4 Make it production-ready, then the capstone" srcset="https://substackcdn.com/image/fetch/$s_!YODM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!YODM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!YODM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!YODM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf85784-2ea3-4822-a621-0b45c4a5f951_3360x1800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A welcome session, four modules, and a capstone.</figcaption></figure></div><h2><strong>Module 1: Consistent Media Generation on Serverless GPUs</strong></h2><p>In the first module, we make media generation work. We give it two weeks, so people who join after the start can still catch up.</p><p>In this module, we:</p><ul><li><p>Get an overview of image, video, and audio models</p></li><li><p>Generate our first image, video, or audio in a few minutes</p></li><li><p>Package a model in a container</p></li><li><p>Run it as an Endpoint for serving and as a Job for batch inference</p></li><li><p>Measure how much one asset costs</p></li></ul><p>In the homework, we:</p><ul><li><p>Generate an asset for our own creative idea</p></li><li><p>Calculate when self-hosting becomes cheaper than paying an API</p></li><li><p>Write a one-page proposal for the capstone</p></li></ul><h2><strong>Module 2: Multi-Agent Media Workflows</strong></h2><p>In the second module, we build the agent. It runs on a CPU and uses the GPU models as tools.</p><p>In this module, we:</p><ul><li><p>Submit long-running generation jobs and poll for the result instead of waiting for minutes</p></li><li><p>Resume a failed run without paying for the same GPU work again</p></li><li><p>Build the agent from a planner, a generator, and a reviewer</p></li><li><p>Check each asset against a versioned rubric and generate it again if it doesn&#8217;t pass</p></li><li><p>Define a creative direction and keep the agents aligned with it</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5js5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5js5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!5js5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!5js5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!5js5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5js5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A CPU agent with planner, generator and reviewer calls GPU models as typed tools: image, voice, video and speech to text. Those tools run on Serverless AI Endpoints and Jobs, and results go to object storage&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A CPU agent with planner, generator and reviewer calls GPU models as typed tools: image, voice, video and speech to text. Those tools run on Serverless AI Endpoints and Jobs, and results go to object storage" title="A CPU agent with planner, generator and reviewer calls GPU models as typed tools: image, voice, video and speech to text. Those tools run on Serverless AI Endpoints and Jobs, and results go to object storage" srcset="https://substackcdn.com/image/fetch/$s_!5js5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!5js5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!5js5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!5js5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e517546-a7bc-46c9-8ebd-8aa95f7e8f51_3360x1800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A CPU agent, with GPU models as tools.</figcaption></figure></div><p>In the homework, we:</p><ul><li><p>Build an agent that makes a coffee ad of about 10 seconds</p></li><li><p>Let it write a plan, make a keyframe (one still image from the video), and review it before it generates the video</p></li><li><p>Trace every step to see how much each one costs</p></li></ul><h2><strong>Module 3: Batch Media Generation at Scale</strong></h2><p>In the third module, we scale the factory to thousands of records. Some workers will fail, and we don&#8217;t want to lose what they already finished.</p><p>In this module, we:</p><ul><li><p>Spread the records across parallel Jobs and save a checkpoint after each item</p></li><li><p>Kill a run on purpose and check that it continues where it stopped</p></li><li><p>Run the Jobs on spot (preemptible) GPUs, which cost $0.80 to $1 per hour last week</p></li><li><p>Put videos in S3-compatible storage and pass links instead of sending the files in API responses</p></li><li><p>See how cold starts, generation time, and the GPU type change the cost of one asset</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!w1zK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!w1zK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!w1zK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!w1zK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!w1zK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!w1zK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Records fan out to spot GPU workers with a per-item checkpoint. A preempted worker returns its items to the queue. Each result is written once to object storage, with a cost report. A killed run resumes and skips items that are already done&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Records fan out to spot GPU workers with a per-item checkpoint. A preempted worker returns its items to the queue. Each result is written once to object storage, with a cost report. A killed run resumes and skips items that are already done" title="Records fan out to spot GPU workers with a per-item checkpoint. A preempted worker returns its items to the queue. Each result is written once to object storage, with a cost report. A killed run resumes and skips items that are already done" srcset="https://substackcdn.com/image/fetch/$s_!w1zK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!w1zK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!w1zK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!w1zK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff278e196-e44e-498c-8831-8a7d93a6a0fa_3360x1800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Parallel Jobs on spot GPUs, with a checkpoint for each item and a cost report.</figcaption></figure></div><p>In the homework, we:</p><ul><li><p>Generate a card text, an image, and a voice track for 200 records, with 10 spot GPU Jobs in parallel</p></li><li><p>Kill the run halfway and resume it without regenerating finished items</p></li><li><p>Calculate how much one asset costs, counting only the assets that passed the quality check</p></li></ul><p>Everyone uses the same records and the same models, so we can compare the costs on a public leaderboard.</p><h2><strong>Optional lesson: Fine-Tuning</strong></h2><p>We&#8217;ll also have an optional lesson about fine-tuning an image model for a specific style.</p><p>A few weeks ago, Nebius ran a hackathon in Berlin about it. People trained LoRA adapters to generate media in a specific style. It took them only a few hours, and the results became much more consistent.</p><p>In the lesson, we&#8217;ll show how to do the same.</p><h2><strong>Module 4: Evaluation, Observability and Production</strong></h2><p>In the last module, we make the system production-ready.</p><p>In this module, we:</p><ul><li><p>Add cheap deterministic checks and a multimodal judge that we calibrate on our own labels</p></li><li><p>Add a golden set to evaluate the system</p></li><li><p>Trace every call to see how much generation costs and how long it takes</p></li><li><p>Define an SLO (a target for speed and reliability) to know when something needs our attention</p></li><li><p>Look at auth, quotas, and abuse</p></li><li><p>Look at prompt injection, content safety, and licensing</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!YmNY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YmNY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!YmNY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!YmNY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!YmNY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YmNY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Module 4: evals at the system, agent and media layers, a trace with cost and time per step, and an SLO with a dashboard and an alert&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Module 4: evals at the system, agent and media layers, a trace with cost and time per step, and an SLO with a dashboard and an alert" title="Module 4: evals at the system, agent and media layers, a trace with cost and time per step, and an SLO with a dashboard and an alert" srcset="https://substackcdn.com/image/fetch/$s_!YmNY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!YmNY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!YmNY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!YmNY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bdfe72a-156b-47f3-8ef9-e004b51ccbb8_3360x1800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Evals, a trace with cost and time per step, and an SLO.</figcaption></figure></div><p>There&#8217;s no homework in this module. Instead, we apply all of it in the capstone project.</p><h2><strong>The Capstone</strong></h2><p>For the capstone, you build your own agent with at least two GPU media models. You run it on Nebius Serverless AI and share it publicly.</p><p>We&#8217;ll grade your system and the engineering behind it, not how pretty the pictures are (but pretty images will be a plus!).</p><p>Your project needs to include:</p><ul><li><p>An agent with 2 or more tools</p></li><li><p>An async GPU workload</p></li><li><p>Assets in object storage</p></li><li><p>An eval suite</p></li><li><p>Tracing</p></li><li><p>An SLO</p></li><li><p>A cost report</p></li><li><p>Crash recovery</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!orhU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!orhU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!orhU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!orhU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!orhU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!orhU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/abe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Capstone requirements: an agent with 2 or more tools, an async GPU workload, assets in object storage, an eval suite, tracing, an SLO, a cost report, and proof that a killed run resumes without redoing finished work&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Capstone requirements: an agent with 2 or more tools, an async GPU workload, assets in object storage, an eval suite, tracing, an SLO, a cost report, and proof that a killed run resumes without redoing finished work" title="Capstone requirements: an agent with 2 or more tools, an async GPU workload, assets in object storage, an eval suite, tracing, an SLO, a cost report, and proof that a killed run resumes without redoing finished work" srcset="https://substackcdn.com/image/fetch/$s_!orhU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!orhU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!orhU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!orhU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabe71e6b-a67c-4bd9-8137-024a898912fd_3360x1800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">What you&#8217;ll build in your capstone.</figcaption></figure></div><p>You can use any data you want. If you don&#8217;t know what to work on, we&#8217;ll share some ideas, like a marketing studio or a podcast producer.</p><p>When you describe your project, don&#8217;t focus only on the demo. Write about the problems you had and how you solved them, like a model that kept failing or costs that were too high. Hiring managers want to see that you understand how the system works.</p><p>Some of you may want to turn the capstone into a product later. For peer review, we need to see the code on GitHub. If you want to keep some parts private, move them into a separate package that your project installs as a dependency. If you fine-tune a model, you don&#8217;t have to share the weights. An endpoint that reviewers can test is enough.</p><h2><strong>The Stack</strong></h2><p>We use Nebius Serverless AI for GPUs and Nebius Token Factory for the LLMs. The code is MIT-licensed, and it only uses OpenAI-compatible and S3-compatible services, so you can also run it elsewhere.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Irdd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Irdd!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!Irdd!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!Irdd!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!Irdd!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Irdd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The course stack: Hugging Face Diffusers, image models such as Qwen-Image, FLUX-class and Sana, Wan for video, Kokoro, Chatterbox and Whisper for audio, Nebius Token Factory for the LLM, raw tool calling and MCP for agents, Nebius Serverless AI for GPU compute, and S3-compatible object storage with FastAPI&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The course stack: Hugging Face Diffusers, image models such as Qwen-Image, FLUX-class and Sana, Wan for video, Kokoro, Chatterbox and Whisper for audio, Nebius Token Factory for the LLM, raw tool calling and MCP for agents, Nebius Serverless AI for GPU compute, and S3-compatible object storage with FastAPI" title="The course stack: Hugging Face Diffusers, image models such as Qwen-Image, FLUX-class and Sana, Wan for video, Kokoro, Chatterbox and Whisper for audio, Nebius Token Factory for the LLM, raw tool calling and MCP for agents, Nebius Serverless AI for GPU compute, and S3-compatible object storage with FastAPI" srcset="https://substackcdn.com/image/fetch/$s_!Irdd!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 424w, https://substackcdn.com/image/fetch/$s_!Irdd!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 848w, https://substackcdn.com/image/fetch/$s_!Irdd!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 1272w, https://substackcdn.com/image/fetch/$s_!Irdd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F666c7774-1691-49ff-8446-7a8dab8642b1_3360x1800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The default stack, with alternatives for each layer.</figcaption></figure></div><p>Each open-weights model comes with its own licence, so check it before you use the model commercially.</p><h2><strong>Prerequisites</strong></h2><p>The course isn&#8217;t for complete beginners.</p><p>We had these people in mind when we were preparing the course:</p><ul><li><p>AI and ML engineers who already work with models and want to add agents or run media pipelines in production</p></li><li><p>Software engineers who integrate open-source models into their applications</p></li><li><p>Researchers who write code. If you know some Python, Git, and Docker, you&#8217;ll be fine</p></li><li><p>Technical founders who pay for media APIs per call and want to know when self-hosting is cheaper</p></li></ul><p>You don&#8217;t need a machine learning background, because the course is about engineering. But you need to be comfortable with Python, Docker basics, HTTP APIs, and the command line. We read and explain a lot of code, so without coding experience, the course will be hard to follow.</p><p>You don&#8217;t need formal work experience to get in, so students are welcome too.</p><p>You also don&#8217;t need a GPU. If your laptop can run Python 3.11, that&#8217;s enough, because the agent runs locally and calls remote endpoints. If your laptop is too old even for that, Nebius DevLabs gives you JupyterLab in the cloud.</p><p>You can also run the models on your own hardware and point the agent to local endpoints. On a laptop with integrated graphics, generation will take a long time, but a recent MacBook Pro or Mac Studio has a better chance. Making the pipeline run on local hardware with smaller models would be a good capstone project.</p><p>We don&#8217;t cover diffusion maths or training foundation models. CUDA, Kubernetes, RAG, and LLM fundamentals aren&#8217;t part of the course either.</p><h2><strong>Applying</strong></h2><p>The course is free, but we have limited seats in the live cohort, so you need to apply. More than 1,160 people have applied so far. We process applications once a day, usually in the evening European time, and you get an email with the decision.</p><p>Our emails may end up in spam or promotions, so please check there. If you use Gmail, drag our email to the primary tab. Then Gmail learns that our emails are important, and you won&#8217;t miss the weekly updates.</p><p>A few people wrote to me after they got a rejection email. If we reject you, the only thing you miss is access to the Nebius platform. All the materials are public, so you can still go through the course, do the homework, and add the project to your portfolio. You&#8217;ll need to find a different way to access GPUs and other resources to run the code.</p><h2><strong>GPU Credits</strong></h2><p>Everyone we accept gets GPU credits through the <a href="https://dev.nebius.com/builders">Nebius AI Builder Program</a>, which is free to join. You get $25 at the start and another $50 after Homework 2.</p><p>Please request the credits after 16 November, not before. Nebius adds compute credits to the program on that day, and we&#8217;ll send you an email with instructions. Nebius may take a couple of days to approve your request.</p><p>If you applied to the program earlier, Nebius can add the credits manually. Ask in Slack if they don&#8217;t show up.</p><p>A regular GPU costs roughly $2 to $3 per hour, and spot instances are much cheaper. So $75 gives you a good amount of GPU time.</p><h2><strong>Dates</strong></h2><p>The deadlines in our course are strict. Once we score a homework form, the answers are public, so we can&#8217;t accept late submissions. I know life happens, and sometimes you need a break for a few weeks. If you fall behind, keep going at your own pace and aim for the second capstone attempt.</p><p>All times are CET, and modules run from Monday to Sunday:</p><ul><li><p>Monday, 16 November, 17:00 to 18:00: launch stream</p></li><li><p>16 to 29 November: Module 1 (two weeks), Homework 1, and the capstone proposal</p></li><li><p>30 November to 6 December: Module 2, Homework 2 due on 6 December</p></li><li><p>7 to 13 December: Module 3, Homework 3 (leaderboard) due on 13 December</p></li><li><p>14 to 20 December: Module 4 and the capstone brief</p></li><li><p>15 January 2027: capstone, attempt 1</p></li><li><p>22 January 2027: peer review, attempt 1</p></li><li><p>29 January 2027: capstone, attempt 2</p></li><li><p>5 February 2027: final peer review, attempt 2</p></li></ul><p>We may still change the materials and the dates.</p><h2><strong>Referrals</strong></h2><p>We also added personal referral links to this course. After you&#8217;re accepted, you get your link in the email. If someone you invite gets accepted, you get a point on the leaderboard. You also get points for correct homework answers and for sharing your progress online.</p><h2><strong>Credits</strong></h2><p>Mikhail Rozhkov, Marouane Khoukh, and Aleksandr Iziumov from the Nebius DevRel team created the course. They&#8217;ll be the main instructors, and I&#8217;ll help where I can. Thanks to the Nebius and Nebius Academy teams for preparing the course and for giving GPU credits to everyone.</p><h2><strong>Links and Questions</strong></h2><p>If you have questions, ask in the <a href="https://datatalks.club/slack.html">DataTalks.Club Slack</a> or come to the weekly office hour. I also shared the slides from the Q&amp;A in Slack.</p><p>Here are all the links:</p><ul><li><p><a href="https://courses.datatalks.club/register/production-ai-agents-generative-media/">Apply for the course</a>. It&#8217;s free, but seats in the live cohort are limited</p></li><li><p><a href="https://github.com/DataTalksClub/production-ai-agents-for-media-generation">Course repository</a> with all the materials under the MIT licence</p></li><li><p><a href="https://luma.com/xey7h50c">Launch stream</a> on Monday, 16 November, at 17:00 CET</p></li><li><p><a href="https://www.youtube.com/watch?v=iACtrkiXndw">Pre-course Q&amp;A recording</a> with more questions about the course, the credits, and the prerequisites</p></li><li><p><a href="https://datatalks.club/slack.html">DataTalks.Club Slack</a> for questions</p></li></ul><p>And please star the course repository - it takes five seconds, and if the repo gets into GitHub trending, more people will find the course.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://courses.datatalks.club/register/production-ai-agents-generative-media/&quot;,&quot;text&quot;:&quot;Join the Course&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://courses.datatalks.club/register/production-ai-agents-generative-media/"><span>Join the Course</span></a></p><p>See you on Monday!</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[Dark Software Factories]]></title><description><![CDATA[Why I stopped reading code from agents]]></description><link>https://aishippingblog.com/p/dark-software-factories</link><guid isPermaLink="false">https://aishippingblog.com/p/dark-software-factories</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 06 Nov 2026 09:43:50 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!arUH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Reading code slows me down and makes me a bottleneck. That&#8217;s why, around six months ago, I stopped doing it routinely. Now I read agent-generated code only when I prepare a workshop or course.</p><p>A few months ago I started hearing the term &#8220;software factories&#8221; and realized I already had one. In April 2026, I published <a href="https://aishippingblog.com/p/i-built-an-ai-agent-team-for-software">I Built an AI Agent Team for Software Development and Tested on 5 Real Projects</a>. I described my agent team before I knew it was called &#8220;a factory&#8221;.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;995bc296-670a-4bf1-a054-d233bbb52510&quot;,&quot;caption&quot;:&quot;Over the past few weeks, I&#8217;ve been trying out a new way of working with agent teams for software development using Claude Code. Instead of just seeing it as a single tool, I&#8217;ve started thinking of the main session as an orchestrator that directs a small team of agents.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;I Built an AI Agent Team for Software Development and Tested on 5 Real Projects&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-04-03T10:46:22.930Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!QESf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F732331c7-693b-46cb-9053-7161a130d3d6_2779x1378.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/i-built-an-ai-agent-team-for-software&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:193047033,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:39,&quot;comment_count&quot;:6,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:false,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p>But more recently I started hearing another term: &#8220;dark software factory&#8221;. It&#8217;s a factory where humans aren&#8217;t allowed. We give agents specifications and check that they follow them, but we don&#8217;t look at how it&#8217;s implemented. We still decide what to build and evaluate the results through tests, demos, and constant monitoring.</p><p>I&#8217;ve been doing it for some time too. But I also wanted to understand what people mean by &#8220;dark software factory&#8221; and how others implement it. That&#8217;s why I did some research, and I&#8217;ll describe what I found in this article.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!arUH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!arUH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 424w, https://substackcdn.com/image/fetch/$s_!arUH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 848w, https://substackcdn.com/image/fetch/$s_!arUH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 1272w, https://substackcdn.com/image/fetch/$s_!arUH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!arUH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png" width="1456" height="745" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:745,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A person feeds a specification checklist into a dark factory where three robotic arms assemble software along a conveyor, with a completed feature emerging at the other end&quot;,&quot;title&quot;:&quot;A person feeds a specification checklist into a dark factory where three robotic arms assemble software along a conveyor, with a completed feature emerging at the other end&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A person feeds a specification checklist into a dark factory where three robotic arms assemble software along a conveyor, with a completed feature emerging at the other end" title="A person feeds a specification checklist into a dark factory where three robotic arms assemble software along a conveyor, with a completed feature emerging at the other end" srcset="https://substackcdn.com/image/fetch/$s_!arUH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 424w, https://substackcdn.com/image/fetch/$s_!arUH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 848w, https://substackcdn.com/image/fetch/$s_!arUH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 1272w, https://substackcdn.com/image/fetch/$s_!arUH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fdf2ab4-b1f7-4ef3-a650-531b0f95b45f_1752x897.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In this article, I&#8217;ll cover:</p><ul><li><p>Why I stopped reading agent-generated code</p></li><li><p>Where the idea of a dark software factory comes from</p></li><li><p>How teams run software factories</p></li><li><p>The practices that keep factories running</p></li><li><p>The technical and comprehension debt factories create</p></li><li><p>Where agent factories still need humans</p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>Not Reading Code</strong></h2><p>When I first started working with agents, I was looking at the code. It was terrible. Especially the tests - they were absolute 100% slop. I wanted the code to look as if I had written it myself.</p><p>I&#8217;d dictate review notes with Google Recorder and give the transcript to my agent teams to correct it. They&#8217;d spend one or two days rewriting the code. Then I&#8217;d review the project again, only to find that some comments were still not addressed. So I&#8217;d record another note, and then another one, and then another one.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cJgi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cJgi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 424w, https://substackcdn.com/image/fetch/$s_!cJgi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 848w, https://substackcdn.com/image/fetch/$s_!cJgi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 1272w, https://substackcdn.com/image/fetch/$s_!cJgi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cJgi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png" width="1200" height="350" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:350,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An agent writes code, a human reviews it, and the agent rewrites it before another human review&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An agent writes code, a human reviews it, and the agent rewrites it before another human review" title="An agent writes code, a human reviews it, and the agent rewrites it before another human review" srcset="https://substackcdn.com/image/fetch/$s_!cJgi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 424w, https://substackcdn.com/image/fetch/$s_!cJgi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 848w, https://substackcdn.com/image/fetch/$s_!cJgi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 1272w, https://substackcdn.com/image/fetch/$s_!cJgi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8369d2d-5664-4ab8-9383-a28dd1b2929d_1200x350.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The repeated code-review and rewrite loop.</figcaption></figure></div><p>At some point I realized I was spending too much time (and tokens) on it without making any meaningful progress.</p><p>And I realized another thing: the actual problem was me, not the code. The original code worked fine, and the tests I disliked were able to catch problems in the code the agents wrote. When I removed some of those tests, things started to break apart.</p><p>Instead of helping, I was making things worse. Yes, the code looked terrible, but I had to either accept it or go back to writing code myself.</p><p>Eventually I left the agents to write whatever they needed, and I focused on the bigger picture:</p><ul><li><p>If two projects need the same functionality, I ask for a shared library.</p></li><li><p>If they need authentication, I ask for single sign-on instead of maintaining an authentication layer in every application.</p></li><li><p>If I want to use the same project across devices, I ask for a common TypeScript codebase instead of a separate Kotlin, desktop, and web implementation. If we catch a bug, we fix it only once.</p></li><li><p>If adding new features or fixing bugs starts taking a long time, I ask to split the code into multiple modules.</p></li><li><p>If the codebase grows too large, I ask for a reimplementation that reduces its size by two to three times. I also specifically ask to remove all the backwards-compatibility layers that agents like to create.</p></li><li><p>If running tests takes too long, I ask to review the tests and/or split the library into multiple modules.</p></li></ul><p>When I work with agents, I&#8217;m still making technical decisions, but I do it by setting the direction and checking the behavior, not by controlling the code.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!plv3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!plv3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 424w, https://substackcdn.com/image/fetch/$s_!plv3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 848w, https://substackcdn.com/image/fetch/$s_!plv3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 1272w, https://substackcdn.com/image/fetch/$s_!plv3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!plv3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png" width="1200" height="350" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:350,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A human sets direction, agents build and verify, and human product checks feed back into better direction&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A human sets direction, agents build and verify, and human product checks feed back into better direction" title="A human sets direction, agents build and verify, and human product checks feed back into better direction" srcset="https://substackcdn.com/image/fetch/$s_!plv3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 424w, https://substackcdn.com/image/fetch/$s_!plv3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 848w, https://substackcdn.com/image/fetch/$s_!plv3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 1272w, https://substackcdn.com/image/fetch/$s_!plv3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58245e0b-3974-4a80-924e-2b304e55b6ac_1200x350.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Setting direction and checking the product.</figcaption></figure></div><p>It was a gradual process, but eventually I arrived at what people now call &#8220;dark software factories&#8221;.</p><p>And since the term keeps appearing, I wanted to understand where it comes from and how people do it.</p><h2><strong>Lights-Out Manufacturing</strong></h2><p>The term &#8220;dark software factory&#8221; is a combination of two ideas:</p><ul><li><p>dark factories, also known as &#8220;lights-out manufacturing&#8221;</p></li><li><p>software factories</p></li></ul><p>In lights-out manufacturing, machines can run without people on the factory floor. <a href="https://www.fanuc.co.jp/en/profile/production/factory1.html">FANUC describes factories</a> where robots put parts into machines so production can continue without people, including nights and weekends.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!f30b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!f30b!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 424w, https://substackcdn.com/image/fetch/$s_!f30b!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 848w, https://substackcdn.com/image/fetch/$s_!f30b!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 1272w, https://substackcdn.com/image/fetch/$s_!f30b!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!f30b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png" width="842" height="422" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/443a1633-a4c8-4536-a39b-1028620fd198_842x422.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:422,&quot;width&quot;:842,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;FANUC robots assembling parts in its CNC factory&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="FANUC robots assembling parts in its CNC factory" title="FANUC robots assembling parts in its CNC factory" srcset="https://substackcdn.com/image/fetch/$s_!f30b!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 424w, https://substackcdn.com/image/fetch/$s_!f30b!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 848w, https://substackcdn.com/image/fetch/$s_!f30b!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 1272w, https://substackcdn.com/image/fetch/$s_!f30b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F443a1633-a4c8-4536-a39b-1028620fd198_842x422.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">FANUC&#8217;s CNC factory. Source: <a href="https://www.fanuc.co.jp/en/profile/production/factory1.html">FANUC</a>.</figcaption></figure></div><h2><strong>Software Factories</strong></h2><p>The &#8220;software factory&#8221; idea is way older than coding agents.</p><p><a href="https://addyosmani.com/blog/software-factories/">Addy Osmani traces it back to 1968</a>. People wanted to make development repeatable through shared tools, reusable components, standards, and quality checks. Writing code wasn&#8217;t automated, but the idea was to have a standard process to ensure good quality. With agents, we&#8217;re now automating these steps, including implementation and review.</p><p>For example, the article &#8220;Mass Customizing Solutions&#8221; (2007) includes a <a href="https://www.methodsandtools.com/PDF/mt200703.pdf#page=31">software factory schema diagram on page 31</a>. It connects roles, tasks, and workstreams to reusable assets. The factory in this case is a structured environment that helps people build related applications from shared components.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fz3s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fz3s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 424w, https://substackcdn.com/image/fetch/$s_!fz3s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 848w, https://substackcdn.com/image/fetch/$s_!fz3s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 1272w, https://substackcdn.com/image/fetch/$s_!fz3s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fz3s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png" width="656" height="340" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c0b7731e-1541-4805-89ec-d0043158c954_656x340.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:340,&quot;width&quot;:656,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:49812,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/223738822?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fz3s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 424w, https://substackcdn.com/image/fetch/$s_!fz3s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 848w, https://substackcdn.com/image/fetch/$s_!fz3s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 1272w, https://substackcdn.com/image/fetch/$s_!fz3s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b7731e-1541-4805-89ec-d0043158c954_656x340.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Illustration from the 2007 article on software factories (<a href="https://www.methodsandtools.com/PDF/mt200703.pdf#page=31">source</a>).</figcaption></figure></div><p>When it comes to agents, the earliest mention I found was <a href="https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/">Dan Shapiro&#8217;s post</a> (January 2026).</p><p>Dan described levels 0 through 5 of a code factory:</p><ul><li><p>Level 0: you write the code and occasionally use AI.</p></li><li><p>Level 1: you delegate small tasks such as tests and docstrings.</p></li><li><p>Level 2: you pair with an agent while staying involved in implementation.</p></li><li><p>Level 3: you mainly review the code agents produce.</p></li><li><p>Level 4: you prepare specifications and checks, then let agents work for hours.</p></li><li><p>Level 5: you operate a system that turns specifications into software without routine human code review.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!k8W5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!k8W5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 424w, https://substackcdn.com/image/fetch/$s_!k8W5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 848w, https://substackcdn.com/image/fetch/$s_!k8W5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 1272w, https://substackcdn.com/image/fetch/$s_!k8W5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!k8W5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg" width="1456" height="989" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:989,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A descending staircase from level 0, writing code, to level 5, running a factory without routine code review&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A descending staircase from level 0, writing code, to level 5, running a factory without routine code review" title="A descending staircase from level 0, writing code, to level 5, running a factory without routine code review" srcset="https://substackcdn.com/image/fetch/$s_!k8W5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 424w, https://substackcdn.com/image/fetch/$s_!k8W5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 848w, https://substackcdn.com/image/fetch/$s_!k8W5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 1272w, https://substackcdn.com/image/fetch/$s_!k8W5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab15dcad-431a-4b0f-a35c-79879d4e747a_1200x815.svg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The six stages in Dan Shapiro&#8217;s levels 0 through 5. Source: <a href="https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/">The Five Levels</a>.</figcaption></figure></div><p>Shapiro placed himself at level 4 and borrowed the dark-factory name from FANUC for level 5.</p><p><a href="https://simonwillison.net/2026/Jan/28/the-five-levels/">Simon Willison wrote a follow-up post about it</a> in January 2026 and described a small team whose agents demonstrated correctness without human code review. (Every time I do term archeology, Simon&#8217;s blog comes up!)</p><p>I was already experimenting with it in January and February with Claude. When it comes to coding, Opus 4.5 was a decent model, but most of us didn&#8217;t have the proper tooling and processes to make it work at Level 5.</p><p>I described some of my earlier experiments and what I learned from them in <a href="https://aishippingblog.com/p/six-projects-that-didnt-make-it">Six Projects That Didn&#8217;t Make It</a>.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;57c1227c-5f5a-4734-bc9b-7d1a215104ff&quot;,&quot;caption&quot;:&quot;I have a lot of projects that I started with some grand idea, spent time on, and then never really did anything with.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Six Projects That Didn&#8217;t Make It&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-06-26T21:07:46.828Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!BQAt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb6e334a-b051-4876-8133-83390b9d3120_1768x1086.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/six-projects-that-didnt-make-it&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:203724093,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:5,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p>Things have changed since January. The tooling is now a lot better, plus we figured out how to organize the process so it actually works: we now create specifications, have agents with different roles, and have an automated way of verifying the results.</p><p>I cover this in <a href="https://aishippingblog.com/p/ai-native-development-specifications">my article about AI-native development</a>, which is the first module of AI Dev Tools Zoomcamp.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;8867fb16-be6d-4dd0-b6fa-5c9b2a68d21d&quot;,&quot;caption&quot;:&quot;This is the first article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;AI-Native Development: Specifications, Loop and Graph Engineering&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-07-22T06:59:57.781Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/youtube/w_728,c_limit/VUJxJGpaDEs&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/ai-native-development-specifications&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:208017204,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:108,&quot;comment_count&quot;:4,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><h2><strong>Teams Running Factories</strong></h2><p>In my courses I teach how I approach agentic engineering. I learned it by trial and error across many projects. But for this article I also wanted to see what other people are doing.</p><p>In my research, I focused on developers sharing their experience and excluded vendors and companies that consult in this area. I also wanted to check recent articles, so I restricted my search to June 2026 onwards. Here&#8217;s what I found:</p><h3><strong>Sandstorm</strong></h3><p><a href="https://brianmcquay.com/blog/going-dark/">Brian McQuay describes Sandstorm</a>, a system he uses for his software factory.</p><p>It&#8217;s a Kanban-driven pipeline:</p><ul><li><p>a human refines a ticket and marks it ready</p></li><li><p>the system prepares an isolated environment for the task to run</p></li><li><p>the task is implemented and reviewed</p></li><li><p>if any problems are identified, they are fixed</p></li><li><p>it opens a PR</p></li><li><p>the PR is merged and conflicts are resolved</p></li></ul><p>Then he uses it for each project.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!TiRf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TiRf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 424w, https://substackcdn.com/image/fetch/$s_!TiRf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 848w, https://substackcdn.com/image/fetch/$s_!TiRf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 1272w, https://substackcdn.com/image/fetch/$s_!TiRf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!TiRf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png" width="1200" height="1055" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1055,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Ready ticket leads to an isolated workspace, implementation, review, a pull request and merge. Problems found in review return to implementation for fixes and another review.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Ready ticket leads to an isolated workspace, implementation, review, a pull request and merge. Problems found in review return to implementation for fixes and another review." title="Ready ticket leads to an isolated workspace, implementation, review, a pull request and merge. Problems found in review return to implementation for fixes and another review." srcset="https://substackcdn.com/image/fetch/$s_!TiRf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 424w, https://substackcdn.com/image/fetch/$s_!TiRf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 848w, https://substackcdn.com/image/fetch/$s_!TiRf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 1272w, https://substackcdn.com/image/fetch/$s_!TiRf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43bbffa5-ca97-4a8f-b025-4f43c626aab5_1200x1055.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A simplified view of Sandstorm&#8217;s ticket-driven workflow. Source: <a href="https://brianmcquay.com/blog/going-dark/">Brian McQuay</a>.</figcaption></figure></div><p>It reminded me of my earlier attempt to create the factory - I called it &#8220;litehive&#8221; (described in <a href="https://aishippingblog.com/p/six-projects-that-didnt-make-it">Six Projects That Didn&#8217;t Make It</a>). I didn&#8217;t manage to make it work, but Brian did.</p><h3><strong>Checks in Lanyard</strong></h3><p>While building an SSH proxy called Lanyard, <a href="https://johnwilger.com/blog/somebody-has-to-design-the-factory/">John Wilger found several ways his agents could appear to finish a task without actually finishing it</a>.</p><p>He added rules to prevent that:</p><ul><li><p>Each task gets its own copy of the code, so agents working on different tasks do not interfere with one another.</p></li><li><p>A deployment is complete only when he can open the live URL and get a response.</p></li><li><p>A command reporting success is not enough. The agent must check that the intended result actually happened.</p></li></ul><p>He also makes agents repeat checks after every code change. Another agent reviews the work in a fresh session and cannot change the code while reviewing it. If the agents cannot pass a required check, they stop and ask John for help instead of skipping it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!U8qT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U8qT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 424w, https://substackcdn.com/image/fetch/$s_!U8qT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 848w, https://substackcdn.com/image/fetch/$s_!U8qT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 1272w, https://substackcdn.com/image/fetch/$s_!U8qT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!U8qT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png" width="1200" height="785" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:785,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An isolated task is tested, reviewed in a fresh session, and checked at its live URL. A code edit sends it back to testing.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An isolated task is tested, reviewed in a fresh session, and checked at its live URL. A code edit sends it back to testing." title="An isolated task is tested, reviewed in a fresh session, and checked at its live URL. A code edit sends it back to testing." srcset="https://substackcdn.com/image/fetch/$s_!U8qT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 424w, https://substackcdn.com/image/fetch/$s_!U8qT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 848w, https://substackcdn.com/image/fetch/$s_!U8qT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 1272w, https://substackcdn.com/image/fetch/$s_!U8qT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28b81f1f-1a20-4316-ad23-2bd7506b782d_1200x785.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Checks apply to the current version of the code. If an agent cannot pass one, it stops and asks Wilger. Source: <a href="https://johnwilger.com/blog/somebody-has-to-design-the-factory/">John Wilger</a>.</figcaption></figure></div><h3><strong>Linear-to-PR</strong></h3><p>I also found an interesting <a href="https://www.reddit.com/r/ycombinator/comments/1vamsls/has_anyone_here_actually_set_up_a_software/">Reddit discussion</a> where people share what they used.</p><p>One of the comments:</p><ul><li><p>prepare stories in Linear</p></li><li><p>launch agent sessions for prepared stories</p></li><li><p>make them follow repository instructions</p></li><li><p>run end-to-end tests</p></li><li><p>produce a PR with a preview URL</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!m-7t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!m-7t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 424w, https://substackcdn.com/image/fetch/$s_!m-7t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 848w, https://substackcdn.com/image/fetch/$s_!m-7t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 1272w, https://substackcdn.com/image/fetch/$s_!m-7t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!m-7t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png" width="1200" height="785" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:785,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A ready Linear story starts an agent session, which produces a pull request with a preview URL before release checks run.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A ready Linear story starts an agent session, which produces a pull request with a preview URL before release checks run." title="A ready Linear story starts an agent session, which produces a pull request with a preview URL before release checks run." srcset="https://substackcdn.com/image/fetch/$s_!m-7t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 424w, https://substackcdn.com/image/fetch/$s_!m-7t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 848w, https://substackcdn.com/image/fetch/$s_!m-7t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 1272w, https://substackcdn.com/image/fetch/$s_!m-7t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0840224b-4dec-48ea-8097-8714016dcb69_1200x785.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The Linear-to-PR workflow described by a developer in a <a href="https://www.reddit.com/r/ycombinator/comments/1vamsls/has_anyone_here_actually_set_up_a_software/">Reddit discussion</a>.</figcaption></figure></div><p></p><h2><strong>Enabling Practices</strong></h2><p>All these stories follow more or less the same flow:</p><ul><li><p>Have a task backlog</p></li><li><p>Use spec-driven development to refine tasks</p></li><li><p>Have agents verify other agents</p></li></ul><p>Having a backlog is important because agents don&#8217;t have long-term memory. Even within one session they can forget that you asked something. So it&#8217;s really important to capture the tasks, be it GitHub issues, Linear, or even plain text files.</p><p>We also need to have clearly defined tasks. Agents are good at writing code. But they are also good at filling the gaps in the specifications. If we ask an agent to implement a feature, it will work. But if we don&#8217;t provide enough details, it won&#8217;t be what we need.</p><p>To address this problem, we follow specification-driven development and turn each task into a spec with clear acceptance criteria.</p><p>Once the feature is implemented, we have another agent that checks the implementation against these criteria. If some of the boxes aren&#8217;t checked, the implementer has to fix it.</p><p>This is the loop I show in <a href="https://aishippingblog.com/p/ai-native-development-specifications">module 1 of AI Dev Tools Zoomcamp</a>:</p><ul><li><p>a product manager agent grooms the issue</p></li><li><p>an engineer implements it</p></li><li><p>a QA agent checks each acceptance criterion</p></li><li><p>on failure, the engineer has to do it again</p></li><li><p>QA only reports what it observed and does not edit the code</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!98fS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!98fS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 424w, https://substackcdn.com/image/fetch/$s_!98fS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 848w, https://substackcdn.com/image/fetch/$s_!98fS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 1272w, https://substackcdn.com/image/fetch/$s_!98fS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!98fS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png" width="1200" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Product manager, engineer, and QA agent workflow from AI Dev Tools Zoomcamp&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Product manager, engineer, and QA agent workflow from AI Dev Tools Zoomcamp" title="Product manager, engineer, and QA agent workflow from AI Dev Tools Zoomcamp" srcset="https://substackcdn.com/image/fetch/$s_!98fS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 424w, https://substackcdn.com/image/fetch/$s_!98fS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 848w, https://substackcdn.com/image/fetch/$s_!98fS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 1272w, https://substackcdn.com/image/fetch/$s_!98fS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6768a8d7-9bde-4a89-b35c-985c5d96c4c2_1200x440.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The product manager grooms the issue, the engineer implements it, and QA checks the result. Source: <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development</a>.</figcaption></figure></div><p>There are some other techniques that we need. One of them is being able to ask questions when the agents aren&#8217;t sure about something.</p><h2><strong>Escalation to Humans</strong></h2><p>We still need to involve humans.</p><p>Agents need to have a way to escalate to humans but only when it&#8217;s absolutely needed. I want to be involved as little as possible - but I also don&#8217;t want to miss important checks and decisions.</p><p>In <a href="https://brianmcquay.com/blog/going-dark/">Brian&#8217;s Sandstorm</a>, higher-risk or ambiguous work requires approval. Lower-risk work can merge automatically and receive application-level QA.</p><p><a href="https://johnwilger.com/blog/somebody-has-to-design-the-factory/">In John&#8217;s Lanyard workflow</a>, agents must stop and ask him for help if they cannot complete a required check. For example, if they cannot confirm that the deployed app responds, they cannot simply mark the task as done.</p><p>Brian and John decide ahead of time when an agent needs to involve a person.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-0w8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-0w8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 424w, https://substackcdn.com/image/fetch/$s_!-0w8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 848w, https://substackcdn.com/image/fetch/$s_!-0w8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 1272w, https://substackcdn.com/image/fetch/$s_!-0w8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-0w8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png" width="1200" height="550" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:550,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Escalation rules let routine work continue and route ambiguous, higher-risk or blocked work to a human decision before reassessment.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Escalation rules let routine work continue and route ambiguous, higher-risk or blocked work to a human decision before reassessment." title="Escalation rules let routine work continue and route ambiguous, higher-risk or blocked work to a human decision before reassessment." srcset="https://substackcdn.com/image/fetch/$s_!-0w8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 424w, https://substackcdn.com/image/fetch/$s_!-0w8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 848w, https://substackcdn.com/image/fetch/$s_!-0w8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 1272w, https://substackcdn.com/image/fetch/$s_!-0w8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1e34b3f-1e1e-4cc1-b7e4-822bf4d9838f_1200x550.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Define when agents can proceed and when they need a human decision.</figcaption></figure></div><p>In my projects, agents use the <code>human</code> tag for GitHub issues. They may decide that it&#8217;s needed when turning a task into a spec, or later when verifying the results.</p><p>Of course, I also need to tune the process. If agents want to involve me when I&#8217;m not needed, I ask them to update the escalation rules. If something wasn&#8217;t escalated but should have been, I do the same. Eventually it becomes clear what should be escalated and what shouldn&#8217;t be.</p><h2><strong>Problems with Factories</strong></h2><p>In the <a href="https://www.reddit.com/r/ycombinator/comments/1vamsls/has_anyone_here_actually_set_up_a_software/">Reddit discussion</a> that I already mentioned, I found another interesting link - a post on <a href="https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/wsff.md">why software factories fail</a>.</p><p>It&#8217;s very long, but the main point I got from there was that maintainability deteriorates with time even when all the tests pass.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!eOWW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!eOWW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 424w, https://substackcdn.com/image/fetch/$s_!eOWW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 848w, https://substackcdn.com/image/fetch/$s_!eOWW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 1272w, https://substackcdn.com/image/fetch/$s_!eOWW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!eOWW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png" width="1456" height="452" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:452,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The full article cut into ten equal consecutive slices, with each slice placed as one column&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The full article cut into ten equal consecutive slices, with each slice placed as one column" title="The full article cut into ten equal consecutive slices, with each slice placed as one column" srcset="https://substackcdn.com/image/fetch/$s_!eOWW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 424w, https://substackcdn.com/image/fetch/$s_!eOWW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 848w, https://substackcdn.com/image/fetch/$s_!eOWW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 1272w, https://substackcdn.com/image/fetch/$s_!eOWW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9e56ea2-8fa8-411e-a440-bbe33bd56c5e_9000x2793.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Software factories will fail - the length of the article should surely convince you.</figcaption></figure></div><p>That&#8217;s true. Tests may pass, but as your codebase grows, adding functionality becomes more and more difficult. Models get rewarded for a quick patch without accounting for the future cost of that design. And since we don&#8217;t look at the code (it&#8217;s a dark factory, after all), the technical debt accumulates.</p><p>It&#8217;s certainly a problem. Agents create code fast and often completely disregard best engineering practices. They write duplicated code and don&#8217;t really care about modularity.</p><p>In some cases it&#8217;s okay. If we&#8217;re only starting a project, it&#8217;s too early to worry about it.</p><p>But if the project doesn&#8217;t get abandoned, over time you will start noticing that adding a feature takes 1 hour instead of 5 minutes. On top of that, some bugs keep coming back, even though you thought you asked to fix them 5 times already.</p><p>When it starts happening, it&#8217;s time to do maintenance work. I ask agents to analyze what slows them down and then find a way to fix it. Sometimes I can also ask them to describe the architecture and see if they can improve it.</p><p>Maintainability is not the only problem. Even with specifications, agents still make a lot of decisions for us.</p><p>I must admit that I sometimes find myself surprised when I discover some features in my own projects. And often the surprise is not a positive one.</p><p><a href="https://addyosmani.com/blog/comprehension-debt/">Addy Osmani calls it &#8220;comprehension debt&#8221;</a>.</p><p>If I have a clear picture of how each feature works, my comprehension debt is low. But if I can&#8217;t predict what happens when I click somewhere or change something, the debt is high.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pZsK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pZsK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 424w, https://substackcdn.com/image/fetch/$s_!pZsK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 848w, https://substackcdn.com/image/fetch/$s_!pZsK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 1272w, https://substackcdn.com/image/fetch/$s_!pZsK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pZsK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png" width="1200" height="690" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:690,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A cycle of prompting, generating code, running it and finding problems produces more code with less understanding&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A cycle of prompting, generating code, running it and finding problems produces more code with less understanding" title="A cycle of prompting, generating code, running it and finding problems produces more code with less understanding" srcset="https://substackcdn.com/image/fetch/$s_!pZsK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 424w, https://substackcdn.com/image/fetch/$s_!pZsK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 848w, https://substackcdn.com/image/fetch/$s_!pZsK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 1272w, https://substackcdn.com/image/fetch/$s_!pZsK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73696490-bf03-4160-afdf-d8710ae338d2_1200x690.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Adapted from <a href="https://www.linkedin.com/pulse/comprehension-debt-hidden-cost-ai-generated-code-amir-mohajerani-phd-nb5be/">Comprehension Debt: The Hidden Cost of AI-Generated Code</a>.</figcaption></figure></div><p>And there&#8217;s no shortcut to fixing it. I need to spend time using the product, checking its behavior, and asking agents to explain all the surprising decisions.</p><p>Later I fix the surprising behaviour. For important features, it often means staying away from the factory format and hand-holding the agent session for the important UX and UI details.</p><h2><strong>Keeping the Factory Useful</strong></h2><p>Yes, there are downsides when it comes to software factories - including the technical debt and comprehension debt that they produce.</p><p>To me, knowing the level of your comprehension debt is way more important than the level of technical debt. That&#8217;s why we can&#8217;t easily scale the factories - otherwise we end up with absolute slop. So you have to spend quality time doing the product work.</p><p>Which is actually good news - we, humans, are still needed.</p><p>But I certainly don&#8217;t want to return to writing or reading code. I want to understand the product well enough to decide what it should do and notice when it goes wrong. That means writing better specifications, having better verifications, and actually using what you create.</p><p>The factory itself also needs maintenance. We need to monitor it to make sure it doesn&#8217;t slow down. And when it does, we try to figure out how to fix it - like changing the process, fixing the code, or splitting projects into smaller ones.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PMyi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PMyi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 424w, https://substackcdn.com/image/fetch/$s_!PMyi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 848w, https://substackcdn.com/image/fetch/$s_!PMyi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 1272w, https://substackcdn.com/image/fetch/$s_!PMyi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PMyi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png" width="1200" height="610" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:610,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Specify, build, verify and use the product, then improve specifications from experience. Maintain the factory when work slows down.&quot;,&quot;title&quot;:&quot;Specify, build, verify and use the product, then improve specifications from experience. Maintain the factory when work slows down.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Specify, build, verify and use the product, then improve specifications from experience. Maintain the factory when work slows down." title="Specify, build, verify and use the product, then improve specifications from experience. Maintain the factory when work slows down." srcset="https://substackcdn.com/image/fetch/$s_!PMyi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 424w, https://substackcdn.com/image/fetch/$s_!PMyi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 848w, https://substackcdn.com/image/fetch/$s_!PMyi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 1272w, https://substackcdn.com/image/fetch/$s_!PMyi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f9045c-0ad3-460a-8a0c-2f00a1451bcd_1200x610.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Product experience improves specifications. Maintenance improves the factory.</figcaption></figure></div><p>So going &#8220;dark&#8221; doesn&#8217;t mean just offloading everything to agents. The more work I delegate, the more carefully I have to design the specifications, checks, and escalation paths around it.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2>Product Shipping with AI</h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!H3Dc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!H3Dc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 424w, https://substackcdn.com/image/fetch/$s_!H3Dc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 848w, https://substackcdn.com/image/fetch/$s_!H3Dc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 1272w, https://substackcdn.com/image/fetch/$s_!H3Dc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!H3Dc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png" width="1018" height="346" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:346,&quot;width&quot;:1018,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:185139,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/223738822?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!H3Dc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 424w, https://substackcdn.com/image/fetch/$s_!H3Dc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 848w, https://substackcdn.com/image/fetch/$s_!H3Dc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 1272w, https://substackcdn.com/image/fetch/$s_!H3Dc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f3a4c7a-d18b-4a54-b6b8-a08985729feb_1018x346.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>I&#8217;m launching a new course where I want to focus on the product work: <a href="https://maven.com/alexey-grigorev/product-shipping">Product Shipping with AI: From Idea to Real Users</a>. You&#8217;ll use AI to build your own project, decide what should be in the first version, put it in front of other people, and improve it based on feedback. I&#8217;ll review your project along the way and help you prioritize things. We&#8217;ll start in in January 2027.</p><p>The course is paid and invite-only, with 20 places in the first cohort. I&#8217;m keeping the group small so I can give each project proper attention. If you have a project you want to ship, register your interest:</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://forms.gle/F3ZiLzq8i2nip2VP6&quot;,&quot;text&quot;:&quot;Fill in the interest form&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://forms.gle/F3ZiLzq8i2nip2VP6"><span>Fill in the interest form</span></a></p><p></p><p> </p>]]></content:encoded></item><item><title><![CDATA[Trends in Agentic Engineering]]></title><description><![CDATA[Dark and light software factories]]></description><link>https://aishippingblog.com/p/trends-in-agentic-engineering</link><guid isPermaLink="false">https://aishippingblog.com/p/trends-in-agentic-engineering</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Tue, 20 Oct 2026 14:06:35 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!GwJM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A few weeks ago <a href="https://substack.com/@hugobowne">Hugo Bowne-Anderson</a> and I ran a webinar about trends in AI engineering.</p><div id="youtube2-whIdxTQsU0A" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;whIdxTQsU0A&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/whIdxTQsU0A?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p>At the beginning, Hugo defined two terms that people often confuse:</p><ul><li><p>AI engineering - AI is part of the product</p></li><li><p>Agentic engineering - agents help you build the product</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xq9d!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xq9d!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xq9d!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xq9d!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xq9d!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!xq9d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg" width="1456" height="821" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:821,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Slide from Hugo's talk comparing agentic engineering, building software with agents where the agent is part of the development team, with AI engineering, building AI-powered products where AI is part of the product&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Slide from Hugo's talk comparing agentic engineering, building software with agents where the agent is part of the development team, with AI engineering, building AI-powered products where AI is part of the product" title="Slide from Hugo's talk comparing agentic engineering, building software with agents where the agent is part of the development team, with AI engineering, building AI-powered products where AI is part of the product" srcset="https://substackcdn.com/image/fetch/$s_!xq9d!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xq9d!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xq9d!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xq9d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66e70343-401e-4420-a80c-fccced6775cc_1600x902.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hugo&#8217;s slide: in agentic engineering the agent is part of the development team</figcaption></figure></div><p></p><p>At the end of the session we decided to write an article about the webinar. When I started writing it, I realized that it makes sense to cover each topic separately. Last week I published <a href="https://aishippingblog.com/p/trends-in-ai-engineering">the first article about AI engineering</a>, and in this article, I focus on agentic engineering.</p><p>In particular, I&#8217;ll cover:</p><ul><li><p>Using one agent to review another&#8217;s code and designs</p></li><li><p>Giving agents rules for coding and writing</p></li><li><p>Organizing agents into teams and larger hierarchies</p></li><li><p>Running software factories where humans focus on outcomes</p></li></ul><h2><strong>Verifying Agent-Generated Work</strong></h2><p>Hugo ran a survey on how people do agentic engineering in data science. 34 people said that AI changed their workflow &#8220;fundamentally&#8221;, and 39 said &#8220;significantly&#8221;.</p><p>Hugo wanted to see what exactly was different, so he followed up and visualized the results with a heatmap.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1uB2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1uB2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 424w, https://substackcdn.com/image/fetch/$s_!1uB2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 848w, https://substackcdn.com/image/fetch/$s_!1uB2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 1272w, https://substackcdn.com/image/fetch/$s_!1uB2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1uB2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png" width="1456" height="790" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:790,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Heat map of verification methods by how much AI agents changed people's workflow. In the group whose workflow changed fundamentally, 59% ask another agent to critique the work and 62% write tests&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Heat map of verification methods by how much AI agents changed people's workflow. In the group whose workflow changed fundamentally, 59% ask another agent to critique the work and 62% write tests" title="Heat map of verification methods by how much AI agents changed people's workflow. In the group whose workflow changed fundamentally, 59% ask another agent to critique the work and 62% write tests" srcset="https://substackcdn.com/image/fetch/$s_!1uB2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 424w, https://substackcdn.com/image/fetch/$s_!1uB2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 848w, https://substackcdn.com/image/fetch/$s_!1uB2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 1272w, https://substackcdn.com/image/fetch/$s_!1uB2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F009c5f5a-bf85-4fbd-a69d-8005efbc854a_1703x924.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hugo&#8217;s survey, redrawn for readability: verification methods grouped by how much agents changed people&#8217;s workflow</figcaption></figure></div><p>Among people whose workflow changed fundamentally, 59% ask another AI tool or agent to critique the work. In the significantly changed group, it&#8217;s 54%.</p><p>The less impact people reported, the less common this practice was. It&#8217;s not causal, but the people who changed their workflow the most also verify using agents the most.</p><p>I can clearly see the reason: the more you delegate your work to agents, the more you need to check the output.</p><h2><strong>Agents Controlling Agents</strong></h2><p>Even frontier models give you pages and pages of absolute slop. With this amount of output it&#8217;s hard to review every line of code ourselves, so we use agents to control agents.</p><p>I call it the &#8220;implementer-reviewer&#8221; approach.</p><p>I have the implementer do the work and the reviewer validate it. I let them iterate until the second agent accepts the output of the first agent.</p><p>For coding, I assign one agent to implement the task as a software engineer. Then I ask another to act as a tester and find problems in the implementation. A feature may take a few hours instead of 20 minutes, but the quality at the end is better.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yf4V!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yf4V!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 424w, https://substackcdn.com/image/fetch/$s_!yf4V!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 848w, https://substackcdn.com/image/fetch/$s_!yf4V!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 1272w, https://substackcdn.com/image/fetch/$s_!yf4V!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yf4V!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png" width="1080" height="360" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:360,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;I ask an implementer agent to build and a reviewer agent to check the result against shared rules such as a design system or Clean Code, sending rejected work back until it accepts it&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="I ask an implementer agent to build and a reviewer agent to check the result against shared rules such as a design system or Clean Code, sending rejected work back until it accepts it" title="I ask an implementer agent to build and a reviewer agent to check the result against shared rules such as a design system or Clean Code, sending rejected work back until it accepts it" srcset="https://substackcdn.com/image/fetch/$s_!yf4V!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 424w, https://substackcdn.com/image/fetch/$s_!yf4V!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 848w, https://substackcdn.com/image/fetch/$s_!yf4V!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 1272w, https://substackcdn.com/image/fetch/$s_!yf4V!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fded229-a370-45b3-8b4a-bac5cdda0115_1080x360.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">I ask one agent to build and another to try to reject the result</figcaption></figure></div><p>It works outside of coding tasks too.</p><p>Agents usually produce terrible and complicated designs, so when I need to work on UI, I use the same implementer-reviewer approach:</p><ul><li><p>The implementer is a designer who creates the interface</p></li><li><p>The reviewer is a senior designer who tries to find reasons to reject the design</p></li></ul><p>I give both agents a design system and a rubric for evaluating the interface. After each iteration I have the senior designer score the interface, and I have them iterate until the score is 10/10.</p><p>I call this <a href="https://github.com/alexeygrigorev/.agents/blob/main/skills/adversarial-design-review/SKILL.md">adversarial design</a>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PZXW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PZXW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 424w, https://substackcdn.com/image/fetch/$s_!PZXW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 848w, https://substackcdn.com/image/fetch/$s_!PZXW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 1272w, https://substackcdn.com/image/fetch/$s_!PZXW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PZXW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png" width="957" height="645" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:645,&quot;width&quot;:957,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;My instructions for an implementer and an independent senior design reviewer in the adversarial design review skill on GitHub&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="My instructions for an implementer and an independent senior design reviewer in the adversarial design review skill on GitHub" title="My instructions for an implementer and an independent senior design reviewer in the adversarial design review skill on GitHub" srcset="https://substackcdn.com/image/fetch/$s_!PZXW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 424w, https://substackcdn.com/image/fetch/$s_!PZXW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 848w, https://substackcdn.com/image/fetch/$s_!PZXW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 1272w, https://substackcdn.com/image/fetch/$s_!PZXW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5f3e69a-45f2-4478-9336-3b7809329b2b_957x645.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The skill that defines the designer and reviewer roles</figcaption></figure></div><blockquote><p>Interested in how I approach design with AI? I described my approach here: </p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;131cad7b-0471-4496-8034-597a44a090d8&quot;,&quot;caption&quot;:&quot;You can usually tell when a website was created by AI. It often happens with &#8220;one-shot designs&#8221;: you ask a coding assistant to create a website, open the result, and immediately see that it is AI-generated.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;AI Design for Non-Designers&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-06-12T08:06:24.151Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!zJM4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc09ad9c0-1b5a-48f9-9ca8-8401db048ea1_1280x547.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/ai-design-for-non-designers&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:201442459,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:8,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div></blockquote><h2><strong>Shared Rules for Coding and Design</strong></h2><p>In these cases both the implementer and reviewer share a common set of rules:</p><ul><li><p>For coding tasks - the acceptance criteria and a coding standards document</p></li><li><p>For design - a design system and an evaluation rubric</p></li></ul><p>But instead of describing the entire system and listing all the criteria, I often take a shortcut and refer to books.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dp61!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dp61!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 424w, https://substackcdn.com/image/fetch/$s_!dp61!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 848w, https://substackcdn.com/image/fetch/$s_!dp61!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!dp61!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dp61!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg" width="335" height="442.6318359375" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1353,&quot;width&quot;:1024,&quot;resizeWidth&quot;:335,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Cover of Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Cover of Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin" title="Cover of Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin" srcset="https://substackcdn.com/image/fetch/$s_!dp61!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 424w, https://substackcdn.com/image/fetch/$s_!dp61!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 848w, https://substackcdn.com/image/fetch/$s_!dp61!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!dp61!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72f34540-474f-4736-912d-10ec8986f3ce_1024x1353.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Clean Code by Robert C. Martin. Cover: <a href="https://www.informit.com/store/clean-code-a-handbook-of-agile-software-craftsmanship-9780132350884">Pearson / InformIT</a></figcaption></figure></div><p>For coding, I can say &#8220;follow the principles from the Clean Code book by Robert Martin&#8221;. I use that instruction to improve code quality.</p><p>In some cases I add details to make it more specific:</p><ul><li><p>Functions no longer than 30 lines</p></li><li><p>Files no longer than 300 lines</p></li></ul><p>I tested these rules on <a href="https://github.com/DataTalksClub/course-management-platform">a fairly large codebase I started in 2023</a>. I developed it with AI, first by copy-pasting from ChatGPT and then fully with coding agents. When the code turned into a pile of slop, I asked the agent to refactor the entire codebase using this approach.</p><p>It took two banked Codex resets, but after that I could read the code without crying.</p><p>This is how <a href="https://github.com/DataTalksClub/course-management-platform/blob/0b9ed3152d5420813e86f56d8f14cb72f82ead4e/courses/homework_score_calculation.py">some functions</a> now look like:</p><pre><code>def submission_total_score(submission: Submission, questions_score_value: int):
    lip_score = update_learning_in_public_score(submission)
    faq_score = update_faq_score(submission)
    return questions_score_value + lip_score + faq_score</code></pre><h2><strong>Rules for Writing</strong></h2><p>Hugo does the same for writing with The Elements of Style from Strunk and White. Its rules include &#8220;omit needless words&#8221; and &#8220;use the active voice&#8221;.</p><p>The agents don&#8217;t always follow them, so he runs three extra verification agents in parallel. Then he has a final agent edit the text.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!AmyJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!AmyJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 424w, https://substackcdn.com/image/fetch/$s_!AmyJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 848w, https://substackcdn.com/image/fetch/$s_!AmyJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 1272w, https://substackcdn.com/image/fetch/$s_!AmyJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!AmyJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png" width="1100" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:1100,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Hugo gives three verification agents the Strunk and White rules in AGENTS.md, runs them in parallel, then has an editor agent edit the text&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Hugo gives three verification agents the Strunk and White rules in AGENTS.md, runs them in parallel, then has an editor agent edit the text" title="Hugo gives three verification agents the Strunk and White rules in AGENTS.md, runs them in parallel, then has an editor agent edit the text" srcset="https://substackcdn.com/image/fetch/$s_!AmyJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 424w, https://substackcdn.com/image/fetch/$s_!AmyJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 848w, https://substackcdn.com/image/fetch/$s_!AmyJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 1272w, https://substackcdn.com/image/fetch/$s_!AmyJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aab3fa0-2881-4f1b-95db-0769164f7f20_1100x440.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hugo&#8217;s editing setup: three checkers in parallel, then one editor</figcaption></figure></div><h2><strong>Organizing Agents Like a Team</strong></h2><p>With an implementer and reviewer we have two agents. But we can scale this idea to a full team by adding a product manager and on-call engineer.</p><p>We give each agent a job and ask them to work together on a project:</p><ul><li><p>PM plans the tasks and adds acceptance criteria</p></li><li><p>Software engineer implements the task</p></li><li><p>QA engineer checks that the code works against the criteria</p></li><li><p>PM runs acceptance tests through a browser</p></li><li><p>On-call engineer makes sure that CI/CD checks pass</p></li></ul><p>I described this approach in <a href="https://aishippingblog.com/p/i-built-an-ai-agent-team-for-software">I Built an AI Agent Team for Software Development</a> and in <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a>.</p><p>When I read <a href="https://openai.com/index/navier-stokes-solution/">OpenAI&#8217;s Navier-Stokes result</a>, I found it interesting. I know little about the mathematics. But the report described 10,000 agents running in parallel, and I tried to imagine how I&#8217;d manage that too.</p><p>It was interesting for me because it&#8217;s far from trivial. It took me quite some time to figure out how to run a single team of five agents. I still haven&#8217;t made a department of teams work reliably, although I hope I&#8217;m pretty close.</p><p>I experimented with scaling beyond one team - most recently while working on <a href="https://alexeygrigorev.com/cloudflare-agent-git/">Cloudflare&#8217;s Git platform competition</a>. Keeping them working was harder than I imagined. The agents were failing, stalling, or waiting for my input or approval. They do that even with <code>--dangerously-skip-permissions</code>, sometimes producing no output for hours or days.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!W75b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!W75b!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!W75b!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!W75b!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!W75b!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!W75b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png" width="603" height="402.13804945054943" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:603,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Conceptual illustration: an unread message waits beside an idle agent while requests are moved onto an open task board.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Conceptual illustration: an unread message waits beside an idle agent while requests are moved onto an open task board." title="Conceptual illustration: an unread message waits beside an idle agent while requests are moved onto an open task board." srcset="https://substackcdn.com/image/fetch/$s_!W75b!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!W75b!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!W75b!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!W75b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d195ba-bf09-43a8-8947-077069858a01_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Illustration from <a href="https://alexeygrigorev.com/cloudflare-agent-git/daily/2026-10-07/">a daily competition report</a>: &#8220;an unread message waits beside an idle agent while requests are moved onto an open task board.&#8221;</figcaption></figure></div><p>I did get some teams working, and now I can run 30-50 agents in parallel working on different projects. I&#8217;ll definitely write more about them soon.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>I&#8217;ll probably never have a problem that requires 10,000 agents, but if I needed that, I&#8217;d organize them into a hierarchy:</p><ul><li><p>A chief scientist setting the direction</p></li><li><p>Team heads coordinating each part of the work</p></li><li><p>Many agents under them exploring different approaches</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!GwJM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GwJM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 424w, https://substackcdn.com/image/fetch/$s_!GwJM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 848w, https://substackcdn.com/image/fetch/$s_!GwJM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 1272w, https://substackcdn.com/image/fetch/$s_!GwJM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GwJM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png" width="1060" height="580" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:580,&quot;width&quot;:1060,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Tree diagram: I would ask a chief scientist agent to assign approaches A, B, and C to three team lead agents, and ask each team lead to direct its own team of agents&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Tree diagram: I would ask a chief scientist agent to assign approaches A, B, and C to three team lead agents, and ask each team lead to direct its own team of agents" title="Tree diagram: I would ask a chief scientist agent to assign approaches A, B, and C to three team lead agents, and ask each team lead to direct its own team of agents" srcset="https://substackcdn.com/image/fetch/$s_!GwJM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 424w, https://substackcdn.com/image/fetch/$s_!GwJM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 848w, https://substackcdn.com/image/fetch/$s_!GwJM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 1272w, https://substackcdn.com/image/fetch/$s_!GwJM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d1bc72-76ca-441f-97eb-9e9cae2164fc_1060x580.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">How I would organize 10,000 agents</figcaption></figure></div><p>With so many agents running in parallel, you can&#8217;t expect humans to be able to review any of this code.</p><h2><strong>Dark Software Factories</strong></h2><p>I recently had a chat with a company that has a policy of &#8220;no handwritten code&#8221;. There, you aren&#8217;t allowed to write code yourself, only through an agent. When some engineers threatened to resign, the company said &#8220;nobody is forcing you to work here, you&#8217;re welcome to explore other options&#8221;.</p><p>Our industry has changed significantly over the last year. Today we write code differently. Even half a year ago I was still checking my code from time to time, but now it&#8217;s over. I do this only when I prepare materials for my workshops and courses.</p><p>Some time ago I heard the term &#8220;software factory&#8221;. It made sense to me: you have a team of agents and they work through your backlog. So I can imagine a conveyor belt of features, each agent is adding something to it, and the result is a product.</p><p>Hugo takes this further and describes a dark software factory. The name comes from lights-out manufacturing: when machines work without people on the factory floor, there&#8217;s no need to keep the lights on. <a href="https://en.wikipedia.org/wiki/Lights_out_%28manufacturing%29#Existing_%22lights-out_factories%22">FANUC&#8217;s factories</a> are a physical example of this kind of automation.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2buf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 424w, https://substackcdn.com/image/fetch/$s_!2buf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 848w, https://substackcdn.com/image/fetch/$s_!2buf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!2buf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2buf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg" width="320" height="213" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:213,&quot;width&quot;:320,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Yellow industrial robots in FANUC's robot factory in Japan&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Yellow industrial robots in FANUC's robot factory in Japan" title="Yellow industrial robots in FANUC's robot factory in Japan" srcset="https://substackcdn.com/image/fetch/$s_!2buf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 424w, https://substackcdn.com/image/fetch/$s_!2buf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 848w, https://substackcdn.com/image/fetch/$s_!2buf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!2buf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07dece2b-82ab-4653-8e15-5627fde4c132_320x213.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">FANUC&#8217;s robot factory. Photo: <a href="https://www.fanuc.co.jp/en/profile/production/factory1.html">FANUC</a>, which describes automatic transport and continuous testing of assembled robots</figcaption></figure></div><p>In a dark software factory, humans aren&#8217;t allowed to look at the code. They get artifacts that let them make business decisions and verify that everything is correct, but they&#8217;re separated from the generated code.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ON2F!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 424w, https://substackcdn.com/image/fetch/$s_!ON2F!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 848w, https://substackcdn.com/image/fetch/$s_!ON2F!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 1272w, https://substackcdn.com/image/fetch/$s_!ON2F!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ON2F!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png" width="1122" height="264" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:264,&quot;width&quot;:1122,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:56251,&quot;alt&quot;:&quot;You give specs to a dark factory. Agents build and verify code without human code review, then return dashboards and cards for you to accept or reject&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="You give specs to a dark factory. Agents build and verify code without human code review, then return dashboards and cards for you to accept or reject" title="You give specs to a dark factory. Agents build and verify code without human code review, then return dashboards and cards for you to accept or reject" srcset="https://substackcdn.com/image/fetch/$s_!ON2F!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 424w, https://substackcdn.com/image/fetch/$s_!ON2F!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 848w, https://substackcdn.com/image/fetch/$s_!ON2F!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 1272w, https://substackcdn.com/image/fetch/$s_!ON2F!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007e7d7c-9d6a-46d5-a3e5-913e11172519_1122x264.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">In a dark software factory you work with specs and artifacts, never with the code</figcaption></figure></div><p>Hugo doesn&#8217;t suggest that everyone should start building a dark software factory today. But he thinks it&#8217;ll soon be economically viable to skip reading code and focus on the business.</p><p>He compares this to trusting compilers without reading the machine code they produce. With agents, we need tests, independent checks and clear specifications to earn that trust.</p><p>I agree with that direction. I&#8217;m a solo founder with limited time, so if I check everything that agents write, I&#8217;ll never be able to finish anything. Once I let the agents run without my involvement, I can spend more time on product and architecture decisions.</p><p>I started reading more about dark software factories and realized that I&#8217;ve been doing some of this for quite some time too. I&#8217;m writing a separate article to explore how other people keep their factories running and what I can learn from them.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>Keeping the Factory Running</strong></h2><p>Humans are still needed to keep the factory running. First of all, it doesn&#8217;t just happen, people build it. And once it&#8217;s running, it needs attention. Even the FANUC factory needs a maintenance every month.</p><p>It&#8217;s the same with agents. You can polish your process and let your agentic teams run unsupervised, but only up to a point. Eventually, you&#8217;ll still become the bottleneck - or else the agents will build something you don&#8217;t need.</p><p>And we&#8217;re still learning how to build it.</p><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/p/trends-in-agentic-engineering?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! This post is public so feel free to share it.</p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/p/trends-in-agentic-engineering?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://aishippingblog.com/p/trends-in-agentic-engineering?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><p></p>]]></content:encoded></item><item><title><![CDATA[Trends in AI Engineering]]></title><description><![CDATA[Webinar with Hugo Bowne-Anderson]]></description><link>https://aishippingblog.com/p/trends-in-ai-engineering</link><guid isPermaLink="false">https://aishippingblog.com/p/trends-in-ai-engineering</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Mon, 12 Oct 2026 14:31:46 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!KDqu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Last week I ran a webinar with <span class="mention-wrap" data-attrs="{&quot;name&quot;:&quot;Hugo Bowne-Anderson&quot;,&quot;id&quot;:63978139,&quot;type&quot;:&quot;user&quot;,&quot;url&quot;:null,&quot;photo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!_vkH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941a1d5e-0a5f-45f8-8400-737c0bf9f85d_400x400.jpeg&quot;,&quot;uuid&quot;:&quot;3eb81c67-1799-41c9-8678-ddd889c0394a&quot;}" data-component-name="MentionToDOM"></span>. Hugo and I met in Berlin about a month before, had a great breakfast, and decided to do something together.</p><p>The first step we decided on was an open ask-us-anything session - which was the webinar.</p><p>We had two goals for it:</p><ul><li><p>help the people who came to the webinar</p></li><li><p>understand what we could create together for you longer term</p></li></ul><div id="youtube2-whIdxTQsU0A" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;whIdxTQsU0A&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/whIdxTQsU0A?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p>At the end of the session, Hugo and I decided to turn the webinar into a Substack article.</p><p>In the session, we talked about both AI engineering and agentic engineering. It was a lot of content, so I decided to split it into two parts. In this article, I focus on AI engineering only.</p><p>In particular, I&#8217;ll cover:</p><ul><li><p>Hugo&#8217;s distinction between AI engineering and agentic engineering</p></li><li><p>What we can see in nine monthly scrapes of AI engineering job postings</p></li><li><p>Which skills will stay valuable, and why working with people still matters</p></li><li><p>What to focus on if you want to become an AI engineer</p></li><li><p>The course Hugo and I want to build</p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>Hugo and Me</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Nu4P!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Nu4P!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Nu4P!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Nu4P!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Nu4P!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Nu4P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg" width="1456" height="1092" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1092,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Alexey Grigorev and Hugo Bowne-Anderson in a cafe in Berlin&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Alexey Grigorev and Hugo Bowne-Anderson in a cafe in Berlin" title="Alexey Grigorev and Hugo Bowne-Anderson in a cafe in Berlin" srcset="https://substackcdn.com/image/fetch/$s_!Nu4P!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Nu4P!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Nu4P!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Nu4P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d4afd7c-0138-4d60-be1a-01f30f76dc2c_1600x1200.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hugo and I at our breakfast in Berlin</figcaption></figure></div><p>Since you&#8217;re reading this newsletter, I hope you already know a few things about me. I like calling myself a product AI engineer: I do whatever it takes to integrate AI from the user&#8217;s perspective. I teach this in my courses.</p><p>Hugo&#8217;s background is in mathematics, physics, and biology. He researched cell biology at the Max Planck Institute in Dresden and then at Yale. About 15 years ago he moved to industry, but he stayed in education. He worked at DataCamp, taught in the SciPy and PyData ecosystems, and did developer relations at startups such as Coiled and Outerbounds. He hosts the <a href="https://vanishinggradients.fireside.fm/">Vanishing Gradients</a> podcast, and that&#8217;s how we first met - we&#8217;ve done a bunch of podcasts together.</p><h2><strong>AI Engineering vs Agentic Engineering</strong></h2><p>We talked about a lot of things in the session. One of them was Hugo&#8217;s distinction between two terms that often get confused:</p><ul><li><p>In agentic engineering, you build software with agents. The agent is part of the dev team. Your job is specification, delegation, review, and orchestration.</p></li><li><p>In AI engineering, you build agentic search, RAG, or an agent. AI is part of the product. You think about models, prompts, evals, harness engineering, and agentic loops.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_xCj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_xCj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_xCj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_xCj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_xCj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_xCj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg" width="1456" height="821" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:821,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two-column slide: in agentic engineering, the agent is part of the development team; in AI engineering, AI is part of the product&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two-column slide: in agentic engineering, the agent is part of the development team; in AI engineering, AI is part of the product" title="Two-column slide: in agentic engineering, the agent is part of the development team; in AI engineering, AI is part of the product" srcset="https://substackcdn.com/image/fetch/$s_!_xCj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_xCj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_xCj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_xCj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32b4eff4-92ba-4126-afef-ec0411708cae_1600x902.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hugo&#8217;s slide comparing agentic engineering and AI engineering</figcaption></figure></div><p>In practice, you do both. You build AI-powered products with agents. And for agentic engineering, you build up your own setup - skills in your AGENTS.md, MCP servers, sandboxes, and guardrails.</p><p>I see this confusion among people who join my courses too: they join a course about AI engineering wanting to learn how to use AI for coding. There&#8217;s a lot of overlap, but the work is different. If you write code with AI, someone else takes care of evaluating and monitoring that tool. You focus on creating the proper specification, making sure the agent follows it, and enforcing the development lifecycle rules.</p><p>In this article, we focus on AI engineering, and in the next one, on agentic engineering.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>The Job Market</strong></h2><p>I maintain the <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide">AI Engineering Field Guide</a> repository.</p><p>Every month since February 2026 I scrape job postings for the query &#8220;AI engineering&#8221; in six locations: Berlin, Amsterdam, London, New York, Los Angeles, and all of India. Then I use LLMs to extract skills, responsibilities, and use cases from each posting.</p><p>This is what one processed job posting looks like:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;yaml&quot;,&quot;nodeId&quot;:&quot;07f6a213-e79c-492f-bf96-c9f69ea2220f&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-yaml">company:
  name: Ecolab
  focus: Water, hygiene and infection prevention solutions
position:
  title: Senior AI Engineer
  ai_type:
    type: ai-first
  responsibilities:
  - Lead development of scalable GenAI-powered systems
  - Design intelligent workflows leveraging LLMs, vector-based retrieval, and multi-agent
    orchestration
  - Drive solution architecture for AI applications
  - Mentor junior engineers
  - Deliver production-ready applications integrated with business processes and platforms
  use_cases:
  - Intelligent business process workflows powered by LLMs
  - Vector-based retrieval and multi-agent orchestration for enterprise applications
  - GenAI applications integrated with business platforms
  skills:
    genai: [Generative AI, LLMs, RAG, Multi-Agent Systems]
# ...</code></pre></div><p>It&#8217;s not enough data to predict the future, but I can already see where the market is going.</p><p>The overall market is growing, and there&#8217;s a lot of demand for AI engineers.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!UVDQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UVDQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!UVDQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!UVDQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!UVDQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UVDQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Bar chart of AI engineering job postings per monthly scrape from February to September 2026, from 895 to 1,086 with a spring dip and a peak of 1,222 in August&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Bar chart of AI engineering job postings per monthly scrape from February to September 2026, from 895 to 1,086 with a spring dip and a peak of 1,222 in August" title="Bar chart of AI engineering job postings per monthly scrape from February to September 2026, from 895 to 1,086 with a spring dip and a peak of 1,222 in August" srcset="https://substackcdn.com/image/fetch/$s_!UVDQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!UVDQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!UVDQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!UVDQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbab5810c-8093-4fa3-a7a5-b7844aa5bce3_1600x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">AI engineering postings per monthly scrape</figcaption></figure></div><p>Forward-deployed engineer (FDE) roles grow even faster than the market overall. The number of FDE postings has roughly doubled since February.</p><blockquote><p>If you want to learn more about FDEs, I have a separate analysis of this part of the AI jobs market:</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;eb92f520-628f-482b-97ec-d4ba3db2af5a&quot;,&quot;caption&quot;:&quot;Forward-deployed engineering became a growing part of the AI engineering market in 2026.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;What AI Forward-Deployed Engineers Do&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-07-25T08:35:43.976Z&quot;,&quot;cover_image&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/022df211-7fed-425c-bccf-fe3a41efd637_1600x817.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/what-ai-forward-deployed-engineers&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:208329624,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:73,&quot;comment_count&quot;:3,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5MWL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5MWL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!5MWL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!5MWL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!5MWL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5MWL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Bar chart of forward-deployed engineer postings per scrape, growing from 18 in February to 34-37 in July through September 2026&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Bar chart of forward-deployed engineer postings per scrape, growing from 18 in February to 34-37 in July through September 2026" title="Bar chart of forward-deployed engineer postings per scrape, growing from 18 in February to 34-37 in July through September 2026" srcset="https://substackcdn.com/image/fetch/$s_!5MWL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!5MWL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!5MWL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!5MWL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce25ae9-1849-4f81-ba54-e6fcff969331_1600x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Forward-deployed engineer postings per scrape</figcaption></figure></div><p>Roles that focus on training models are shrinking. Before, companies hired more people to train models with PyTorch. Now they post fewer of these roles. And I can see fewer inference and serving roles in the data too.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!wgkx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wgkx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!wgkx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!wgkx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!wgkx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!wgkx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart: the share of AI-first postings that mention inference and serving skills fell from 20% in February to 16% in September 2026, with a dip to 10% in August&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart: the share of AI-first postings that mention inference and serving skills fell from 20% in February to 16% in September 2026, with a dip to 10% in August" title="Line chart: the share of AI-first postings that mention inference and serving skills fell from 20% in February to 16% in September 2026, with a dip to 10% in August" srcset="https://substackcdn.com/image/fetch/$s_!wgkx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!wgkx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!wgkx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!wgkx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1438780c-60c5-4bff-a93c-23ffd3734be6_1600x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Share of AI-first postings that mention inference and serving skills</figcaption></figure></div><p>The data suggests that companies now mostly use &#8220;AI engineer&#8221; to mean someone who calls models through APIs and builds things around them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KDqu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KDqu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!KDqu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!KDqu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!KDqu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KDqu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart: the share of AI-first postings that mention the integrator stack grew from 80% to 91%, while the trainer stack fell from 38% to 30%&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart: the share of AI-first postings that mention the integrator stack grew from 80% to 91%, while the trainer stack fell from 38% to 30%" title="Line chart: the share of AI-first postings that mention the integrator stack grew from 80% to 91%, while the trainer stack fell from 38% to 30%" srcset="https://substackcdn.com/image/fetch/$s_!KDqu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!KDqu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!KDqu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!KDqu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f9eaf1-1974-4df9-b30b-e770d4784a52_1600x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Share of AI-first postings that mention the integrator stack (RAG, agents, LLM APIs) vs the trainer stack (PyTorch, fine-tuning, CUDA)</figcaption></figure></div><p>Companies build agents and RAG around these models, and both keep growing. Agents went from 50% to 60% of all postings, RAG from 33% to 45%.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KqJE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KqJE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!KqJE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!KqJE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!KqJE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KqJE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart: the share of postings that mention agents grew from 50% to 60%, and RAG from 33% to 45%, between February and September 2026&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart: the share of postings that mention agents grew from 50% to 60%, and RAG from 33% to 45%, between February and September 2026" title="Line chart: the share of postings that mention agents grew from 50% to 60%, and RAG from 33% to 45%, between February and September 2026" srcset="https://substackcdn.com/image/fetch/$s_!KqJE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!KqJE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!KqJE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!KqJE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F839c18ef-2578-4bc0-b9f1-154fc078ac0d_1600x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Share of all postings that mention agents and RAG</figcaption></figure></div><p>There are almost no junior positions: only about 1% of job titles say junior or entry level. Half of the titles don&#8217;t state a level, but among those that do, almost all are senior or above. Starting your career as an AI engineer is hard.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zlJ6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zlJ6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!zlJ6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!zlJ6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!zlJ6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zlJ6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Horizontal bar chart of seniority from job titles in the September 2026 scrape: junior or entry 1.1%, no level in title 49.8%, senior 27.8%, lead 7.0%, staff or principal 13.1%, manager or director 1.1%&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Horizontal bar chart of seniority from job titles in the September 2026 scrape: junior or entry 1.1%, no level in title 49.8%, senior 27.8%, lead 7.0%, staff or principal 13.1%, manager or director 1.1%" title="Horizontal bar chart of seniority from job titles in the September 2026 scrape: junior or entry 1.1%, no level in title 49.8%, senior 27.8%, lead 7.0%, staff or principal 13.1%, manager or director 1.1%" srcset="https://substackcdn.com/image/fetch/$s_!zlJ6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!zlJ6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!zlJ6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!zlJ6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c19de25-a6a3-469a-aa76-a58fda687b1f_1600x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Seniority from the job title, September 2026 scrape</figcaption></figure></div><p>So AI engineering is not a junior role.</p><h2><strong>Relevant Skills</strong></h2><p>We can observe the current situation on the market through data and by talking to friends and our network. But we don&#8217;t have a crystal ball to see what roles, hiring processes, or even tools will look like in a year or two.</p><p>But based on our past experience, we can try to make an educated guess about the skills that will stay relevant and valuable in the future.</p><p>Hugo believes that it&#8217;s a combination of:</p><ul><li><p>A hacker mentality - playing with new models when they come out, exploring what&#8217;s possible and seeing how to use them to deliver business value or help people</p></li><li><p>Data science skills - statistics, data analysis, using data to improve products</p></li><li><p>Domain expertise - SRE, DevOps, marketing, or anything else</p></li></ul><p>I can&#8217;t argue with that, but I would also add the problem discovery skill.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WAzi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WAzi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 424w, https://substackcdn.com/image/fetch/$s_!WAzi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 848w, https://substackcdn.com/image/fetch/$s_!WAzi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 1272w, https://substackcdn.com/image/fetch/$s_!WAzi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!WAzi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png" width="780" height="300" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:300,&quot;width&quot;:780,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A 2x2 grid of four skills that stay valuable: hacker mentality, data science, domain expertise, and problem discovery&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A 2x2 grid of four skills that stay valuable: hacker mentality, data science, domain expertise, and problem discovery" title="A 2x2 grid of four skills that stay valuable: hacker mentality, data science, domain expertise, and problem discovery" srcset="https://substackcdn.com/image/fetch/$s_!WAzi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 424w, https://substackcdn.com/image/fetch/$s_!WAzi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 848w, https://substackcdn.com/image/fetch/$s_!WAzi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 1272w, https://substackcdn.com/image/fetch/$s_!WAzi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fff884f-e533-4a20-9c85-747dfaa4a229_780x300.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Four skills to stay valuable</figcaption></figure></div><p>There are so many problems around us waiting to be discovered and solved. Look at your own everyday life. You repeat many things, and many things are mildly annoying, but you aren&#8217;t doing anything about them yet.</p><p>With AI, I can finally approach these problems. But before I can even approach them, I need to know they exist and identify them. Only then can I solve them with AI. AI can&#8217;t find these problems for you, and you can stand out by learning how to find them.</p><p>And finally, we need the human skills.</p><h2><strong>Human Skills: Working With People</strong></h2><p>We use agents to write code, and we write code to solve problems for our customers. Our customers are people, not agents. We still need the human connection, user interviews, and everything else that product managers and product designers have been doing for a long time.</p><p>As a solo founder, I can go far with AI. But my background is engineering, and I don&#8217;t have marketing or sales skills. For me, it&#8217;s hard to market and sell what I build.</p><p>Freelancers have the same problem. You need to explain to clients why you charge $2,000 per day when a Claude subscription costs $200 per month. At work, you still need to convince your manager to give you that promotion.</p><p>Hugo believes that the work of engineers is speaking with people, understanding their problems, and solving them in a way that gives them more value than they pay. Even this session exists because we (humans) met in Berlin and had breakfast.</p><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/p/trends-in-ai-engineering?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! This post is public so feel free to share it.</p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/p/trends-in-ai-engineering?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://aishippingblog.com/p/trends-in-ai-engineering?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h2><strong>Becoming an AI Engineer</strong></h2><p>There are also technical skills that you need. If you want to become an AI engineer and have a limited amount of time, it makes sense to focus on the areas with the most return on investment: RAG, agents, and evaluations.</p><p>Evals are the most important part. Systems built on LLMs aren&#8217;t deterministic, so we can&#8217;t simply trust them. We need a framework to make sure they can be relied on - for example, an evaluation set built from real user data.</p><p>Most jobs assume you have evaluation skills, but we can also see that more and more positions explicitly list them as a requirement.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ADO8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ADO8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!ADO8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!ADO8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!ADO8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ADO8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart: the share of AI engineering postings that mention LLM evaluation grew from 20% in February to 25% in September 2026&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart: the share of AI engineering postings that mention LLM evaluation grew from 20% in February to 25% in September 2026" title="Line chart: the share of AI engineering postings that mention LLM evaluation grew from 20% in February to 25% in September 2026" srcset="https://substackcdn.com/image/fetch/$s_!ADO8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 424w, https://substackcdn.com/image/fetch/$s_!ADO8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 848w, https://substackcdn.com/image/fetch/$s_!ADO8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 1272w, https://substackcdn.com/image/fetch/$s_!ADO8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa914cd3b-063d-4ca3-bb8e-8c9e64272ab0_1600x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Share of AI engineering postings that mention LLM evaluation</figcaption></figure></div><p>You learn evals by building a project: an agent or an AI-powered application with real users.</p><p>There are also things you can skip. The data shows that you don&#8217;t need to focus on training and serving models. Hugo compared training models to working on an internal combustion engine. If you&#8217;re a Formula One driver, it&#8217;s good to know how it works, but most people who use a car don&#8217;t need to. An AI engineer works on a product.</p><p>You don&#8217;t need to know LLM internals to start working as an AI engineer. At the beginning, it&#8217;s enough to know how to send a request to an LLM provider, build everything around it, and have the data science mindset. Some companies do lower-level work and may ask for it, but they&#8217;re a minority in my data, and the share keeps falling.</p><p>But if you want to learn more about these things, I&#8217;d recommend Inference Engineering by Philip Kiely. Recently <a href="https://aishippinglabs.com/books/inference-engineering">we hosted a book club about it</a>, and it&#8217;s a wonderful book if you want to know what happens after you send a request. But if your goal is to become an AI engineer, read it later and focus on building projects now. If you want to become an inference engineer, that&#8217;s the book to read.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!j2Df!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!j2Df!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 424w, https://substackcdn.com/image/fetch/$s_!j2Df!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 848w, https://substackcdn.com/image/fetch/$s_!j2Df!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 1272w, https://substackcdn.com/image/fetch/$s_!j2Df!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!j2Df!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png" width="572" height="302.01097694840837" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/23703efb-5c17-4b94-a141-d11a40975b96_911x481.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:481,&quot;width&quot;:911,&quot;resizeWidth&quot;:572,&quot;bytes&quot;:313037,&quot;alt&quot;:&quot;Cover of Inference Engineering by Philip Kiely&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Cover of Inference Engineering by Philip Kiely" title="Cover of Inference Engineering by Philip Kiely" srcset="https://substackcdn.com/image/fetch/$s_!j2Df!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 424w, https://substackcdn.com/image/fetch/$s_!j2Df!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 848w, https://substackcdn.com/image/fetch/$s_!j2Df!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 1272w, https://substackcdn.com/image/fetch/$s_!j2Df!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23703efb-5c17-4b94-a141-d11a40975b96_911x481.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Inference Engineering by Philip Kiely</figcaption></figure></div><p></p><h2><strong>Agentic Engineering</strong></h2><p>That was only half of our webinar with Hugo. In the second half, we talked about agentic engineering - using AI assistants to create code. I&#8217;m working on an article about that and will publish it soon.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4P0X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4P0X!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 424w, https://substackcdn.com/image/fetch/$s_!4P0X!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 848w, https://substackcdn.com/image/fetch/$s_!4P0X!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!4P0X!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4P0X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg" width="1456" height="782" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:782,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Hugo's slide of the agentic software lifecycle: a spec goes to an agent-driven build, then to verification by human review, agent review, LLM as judge, and automated tests, and only working code ships&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Hugo's slide of the agentic software lifecycle: a spec goes to an agent-driven build, then to verification by human review, agent review, LLM as judge, and automated tests, and only working code ships" title="Hugo's slide of the agentic software lifecycle: a spec goes to an agent-driven build, then to verification by human review, agent review, LLM as judge, and automated tests, and only working code ships" srcset="https://substackcdn.com/image/fetch/$s_!4P0X!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 424w, https://substackcdn.com/image/fetch/$s_!4P0X!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 848w, https://substackcdn.com/image/fetch/$s_!4P0X!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!4P0X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57f338f2-aa85-4fdd-a0fe-f17ce992d1a9_1600x859.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hugo&#8217;s slide on the agentic software lifecycle</figcaption></figure></div><p>In it, I&#8217;ll cover:</p><ul><li><p>How agents changed the way we write code</p></li><li><p>Why verification is now the bottleneck</p></li><li><p>The implementer-reviewer pattern: letting one agent check another</p></li><li><p>How to deal with sloppy agent code</p></li><li><p>The rules for better code and better writing</p></li><li><p>How to organize running many agents in parallel</p></li><li><p>Dark software factories</p></li><li><p>How I stay out of the loop in my own projects</p></li></ul><p>Subscribe to make sure you don&#8217;t miss it.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>Our Next Course</strong></h2><p>The main goal of our session was to find out how we can help you achieve your goals and what we can build for you through a course that we want to run together. There are infinitely many ideas, so we want to understand what the best direction is.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!K7j-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!K7j-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 424w, https://substackcdn.com/image/fetch/$s_!K7j-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 848w, https://substackcdn.com/image/fetch/$s_!K7j-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 1272w, https://substackcdn.com/image/fetch/$s_!K7j-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!K7j-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png" width="1456" height="973" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:973,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Intro of the questionnaire Help Shape a New Practical AI Engineering Course by Hugo Bowne-Anderson and Alexey Grigorev: a paid hands-on course, tuition of about US$1,500, a 4 to 5 minute survey, and a 30% discount for respondents&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Intro of the questionnaire Help Shape a New Practical AI Engineering Course by Hugo Bowne-Anderson and Alexey Grigorev: a paid hands-on course, tuition of about US$1,500, a 4 to 5 minute survey, and a 30% discount for respondents" title="Intro of the questionnaire Help Shape a New Practical AI Engineering Course by Hugo Bowne-Anderson and Alexey Grigorev: a paid hands-on course, tuition of about US$1,500, a 4 to 5 minute survey, and a 30% discount for respondents" srcset="https://substackcdn.com/image/fetch/$s_!K7j-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 424w, https://substackcdn.com/image/fetch/$s_!K7j-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 848w, https://substackcdn.com/image/fetch/$s_!K7j-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 1272w, https://substackcdn.com/image/fetch/$s_!K7j-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16a0b9ac-a8d5-4272-b3af-52a441586903_1676x1120.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The questionnaire we shared at the end of the session</figcaption></figure></div><p>Tell us where you stand, what you want to build, where you&#8217;re stuck, and what format you prefer. We want to hear both what frustrates you and what excites you about AI.</p><p>If you want to help us, fill in <a href="https://forms.gle/EjtBzfmZft15dCvY9">this questionnaire</a>. It takes about 5 minutes. As a thank you, people who fill it in will get a discount on the future course when we announce it.</p><p>Of course, I&#8217;ll announce it here in this newsletter.</p>]]></content:encoded></item><item><title><![CDATA[AI Agents Can Do Everything Now. Do We Still Need Kubernetes?]]></title><description><![CDATA[Yes, we do!]]></description><link>https://aishippingblog.com/p/ai-agents-can-do-everything-now-do</link><guid isPermaLink="false">https://aishippingblog.com/p/ai-agents-can-do-everything-now-do</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Wed, 07 Oct 2026 16:44:48 GMT</pubDate><enclosure url="https://substackcdn.com/image/youtube/w_728,c_limit/uBpXLrpc3IY" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>AI agents write code, trigger deployments, watch dashboards, and handle on-call problems. Should we still care about Kubernetes? Yes, we should.</p><p>The agents need a place to run. They need:</p><ul><li><p>Compute</p></li><li><p>GPUs</p></li><li><p>Networking</p></li><li><p>Access to data</p></li><li><p>A way to scale</p></li></ul><p>And when something fails, they need to recover.</p><p>Kubernetes is the layer that coordinates all of that.</p><div id="youtube2-uBpXLrpc3IY" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;uBpXLrpc3IY&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/uBpXLrpc3IY?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p>Next Thursday, October 15, is Kubernetes Automation Day Berlin. Several of its talks answer exactly this question.</p><p>In this article, I go through the talks I find most interesting. They show why Kubernetes is still relevant and why we should care about it.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>An Agent on a Laptop Isn&#8217;t Production</strong></h2><p>Giant Swarm, a company that builds and runs Kubernetes platforms for enterprises, built an SRE agent that helps them handle real incidents. At first, it ran on the laptop of the engineer who built it.</p><p>When an alert fired at 3 AM, a human still had to wake up just to watch the agent work.</p><p>They moved this agent to Kubernetes, so now it runs in the background, reacts to alerts on its own, and wakes an engineer only when it can&#8217;t fix the incident.</p><p>In his talk, Dominik Schmidle from Giant Swarm will describe how they did it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9gS_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9gS_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 424w, https://substackcdn.com/image/fetch/$s_!9gS_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 848w, https://substackcdn.com/image/fetch/$s_!9gS_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 1272w, https://substackcdn.com/image/fetch/$s_!9gS_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9gS_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png" width="980" height="640" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:640,&quot;width&quot;:980,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;SRE agent running inside a Kubernetes cluster with a ServiceAccount, the Kubernetes API, and monitoring&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="SRE agent running inside a Kubernetes cluster with a ServiceAccount, the Kubernetes API, and monitoring" title="SRE agent running inside a Kubernetes cluster with a ServiceAccount, the Kubernetes API, and monitoring" srcset="https://substackcdn.com/image/fetch/$s_!9gS_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 424w, https://substackcdn.com/image/fetch/$s_!9gS_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 848w, https://substackcdn.com/image/fetch/$s_!9gS_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 1272w, https://substackcdn.com/image/fetch/$s_!9gS_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6406e465-9c25-4c1c-bb6d-bf5d0ecd5aa3_980x640.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Once the agent runs in the cluster, it gets an identity, a reviewed deploy path from Git, and monitoring</figcaption></figure></div><h2><strong>The Night Ops</strong></h2><p>Mehul Patel from BuildingMinds tackles the same problem. He&#8217;ll talk about an autonomous SRE agent that he built with Google&#8217;s ADK. It has two components: an agent for triage and an agent for investigation.</p><p>But you don&#8217;t want to let the agent have unlimited access to the production environment. <a href="https://aishippingblog.com/p/how-i-dropped-our-production-database">Bad things can happen</a> when you do that. Instead, you need guardrails that stop the agent when it tries to do something wrong.</p><p>In the talk, Mehul will present his open-source project <a href="https://github.com/nomadicmehul/TheNightOps">TheNightOps</a> and show what these guardrails can look like. In this case, it&#8217;s a policy file that describes which actions the agent may take on its own.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XNMr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XNMr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 424w, https://substackcdn.com/image/fetch/$s_!XNMr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 848w, https://substackcdn.com/image/fetch/$s_!XNMr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 1272w, https://substackcdn.com/image/fetch/$s_!XNMr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XNMr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png" width="940" height="740" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:740,&quot;width&quot;:940,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Agent actions pass a policy file that auto-runs, asks a human, or blocks them before RBAC in the Kubernetes API&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Agent actions pass a policy file that auto-runs, asks a human, or blocks them before RBAC in the Kubernetes API" title="Agent actions pass a policy file that auto-runs, asks a human, or blocks them before RBAC in the Kubernetes API" srcset="https://substackcdn.com/image/fetch/$s_!XNMr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 424w, https://substackcdn.com/image/fetch/$s_!XNMr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 848w, https://substackcdn.com/image/fetch/$s_!XNMr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 1272w, https://substackcdn.com/image/fetch/$s_!XNMr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09dc50bd-7e00-4553-b74a-7b4ce8b72ae0_940x740.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The policy file decides who acts, and the API server enforces RBAC whatever the prompt says</figcaption></figure></div><p></p><h2><strong>Agents Aren&#8217;t Microservices</strong></h2><p>Abdel Sghiouar from Google Cloud will talk about deploying agents.</p><p>For anyone who knows a bit of Kubernetes, the first idea is to give each agent its own pod. That&#8217;s what we usually do with microservices.</p><blockquote><p>If you don&#8217;t know what a pod is and want to learn more about Kubernetes, I teach it in <a href="https://github.com/DataTalksClub/machine-learning-zoomcamp/tree/main/10-kubernetes">Machine Learning Zoomcamp Module 10</a>.</p></blockquote><p>But agents aren&#8217;t microservices. An agent runs in a pod for a task, and it can spawn multiple subagents along the way. Then it can sit idle for a while, waiting for the subagents to finish.</p><p>Agents can spend up to 90% of their time idle, so giving each agent a dedicated pod may waste resources. It also hits scale limits.</p><p>In the talk, Abdel describes <a href="https://github.com/agent-substrate/substrate">Agent Substrate</a> - an open-source system that separates the agent session from the pod it runs on. With it, you can host many agents on very few pods.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!48jO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!48jO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 424w, https://substackcdn.com/image/fetch/$s_!48jO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 848w, https://substackcdn.com/image/fetch/$s_!48jO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 1272w, https://substackcdn.com/image/fetch/$s_!48jO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!48jO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png" width="980" height="520" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:520,&quot;width&quot;:980,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Substrate restores an idle agent from a snapshot onto a free worker pod&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Substrate restores an idle agent from a snapshot onto a free worker pod" title="Substrate restores an idle agent from a snapshot onto a free worker pod" srcset="https://substackcdn.com/image/fetch/$s_!48jO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 424w, https://substackcdn.com/image/fetch/$s_!48jO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 848w, https://substackcdn.com/image/fetch/$s_!48jO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 1272w, https://substackcdn.com/image/fetch/$s_!48jO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1360198b-e3f1-435f-b9aa-085faa8be15f_980x520.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Idle agents wait as snapshots, so about 250 agents share eight worker pods</figcaption></figure></div><h2><strong>Agents Ship More Code to Production</strong></h2><p>We all know that agents produce a lot of code.</p><p>GitHub&#8217;s COO Kyle Daigle <a href="https://x.com/kdaigle/status/2040164759836778878">shared the numbers</a> in April. There were 1 billion commits on GitHub in 2025. For 2026, they&#8217;re on pace for 14 billion.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Hwdj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Hwdj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 424w, https://substackcdn.com/image/fetch/$s_!Hwdj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 848w, https://substackcdn.com/image/fetch/$s_!Hwdj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 1272w, https://substackcdn.com/image/fetch/$s_!Hwdj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Hwdj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png" width="1600" height="708" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:708,&quot;width&quot;:1600,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:79470,&quot;alt&quot;:&quot;Bar chart of commits on GitHub: 1 billion in 2025 and about 14 billion projected for 2026&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Bar chart of commits on GitHub: 1 billion in 2025 and about 14 billion projected for 2026" title="Bar chart of commits on GitHub: 1 billion in 2025 and about 14 billion projected for 2026" srcset="https://substackcdn.com/image/fetch/$s_!Hwdj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 424w, https://substackcdn.com/image/fetch/$s_!Hwdj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 848w, https://substackcdn.com/image/fetch/$s_!Hwdj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 1272w, https://substackcdn.com/image/fetch/$s_!Hwdj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8181ce4b-e96e-472e-9ae7-24be7f2d7182_1600x708.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">This year we will have 14 times more code than last year. All of that code needs to run somewhere.</figcaption></figure></div><p>We&#8217;re drowning in code. I also contribute to it by teaching agentic engineering in <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a>.</p><p>Adi Shacham-Shavit, who leads the platform team at Enpal, will open the event with a keynote about it: someone has to run, secure, and scale the flood of &#8220;vibe-coded&#8221; software.</p><p>From her abstract:</p><blockquote><p>We used to teach engineers how the infrastructure worked. Now platform teams design for people and agents who have no idea what production looks like.</p></blockquote><p>The barrier to entry is much lower in 2026, but we still need to follow good engineering practices. Platforms can help with that.</p><p></p><h2><strong>KubeAuto Day Berlin</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5mJ5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5mJ5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 424w, https://substackcdn.com/image/fetch/$s_!5mJ5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 848w, https://substackcdn.com/image/fetch/$s_!5mJ5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 1272w, https://substackcdn.com/image/fetch/$s_!5mJ5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5mJ5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png" width="1456" height="929" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:929,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2707496,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/219295051?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5mJ5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 424w, https://substackcdn.com/image/fetch/$s_!5mJ5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 848w, https://substackcdn.com/image/fetch/$s_!5mJ5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 1272w, https://substackcdn.com/image/fetch/$s_!5mJ5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167f039b-2774-4be8-8379-f3d41e55b4e5_2400x1532.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>If you&#8217;re in Berlin, come to <a href="https://kubeauto.day/berlin?utm_source=substack&amp;utm_medium=newsletter&amp;utm_campaign=kubeauto_day_berlin_2026&amp;utm_content=article">Kubernetes Automation Day Berlin</a> on Thursday, October 15. There will be these and many other talks:</p><ul><li><p>building Kubernetes operators</p></li><li><p>attacking misconfigured clusters</p></li><li><p>optimizing workloads</p></li><li><p>a panel on DevOps and platform engineering in the AI world</p></li></ul><p>It&#8217;s a free community event at Alte Turnhalle in Friedrichshain, from 3:30 PM to 10 PM, with 200+ attendees. Tickets are limited. </p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://kubeauto.day/berlin?utm_source=substack&amp;utm_medium=newsletter&amp;utm_campaign=kubeauto_day_berlin_2026&amp;utm_content=article&quot;,&quot;text&quot;:&quot;Register here&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://kubeauto.day/berlin?utm_source=substack&amp;utm_medium=newsletter&amp;utm_campaign=kubeauto_day_berlin_2026&amp;utm_content=article"><span>Register here</span></a></p><p></p><p>If you&#8217;re not in Berlin, there are KubeAuto Days in Paris, Salt Lake City, and Las Vegas. Tel Aviv and S&#227;o Paulo come next year. The full list is on <a href="https://kubeauto.day/">kubeauto.day</a>.</p><p></p><p>This post is made in collaboration with KubeAuto Day Berlin. Thank you for supporting our community!</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Aplexer - Why I Stopped Using Tmux and Created My Own Terminal Multiplexer]]></title><description><![CDATA[Advantages and disadvantages of tmux and my new agent multiplexer]]></description><link>https://aishippingblog.com/p/aplexer-why-i-stopped-using-tmux</link><guid isPermaLink="false">https://aishippingblog.com/p/aplexer-why-i-stopped-using-tmux</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 02 Oct 2026 13:21:56 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!x_Gi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>tmux is a terminal multiplexer that I use to run agents on a remote machine. I wrote about using it in my article <a href="https://aishippingblog.com/p/the-system-i-built-to-ship-code-from">The System I Built to Ship Code From a Phone</a>.</p><p>When you connect to a remote machine with SSH, the processes you launch are bound to your SSH session. That means when you disconnect, or the connection drops, the processes die together with the session.</p><p>A terminal multiplexer is a tool that detaches the processes you start from the SSH session, so they can keep running after the disconnect. Screen and tmux are the most well-known terminal multiplexers, but tmux is more modern and generally preferred among developers.</p><p>I&#8217;ve used tmux for ages, but with agents running 24/7 on my remote dev box, it no longer works for me. To replace it, I created <a href="https://github.com/PocketShell-io/aplexer">aplexer</a>. In this article I&#8217;ll tell you why.</p><p>In particular, we&#8217;ll cover:</p><ul><li><p>What tmux is and how it works</p></li><li><p>What&#8217;s PTY and how it&#8217;s related to SSH connections </p></li><li><p>OOM killers and cgroups</p></li><li><p>Aplexer as the solution to my current problems with tmux</p></li></ul><p>Let&#8217;s start!</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>Terminal multiplexer</strong></h2><p>The <a href="https://github.com/tmux/tmux">tmux README</a> says:</p><blockquote><p>tmux is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.</p></blockquote><p>It&#8217;s a layer between you (your SSH session) and the processes you run.</p><p>After I ssh into my dev box, I can start a tmux session:</p><pre><code>tmux new-session -s ai-shipping-labs</code></pre><p>Under the hood, multiple things happen:</p><ul><li><p>My terminal starts a tmux client process</p></li><li><p>The client connects to a tmux server</p></li><li><p>If the server is not running, the command starts it</p></li></ul><p>The client connects to the server through a socket. The server creates a session and a PTY for that session.</p><p>A PTY is a pseudo-terminal created by the kernel. It has two sides:</p><ul><li><p>the master (user side) - sends the input from the user to whatever is running in the terminal.</p></li><li><p>the slave (shell side) - receives the input from the master and sends it to the shell or whatever program is running in the terminal.</p></li></ul><p>When we start the standard terminal emulator app on any Linux, it creates a PTY. The master side connects to the terminal app, and the slave side connects to a shell (usually bash). It&#8217;s a bit more complicated than that but I won&#8217;t go into more details here.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!S1Zy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 424w, https://substackcdn.com/image/fetch/$s_!S1Zy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 848w, https://substackcdn.com/image/fetch/$s_!S1Zy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 1272w, https://substackcdn.com/image/fetch/$s_!S1Zy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!S1Zy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png" width="1080" height="158" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:158,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:35310,&quot;alt&quot;:&quot;A terminal emulator connects to the master end of one kernel PTY while Bash connects to the slave end&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A terminal emulator connects to the master end of one kernel PTY while Bash connects to the slave end" title="A terminal emulator connects to the master end of one kernel PTY while Bash connects to the slave end" srcset="https://substackcdn.com/image/fetch/$s_!S1Zy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 424w, https://substackcdn.com/image/fetch/$s_!S1Zy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 848w, https://substackcdn.com/image/fetch/$s_!S1Zy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 1272w, https://substackcdn.com/image/fetch/$s_!S1Zy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d78914-bb3e-4836-adc5-9d0339896570_1080x158.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">The terminal emulator uses the master end, and Bash uses the slave end.</figcaption></figure></div><p>When we start a process from a PTY, it gets attached to the shell. When the PTY stops, the shell stops too, and all the connected processes follow.</p><p>If we take the terminal app in Ubuntu, each tab in the app is a PTY. When I start a process in a tab, and then close that tab, the PTY closes too, and the process follows.</p><p>A similar thing happens when we use SSH. The SSH client on our computer connects to the SSH server (sshd) on the remote machine, and sshd creates a PTY. The master is on the sshd side, and the slave is on the shell side.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ngJ1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ngJ1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 424w, https://substackcdn.com/image/fetch/$s_!ngJ1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 848w, https://substackcdn.com/image/fetch/$s_!ngJ1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 1272w, https://substackcdn.com/image/fetch/$s_!ngJ1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ngJ1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png" width="1180" height="400" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:400,&quot;width&quot;:1180,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An SSH connection crosses from an SSH client on the local computer to sshd on the remote devbox, where sshd opens a separate PTY for Bash&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An SSH connection crosses from an SSH client on the local computer to sshd on the remote devbox, where sshd opens a separate PTY for Bash" title="An SSH connection crosses from an SSH client on the local computer to sshd on the remote devbox, where sshd opens a separate PTY for Bash" srcset="https://substackcdn.com/image/fetch/$s_!ngJ1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 424w, https://substackcdn.com/image/fetch/$s_!ngJ1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 848w, https://substackcdn.com/image/fetch/$s_!ngJ1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 1272w, https://substackcdn.com/image/fetch/$s_!ngJ1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7795d97-25e7-40dc-8cb4-2443d332417d_1180x400.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">SSH uses a separate PTY on the remote machine.</figcaption></figure></div><p>When SSH disconnects, sshd stops the PTY that was created for this session, and all the processes die with it.</p><p>Terminal multiplexers add one more hop: tmux server creates another PTY, and the shell that&#8217;s connected to that PTY is the parent for all these processes.</p><p>That&#8217;s why the processes that we start in tmux continue running - they are attached to tmux server&#8217;s PTY. Only the tmux client dies when the SSH connection drops.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mvjM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mvjM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 424w, https://substackcdn.com/image/fetch/$s_!mvjM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 848w, https://substackcdn.com/image/fetch/$s_!mvjM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 1272w, https://substackcdn.com/image/fetch/$s_!mvjM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mvjM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png" width="728" height="300.44444444444446" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:520,&quot;width&quot;:1260,&quot;resizeWidth&quot;:728,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;After an SSH disconnect, sshd, its PTY, and the tmux client stop while the tmux server keeps its pane PTY and agent running&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="After an SSH disconnect, sshd, its PTY, and the tmux client stop while the tmux server keeps its pane PTY and agent running" title="After an SSH disconnect, sshd, its PTY, and the tmux client stop while the tmux server keeps its pane PTY and agent running" srcset="https://substackcdn.com/image/fetch/$s_!mvjM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 424w, https://substackcdn.com/image/fetch/$s_!mvjM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 848w, https://substackcdn.com/image/fetch/$s_!mvjM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 1272w, https://substackcdn.com/image/fetch/$s_!mvjM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b250a24-590c-4761-ad89-b46c5683b51a_1260x520.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">After SSH disconnects, the tmux client exits. The server keeps the agent&#8217;s PTY open.</figcaption></figure></div><p>There are multiple important organizational primitives in tmux:</p><ul><li><p>A pane is the basic unit of tmux. Each pane has its own PTY.</p></li><li><p>A window has one or multiple panes.</p></li><li><p>A session holds one or multiple windows.</p></li><li><p>The tmux server can have many sessions.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6SC2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6SC2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 424w, https://substackcdn.com/image/fetch/$s_!6SC2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 848w, https://substackcdn.com/image/fetch/$s_!6SC2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 1272w, https://substackcdn.com/image/fetch/$s_!6SC2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6SC2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png" width="728" height="419.6111111111111" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:830,&quot;width&quot;:1440,&quot;resizeWidth&quot;:728,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A tmux window with three panes: htop monitors the devbox, an agent session runs in the top-right pane, and a shell lists other sessions in the bottom pane&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A tmux window with three panes: htop monitors the devbox, an agent session runs in the top-right pane, and a shell lists other sessions in the bottom pane" title="A tmux window with three panes: htop monitors the devbox, an agent session runs in the top-right pane, and a shell lists other sessions in the bottom pane" srcset="https://substackcdn.com/image/fetch/$s_!6SC2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 424w, https://substackcdn.com/image/fetch/$s_!6SC2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 848w, https://substackcdn.com/image/fetch/$s_!6SC2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 1272w, https://substackcdn.com/image/fetch/$s_!6SC2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F788a4636-7a06-47fc-bea4-f488088dd680_1440x830.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">One tmux window with 3 panes: htop, claude and shell.</figcaption></figure></div><p>When I create a new session with a command like this:</p><pre><code>tmux new-session -s ai-shipping-labs</code></pre><p>It creates a session with one window with one pane inside it, and this pane has a PTY. I can add a new window or split the current one horizontally or vertically into multiple panes, and each pane would be a separate PTY.</p><p>I usually don&#8217;t do that, though. I use tmux mostly for keeping my agents running when I disconnect, so I don&#8217;t need any window arrangement capabilities. For me it&#8217;s always one session - one PTY, and different agents are running in different sessions.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gzfx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gzfx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 424w, https://substackcdn.com/image/fetch/$s_!gzfx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 848w, https://substackcdn.com/image/fetch/$s_!gzfx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 1272w, https://substackcdn.com/image/fetch/$s_!gzfx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gzfx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png" width="1040" height="600" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:600,&quot;width&quot;:1040,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;One tmux server fans out to three sessions, each with one pane, one PTY, and an agent or shell&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="One tmux server fans out to three sessions, each with one pane, one PTY, and an agent or shell" title="One tmux server fans out to three sessions, each with one pane, one PTY, and an agent or shell" srcset="https://substackcdn.com/image/fetch/$s_!gzfx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 424w, https://substackcdn.com/image/fetch/$s_!gzfx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 848w, https://substackcdn.com/image/fetch/$s_!gzfx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 1272w, https://substackcdn.com/image/fetch/$s_!gzfx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13e54fac-d62a-4a50-88b8-bbe662997690_1040x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">I use one pane per session, so each agent gets its own PTY.</figcaption></figure></div><p>We can do a lot of things programmatically with tmux. We can create all these windows and panes using the tmux CLI, or we can send input to any of the panes as if it was typed by a human. In fact, the earlier screenshot with three panes was created by <a href="https://gist.github.com/alexeygrigorev/8df0400d8814914291487ada31b4a119">this script</a>.</p><p>Because of that, with a bit of scripting, you can teach agents running in different tmux sessions to talk to each other.</p><h2><strong>Problems with tmux</strong></h2><p>I like tmux, but as I started running more and more agents, session management became more difficult.</p><p>I really struggle with the CLI. The commands are:</p><pre><code>tmux new-session -s ai-shipping-labs
tmux list-sessions
tmux attach-session -t ai-shipping-labs</code></pre><p>I always forget these commands. Typing all that, even with autocomplete, is always complicated. I never seem to remember what to type and need to look it up.</p><p>Also, you have to remember that <code>-s</code> is for new session and <code>-t</code> is for attach. To make it even more confusing, <code>new-session</code> also has the <code>-t</code> parameter, but it&#8217;s not the same as <code>-s</code>.</p><p>Eventually I solved this problem with <a href="https://github.com/alexeygrigorev/tmuxctl">tmuxctl</a> - a wrapper around tmux.</p><p>I created an executable <code>tmuxctl</code> and an alias <code>t</code> for it to save typing time. With it, I can simply run</p><pre><code>t -</code></pre><p>It will:</p><ul><li><p>create a tmux session in the current directory</p></li><li><p>name the session after the directory</p></li><li><p>if a session already exists, attach to it</p></li></ul><p>If I run just <code>t</code> without any arguments, I&#8217;ll see a numbered list of recent sessions:</p><pre><code><code>IDX  SESSION               CREATED
1    codex                 2026-04-03 15:56:59
2    backend-worker        2026-04-03 15:22:10
3    docs                  2026-04-03 14:10:31</code></code></pre><p>And if I want to connect to any particular session from that list, I simply run</p><pre><code>t 2</code></pre><p>That made the process more convenient and also saved a lot of time when jumping between sessions.</p><h2><strong>OOM and cgroups</strong></h2><p>But there&#8217;s another problem with tmux - its server is the single point of failure.</p><p>On my dev box, I run many things in parallel.</p><p>At the same time, it could be:</p><ul><li><p>Compiling a project in Rust (like <a href="https://github.com/alexeygrigorev/codex-zcode/">Codex-ZCode bridge</a>)</p></li><li><p>Running Android emulator tests for <a href="https://pocketshell.io/">PocketShell</a></p></li><li><p>Running e2e tests with Playwright for <a href="https://aishippinglabs.com/">AI Shipping Labs</a></p></li></ul><p>If I&#8217;m unlucky and all these things run at the same time, my machine runs out of memory.</p><p>It&#8217;s usually not a problem for Android emulators or Playwright - they are simply killed when it happens. But if I run something like <code>cargo clippy</code> (a linter in Rust), it&#8217;s more dangerous.</p><p>OOM can not only kill the linter process, but also bring down the agent that&#8217;s running it, the shell that&#8217;s running the agent, the session that&#8217;s running the shell, and the tmux server too. When the tmux server dies, all the other sessions go with it.</p><p>So an OOM in Rust can wipe out all the tmux sessions on the machine. All of them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SknL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SknL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 424w, https://substackcdn.com/image/fetch/$s_!SknL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 848w, https://substackcdn.com/image/fetch/$s_!SknL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 1272w, https://substackcdn.com/image/fetch/$s_!SknL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SknL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png" width="1040" height="600" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:600,&quot;width&quot;:1040,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A cargo clippy workload exhausts memory. If the OOM also kills the shared tmux server, sessions A, B, and C all lose their PTYs&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A cargo clippy workload exhausts memory. If the OOM also kills the shared tmux server, sessions A, B, and C all lose their PTYs" title="A cargo clippy workload exhausts memory. If the OOM also kills the shared tmux server, sessions A, B, and C all lose their PTYs" srcset="https://substackcdn.com/image/fetch/$s_!SknL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 424w, https://substackcdn.com/image/fetch/$s_!SknL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 848w, https://substackcdn.com/image/fetch/$s_!SknL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 1272w, https://substackcdn.com/image/fetch/$s_!SknL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5179dca-882e-4edb-a262-cbe7075b461b_1040x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">If an OOM kills the tmux server, all three sessions lose their terminals.</figcaption></figure></div><p>An OOM kill doesn&#8217;t propagate from cargo to tmux. The kernel (&#8221;OOM killer&#8221;) decides who should receive <code>SIGKILL</code> when OOM happens. Usually it&#8217;s the process that caused the actual OOM. But <a href="https://github.com/tmux/tmux/issues/4151">sometimes</a> it may decide to kill the entire memory cgroup where it&#8217;s working, which includes the process, the shell, and the tmux server (bringing down all the sessions too).</p><p>A cgroup (control group) is a container around a group of processes that Linux manages like a single unit. It&#8217;s used to limit the CPU, memory and other resources each group can use. If a process in the group exceeds its memory limit, the kernel will target the processes inside that group for <code>SIGKILL</code>.</p><p>So a natural solution is to <a href="https://github.com/alexeygrigorev/rustkyll/blob/main/scripts/cargo-safe">run unsafe operations in an isolated cgroup</a>, so the OOM killer won&#8217;t touch anything outside.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kVz3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kVz3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 424w, https://substackcdn.com/image/fetch/$s_!kVz3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 848w, https://substackcdn.com/image/fetch/$s_!kVz3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 1272w, https://substackcdn.com/image/fetch/$s_!kVz3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kVz3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png" width="920" height="610" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:610,&quot;width&quot;:920,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Agent B hits its cgroup memory limit and is killed. The tmux server, both shells, and agent A keep running.&quot;,&quot;title&quot;:&quot;Agent B hits its cgroup memory limit and is killed. The tmux server, both shells, and agent A keep running.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Agent B hits its cgroup memory limit and is killed. The tmux server, both shells, and agent A keep running." title="Agent B hits its cgroup memory limit and is killed. The tmux server, both shells, and agent A keep running." srcset="https://substackcdn.com/image/fetch/$s_!kVz3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 424w, https://substackcdn.com/image/fetch/$s_!kVz3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 848w, https://substackcdn.com/image/fetch/$s_!kVz3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 1272w, https://substackcdn.com/image/fetch/$s_!kVz3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc563d87f-8e8d-4caa-943a-4cb3ea43f182_920x610.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Agent B is killed at its cgroup memory limit, while the tmux server and workload A keep running. Machine-wide OOM can still affect other groups.</figcaption></figure></div><p>However, you can&#8217;t know in advance which process will cause the OOM collapse. It kept happening to me over and over again. When I run 20-30 sessions in parallel, it&#8217;s very frustrating. </p><p>Eventually, I decided to run each tmux session in its own cgroup. Since I was already using tmuxctl as a wrapper around tmux, <a href="https://github.com/alexeygrigorev/tmuxctl/blob/6a20db9ac1b42b5c01fbce1e27fadcbda4fe0605/tmuxctl/robust.py#L326-L347">I added support for cgroups there</a>.</p><p>It worked well, but unfortunately it didn&#8217;t solve the main problem - the tmux server was still the single point of failure. Even with each session running in a cgroup, the server would die occasionally, bringing down all the sessions along with it.</p><p>After consulting Fable, we did the next logical thing: for each new tmux session start a separate server. So if one server dies, the rest keep running.</p><p>All that logic went into tmuxctl, which by that time became a Frankenstein monster, not just a simple wrapper around tmux CLI.</p><p>I got really tired of patching it up, and it felt like I was fighting tmux instead of using it. So I opened ChatGPT and asked &#8220;how difficult is it to write my own terminal multiplexer?&#8221;. It said it would be a few weeks of work. Then I turned on Pro mode and asked it to implement it. It thought for 10 minutes and gave me a first version written in Rust (which kind of worked).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!VcKP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!VcKP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 424w, https://substackcdn.com/image/fetch/$s_!VcKP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 848w, https://substackcdn.com/image/fetch/$s_!VcKP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 1272w, https://substackcdn.com/image/fetch/$s_!VcKP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!VcKP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png" width="1456" height="1099" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1099,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;My dictated ChatGPT prompt asking for a Linux terminal multiplexer in Rust that keeps other sessions alive when one runs out of memory&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="My dictated ChatGPT prompt asking for a Linux terminal multiplexer in Rust that keeps other sessions alive when one runs out of memory" title="My dictated ChatGPT prompt asking for a Linux terminal multiplexer in Rust that keeps other sessions alive when one runs out of memory" srcset="https://substackcdn.com/image/fetch/$s_!VcKP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 424w, https://substackcdn.com/image/fetch/$s_!VcKP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 848w, https://substackcdn.com/image/fetch/$s_!VcKP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 1272w, https://substackcdn.com/image/fetch/$s_!VcKP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2f018c-ba00-40cb-918a-191b9106ef6c_2064x1558.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">My brain dump, captured via dictation, that became the first version of aplexer.</figcaption></figure></div><p>I decided to call it &#8220;aplexer&#8221; which stands for &#8220;Agent Multiplexer&#8221;. &#8220;Amux&#8221; was already taken - I counted 3 products with this name, and none of them were doing what I needed. It&#8217;s very difficult to find a name that&#8217;s available on PyPI these days!</p><h2><strong>Aplexer&#8217;s initial requirements</strong></h2><p>What I needed from it:</p><ul><li><p>Simple commands to create sessions, list sessions, attach and detach</p></li><li><p>No server - no single point of failure.</p></li><li><p>Dealing with OOM errors without having to worry about cgroups</p></li></ul><p>Plus I wanted to have the same features that tmux had:</p><ul><li><p>Working after ssh disconnects</p></li><li><p>Attaching and detaching sessions</p></li><li><p>Sending input to each session</p></li><li><p>Seeing the history (scrolling up to see the previous messages from the agents)</p></li></ul><p>The main focus was on running agents, so I also wanted:</p><ul><li><p>Seeing which folder (&#8221;workspace&#8221;) each session is running in</p></li><li><p>Seeing what&#8217;s running inside each session - which agent</p></li><li><p>Letting agents send messages to each other natively</p></li></ul><p>I always follow the <a href="https://aishippingblog.com/p/ai-native-development-specifications">spec-driven development approach</a>, so I discussed the requirements with ChatGPT, got the <a href="https://github.com/PocketShell-io/aplexer/blob/main/spec.md">first specification</a> out, and started developing it.</p><h2><strong>Eat your own dog food</strong></h2><p>I don&#8217;t know why this approach is called that. But the idea is to start using the tool you develop as soon as possible.</p><p>This approach works extremely well and forces you to find and fix all the inconvenient points.</p><p>That&#8217;s why the focus for v0 of aplexer was to let me start using it instead of tmux, and then iterate and polish all the rough edges.</p><p>The main acceptance criteria for v0 were:</p><ul><li><p>I can use it for running agents</p></li><li><p>Agents run in detachable sessions that don&#8217;t stop after SSH disconnect</p></li><li><p>OOM in one session doesn&#8217;t affect any other session</p></li></ul><p>The first test that I implemented was causing OOM in one session and making sure the others weren&#8217;t affected. The <a href="https://github.com/PocketShell-io/aplexer/blob/main/tests/oom_isolation.rs">OOM isolation test</a> starts three sessions with 128 MB memory limits, exhausts the memory in session B, and checks that sessions A and C still respond to commands.</p><p>It took one week to have a stable version that works as v0, and then another couple of weeks to polish and add the features that I needed. (ChatGPT was right!)</p><p>I like short aliases, so I use <code>a</code> for aplexer in my terminal.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!GhdB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GhdB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 424w, https://substackcdn.com/image/fetch/$s_!GhdB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 848w, https://substackcdn.com/image/fetch/$s_!GhdB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 1272w, https://substackcdn.com/image/fetch/$s_!GhdB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GhdB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png" width="1456" height="1223" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1223,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Aplexer lists workspaces with agent sessions, including Claude, Codex, Grok, and OpenCode, with running and idle states&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Aplexer lists workspaces with agent sessions, including Claude, Codex, Grok, and OpenCode, with running and idle states" title="Aplexer lists workspaces with agent sessions, including Claude, Codex, Grok, and OpenCode, with running and idle states" srcset="https://substackcdn.com/image/fetch/$s_!GhdB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 424w, https://substackcdn.com/image/fetch/$s_!GhdB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 848w, https://substackcdn.com/image/fetch/$s_!GhdB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 1272w, https://substackcdn.com/image/fetch/$s_!GhdB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdc834b-4a06-4446-b851-2d84c925df77_1684x1414.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">My sessions grouped by workspace, with the agent and its current state.</figcaption></figure></div><p>By now it has fully replaced tmux in my workflow, and I haven&#8217;t had a problem with OOM wiping out all my sessions since then.</p><h2><strong>Aplexer&#8217;s architecture</strong></h2><p>Each session in aplexer is independent of the others, and there&#8217;s no shared server. Instead, each session has a PTY worker that&#8217;s assigned only to that session.</p><p>When a new session is created, the aplexer client creates a worker, and the worker manages the PTY. Optionally, the process can be launched in a cgroup too.</p><p>Aplexer doesn&#8217;t need a central process for keeping track of all the sessions: it stores them in the filesystem.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ECnz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ECnz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 424w, https://substackcdn.com/image/fetch/$s_!ECnz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 848w, https://substackcdn.com/image/fetch/$s_!ECnz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 1272w, https://substackcdn.com/image/fetch/$s_!ECnz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ECnz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png" width="1240" height="620" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:620,&quot;width&quot;:1240,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The aplexer client routes to three independent session boundaries, each containing a worker and workload, with session B shown in red to illustrate an independent failure&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The aplexer client routes to three independent session boundaries, each containing a worker and workload, with session B shown in red to illustrate an independent failure" title="The aplexer client routes to three independent session boundaries, each containing a worker and workload, with session B shown in red to illustrate an independent failure" srcset="https://substackcdn.com/image/fetch/$s_!ECnz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 424w, https://substackcdn.com/image/fetch/$s_!ECnz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 848w, https://substackcdn.com/image/fetch/$s_!ECnz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 1272w, https://substackcdn.com/image/fetch/$s_!ECnz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18b78e1-9ca9-4349-b978-a1c048f3f857_1240x620.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">aplexer gives each session its own worker, PTY, lifecycle, and optional workload cgroup.</figcaption></figure></div><p>I don&#8217;t need panes, windows and other things, so for me one session is one PTY.</p><p>But I still want to organize the sessions. I group sessions by workspace - the folder where the agents are running.</p><p>When I run <code>a</code>, I get a list of all the workspaces and a list of sessions in each.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!n0ky!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!n0ky!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 424w, https://substackcdn.com/image/fetch/$s_!n0ky!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 848w, https://substackcdn.com/image/fetch/$s_!n0ky!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 1272w, https://substackcdn.com/image/fetch/$s_!n0ky!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!n0ky!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png" width="1456" height="651" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:651,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Workspace 7, dapier, contains six sessions including designer at index 2; community-base and ai-shipping-labs appear below it&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Workspace 7, dapier, contains six sessions including designer at index 2; community-base and ai-shipping-labs appear below it" title="Workspace 7, dapier, contains six sessions including designer at index 2; community-base and ai-shipping-labs appear below it" srcset="https://substackcdn.com/image/fetch/$s_!n0ky!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 424w, https://substackcdn.com/image/fetch/$s_!n0ky!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 848w, https://substackcdn.com/image/fetch/$s_!n0ky!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 1272w, https://substackcdn.com/image/fetch/$s_!n0ky!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb960c1c-bc0d-496b-abf5-b8b531ef4b82_1710x764.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A workspace can contain several sessions, each with its own tag and agent.</figcaption></figure></div><p>Now if I want to attach to any of the sessions, I can type</p><pre><code>a 7 2</code></pre><p>This will attach to the workspace number 7 (<code>~/git/dapier</code>), session 2 (<code>designer</code>). I can refer to them by names too, but that&#8217;s too much to type.</p><h2><strong>Making it agent-aware</strong></h2><p>I also wanted aplexer to keep track of what&#8217;s running inside each session. It shows which agent is there, whether it is running or idle, and when it was last active.</p><p>If I want to start a Codex session with the tag &#8220;code-refactor&#8221; in the current directory, I simply type:</p><pre><code>a - codex code-refactor</code></pre><p>Of course, that&#8217;s all documented in the <a href="https://github.com/PocketShell-io/aplexer/">README</a>.</p><h2><strong>Communication</strong></h2><p>In tmux my agents were already talking to each other, so I wanted to have the same functionality in aplexer too.</p><p>This is what it looks like:</p><pre><code>a send code-refactor "Please review the current diff and report any bugs." --enter</code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3uir!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3uir!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 424w, https://substackcdn.com/image/fetch/$s_!3uir!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 848w, https://substackcdn.com/image/fetch/$s_!3uir!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!3uir!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3uir!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg" width="1280" height="880" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:880,&quot;width&quot;:1280,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Codex receives a cross-workspace coordination request through aplexer&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Codex receives a cross-workspace coordination request through aplexer" title="Codex receives a cross-workspace coordination request through aplexer" srcset="https://substackcdn.com/image/fetch/$s_!3uir!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 424w, https://substackcdn.com/image/fetch/$s_!3uir!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 848w, https://substackcdn.com/image/fetch/$s_!3uir!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!3uir!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff474c197-7d62-44c5-95c2-4d6aa59f8f95_1280x880.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The agents coordinate a review and integration across workspaces</figcaption></figure></div><p>And check what&#8217;s on the screen:</p><pre><code>a capture code-refactor --screen</code></pre><p>These commands send text as the prompt. But there&#8217;s also a message bus that the agents can use for asynchronous communication.</p><p>Within a session it looks like this:</p><pre><code>a message send --to code-refactor "Implementation is ready. Please review the diff."
a message send --all "The tests pass. I&#8217;m ready for review."</code></pre><p>The recipient can read its inbox, acknowledge it, and then reply when it&#8217;s ready:</p><pre><code>a message inbox
a message ack &lt;message-id&gt;
a message reply &lt;message-id&gt; &lt;message&gt;</code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1Ky6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1Ky6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1Ky6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1Ky6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1Ky6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1Ky6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg" width="1080" height="386" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:386,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Agent sends an integration push notice to its peer using aplexer message reply&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Agent sends an integration push notice to its peer using aplexer message reply" title="Agent sends an integration push notice to its peer using aplexer message reply" srcset="https://substackcdn.com/image/fetch/$s_!1Ky6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1Ky6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1Ky6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1Ky6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa688b766-24b7-43a3-8234-737c98a07376_1080x386.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The agent replies after coordinating an isolated worktree integration</figcaption></figure></div><h2><strong>Aplexer and PocketShell</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!x_Gi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!x_Gi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 424w, https://substackcdn.com/image/fetch/$s_!x_Gi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 848w, https://substackcdn.com/image/fetch/$s_!x_Gi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 1272w, https://substackcdn.com/image/fetch/$s_!x_Gi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!x_Gi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png" width="1456" height="978" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:978,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:205112,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/218398320?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!x_Gi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 424w, https://substackcdn.com/image/fetch/$s_!x_Gi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 848w, https://substackcdn.com/image/fetch/$s_!x_Gi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 1272w, https://substackcdn.com/image/fetch/$s_!x_Gi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83e0ecf5-010f-4928-b6aa-13c2a9ee7750_1878x1262.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>tmux is excellent at what it does, but it stopped working for me for running agents on my dev box. So I created aplexer. Aplexer is an agent multiplexer written in Rust that doesn&#8217;t require a single shared server. It groups my sessions in workspaces, and I can tag them and see their state.</p><p>If you run agents on a Linux dev box and want to try it, the <a href="https://github.com/PocketShell-io/aplexer/blob/main/README.md">README</a> has installation instructions and the full command reference.</p><p>Now I rarely use aplexer directly - I mostly use it via <a href="https://pocketshell.io/">PocketShell</a> to manage my sessions. It&#8217;s an app that lets me connect to my dev box from my Android phone, from my laptop or from the web. It lists all the sessions there and lets me switch easily between them.</p><p>I started working on it in May and I like how it&#8217;s coming along. I&#8217;ll polish it a bit more and soon will write another blog post on how I use it for my work with agents, so you can try it too.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mMXv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mMXv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 424w, https://substackcdn.com/image/fetch/$s_!mMXv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 848w, https://substackcdn.com/image/fetch/$s_!mMXv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 1272w, https://substackcdn.com/image/fetch/$s_!mMXv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mMXv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png" width="1456" height="918" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3242a0ca-c318-4432-a497-000693a66319_2596x1636.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:918,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;PocketShell&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="PocketShell" title="PocketShell" srcset="https://substackcdn.com/image/fetch/$s_!mMXv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 424w, https://substackcdn.com/image/fetch/$s_!mMXv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 848w, https://substackcdn.com/image/fetch/$s_!mMXv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 1272w, https://substackcdn.com/image/fetch/$s_!mMXv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3242a0ca-c318-4432-a497-000693a66319_2596x1636.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Managing agent sessions with PocketShell. Each agent runs in aplexer, and the list of sessions on the side bar is coming from aplexer too.</figcaption></figure></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><p></p>]]></content:encoded></item><item><title><![CDATA[Last Call: AI Engineering Buildcamp Cohort 4 ]]></title><description><![CDATA[The enrollment deadline is next week on Monday]]></description><link>https://aishippingblog.com/p/last-call-ai-engineering-buildcamp</link><guid isPermaLink="false">https://aishippingblog.com/p/last-call-ai-engineering-buildcamp</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Wed, 23 Sep 2026 11:26:48 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!bRe3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>We started the new cohort of AI Engineering Buildcamp this Monday, but you can still join. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bRe3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bRe3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 424w, https://substackcdn.com/image/fetch/$s_!bRe3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 848w, https://substackcdn.com/image/fetch/$s_!bRe3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!bRe3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!bRe3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg" width="1280" height="777" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:777,&quot;width&quot;:1280,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:152315,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/217046991?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bRe3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 424w, https://substackcdn.com/image/fetch/$s_!bRe3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 848w, https://substackcdn.com/image/fetch/$s_!bRe3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!bRe3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebf638a-b71b-4ae4-bd40-34a68d083e27_1280x777.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We had our first live session, where I went through the course structure, talked about projects, and answered questions from the participants.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>In this post, I want to focus on what we discussed during the launch session.</p><p>In particular, I&#8217;ll cover:</p><ul><li><p>Course curriculum</p></li><li><p>How the capstone project works</p></li><li><p>How to come up with a project idea</p></li><li><p>How I&#8217;ll help with the project scope</p></li><li><p>How we&#8217;ll use coding agents</p></li><li><p>Common questions</p></li></ul><p>Registration is still open until next Monday, September 28, and there&#8217;s one buffer week to catch up if you&#8217;re joining late.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://maven.com/alexey-grigorev/from-rag-to-agents&quot;,&quot;text&quot;:&quot;Join AI Engineering Buildcamp&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://maven.com/alexey-grigorev/from-rag-to-agents"><span>Join AI Engineering Buildcamp</span></a></p><h2>Course Overview</h2><p>I already wrote about the course in detail in my previous post. I covered the curriculum, homework projects, projects from the previous cohorts, workload, and other details in that post.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;1ed284d7-bf95-4b00-8ea4-aea1ce212666&quot;,&quot;caption&quot;:&quot;Next Monday (September 21), I start the next cohort AI Engineering Buildcamp. It&#8217;s going to be the forth cohort of the course.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;AI Engineering Buildcamp Starts Next Week&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-09-14T17:21:42.752Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!CsUw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/ai-engineering-buildcamp-starts-next&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:215695735,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:14,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p>Let me do a quick recap: </p><ul><li><p>Week 1 is about the foundations. We prepare the environment, learn how to work with LLMs, and then focus on RAG and search.</p></li><li><p>Week 2 is a buffer week. I want to make sure everyone has enough time to prepare the environment. If you&#8217;re joining a bit late, you will use this time to catch up. </p></li><li><p>Week 3: agents. We go from regular RAG to agentic RAG, learn about tool calling, and implement the agent loop ourselves, and then use an agentic framework.</p></li><li><p>Week 4: testing. We add tests to our agent and make sure it behaves as expected.</p></li><li><p>Week 5: monitoring. We log all the interactions between the users and the agent.</p></li><li><p>Week 6: evaluation. We create evaluation datasets, use LLM judges, and systematically evaluate the system.</p></li><li><p>Weeks 7 and 8: focused project work.</p></li><li><p>Week 9: peer review and demo day.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!TqWv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TqWv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 424w, https://substackcdn.com/image/fetch/$s_!TqWv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 848w, https://substackcdn.com/image/fetch/$s_!TqWv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 1272w, https://substackcdn.com/image/fetch/$s_!TqWv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!TqWv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png" width="1672" height="721" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:721,&quot;width&quot;:1672,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1211922,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/217046991?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c4297ed-e8b8-46eb-80ee-4dce21efc318_1672x941.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!TqWv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 424w, https://substackcdn.com/image/fetch/$s_!TqWv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 848w, https://substackcdn.com/image/fetch/$s_!TqWv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 1272w, https://substackcdn.com/image/fetch/$s_!TqWv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc45ab45e-b901-4d49-98cf-b7c4a2448954_1672x721.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The course takes 9 weeks: 6 week of core work and 3 weeks of project work.</p><h2>The Capstone Project</h2><p>The project is the most important part of the course. I want everyone to finish with a project that you can add to your project portfolio, use at work or show it to potential clients. </p><p>For the capstone, you need to come up with a project idea and implement it end-to-end. It&#8217;ll be a RAG application, an agent, or something in between. You will cover it with tests, add monitoring, evaluate it and deploy to the cloud. </p><p>In our curriculum, weeks 7 and 8 are specifically dedicated to project work, but I want you to start working on your projects from week 1. This way, by the end of week 6 you have something working, and you can spend weeks 7 and 8 polishing it and finishing the missing parts.</p><p>I you already have an idea - that&#8217;s great, I&#8217;ll help you refine it.</p><p>But if you don&#8217;t, I&#8217;ll give you a project ideation process that you can follow to select a meaningful idea to work on. This approach uses design thinking to discover problems you can solve with AI.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LLyp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LLyp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 424w, https://substackcdn.com/image/fetch/$s_!LLyp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 848w, https://substackcdn.com/image/fetch/$s_!LLyp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 1272w, https://substackcdn.com/image/fetch/$s_!LLyp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LLyp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png" width="1456" height="786" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/deb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:786,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1423820,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/217046991?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!LLyp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 424w, https://substackcdn.com/image/fetch/$s_!LLyp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 848w, https://substackcdn.com/image/fetch/$s_!LLyp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 1272w, https://substackcdn.com/image/fetch/$s_!LLyp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb5b3b8-a6e1-425e-b62f-53d4b6c819eb_1707x921.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Coding Agents</h2><p>I also want to encourage you to use coding agents during the course.</p><p>During the lessons, we write code together. When I cover something in the coding sessions, I want you to understand how things work. For example we first implement the tool-calling loop ourselves before using any agent framework.</p><p>But once you understand the core concepts and the code from the lessons, you can give it to a coding agent:</p><blockquote><p>Based on the code we created in the lesson, I want to create &lt;YOUR PROJECT IDEA&gt;. Please use this implementation as a reference and adapt it to my project.</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8aOo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8aOo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 424w, https://substackcdn.com/image/fetch/$s_!8aOo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 848w, https://substackcdn.com/image/fetch/$s_!8aOo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 1272w, https://substackcdn.com/image/fetch/$s_!8aOo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8aOo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png" width="1952" height="1243" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1243,&quot;width&quot;:1952,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:244253,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/217046991?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a84025e-5c83-480e-8f07-9d3dc8618403_1952x1318.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8aOo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 424w, https://substackcdn.com/image/fetch/$s_!8aOo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 848w, https://substackcdn.com/image/fetch/$s_!8aOo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 1272w, https://substackcdn.com/image/fetch/$s_!8aOo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46add96-5d0b-4421-a38a-4ce62d66ee68_1952x1243.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>It makes the project development much faster. Instead of starting from zero, the coding agent already has an example that it will use for your own project. You already understand the original implementation, so understanding the code for your project will be much easier. </p><p>This is how I recommend using coding agents: not as a replacement for learning, but as a tool that can get us from an idea to a working implementation faster.</p><p></p><p>See what people from Cohort 3 implemented:</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;bc9cdb3c-7601-40cc-ae87-800f1461e48b&quot;,&quot;caption&quot;:&quot;The third cohort of my AI Engineering course wrapped this summer, and in this post I want to share what we built.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;13 AI Projects from AI Engineering Buildcamp Cohort 3&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-09-04T18:21:30.585Z&quot;,&quot;cover_image&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/04db2d9f-7355-49f4-bce5-58a862c6364e_1456x1048.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/13-ai-projects-from-ai-engineering&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:214196526,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:15,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p></p><h2>Questions From the Launch Session</h2><p>We had a few questions during the launch stream and I think they are quite interesting, so I wanted to cover them here too.</p><h3>Is RAG Dead?</h3><p>No, it&#8217;s not dead. On the contrary, according to my analysis for <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide">AI Engineering Field Guide</a>, the majority of jobs require it. That&#8217;s why we start the course from RAG and cover it in detail.</p><p>The reasons we need it:</p><ul><li><p>The model doesn&#8217;t have access to proprietary data. If your company has an internal knowledge base, the model has never seen it. With RAG, you can search this knowledge base, retrieve the relevant information, and include it in the context.</p></li><li><p>These knowledge bases are usually not that large. With models capable of processing 128 million tokens and more, we can fit the entire dataset into the context window. The RAG is still needed though: if you send the entire dataset to the model, it will have hard time figuring out what&#8217;s useful. But if you retrieve only the relevant pieces, you send less data. This makes it faster and cheaper, and it also gives the model more focused context.</p><p></p></li></ul><h3>Are We Going to Cover Fine-Tuning?</h3><p>No. Fine-tuning is slow, expensive, and often not needed.</p><p>If we have our internal documentation, we can prepare a training dataset in a special format, take a model with open weights, rent a GPU, and spend a day or two fine-tuning it, so it actually learns from our data. Next time we use it, we don&#8217;t need to inject the context with RAG: the model already has our knowledge base internally in its weights. </p><p>There are a few problems with fine-tuning though: </p><ul><li><p>You need to have a lot of data and prepare it in a special way</p></li><li><p>GPUs aren&#8217;t cheap and not always available</p></li><li><p>It requires time to tune the model </p></li><li><p>It&#8217;s difficult to enforce answering with the learned information instead of the general knowledge it had </p></li><li><p>The worst part: you update the dataset by including one new page and you need to fine-tune again</p></li></ul><p>Now compare it with RAG: we just update the documents in the search index and that&#8217;s it. These problems make fine-tuning impractical for the majority of use-cases. And my job analysis shows that most companies don&#8217;t actually require it.</p><p>That&#8217;s why we don&#8217;t cover it in the course. </p><h3>Do We Need Multiple Agents?</h3><p>In most cases you don&#8217;t. In many cases you don&#8217;t even need an agent, just RAG is enough.</p><p>I will help you understand when you need what, and scope your project in a way that it&#8217;s focused, manageable - and also possible to implement during the course.</p><p>We do have a few places where I cover multi-agent setup. For example, the Deep Research Agent uses multiple agents for different parts of the research process, or the Coding Agent that uses the plan-then-implement pattern. </p><p>But it&#8217;s not the focus of the course. It&#8217;s more important to first understand how to create one agent well, test it, add monitoring and evaluate it to make sure it works properly. And then, only if the problem actually requires more complexity, add it. (In many cases it doesn&#8217;t.)</p><h2>Can I Use Another Technology Stack?</h2><p>In the course we use OpenAI, Python, Pydantic AI and other technologies to implement agents.</p><p>But you don&#8217;t have to use these specific technologies in your capstone. If you want to build your project in TypeScript - that&#8217;s fine. You want to learn LangChain and use it instead of Pydantic AI - totally fine too. Anthropic, Gemini, or another provider work well instead of Open AI.</p><p>If you don&#8217;t have a strong reason to choose something else, I recommend using the same stack as in the course. It will be easier for me to help you.</p><p>But it&#8217;s not a requirement.</p><h2>Do I Need an Expensive Coding Agent Subscription?</h2><p>No. In fact, even a free subscription will be sufficient. There are a few places in the course where I use a coding agent to generate code - for example, for creating a streamlit tool for evals. But for that I use free Antigravity, so everyone can reproduce it. Some students prefer not to use any coding agent at all and it works too. </p><p>I&#8217;d still recommend investing at least $10 in getting an OpenCode subscription or $20 for Claude Code Pro or ChatGPT Plus, so you don&#8217;t need to worry about hitting the usage limits with the free subscriptions.</p><p>Also check my recent post about free and affordable coding agents:</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;86e0abeb-4c28-467b-a141-b8a9ee468f24&quot;,&quot;caption&quot;:&quot;I recently posted on X that we have access to so many tokens that I&#8217;m running out of ideas what to build.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Free and Affordable AI-Native Development&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-09-11T23:02:46.666Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!zfRh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/free-and-affordable-ai-native-development&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:215293864,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:16,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p></p><h2>What If I Can&#8217;t Attend the Live Sessions?</h2><p>The course is designed to be asynchronous. </p><p>Each week we have a live office hour session. On these sessions we discuss questions, project ideas, architecture decisions, and other things that come up during the week. </p><p>If you can&#8217;t attend, it&#8217;s recorded and shared shortly after. You can also ask your questions in advance and we will cover them during the session. </p><p>Most of the content is already recorded as videos (with text articles along them), which you can watch whenever it&#8217;s convenient for you.</p><h2>Can It Help me Transition into AI Engineering? </h2><p>Yes.</p><p>In previous cohorts we had participants from different backgrounds: software engineers, DevOps engineers, data scientists, researchers, and others. For many people the course helped them start doing AI engineering work in their existing company. Others used what they learned when looking for a new role. Some use these skills for freelancing. </p><p>If finding an AI engineering job after this course is one of your goals, I recommend thinking about the capstone as a portfolio project from the beginning. Build something you can include in your CV.</p><p>If you need help with that, I&#8217;ll be very happy to support.</p><p></p><p>I always enjoy our launch streams and this one was not an exception. It&#8217;s going to be a lot of fun! </p><h1>Join Cohort 4</h1><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!maJv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!maJv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 424w, https://substackcdn.com/image/fetch/$s_!maJv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 848w, https://substackcdn.com/image/fetch/$s_!maJv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 1272w, https://substackcdn.com/image/fetch/$s_!maJv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!maJv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png" width="1456" height="590" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:590,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:859501,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/217046991?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!maJv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 424w, https://substackcdn.com/image/fetch/$s_!maJv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 848w, https://substackcdn.com/image/fetch/$s_!maJv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 1272w, https://substackcdn.com/image/fetch/$s_!maJv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb31eb9d6-bd76-4aaa-9d29-ab844b168f42_2762x1120.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Registration closes next Monday, September 28. Use code &#8220;SUBSTACK&#8221; to get 20% off. </p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://maven.com/alexey-grigorev/from-rag-to-agents&quot;,&quot;text&quot;:&quot;Join Cohort 4&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://maven.com/alexey-grigorev/from-rag-to-agents"><span>Join Cohort 4</span></a></p><p>See you on Monday! </p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Jev: A Decision Model That Does Not Generate Text]]></title><description><![CDATA[If you&#8217;ve been on X in the last few days, chances are that you&#8217;ve seen posts about Jev from TypeSave AI.]]></description><link>https://aishippingblog.com/p/jev-a-decision-model-that-does-not</link><guid isPermaLink="false">https://aishippingblog.com/p/jev-a-decision-model-that-does-not</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 18 Sep 2026 12:36:51 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!U0Eg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>If you&#8217;ve been on X in the last few days, chances are that you&#8217;ve seen posts about Jev from TypeSave AI. Every time I open my timeline, I see people posting about it.</p><div class="twitter-embed" data-attrs="{&quot;url&quot;:&quot;https://x.com/CompleteSkeptic/status/2099925682726002904&quot;,&quot;full_text&quot;:&quot;After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?\n\nI&#8217;ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev\n\n&#8226; 20-200x faster\n&#8226; 40-400x &#8230;&quot;,&quot;username&quot;:&quot;CompleteSkeptic&quot;,&quot;name&quot;:&quot;Diogo Almeida&quot;,&quot;profile_image_url&quot;:&quot;https://pbs.substack.com/profile_images/1650708125685800960/7k6r0UZg_normal.jpg&quot;,&quot;date&quot;:&quot;2026-09-15T18:17:52.000Z&quot;,&quot;photos&quot;:[{&quot;img_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!z-75!,w_1028,c_limit,f_auto,q_auto:best,fl_progressive:steep/l_play_button_usfui2,w_88,e_colorize:0/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F__ss-rehost__tw-video-preview-13_2099925575637057536.jpg&quot;,&quot;link_url&quot;:&quot;https://t.co/JSybNG2BKJ&quot;}],&quot;quoted_tweet&quot;:{},&quot;reply_count&quot;:3509,&quot;retweet_count&quot;:7023,&quot;like_count&quot;:66618,&quot;impression_count&quot;:31687933,&quot;expanded_url&quot;:null,&quot;video_url&quot;:&quot;https://video.twimg.com/amplify_video/2099925575637057536/vid/avc1/1280x720/ZoKJ_BS5SNaqG45k.mp4&quot;,&quot;video_preview_media_key&quot;:&quot;13_2099925575637057536&quot;,&quot;belowTheFold&quot;:false}" data-component-name="Twitter2ToDOM"></div><p>The model <a href="https://openrouter.ai/typesafe/jev-1.13">was recently released on OpenRouter</a>, so I decided to give it a try.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h2><strong>Jev and TypeSafe AI</strong></h2><p>I saw the announcement of Jev on X a few days. It&#8217;s interesting because it doesn&#8217;t generate text. When I first read about it, I thought, &#8220;Huh? No text?&#8221; After seeing a few more posts, I realized that it could be useful for quick decision-making.</p><p>Jev is a very fast one-shot text classifier. You give it text, and it makes a decision. You don&#8217;t need to train or fine-tune it. </p><p>That makes it useful as a router:</p><ul><li><p>Should I send this request to a heavy model or a light one?</p></li><li><p>Which agent should I select?</p></li><li><p>Should I handle this request automatically or send it to a human?</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6XLF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6XLF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 424w, https://substackcdn.com/image/fetch/$s_!6XLF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 848w, https://substackcdn.com/image/fetch/$s_!6XLF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 1272w, https://substackcdn.com/image/fetch/$s_!6XLF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6XLF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png" width="1456" height="847" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:847,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:353079,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/216290283?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!6XLF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 424w, https://substackcdn.com/image/fetch/$s_!6XLF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 848w, https://substackcdn.com/image/fetch/$s_!6XLF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 1272w, https://substackcdn.com/image/fetch/$s_!6XLF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e455134-49ce-47e5-a98d-11f7fb643661_2428x1412.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev">TypeSafe calls Jev a &#8220;System One Model&#8221;</a>. The name comes from &#8220;Thinking, Fast and Slow&#8221;. Yes, that&#8217;s this thick, long and boring book that I tried to read but gave up (I managed to finish the audio version though).</p><ul><li><p>System 1 is the fast, intuitive judgment your brain makes without effort.</p></li><li><p>System 2 is slow, deliberate reasoning.</p></li></ul><p>So, Jev is System 1, and LLMs are System 2. </p><p>Jev isn&#8217;t a replacement for LLMs. It&#8217;s a layer that sits in front of them.</p><p>You can also use it anywhere you previously relied on a cheap, fast LLM with structured output to make a decision.</p><p>For example, it can act as a guardrail. In <a href="https://aishippinglabs.com/workshops/agent-with-guardrails">my AI Shipping Labs workshop about guardrails</a>, I used <code>gpt-4o-mini</code> as a quick classifier to decide whether to continue generation. Jev can replace that binary LLM classifier.</p><h2><strong>The API</strong></h2><p>You send Jev one request containing two things:</p><ul><li><p>a state (unstructured text or a JSON object describing the situation)</p></li><li><p>a set of typed questions</p></li></ul><p>Jev answers all questions in parallel and returns structured JSON with probability distributions.</p><p>There are three question types (<a href="https://docs.typesafe.ai/primitives">TypeSafe calls them &#8220;primitives&#8221;</a>):</p><ul><li><p>Noul: a yes/no question. Returns a single probability from 0 to 1. &#8220;Does this message request a refund?&#8221; might return <code>0.93</code>. Think binary classifier.</p></li><li><p>Choice: pick one option from a set of options. Returns the selected option, a probability distribution across all options, and a confidence score. Up to 255 options per question. Think multiclass classification.</p></li><li><p>Score: rate the state on an ordered scale. Returns a weighted score, the distribution across levels, and confidence. Between 2 and 10 levels. Think ordinal classification.</p></li></ul><p>Here&#8217;s a request that includes all three question types for one state:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;json&quot;,&quot;nodeId&quot;:&quot;8c73393b-d41a-4317-b39e-c81ea174b2b4&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-json">{
  "state": "Help! My payouts have been failing for 3 days.",
  "model": "jev-latest",
  "questions": {
    "is_urgent": {
      "type": "noul",
      "instructions": "Does this convey urgency?"
    },
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Payments, invoicing, refunds",
        "technical": "Bugs, outages, integrations",
        "sales": "Pricing, upgrades, new accounts"
      }
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["Calm", "Frustrated", "Very angry"]
    }
  }
}</code></pre></div><p>The <code>state</code> is the text that we analyze (it could also be a JSON object). Each entry in <code>questions</code> is one question with whatever key we choose. Inside, <code>instructions</code> is the question that the model needs to answer, and <code>criteria</code> describes the possible answers:</p><ul><li><p>a map of options for Choice</p></li><li><p>an ordered list of levels for Score</p></li><li><p>an optional explanation of yes and no for Noul.</p></li></ul><p>Here&#8217;s what a response looks like:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;json&quot;,&quot;nodeId&quot;:&quot;592b13c5-75df-4d42-b403-2b38ebf5c556&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-json">{
  "model": "jev-1.13.0",
  "answers": {
    "is_urgent": { "type": "noul", "noul": 0.93 },
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.84, "technical": 0.15, "sales": 0.0, "other": 0.01 },
      "confidence": 0.6
    },
    "frustration": {
      "type": "score",
      "score": 1.3,
      "probabilities": { "0": 0.0, "1": 0.7, "2": 0.3 },
      "confidence": 0.54
    }
  }
}</code></pre></div><p>The response contains one typed answer for each question. The <code>noul</code> value is the probability that the message is urgent, <code>choice</code> is the selected department, and <code>score</code> is the weighted frustration level.</p><p>We can now use these answers in our code:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:&quot;253a1faa-efb6-45a0-84f9-66b806874379&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">urgency = answers["is_urgent"]["noul"]
department = answers["department"]["choice"]
department_confidence = answers["department"]["confidence"]

if urgency &gt; 0.8 and department == "billing":
    escalate_to_billing()
elif department_confidence &lt; 0.5:
    route_to_human()</code></pre></div><h2><strong>Benchmarks</strong></h2><p><a href="https://evals.typesafe.ai/">TypeSafe built a benchmark suite</a> of four business workflows to show how it works:</p><ul><li><p>security incident triage</p></li><li><p>agent trace observability</p></li><li><p>invoice processing</p></li><li><p>customer service</p></li></ul><p>Each model ran through the same workflow harness, and the results were compared against a reference policy derived from the average judgments of GPT-6 Astra and Claude Fable 5.1.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!T64v!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!T64v!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 424w, https://substackcdn.com/image/fetch/$s_!T64v!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 848w, https://substackcdn.com/image/fetch/$s_!T64v!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 1272w, https://substackcdn.com/image/fetch/$s_!T64v!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!T64v!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png" width="1456" height="412" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/43717639-c157-47d8-b383-de02faf87860_2084x590.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:412,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:126915,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/216290283?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!T64v!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 424w, https://substackcdn.com/image/fetch/$s_!T64v!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 848w, https://substackcdn.com/image/fetch/$s_!T64v!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 1272w, https://substackcdn.com/image/fetch/$s_!T64v!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43717639-c157-47d8-b383-de02faf87860_2084x590.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A separate <a href="https://github.com/gemanor/jev-code-review-benchmark">code-review benchmark by an independent developer</a> compared Jev against Gemini Flash and Claude Fable on Python linting rules.</p><ul><li><p>Jev cost 45 times less than Gemini Flash and 274 times less than Claude Fable.</p></li><li><p>Its median response time was 0.75 seconds (versus 3.59 seconds for Flash and 4.31 seconds for Fable).</p></li><li><p>Jev scored 98% correctness versus 100% for both comparators.</p></li></ul><h2><strong>Applications</strong></h2><p>I&#8217;ve seen a lot of posts on X about what people have built with Jev over the last few days.</p><h3><strong>Self-Driving Simulations</strong></h3><ul><li><p>A 2D top-down browser simulation where Jev acts as the driving classifier, receiving road state and making steering decisions. <a href="https://github.com/vinilana/live-jev">Source</a></p></li><li><p>A 3D Three.js driving simulator. Jev receives a compact JSON snapshot of current speed, speed limit, next turn distance, nearby vehicles and pedestrians with relative positions and speeds, traffic signals, and predicted hazards. It returns a driving decision. <a href="https://github.com/standardagents/jevdrive">Source</a></p></li><li><p>An interactive playground with 3D driving simulations showing real AI decisions with visible sensor inputs. <a href="https://github.com/kavehmz/typesafe-playground">Source</a></p></li></ul><h3><strong>Games</strong></h3><p>TypeSafe&#8217;s launch post shows Jev playing Doom at about 10 queries per second, at a cost of roughly $7 per hour.</p><p>People also made it play chess:</p><ul><li><p>Jev versus an OpenAI model. <a href="https://github.com/wondertwins/jev-benchmark">Source</a></p></li><li><p>Two Jev instances playing against each other. <a href="https://github.com/TholeG/typesafe-chess">Source</a></p></li></ul><p>Other games:</p><ul><li><p>An NPC demo tested whether Jev could decide which characters should respond when a player speaks. The game engine sends the transcript and NPC descriptions, and Jev decides which NPC the player is addressing. <a href="https://github.com/wondertwins/jev-benchmark">Source</a></p></li><li><p>Wikiracing, where Jev navigates from one Wikipedia page to another using only links on each page. Jev had to choose among hundreds or thousands of links per page. <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev">Source</a></p></li></ul><h2><strong>Using Jev as an Agent Router for My Work</strong></h2><p>I wanted to give it a try too. Since Jev became available via OpenRouter today, I could skip the waiting list.</p><p>Currently, I&#8217;m automating my work at <a href="https://datatalks.club/">DataTalks.Club</a>, <a href="https://aishippinglabs.com/">AI Shipping Labs</a> and I also have a lot of other operational work.</p><p>Because each piece of work is different and requires different tools, I need a different agent for each task:</p><ul><li><p>If a task is about DataTalks.Club course operations (cohorts, homeworks, projects, leaderboards, certificates) or DataTalks.Club events (workshops or podcast interviews), it should go to the DTC agent.</p></li><li><p>If a task is about AI Shipping Labs (workshops, events, membership and payments, sprints, marketing pages), it should go to the AISL agent.</p></li><li><p>Otherwise, it should go to the admin agent.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!U0Eg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U0Eg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 424w, https://substackcdn.com/image/fetch/$s_!U0Eg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 848w, https://substackcdn.com/image/fetch/$s_!U0Eg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 1272w, https://substackcdn.com/image/fetch/$s_!U0Eg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!U0Eg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png" width="1020" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:1020,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Jev routes one request to one of three specialist agents&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Jev routes one request to one of three specialist agents" title="Jev routes one request to one of three specialist agents" srcset="https://substackcdn.com/image/fetch/$s_!U0Eg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 424w, https://substackcdn.com/image/fetch/$s_!U0Eg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 848w, https://substackcdn.com/image/fetch/$s_!U0Eg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 1272w, https://substackcdn.com/image/fetch/$s_!U0Eg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416fad25-b9ac-40c8-a9ba-1340a6f162db_1020x440.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">I use Jev to route the task to the appropriate specialist agent</figcaption></figure></div><p></p><h3><strong>Routing with structured output</strong></h3><p>Previously, I&#8217;d use a simple model like <code>gpt-4o-mini</code> for routing with a structured output schema:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:&quot;2c694d4a-ad0c-4293-a6d8-7546ca037f3d&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">from enum import Enum
from pydantic import BaseModel, Field

class Destination(str, Enum):
    DATATALKS_CLUB = "datatalks_club"
    AI_SHIPPING_LABS = "ai_shipping_labs"
    ADMIN = "admin"

class RouteDecision(BaseModel):
    destination: Destination = Field(
        description=(
            "Which specialist should handle this request? Choose datatalks_club for "
            "DataTalks.Club course operations: cohorts, homeworks, learner projects "
            "and peer review, leaderboards, certificates, registrations, and meetups. "
            "Choose ai_shipping_labs for AI Shipping Labs operations: workshops and "
            "the membership tiers that gate them (main+ and above), events, plans and "
            "payments, shipping sprints, the book club, marketing pages, campaigns, "
            "and member CRM notes. Choose admin for personal task tracking in a local "
            "task list: capturing a task or reminder, listing tasks, marking one done."
        )
    )</code></pre></div><p>The enum describes the directions to route the agent, so I first run a classifier to decide which agent to invoke, and then invoke that agent.</p><p>In code it can look like this:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:&quot;d4cafd4a-15a9-4cee-8722-a811a4149b7b&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">from pydantic_ai import Agent
from site_agents.llm import openrouter_model

router = Agent(
    openrouter_model("openai/gpt-4o-mini"),
    instructions=GPT_ROUTER_INSTRUCTIONS,
    output_type=RouteDecision,
)

result = await router.run(request)
routing_decision = result.output

agents[routing_decision].run(request)</code></pre></div><h3><strong>Routing with Jev</strong></h3><p>With Jev, the same routing decision is one <code>Choice</code> question:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;javascript&quot;,&quot;nodeId&quot;:&quot;da70ef21-d856-4571-8e95-e3207a1c4798&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-javascript">ROUTER_INSTRUCTIONS = """
You are the dispatch layer for a small operations assistant.
Classify the user&#8217;s request into the single specialist that can do the work.
Do not answer the request and do not invent a fourth category.
""".strip()

ROUTING_CRITERIA = {
    "datatalks_club": (
        "DataTalks.Club course operations: cohorts and courses, homeworks and their "
        "questions, learner projects and peer review, leaderboards, certificates, "
        "graduates, course registrations, and community meetups."
    ),
    "ai_shipping_labs": (
        "AI Shipping Labs operations: workshops and the membership tiers that gate "
        "them (main+ and above), events, plans, payments and tier overrides, shipping "
        "sprints and their weekly checkpoints, the book club, marketing pages and "
        "site navigation, email campaigns, and member CRM notes."
    ),
    "admin": (
        "Personal task tracking in a local task list: capturing a task or reminder, "
        "listing open or completed tasks, and marking a task done. No website, course, "
        "workshop, or membership system is involved."
    ),
}</code></pre></div><p>Now we prepare the payload and send a request:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:&quot;50ba89b6-5c9e-414c-b641-29642e4e7507&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">payload = {
    "model": "~typesafe/jev-latest",
    "state": request,
    "questions": {
        "destination": {
            "type": "choice",
            "instructions": (
                f"{ROUTER_INSTRUCTIONS}\n\n"
                "Which specialist should handle this request? "
                "Choose exactly one option."
            ),
            "criteria": ROUTING_CRITERIA,
        }
    },
}

response = await client.post(
    "/api/alpha/decisions",
    headers={"Authorization": f"Bearer {api_key}"},
    json=payload,
)</code></pre></div><p>The answer comes back in <code>destination</code>. Since the model can only return one of my three options, I can map it to <code>Literal</code>:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:&quot;06446533-755d-4744-909b-6970fa1e9528&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">from typing import Literal

class JevChoiceAnswer(BaseModel):
    type: Literal["choice"]
    choice: Destination
    probabilities: dict[str, float]
    confidence: float</code></pre></div><p>From there, dispatching is the same:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:&quot;f9236e6b-10bd-4b45-8eaf-19e9b7777cb5&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">route = await route_request(request)
routing_decision = result.destination
agents[routing_decision].run(request)</code></pre></div><h3><strong>Comparing the Two Routers</strong></h3><p>To compare Jev with <code>gpt-4o-mini</code>, I created 50 short, hand-labeled requests and marked the destination for each one. They cover the work each site actually has: cohorts, homeworks, projects and leaderboards for DataTalks.Club, workshops, membership tiers, sprints and marketing pages for AI Shipping Labs, and plain task capture for the admin list.</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;json&quot;,&quot;nodeId&quot;:&quot;50a5226d-8fa8-4d63-be94-da141156ab98&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-json">[
  {
    "id": "aisl-01",
    "request": "Create a workshop Tuesday 16:30 about JEV, main+ only.",
    "expected": "ai_shipping_labs"
  },
  {
    "id": "admin-02",
    "request": "Remind me tomorrow to submit the VAT receipts.",
    "expected": "admin"
  }
]</code></pre></div><p>These are the results:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!N9zR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 424w, https://substackcdn.com/image/fetch/$s_!N9zR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 848w, https://substackcdn.com/image/fetch/$s_!N9zR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 1272w, https://substackcdn.com/image/fetch/$s_!N9zR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!N9zR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png" width="1456" height="318" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:318,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Benchmark results: Jev 50/50 at 0.32 s median, 0.55 s p95 and $0.001089, gpt-4o-mini 50/50 at 0.76 s median, 1.17 s p95 and $0.003704&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Benchmark results: Jev 50/50 at 0.32 s median, 0.55 s p95 and $0.001089, gpt-4o-mini 50/50 at 0.76 s median, 1.17 s p95 and $0.003704" title="Benchmark results: Jev 50/50 at 0.32 s median, 0.55 s p95 and $0.001089, gpt-4o-mini 50/50 at 0.76 s median, 1.17 s p95 and $0.003704" srcset="https://substackcdn.com/image/fetch/$s_!N9zR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 424w, https://substackcdn.com/image/fetch/$s_!N9zR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 848w, https://substackcdn.com/image/fetch/$s_!N9zR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 1272w, https://substackcdn.com/image/fetch/$s_!N9zR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee39d0a-b3ad-486c-9bf7-fbaf3f3e1014_1594x348.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Both routers get every case right. It&#8217;s not a suprsize, though. But speed and price are different: Jev was 2.4 times faster and 3.4 times cheaper.</p><p>I ran the benchmark several times and the numbers barely moved - Jev&#8217;s median stayed between 0.32 and 0.34 seconds, gpt-4o-mini&#8217;s between 0.73 and 0.76 second. The cost was the same on every run.</p><p>I think it&#8217;s a very promising technology and I&#8217;ll most likely integrate it in my workflows.</p><h3><strong>Workshop on Tuesday</strong></h3><p>I want to do a more detailed workshop about Jev next Tuesday at 16:30 Berlin time on AI Shipping Labs. If you&#8217;re member, come join me!</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://aishippinglabs.com/events/56/building-with-jev&quot;,&quot;text&quot;:&quot;Sign up for workshop&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://aishippinglabs.com/events/56/building-with-jev"><span>Sign up for workshop</span></a></p><p>If you have a use case, bring it along too. See you there!</p><h2><strong>Tools</strong></h2><p>This week I bookmarked a few interesting tools that I want to try.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!EmdW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EmdW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 424w, https://substackcdn.com/image/fetch/$s_!EmdW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 848w, https://substackcdn.com/image/fetch/$s_!EmdW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 1272w, https://substackcdn.com/image/fetch/$s_!EmdW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!EmdW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png" width="1456" height="1050" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1050,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Rebuno dashboard showing an agent execution with durable steps, approvals, and execution statuses&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Rebuno dashboard showing an agent execution with durable steps, approvals, and execution statuses" title="Rebuno dashboard showing an agent execution with durable steps, approvals, and execution statuses" srcset="https://substackcdn.com/image/fetch/$s_!EmdW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 424w, https://substackcdn.com/image/fetch/$s_!EmdW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 848w, https://substackcdn.com/image/fetch/$s_!EmdW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 1272w, https://substackcdn.com/image/fetch/$s_!EmdW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f95dd1b-860d-4381-a752-9ca5f4be6d90_1836x1324.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Rebuno&#8217;s execution dashboard shows durable agent steps, approvals, and execution status.</span></p><ul><li><p><a href="https://github.com/rebuno/rebuno">rebuno</a>: a library for adding durable step-level execution to agents: every tool and LLM call is recorded to a Postgres-backed event log, interrupted runs resume from the last recorded step, and YAML guardrails allow, deny, or hold any step for human approval. It works with LangChain, CrewAI, Pydantic AI, Mastra, and Vercel AI SDK.</p></li><li><p><a href="https://github.com/pizza-bot-app/pizza-bot">pizza-bot</a>: an inbox for long-running AI agents. It puts work items that wait on your decision in Unread, but can continue to run. It&#8217;s something I wanted to build in <a href="https://github.com/alexeygrigorev/codehive">CodeHive</a>, but never managed.</p></li><li><p><a href="https://github.com/ordewell/ordewell">ordewell</a>: a multi-agent task orchestrator that turns one goal into an ordered plan of coding-agent tasks, each with its own runner, model, thinking effort, and mode.</p></li><li><p><a href="https://github.com/greentfrapp/panel">panel</a>: a research workspace where the agent works together with you. It shows all the files, PDFs, and notebooks in one dock. You interact with the agent in the same environment and don&#8217;t need back-and-forth switching.</p></li></ul><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[AI Engineering Buildcamp Starts Next Week]]></title><description><![CDATA[Next Monday (September 21), I start the next cohort AI Engineering Buildcamp. It&#8217;s going to be the forth cohort of the course.]]></description><link>https://aishippingblog.com/p/ai-engineering-buildcamp-starts-next</link><guid isPermaLink="false">https://aishippingblog.com/p/ai-engineering-buildcamp-starts-next</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Mon, 14 Sep 2026 17:21:42 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!CsUw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Next Monday (September 21), I start the next cohort <a href="https://maven.com/alexey-grigorev/from-rag-to-agents">AI Engineering Buildcamp</a>. It&#8217;s going to be the forth cohort of the course.</p><p>The course is a practical and hands-on introduction to AI Engineering. After finishing the course, you&#8217;ll be able to create a project independently with tests, monitoring and a proper evaluation set. There are no slides in the course. Instead, we focus on coding a lot - also with help of AI assistants.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>I selected the topics for the course to give you a solid foundation to work as an AI Engineer and be able to integrate AI into your product.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CsUw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CsUw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 424w, https://substackcdn.com/image/fetch/$s_!CsUw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 848w, https://substackcdn.com/image/fetch/$s_!CsUw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 1272w, https://substackcdn.com/image/fetch/$s_!CsUw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!CsUw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png" width="1456" height="683" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:683,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;AI Engineering Buildcamp course page showing 4.7 rating and Sep 21 to Nov 22 cohort dates&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="AI Engineering Buildcamp course page showing 4.7 rating and Sep 21 to Nov 22 cohort dates" title="AI Engineering Buildcamp course page showing 4.7 rating and Sep 21 to Nov 22 cohort dates" srcset="https://substackcdn.com/image/fetch/$s_!CsUw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 424w, https://substackcdn.com/image/fetch/$s_!CsUw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 848w, https://substackcdn.com/image/fetch/$s_!CsUw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 1272w, https://substackcdn.com/image/fetch/$s_!CsUw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe30a1510-7b48-4e8f-9b58-1dffc0105185_2772x1300.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">AI Engineering Buildcamp cohort 4 runs Sep 21 to Nov 22, 2026</figcaption></figure></div><p>In this post, I&#8217;ll talk more about the course content, the projects from past cohorts, what participants say, and how to join.</p><p>In particular, I&#8217;ll cover:</p><ul><li><p>What is AI Engineering Buildcamp</p></li><li><p>The running example, optional projects, and homework</p></li><li><p>How the capstone works</p></li><li><p>What people in the past cohorts built</p></li><li><p>What participants say</p></li><li><p>How to join Cohort 4</p></li></ul><p></p><h2><strong>AI Engineering Buildcamp</strong></h2><p>In AI Engineering Buildcamp you learn AI engineering by building. We take one step at at a time, and at the end arrive at a fully-working system.</p><p>You&#8217;ll work on many projects during the course:</p><ul><li><p>The key running example project that evolves through the course - the documentation agent.</p></li><li><p>Each week, there will be a new homework project. You will have to complete it independently to solidify the learning from the lessons.</p></li><li><p>Optional example projects that show how the same ideas apply to other RAG and agent use cases</p></li><li><p>Your capstone project, where you apply everything to a use case you choose</p></li></ul><p></p><h2><strong>1. A Running Example Project</strong></h2><p>The running example through the course is the Documentation Agent. We index the Evidently documentation and build an AI agent on top of it.</p><p>This is a very practical application. Many companies have internal knowledge databases and need to be able to access them. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ycBZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ycBZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 424w, https://substackcdn.com/image/fetch/$s_!ycBZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 848w, https://substackcdn.com/image/fetch/$s_!ycBZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 1272w, https://substackcdn.com/image/fetch/$s_!ycBZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ycBZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png" width="1456" height="704" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:704,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Docs Assistant answering from Evidently documentation with agent activity and cited files&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Docs Assistant answering from Evidently documentation with agent activity and cited files" title="Docs Assistant answering from Evidently documentation with agent activity and cited files" srcset="https://substackcdn.com/image/fetch/$s_!ycBZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 424w, https://substackcdn.com/image/fetch/$s_!ycBZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 848w, https://substackcdn.com/image/fetch/$s_!ycBZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 1272w, https://substackcdn.com/image/fetch/$s_!ycBZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd763b8c0-5c74-4840-8976-c62f0127d3bf_1804x872.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Docs Assistant answers from Evidently docs with search activity and cited files</figcaption></figure></div><p>We start from RAG and add on top of it each week:</p><ul><li><p>Week 1 lays the foundation. It covers LLMs, the OpenAI API, RAG, and search. We index the docs and build a first RAG app to answer questions about them. From week 1, you already have a usable AI system.</p></li><li><p>Week 2 helps you catch up on week 1 materials. It adds more examples to strengthen your understanding of RAG.</p></li><li><p>Week 3 adds agentic capabilities. We turn the RAG app into an agent that can explore the docs database.</p></li><li><p>Week 4 focuses on reliability. We add tests to verify that the agent works as intended. This includes classical unit tests and LLM-based judges.</p></li><li><p>Week 5 focuses on monitoring. You see how the agent behaves in production. You collect the info you need to observe and debug it.</p></li><li><p>Week 6 adds systematic evaluation. You learn to evaluate agents with a human in the loop. You create judges that mimic human evaluators and use synthetic data.</p></li></ul><p>After six week, we&#8217;ll go through the same progression as any serious AI application: from simple RAG to a tool-using agent that we test, monitor and evaluate.</p><h2><strong>2. Optional Projects</strong></h2><p>If the main example is not enough, there are more optional projects that you can learn from.</p><p>For RAG, we&#8217;ll cover:</p><ul><li><p>FAQ Assistant: a chatbot for course FAQ data that uses RAG with boosting and filtering</p></li><li><p>YouTube Transcript Summarizer: a system that extracts summaries and chapter structure from YouTube videos with structured output</p></li><li><p>PDF Book Processor: a workflow for parsing a complex PDF, such as a book with math formulas and non-trivial layout. This kind of task often appears in take-home assignments and interviews.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SaES!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SaES!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 424w, https://substackcdn.com/image/fetch/$s_!SaES!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 848w, https://substackcdn.com/image/fetch/$s_!SaES!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 1272w, https://substackcdn.com/image/fetch/$s_!SaES!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SaES!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png" width="1456" height="768" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/66368a6d-1035-4958-942f-867360f2afbf_1727x911.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:768,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;PDF Book Processor notebook parsing a math book with PyMuPDF and page image encoding&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="PDF Book Processor notebook parsing a math book with PyMuPDF and page image encoding" title="PDF Book Processor notebook parsing a math book with PyMuPDF and page image encoding" srcset="https://substackcdn.com/image/fetch/$s_!SaES!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 424w, https://substackcdn.com/image/fetch/$s_!SaES!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 848w, https://substackcdn.com/image/fetch/$s_!SaES!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 1272w, https://substackcdn.com/image/fetch/$s_!SaES!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66368a6d-1035-4958-942f-867360f2afbf_1727x911.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">PDF Book Processor parses a math book page by page for RAG</figcaption></figure></div><p>We&#8217;ll have a few agent applications too:</p><ul><li><p>Web Search Agent: an agent that uses web search tools to find and filter and synthesize info from the internet</p></li><li><p>YouTube Researcher: an agent that searches YouTube and fetches transcripts and produces structured research summaries</p></li><li><p>Coding Agent: a functional coding agent that scaffolds Django applications</p></li><li><p>Code Analysis Agent: an agent for analyzing and understanding codebases</p></li><li><p>Deep Research Agent: a multi-stage research system that starts with broad search and expands into follow-up queries. It goes deeper into promising directions and fact-checks findings and generates a final article.</p></li></ul><h2><strong>3. Homework Projects</strong></h2><p>The core weeks (weeks 1-6) end with a separate homework mini-project.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CtLg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CtLg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 424w, https://substackcdn.com/image/fetch/$s_!CtLg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 848w, https://substackcdn.com/image/fetch/$s_!CtLg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 1272w, https://substackcdn.com/image/fetch/$s_!CtLg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CtLg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png" width="1456" height="1018" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1018,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Homework mini-project building a SQL agent over NYC taxi data with tests and LLM judges&quot;,&quot;title&quot;:&quot;Homework mini-project building a SQL agent over NYC taxi data with tests and LLM judges&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Homework mini-project building a SQL agent over NYC taxi data with tests and LLM judges" title="Homework mini-project building a SQL agent over NYC taxi data with tests and LLM judges" srcset="https://substackcdn.com/image/fetch/$s_!CtLg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 424w, https://substackcdn.com/image/fetch/$s_!CtLg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 848w, https://substackcdn.com/image/fetch/$s_!CtLg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 1272w, https://substackcdn.com/image/fetch/$s_!CtLg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95c66712-66c3-4cfc-9ce1-f3d66ff2b6bb_1722x1204.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Homework mini-project with a SQL agent over NYC taxi data</figcaption></figure></div><p></p><p>To solidify what you learned in the lessons, you&#8217;ll need to build another system yourself:</p><ul><li><p>Weeks 1 to 2: document processing with AI. You download books and extract PDF text and chunk docs and build a full RAG pipeline.</p></li><li><p>Week 3: Wikipedia Agent. You implement search and page-fetching tools and build an agent with a framework of your choice.</p></li><li><p>Week 4: testing through the DuckDB SQL Agent. You build an agent that translates natural language into SQL. You write pytest tests and add LLM judges and track costs.</p></li><li><p>Week 5: Trivia Quizmaster Agent. You build an interactive trivia system and add monitoring it.</p></li><li><p>Week 6: systematic evaluation through the Recipe Assistant Evaluation. You design 20 plus evaluation scenarios and run batch tests and detect hallucinations.</p></li></ul><p></p><h2><strong>4. Capstone Project</strong></h2><p>On top of that, you work on your capstone during the course.</p><p>I want to help you to start with your capstone from day one and start building your project layer by layer.:</p><ul><li><p>Weeks 1 to 2: define the use case and build the first working version with RAG</p></li><li><p>Week 3: make it agentic by adding tools. At this stage, your project moves beyond answering questions. It becomes a system that can make decisions.</p></li><li><p>Week 4: make it reliable by adding testing</p></li><li><p>Week 5: monitor it and start collecting logs from user interactions</p></li><li><p>Week 6: evaluate the project based on collected interactions and user scenarios you generated</p></li><li><p>Weeks 7 and 8: polish and extend and deploy the project</p></li><li><p>Week 9: present the result and get peer feedback and continue developing it</p></li></ul><p>My main focus for this cohort will be to make sure more people finish with a project. In the previous cohorts, some participants waited too long to decide what to build, and in the end they run out of time without submitting a project.</p><p>I want to address that directly. I will encourage to work on your projects from day 1.</p><p>If you already have an idea, I&#8217;ll share a framework and a set of instructions for your coding agent to implement it as quickly as possible.</p><p>If you don&#8217;t have an idea yet, I&#8217;ll share my design thinking approach to help you find something meaningful to solve.</p><p>I already used this approach in the last cohort, and one of the course graduates, Amar Agrawal, created a special app based on this approach: <a href="https://project-ideation-tool.streamlit.app/">Project Ideation Tool</a>.</p><p></p><h2><strong>Certificate</strong></h2><p>If you finish the project and do 3 peer reviews, you will get a certificate signed by me.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gQQk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gQQk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!gQQk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!gQQk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!gQQk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gQQk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Sample AI Engineering Buildcamp certificate of completion&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Sample AI Engineering Buildcamp certificate of completion" title="Sample AI Engineering Buildcamp certificate of completion" srcset="https://substackcdn.com/image/fetch/$s_!gQQk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!gQQk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!gQQk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!gQQk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d0c4a37-9b14-439a-97e8-02b7e475ff6f_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Sample certificate of completion for AI Engineering Buildcamp</figcaption></figure></div><p>The certificate confirms that you shipped a capstone.</p><h2><strong>Past projects</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZJX6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZJX6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 424w, https://substackcdn.com/image/fetch/$s_!ZJX6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 848w, https://substackcdn.com/image/fetch/$s_!ZJX6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 1272w, https://substackcdn.com/image/fetch/$s_!ZJX6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZJX6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png" width="1456" height="1048" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1048,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Collage of 12 cohort 3 capstone project interfaces&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Collage of 12 cohort 3 capstone project interfaces" title="Collage of 12 cohort 3 capstone project interfaces" srcset="https://substackcdn.com/image/fetch/$s_!ZJX6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 424w, https://substackcdn.com/image/fetch/$s_!ZJX6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 848w, https://substackcdn.com/image/fetch/$s_!ZJX6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 1272w, https://substackcdn.com/image/fetch/$s_!ZJX6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f1ed4c1-749f-41e1-8902-9eb4ae325e8c_1456x1048.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Cohort 3 capstones, from exam generation and job tools to chess coaching and learning apps</figcaption></figure></div><p>I have already described what people from the past cohorts build.</p><p>Here are the articles:</p><ul><li><p>Cohort 1: <a href="https://aishippingblog.com/p/5-ideas-for-ai-agents-and-openais">5 ideas for AI agents</a>.</p></li><li><p>Cohort 2: <a href="https://aishippingblog.com/p/9-real-life-ai-projects-from-ai-engineering">9 Real-Life AI Projects from AI Engineering Buildcamp Graduates</a>.</p></li><li><p>Cohort 3: <a href="https://aishippingblog.com/p/13-ai-projects-from-ai-engineering">13 AI Projects from AI Engineering Buildcamp</a>.</p></li></ul><p>I&#8217;m looking forward to seeing what you will build in Cohort 4!</p><h2><strong>What participants say</strong></h2><p>Here is what past participants say about the course.</p><p>Vishnu, Data Scientist at Humana, joined cohort 2.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!siKq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!siKq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 424w, https://substackcdn.com/image/fetch/$s_!siKq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 848w, https://substackcdn.com/image/fetch/$s_!siKq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 1272w, https://substackcdn.com/image/fetch/$s_!siKq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!siKq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png" width="1130" height="544" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:544,&quot;width&quot;:1130,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Vishnu review from April 2026 praising week-one project start and help with shaping ideas&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Vishnu review from April 2026 praising week-one project start and help with shaping ideas" title="Vishnu review from April 2026 praising week-one project start and help with shaping ideas" srcset="https://substackcdn.com/image/fetch/$s_!siKq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 424w, https://substackcdn.com/image/fetch/$s_!siKq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 848w, https://substackcdn.com/image/fetch/$s_!siKq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 1272w, https://substackcdn.com/image/fetch/$s_!siKq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e777ccf-7185-40a6-91e4-bd5e4d5238df_1130x544.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Vishnu on starting in week one and shaping the project idea</figcaption></figure></div><p>Carmine, Senior Coach, Data Science and ML Engineer, joined cohort 3.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jKpR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jKpR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 424w, https://substackcdn.com/image/fetch/$s_!jKpR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 848w, https://substackcdn.com/image/fetch/$s_!jKpR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 1272w, https://substackcdn.com/image/fetch/$s_!jKpR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jKpR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png" width="1112" height="824" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:824,&quot;width&quot;:1112,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Carmine review from June 2026 praising structure and building from day one&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Carmine review from June 2026 praising structure and building from day one" title="Carmine review from June 2026 praising structure and building from day one" srcset="https://substackcdn.com/image/fetch/$s_!jKpR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 424w, https://substackcdn.com/image/fetch/$s_!jKpR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 848w, https://substackcdn.com/image/fetch/$s_!jKpR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 1272w, https://substackcdn.com/image/fetch/$s_!jKpR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4845932f-91bf-42eb-bf4a-bf26f3c55242_1112x824.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Carmine on structure and building from day one</figcaption></figure></div><p>Lars, Data Scientist at NS, joined cohort 3.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!En8O!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!En8O!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 424w, https://substackcdn.com/image/fetch/$s_!En8O!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 848w, https://substackcdn.com/image/fetch/$s_!En8O!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 1272w, https://substackcdn.com/image/fetch/$s_!En8O!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!En8O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png" width="1112" height="532" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:532,&quot;width&quot;:1112,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Lars review from July 2026 praising structure and project-based learning&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Lars review from July 2026 praising structure and project-based learning" title="Lars review from July 2026 praising structure and project-based learning" srcset="https://substackcdn.com/image/fetch/$s_!En8O!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 424w, https://substackcdn.com/image/fetch/$s_!En8O!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 848w, https://substackcdn.com/image/fetch/$s_!En8O!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 1272w, https://substackcdn.com/image/fetch/$s_!En8O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F682a77c8-c091-43c3-8e99-dfa5627e3680_1112x532.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Lars on learning RAG and agents through his own project</figcaption></figure></div><p></p><p>You can find more reviews in the testimonial section of <a href="https://maven.com/alexey-grigorev/from-rag-to-agents">the course page</a>.</p><h2><strong>Weekly Workload</strong></h2><p>The course lasts about nine weeks. It runs September 21 to November 22, 2026. It is designed for working professionals, but it still requires a meaningful commitment.</p><p>Most of the work is asynchronous. Each week features pre-recorded lessons and practical exercises and homework. Depending on your background and pace, this usually takes 3 to 10 hours per week. Project work takes 10 to 20 hours per week, distributed through the course.</p><p>There is one live session each week, mainly for office hours. This is when you can ask technical questions and discuss architecture decisions and receive feedback on your projects from me. I record these sessions and share them with everyone, along with a written summary.</p><p>This is not a passive course. It is intended for people who want to spend real time building and debugging, rather than just watching explanations and taking notes.</p><h2><strong>Scholarship</strong></h2><p>The scholarship form closed today. I received over 2,100 submissions. Selecting recipients was extremely difficult.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pIuO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pIuO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 424w, https://substackcdn.com/image/fetch/$s_!pIuO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 848w, https://substackcdn.com/image/fetch/$s_!pIuO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 1272w, https://substackcdn.com/image/fetch/$s_!pIuO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pIuO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png" width="1456" height="376" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:376,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Scholarship application form for cohort 4 showing 2,143 responses&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Scholarship application form for cohort 4 showing 2,143 responses" title="Scholarship application form for cohort 4 showing 2,143 responses" srcset="https://substackcdn.com/image/fetch/$s_!pIuO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 424w, https://substackcdn.com/image/fetch/$s_!pIuO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 848w, https://substackcdn.com/image/fetch/$s_!pIuO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 1272w, https://substackcdn.com/image/fetch/$s_!pIuO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb84147e-a9c5-47c4-aaa6-dcb4aa1050f5_1768x456.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Scholarship applications for cohort 4 closed with 2,143 responses</figcaption></figure></div><p>Today I contacted the people who were selected. If you do not see an email from me, you were not selected this time. I can not write to everyone individually.</p><p>If you applied and opted in for updates, I added you to this newsletter. That is why you see this email now. Thanks for all your applications.</p><p>When deciding, I focused on existing impact. I looked for people already doing something and helping their local community, and I prioritized regions where the course price is hard to afford. </p><h2><strong>Ways to Join</strong></h2><p>The standard price of the course is 1,799 USD.</p><ol><li><p>You can use your company learning budget to cover the cost. I wrote a <a href="https://aishippinglabs.com/blog/how-to-join-ai-engineering-buildcamp">short guide</a> with an email template you can use to reach out to your manager. Maven also has <a href="https://maven.com/expense">their own guide</a>.</p></li><li><p>You can get a team discount. Teams of 2 to 9 get 20 percent off. Teams of 10 plus get 25 percent off.</p></li><li><p>I also provide full and partial scholarships for each cohort. For cohort 4, the form is already closed. See the Scholarship section above for details.</p></li><li><p>If none of the options above work for you, you can use the &#8220;SUBSTACK&#8221; code to get a 20% discount.</p></li></ol><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://maven.com/alexey-grigorev/from-rag-to-agents&quot;,&quot;text&quot;:&quot;Enroll now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://maven.com/alexey-grigorev/from-rag-to-agents"><span>Enroll now</span></a></p><p>If you need a student discount or PPP pricing or have any questions about the joining options, contact me at <a href="mailto:alexey@datatalks.club">alexey@datatalks.club</a>, and I&#8217;ll help you.</p><h2><strong>Join the Course</strong></h2><p>AI Engineering Buildcamp is designed for hands-on learning. You will develop your own capstone project and complete mini-projects for homework and engage in optional practical projects. You will gain experience with RAG and agents and testing and observability and evaluation and deployment.</p><p>If you have been thinking about joining, registration is open until September 21.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Free and Affordable AI-Native Development]]></title><description><![CDATA[Build more, learn more!]]></description><link>https://aishippingblog.com/p/free-and-affordable-ai-native-development</link><guid isPermaLink="false">https://aishippingblog.com/p/free-and-affordable-ai-native-development</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 11 Sep 2026 23:02:46 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!zfRh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I recently posted on X that we have access to so many tokens that I&#8217;m running out of ideas what to build.</p><div class="twitter-embed" data-attrs="{&quot;url&quot;:&quot;https://x.com/Al_Grigor/status/2096543638687756663&quot;,&quot;full_text&quot;:&quot;Now we can have\n\n- 3 codex resets\n- free Muse 3 on OpenCode Zen \n- Very cheap Muse 3 and Omen Alpha on OpenCode Go\n- Unlimited glm-5.3-flash on <a class=\&quot;tweet-url\&quot; href=\&quot;https://z.ai\&quot;>z.ai</a> plans from 5 pm every day for 10 hours\n- Hy3 free on multiple platforms \n\nI'm running out of things to build&#8230;&quot;,&quot;username&quot;:&quot;Al_Grigor&quot;,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;profile_image_url&quot;:&quot;https://pbs.substack.com/profile_images/1793934377547644928/n0X1VyYw_normal.jpg&quot;,&quot;date&quot;:&quot;2026-09-06T10:18:50.000Z&quot;,&quot;photos&quot;:[],&quot;quoted_tweet&quot;:{&quot;full_text&quot;:&quot;what do I do to have unlimited tokens&quot;,&quot;username&quot;:&quot;enjojoyy&quot;,&quot;name&quot;:&quot;albina&quot;,&quot;profile_image_url&quot;:&quot;https://pbs.substack.com/profile_images/1997389601216880640/o6jlOaId_normal.jpg&quot;},&quot;reply_count&quot;:89,&quot;retweet_count&quot;:147,&quot;like_count&quot;:3314,&quot;impression_count&quot;:237432,&quot;expanded_url&quot;:null,&quot;video_url&quot;:null,&quot;video_preview_media_key&quot;:null,&quot;belowTheFold&quot;:false}" data-component-name="Twitter2ToDOM"></div><p>At DataTalks.Club we&#8217;re running <a href="https://aishippingblog.com/p/ai-dev-tools-zoomcamp-2026-starts">AI Dev Tools Zoomcamp</a> and people are asking me where they can get access to coding agents for free or without paying a lot of money. So I figured I should take that post from X and turn it into an article.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>While I do recommend getting a paid Codex or Claude subscription and use it for the course, there are some other providers that can get you a lot of tokens for less money or even for free.</p><p>In this post, I&#8217;ll share some of them. We&#8217;ll cover</p><ul><li><p>Free coding plans</p></li><li><p>Cheap coding plans</p></li><li><p>Current promotions</p></li></ul><p>Let&#8217;s go!</p><h2><strong>Free coding plans</strong></h2><p>There are multiple platforms that currently offer free coding plans.</p><p>One of them is Tencent. They recently released Hy3 and now offer a free tier through its <a href="https://www.workbuddy.ai/">WorkBuddy</a> and <a href="https://www.codebuddy.ai/">CodeBuddy</a> assistants. It&#8217;s not available in Germany, so I couldn&#8217;t try it - but maybe it&#8217;ll work for you.</p><p>Another one is <a href="https://opencode.ai/zen">OpenCode Zen</a>. They have a variety of models absolutely for free. One of these models is Muse Spark 1.3 from Meta.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!RJSN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RJSN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 424w, https://substackcdn.com/image/fetch/$s_!RJSN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 848w, https://substackcdn.com/image/fetch/$s_!RJSN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 1272w, https://substackcdn.com/image/fetch/$s_!RJSN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RJSN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png" width="1456" height="865" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:865,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;OpenCode Zen pricing table showing free models including Big Pickle, MiMo-V2.5, Ling 3.0 Flash, Nemotron 3 Ultra, Nemotron 3.5 Lightning, and Muse Spark 1.3 Contributor Free, all with free input, output and cached read pricing&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="OpenCode Zen pricing table showing free models including Big Pickle, MiMo-V2.5, Ling 3.0 Flash, Nemotron 3 Ultra, Nemotron 3.5 Lightning, and Muse Spark 1.3 Contributor Free, all with free input, output and cached read pricing" title="OpenCode Zen pricing table showing free models including Big Pickle, MiMo-V2.5, Ling 3.0 Flash, Nemotron 3 Ultra, Nemotron 3.5 Lightning, and Muse Spark 1.3 Contributor Free, all with free input, output and cached read pricing" srcset="https://substackcdn.com/image/fetch/$s_!RJSN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 424w, https://substackcdn.com/image/fetch/$s_!RJSN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 848w, https://substackcdn.com/image/fetch/$s_!RJSN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 1272w, https://substackcdn.com/image/fetch/$s_!RJSN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab1f7959-5176-4e35-925f-42aa09ea5ba1_1478x878.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">OpenCode Zen free models - Muse Spark 1.3 Contributor Free included</figcaption></figure></div><p>The full name for this model is &#8220;Muse Spark 1.3 Contributor Free&#8221;. The &#8220;contributor&#8221; part here means that you&#8217;re &#8220;contributing&#8221; your data to Meta. That is, they can use your data to improve their model. That&#8217;s why it&#8217;s free - you pay for it with your data.</p><p>For me it&#8217;s not a concern. It&#8217;s quite a powerful model, it feels like Opus 5 level. I can have one or two heavy coding sessions with Muse 1.3 in OpenCode Zen absolutely for free.</p><p>Once I hit the limits, I can stay in OpenCode and switch the session to OpenCode Go - their coding plan.</p><h2><strong>Affordable coding plans</strong></h2><p><a href="https://opencode.ai/go">OpenCode Go</a> costs $10 per month. For this $10 you get a lot. Specifically, you get the same Muse 1.3 Contributor, although without the &#8220;free&#8221; part.</p><p>Again, it&#8217;s a contributor model, so Meta is subsidizing 90% of the cost - at least this is what they say. And you can get a lot done with this model. Combine it with 1-2 free sessions from OpenCode Zen, and you&#8217;ll get 5+ hours of work per day with this plan.</p><p>And there are other models, like DeepSeek V4.1 Flash, which currently offers 75% discount (i.e. 4x more usage).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!E9Fe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!E9Fe!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 424w, https://substackcdn.com/image/fetch/$s_!E9Fe!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 848w, https://substackcdn.com/image/fetch/$s_!E9Fe!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 1272w, https://substackcdn.com/image/fetch/$s_!E9Fe!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!E9Fe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png" width="1456" height="863" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:863,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;OpenCode Go usage chart showing estimated requests per 5 hours: Kimi K3 at 110, Kimi K2.7 Code at 1,350, GPT 5.6 Luna at 2,050, MiniMax M3 at 3,200, Qwen3.7 Plus at 4,300, GLM-5.3-Flash at 6,320, DeepSeek V4 Flash at 13,000, DeepSeek V4.1 Flash at 26,000 with 4x usage tag, MiMo-V2.5 at 30,100, Muse Spark 1.3 Contributor at 45,300&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="OpenCode Go usage chart showing estimated requests per 5 hours: Kimi K3 at 110, Kimi K2.7 Code at 1,350, GPT 5.6 Luna at 2,050, MiniMax M3 at 3,200, Qwen3.7 Plus at 4,300, GLM-5.3-Flash at 6,320, DeepSeek V4 Flash at 13,000, DeepSeek V4.1 Flash at 26,000 with 4x usage tag, MiMo-V2.5 at 30,100, Muse Spark 1.3 Contributor at 45,300" title="OpenCode Go usage chart showing estimated requests per 5 hours: Kimi K3 at 110, Kimi K2.7 Code at 1,350, GPT 5.6 Luna at 2,050, MiniMax M3 at 3,200, Qwen3.7 Plus at 4,300, GLM-5.3-Flash at 6,320, DeepSeek V4 Flash at 13,000, DeepSeek V4.1 Flash at 26,000 with 4x usage tag, MiMo-V2.5 at 30,100, Muse Spark 1.3 Contributor at 45,300" srcset="https://substackcdn.com/image/fetch/$s_!E9Fe!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 424w, https://substackcdn.com/image/fetch/$s_!E9Fe!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 848w, https://substackcdn.com/image/fetch/$s_!E9Fe!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 1272w, https://substackcdn.com/image/fetch/$s_!E9Fe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F589284fc-ee3f-41c9-8224-12b85d8ea66d_2004x1188.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">OpenCode Go usage estimates - DeepSeek V4.1 Flash at 26,000 requests per 5 hours with 4x usage, Muse Spark 1.3 Contributor at 45,300</figcaption></figure></div><p>I used to recommend GitHub Copilot as the first coding plan. Not anymore - now it&#8217;s OpenCode Go.</p><p>Other cheap alternatives in the same $8-20 range:</p><ul><li><p><a href="https://openai.com/chatgpt/pricing/">ChatGPT Plus</a> ($20)</p></li><li><p><a href="https://z.ai/subscribe">Z.ai GLM Coding Plan Lite</a> ($18)</p></li><li><p><a href="https://platform.minimax.io/subscribe/token-plan">MiniMax Token Plan</a> ($20)</p></li><li><p><a href="https://commandcode.ai/docs/plans/goat">Command Code GOAT</a> ($10)</p></li></ul><p>Prices and quotas change fast, so check the current plan pages.</p><h2><strong>Cheap API access</strong></h2><p>So far we talked about coding plans. Instead, you can pay for API calls yourself and plug any API LLM platform to your coding agent.</p><p>If you do that directly with OpenAI or Anthropic and put $5 into your account, they will disappear before you can say &#8220;hello&#8221; to your coding agent.</p><p>But if you use <a href="https://api-docs.deepseek.com/quick_start/pricing">DeepSeek&#8217;s V4.1 Flash</a> during off-peak hours, you&#8217;ll be getting millions of tokens, which is enough to do multiple coding sessions.</p><p>Also, <a href="https://openrouter.ai/discover">OpenRouter</a> gives access to many free and cheap models. Any model ID ending in <code>:free</code> costs zero tokens. The rate limits are very strict though, but once you pay, the limits are lifted.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vvgC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vvgC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 424w, https://substackcdn.com/image/fetch/$s_!vvgC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 848w, https://substackcdn.com/image/fetch/$s_!vvgC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 1272w, https://substackcdn.com/image/fetch/$s_!vvgC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vvgC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png" width="1254" height="836" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:836,&quot;width&quot;:1254,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;OpenRouter most-used free models ranking by weekly tokens: Nemotron 3 Ultra free at 3.6T tokens per week with 1M context, Laguna S 2.1 free at 1.2T with 262K context, Nemotron 3.5 Lightning free at 958.4B with 1M context, Ling 3.0 Flash Fin free at 881.5B with 262K context, Dots3-Note Preview free at 397.9B with 512K context&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="OpenRouter most-used free models ranking by weekly tokens: Nemotron 3 Ultra free at 3.6T tokens per week with 1M context, Laguna S 2.1 free at 1.2T with 262K context, Nemotron 3.5 Lightning free at 958.4B with 1M context, Ling 3.0 Flash Fin free at 881.5B with 262K context, Dots3-Note Preview free at 397.9B with 512K context" title="OpenRouter most-used free models ranking by weekly tokens: Nemotron 3 Ultra free at 3.6T tokens per week with 1M context, Laguna S 2.1 free at 1.2T with 262K context, Nemotron 3.5 Lightning free at 958.4B with 1M context, Ling 3.0 Flash Fin free at 881.5B with 262K context, Dots3-Note Preview free at 397.9B with 512K context" srcset="https://substackcdn.com/image/fetch/$s_!vvgC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 424w, https://substackcdn.com/image/fetch/$s_!vvgC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 848w, https://substackcdn.com/image/fetch/$s_!vvgC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 1272w, https://substackcdn.com/image/fetch/$s_!vvgC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff732ec7f-e515-4763-8dd6-3d30ea98838c_1254x836.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Most-used free models on OpenRouter by weekly tokens - Nemotron 3 Ultra leads at 3.6T.</figcaption></figure></div><h2><strong>Current promotions</strong></h2><p>I really like Z.ai. I have used them since September last year through their API, and I&#8217;ve been using their coding plan since December.</p><p>Now they are regularly running promotions and giving away 300 million tokens for glm-5.3-flash. You have to use their <a href="https://zcode.z.ai/en">Zcode harness</a>, and the promotions are typically timed for weekends. It&#8217;s a lot of tokens - many people complain that these tokens expire before they can spend them.</p><div class="twitter-embed" data-attrs="{&quot;url&quot;:&quot;https://x.com/zcode_ai/status/2098396306750517317&quot;,&quot;full_text&quot;:&quot;WEEKEND BUILD V is arriving &#129370;&#10024;\n300M tokens per person, free.Bring your next idea to life.\nSep 11, 8 AM &#8211; Sep 13, 6 PM PDT.\nTokens activate when the event starts.\nUpdate ZCode to the latest version, then tap the bottom-left event card when it opens.\n<a class=\&quot;tweet-url\&quot; href=\&quot;https://zcode.z.ai\&quot;>zcode.z.ai</a> &quot;,&quot;username&quot;:&quot;zcode_ai&quot;,&quot;name&quot;:&quot;ZCode&quot;,&quot;profile_image_url&quot;:&quot;https://pbs.substack.com/profile_images/2066234804325289984/oEcGryOS_normal.jpg&quot;,&quot;date&quot;:&quot;2026-09-11T13:00:40.000Z&quot;,&quot;photos&quot;:[{&quot;img_url&quot;:&quot;https://pbs.substack.com/media/HR7-jS8asAAFwVi.jpg&quot;,&quot;link_url&quot;:&quot;https://t.co/nsfg7NtxQx&quot;}],&quot;quoted_tweet&quot;:{},&quot;reply_count&quot;:113,&quot;retweet_count&quot;:24,&quot;like_count&quot;:400,&quot;impression_count&quot;:34469,&quot;expanded_url&quot;:null,&quot;video_url&quot;:null,&quot;video_preview_media_key&quot;:null,&quot;belowTheFold&quot;:true}" data-component-name="Twitter2ToDOM"></div><p>Additionally, right now they are running <a href="https://docs.z.ai/devpack/notice/event-glm-5.3-flash">a promo for all Z.ai coding plan subscribers</a>. If you&#8217;re one of them, you get unlimited glm-5.3-flash tokens from 5pm to 3am CET.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rC9N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rC9N!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 424w, https://substackcdn.com/image/fetch/$s_!rC9N!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 848w, https://substackcdn.com/image/fetch/$s_!rC9N!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 1272w, https://substackcdn.com/image/fetch/$s_!rC9N!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rC9N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png" width="1418" height="590" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:590,&quot;width&quot;:1418,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Z.ai campaign details: every day from 23:00 to 09:00 the following day, GLM-5.3-Flash through GLM Coding Plan - use via ZCode means zero quota consumption for unlimited usage, use via other supported agents means available quota is doubled&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Z.ai campaign details: every day from 23:00 to 09:00 the following day, GLM-5.3-Flash through GLM Coding Plan - use via ZCode means zero quota consumption for unlimited usage, use via other supported agents means available quota is doubled" title="Z.ai campaign details: every day from 23:00 to 09:00 the following day, GLM-5.3-Flash through GLM Coding Plan - use via ZCode means zero quota consumption for unlimited usage, use via other supported agents means available quota is doubled" srcset="https://substackcdn.com/image/fetch/$s_!rC9N!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 424w, https://substackcdn.com/image/fetch/$s_!rC9N!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 848w, https://substackcdn.com/image/fetch/$s_!rC9N!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 1272w, https://substackcdn.com/image/fetch/$s_!rC9N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a9099-9936-4c07-8a09-4077bcd136e9_1418x590.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Z.ai evening promo rules - zero quota consumption via ZCode, doubled quota via other agents (time in SG timezone)</figcaption></figure></div><p>Well, they aren&#8217;t completely &#8220;unlimited&#8221; - there are still rate-limits. But I don&#8217;t hit these limits unless I run 20+ agents in parallel.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zfRh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zfRh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 424w, https://substackcdn.com/image/fetch/$s_!zfRh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 848w, https://substackcdn.com/image/fetch/$s_!zfRh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 1272w, https://substackcdn.com/image/fetch/$s_!zfRh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zfRh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png" width="1456" height="752" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:752,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Z.ai usage trends bar chart for September 5-11 showing total 9.7B tokens, almost all GLM-5.3-Flash at 9.7B with GLM-5.3 at only 10.5M tokens&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Z.ai usage trends bar chart for September 5-11 showing total 9.7B tokens, almost all GLM-5.3-Flash at 9.7B with GLM-5.3 at only 10.5M tokens" title="Z.ai usage trends bar chart for September 5-11 showing total 9.7B tokens, almost all GLM-5.3-Flash at 9.7B with GLM-5.3 at only 10.5M tokens" srcset="https://substackcdn.com/image/fetch/$s_!zfRh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 424w, https://substackcdn.com/image/fetch/$s_!zfRh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 848w, https://substackcdn.com/image/fetch/$s_!zfRh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 1272w, https://substackcdn.com/image/fetch/$s_!zfRh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9baf1c0-2cf2-4974-86f0-492df3a7d22a_1738x898.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">My Z.ai usage for the week - 9.7B GLM-5.3-Flash tokens</figcaption></figure></div><p>That plus 300m tokens plus their quota reduction if you use Zcode - that combined gives you a lot of tokens.</p><p>One downside of Zcode is that it&#8217;s GUI only, no TUI. But I figured out how to <a href="https://github.com/alexeygrigorev/codex-zcode/">connect it to Codex</a>, so I can run Zcode from my terminal.</p><p>Z.ai is not the only company that runs promos. There are others:</p><ul><li><p>Tencent Cloud TokenHub gives every new account 1M free tokens per model, covering DeepSeek V4 Pro/Flash, GLM-5, MiniMax M3 and Kimi K2.6.</p></li><li><p>Verdant currently offers GLM 5.3 Flash and DeepSeek V4 Flash for free within 5-hour and weekly limits.</p></li><li><p>Cursor doubled included usage for its first-party models (Auto, Composer 2.5) on paid plans.</p></li><li><p>Cognition (the company behind Devin) just released a new model SWE-2 and they made it unlimited in Devin paid plans.</p></li></ul><h2><strong>Build more</strong></h2><p>With all these free models and promotions I no longer have a problem of having no tokens. My main problem is deciding what to build next.</p><p>I hope you&#8217;ll have the same problem. Have fun and enjoy building!</p><h2><strong>Tools</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!a7yQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!a7yQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 424w, https://substackcdn.com/image/fetch/$s_!a7yQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 848w, https://substackcdn.com/image/fetch/$s_!a7yQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 1272w, https://substackcdn.com/image/fetch/$s_!a7yQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!a7yQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png" width="1456" height="902" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:902,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;OtoDock GitHub hero: blue robot logo, 'OtoDock - Collaborative Agents', tagline 'The agentic company OS', subtitle 'The brains of your company, built on Claude Code and Codex, working on your Anthropic and OpenAI subscriptions', badges for FSL-1.1-Apache-2.0 license, version 1.6.0, passing CI, docs and website links&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="OtoDock GitHub hero: blue robot logo, 'OtoDock - Collaborative Agents', tagline 'The agentic company OS', subtitle 'The brains of your company, built on Claude Code and Codex, working on your Anthropic and OpenAI subscriptions', badges for FSL-1.1-Apache-2.0 license, version 1.6.0, passing CI, docs and website links" title="OtoDock GitHub hero: blue robot logo, 'OtoDock - Collaborative Agents', tagline 'The agentic company OS', subtitle 'The brains of your company, built on Claude Code and Codex, working on your Anthropic and OpenAI subscriptions', badges for FSL-1.1-Apache-2.0 license, version 1.6.0, passing CI, docs and website links" srcset="https://substackcdn.com/image/fetch/$s_!a7yQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 424w, https://substackcdn.com/image/fetch/$s_!a7yQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 848w, https://substackcdn.com/image/fetch/$s_!a7yQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 1272w, https://substackcdn.com/image/fetch/$s_!a7yQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc620908b-f3f3-424d-8e61-945fbe248aa6_1872x1160.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>I haven&#8217;t included the section with new tools for quite some time. Now I want to fix it.</p><ul><li><p><strong><a href="https://github.com/timgordontg/engrim">engrim</a></strong>: a local-first SQLite memory engine that keeps one project-scoped episodic store across Antigravity, Claude Code, Cursor, Windsurf, and Codex. With their approach it&#8217;s very easy to switch models in the middle of a session, and you don&#8217;t need to send the whole session transcript when doing it.</p></li><li><p><strong><a href="https://github.com/OtoDock/oto-dock">oto-dock</a></strong>: &#8220;The brains of your company, built on Claude Code &amp; Codex, working on your Anthropic and OpenAI subscriptions.&#8221; It&#8217;s a self-hosted company OS where agents organized in departments connect to company tools, delegate to each other, and keep working unattended. I&#8217;m experimenting with automating as much of DataTalks.Club ops as I can, so this project looks promising.</p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[13 AI Projects from AI Engineering Buildcamp Cohort 3]]></title><description><![CDATA[See what the course participants built]]></description><link>https://aishippingblog.com/p/13-ai-projects-from-ai-engineering</link><guid isPermaLink="false">https://aishippingblog.com/p/13-ai-projects-from-ai-engineering</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 04 Sep 2026 18:21:30 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/04db2d9f-7355-49f4-bce5-58a862c6364e_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The third cohort of my AI Engineering course wrapped this summer, and in this post I want to share what we built.</p><p>The projects are real-life and include an exam questions generator, a chess coach, a RAG pipeline, job tools, and research assistants.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>This is not the first post about the projects that AI Buildcamp graduates built. I wrote about the previous two cohorts earlier:</p><ul><li><p>First cohort: <a href="https://aishippingblog.com/p/5-ideas-for-ai-agents-and-openais">5 ideas for AI agents</a></p></li><li><p>Second cohort: <a href="https://aishippingblog.com/p/9-real-life-ai-projects-from-ai-engineering">9 Real-Life AI Projects from AI Engineering Buildcamp Graduates</a></p></li></ul><p>In this post I want to describe some of these projects.</p><p>Some of the course participants presented their projects live on <a href="https://youtu.be/kKUDZwdyuP4">Demo Day</a>, so we&#8217;ll start with them.</p><p>Let&#8217;s start!</p><h2><strong>1) Exam Questions Generator by Salma Bouzid</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!L-6J!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!L-6J!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 424w, https://substackcdn.com/image/fetch/$s_!L-6J!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 848w, https://substackcdn.com/image/fetch/$s_!L-6J!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 1272w, https://substackcdn.com/image/fetch/$s_!L-6J!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!L-6J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png" width="1456" height="884" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:884,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Quizgen landing page inviting professors to generate exam questions from their own lecture slides&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Quizgen landing page inviting professors to generate exam questions from their own lecture slides" title="Quizgen landing page inviting professors to generate exam questions from their own lecture slides" srcset="https://substackcdn.com/image/fetch/$s_!L-6J!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 424w, https://substackcdn.com/image/fetch/$s_!L-6J!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 848w, https://substackcdn.com/image/fetch/$s_!L-6J!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 1272w, https://substackcdn.com/image/fetch/$s_!L-6J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316ebe9a-9bf3-4d8e-a530-2cee26c77991_1610x977.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Quizgen generates exam questions from the lecture slides</figcaption></figure></div><p>Salma is a data analyst who took the course to get better at building an AI app. She&#8217;s leaving her job to work as a founder. Her project helps university professors generate exams from their own slide decks.</p><p>You upload a course PDF and pick how many questions you want. You also choose a mix of case studies, theory, and applied-concept questions. A professor can approve, edit, or reject each question, then export the exam. In the live demo she asked the model to tailor an HR-management exam to a business student in Berlin.</p><ul><li><p>The frontend is Next.js</p></li><li><p>Supabase stores user actions plus API cost, PDF size, and regeneration prompts</p></li><li><p>She started with Claude and later moved to OpenAI</p></li><li><p>An LLM judge checks that a question is grounded in the course and not answerable without it</p></li></ul><p><a href="https://www.linkedin.com/in/salma-bouzid-45634762/">Salma&#8217;s LinkedIn</a></p><h2><strong>2) ATS Gap Analyser by Amar Agrawal</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mOUn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mOUn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 424w, https://substackcdn.com/image/fetch/$s_!mOUn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 848w, https://substackcdn.com/image/fetch/$s_!mOUn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 1272w, https://substackcdn.com/image/fetch/$s_!mOUn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mOUn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png" width="1140" height="720" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:720,&quot;width&quot;:1140,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;ATS Gap Analyser example result showing a 75 out of 100 moderate match, missing keywords, and improvement suggestions&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="ATS Gap Analyser example result showing a 75 out of 100 moderate match, missing keywords, and improvement suggestions" title="ATS Gap Analyser example result showing a 75 out of 100 moderate match, missing keywords, and improvement suggestions" srcset="https://substackcdn.com/image/fetch/$s_!mOUn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 424w, https://substackcdn.com/image/fetch/$s_!mOUn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 848w, https://substackcdn.com/image/fetch/$s_!mOUn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 1272w, https://substackcdn.com/image/fetch/$s_!mOUn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fd9d734-6a2e-47a1-a94e-269ca362b9e5_1140x720.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">ATS Gap Analyser&#8217;s example result with a 75/100 match, missing keywords, and improvement suggestions</figcaption></figure></div><p>Amar&#8217;s <a href="https://github.com/Amar-Ag/ats-gap-analyser">ATS Gap Analyser</a> is for people who apply to jobs and never find out what went wrong.</p><p>You paste a job URL and your CV. The app returns a match score out of 100, missing keywords, improvement suggestions, and a three-paragraph cover letter.</p><p>The app is live on <a href="https://ats-gap-analyser.streamlit.app/">Streamlit</a>. It doesn&#8217;t store the CV or the job description, not even in the logs.</p><p>The agent has four tools.</p><ul><li><p>Extract job requirements</p></li><li><p>Score the CV</p></li><li><p>Suggest improvements</p></li><li><p>Generate the cover letter</p></li></ul><p>Suggestions retrieve from 50 ATS best-practice documents with minsearch.</p><ul><li><p>Eval used 50 manually written cases.</p></li><li><p>Version 1 had 60% recall and 30% accuracy, and version 6 reached 88% recall.</p></li><li><p>Amar kept accuracy lower on purpose: he would rather flag a missing skill than miss it.</p></li></ul><p><a href="https://www.linkedin.com/in/amar-agrawal-/">Amar&#8217;s LinkedIn</a></p><h2><strong>3) Chess Coach Agent by Leo Cabibihan</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!klGD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!klGD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 424w, https://substackcdn.com/image/fetch/$s_!klGD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 848w, https://substackcdn.com/image/fetch/$s_!klGD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 1272w, https://substackcdn.com/image/fetch/$s_!klGD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!klGD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png" width="1226" height="835" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:835,&quot;width&quot;:1226,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Chess Coach practice screen showing a chess position, an engine-approved move, and the next scheduled review date&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Chess Coach practice screen showing a chess position, an engine-approved move, and the next scheduled review date" title="Chess Coach practice screen showing a chess position, an engine-approved move, and the next scheduled review date" srcset="https://substackcdn.com/image/fetch/$s_!klGD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 424w, https://substackcdn.com/image/fetch/$s_!klGD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 848w, https://substackcdn.com/image/fetch/$s_!klGD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 1272w, https://substackcdn.com/image/fetch/$s_!klGD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F143b1974-6104-4494-8d6c-7c8ab4afa385_1226x835.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Chess Coach practice screen with a Stockfish-backed move recommendation and a scheduled retry</figcaption></figure></div><p>Leo&#8217;s first project was a chess-coach chat, but it wasn&#8217;t useful. <a href="https://github.com/leo-cabibihan/chess-coach-agent">Chess Coach Agent</a> starts from your real games instead. You import Lichess, Chess.com, or a PGN file. The app finds weak moments, explains them, and drills those positions.</p><p>Leo imported about 2,700 of his own Lichess games for the demo.</p><p>The stack includes FastAPI, React, Postgres with pgvector, and PydanticAI. It also uses Stockfish, BM25 lessons, Logfire, and it&#8217;s <a href="https://chess-coach-agent.onrender.com/">deployed on Render</a>.</p><p>Practice uses spaced repetition at 1, 3, and 7 days, doubling up to 30. If there&#8217;s no API key, deterministic templates still run. CI uses a TestModel so tests stay offline.</p><p><a href="https://www.linkedin.com/in/leo-cabibihan/">Leo&#8217;s LinkedIn</a></p><h2><strong>4) Document Preparation Agent by Paulien Out and Alena Fojtik</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!aI9w!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aI9w!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 424w, https://substackcdn.com/image/fetch/$s_!aI9w!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 848w, https://substackcdn.com/image/fetch/$s_!aI9w!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 1272w, https://substackcdn.com/image/fetch/$s_!aI9w!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!aI9w!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png" width="740" height="300" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:300,&quot;width&quot;:740,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;ArXiv papers, manuals, and EU documents feed an LLM for parsing, classification, metadata extraction, and chunking before becoming JSON manifests&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="ArXiv papers, manuals, and EU documents feed an LLM for parsing, classification, metadata extraction, and chunking before becoming JSON manifests" title="ArXiv papers, manuals, and EU documents feed an LLM for parsing, classification, metadata extraction, and chunking before becoming JSON manifests" srcset="https://substackcdn.com/image/fetch/$s_!aI9w!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 424w, https://substackcdn.com/image/fetch/$s_!aI9w!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 848w, https://substackcdn.com/image/fetch/$s_!aI9w!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 1272w, https://substackcdn.com/image/fetch/$s_!aI9w!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ce8f757-47fe-4f49-9f1a-765972d504e4_740x300.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">ArXiv papers, manuals, and EU documents are parsed by LLM so they can be used in a RAG pipeline</figcaption></figure></div><p>Paulien and Alena build on-premise AI systems. They created <a href="https://github.com/PaulienOut/rag-dataprep-agent">Document Preparation Agent</a>, which is RAG with extra metadata. This metadata is needed for filtering: their clients often want to add query qualifiers like &#8220;only the latest documents&#8221; or &#8220;only files written by person X&#8221;.</p><p>Because the customer data is private, the demo they used public arXiv papers, manuals, and EU documents. For v1 they focused only on PDFs only. There&#8217;s also no UI, only CLI.</p><p>Each manifest has author and document type, plus a summary, keywords, and numbered chunks.</p><p><a href="https://www.linkedin.com/in/paulien-out-023a708/">Paulien&#8217;s LinkedIn</a> and <a href="https://www.linkedin.com/in/alena-fojtik-ab439999/">Alena&#8217;s LinkedIn</a></p><h2><strong>5) Research Radar by Hana Ben Ali</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!YBck!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YBck!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 424w, https://substackcdn.com/image/fetch/$s_!YBck!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 848w, https://substackcdn.com/image/fetch/$s_!YBck!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 1272w, https://substackcdn.com/image/fetch/$s_!YBck!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YBck!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png" width="1396" height="1127" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1127,&quot;width&quot;:1396,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Research Radar dashboard showing paper rating distribution and papers saved for later&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Research Radar dashboard showing paper rating distribution and papers saved for later" title="Research Radar dashboard showing paper rating distribution and papers saved for later" srcset="https://substackcdn.com/image/fetch/$s_!YBck!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 424w, https://substackcdn.com/image/fetch/$s_!YBck!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 848w, https://substackcdn.com/image/fetch/$s_!YBck!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 1272w, https://substackcdn.com/image/fetch/$s_!YBck!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d24aa92-477d-420b-9a4e-b24eb83746e9_1396x1127.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Research Radar dashboard showing star-rating distribution and papers saved for later</figcaption></figure></div><p>Hana is a structural engineer and a PhD student. She needs to stay of top of new research, but it&#8217;s very difficult: there are 100&#8211;300 new papers per day on arXiv. Scanning all the titles and abstracts manually takes hours, and relevant work is easy to miss.</p><p>Her <a href="https://github.com/55382/Research_Radar_agent_phd_assistant">Research Radar</a> solves this problem. It fetches the new papers every morning, ranks them against her interests, and emails her the best matches.</p><p>The app also builds her interest profile based on the papers she already read, and the new recommendations are based on it.</p><p>It works in two stages:</p><ol><li><p>Semantic search with cosine similarity and an author boost, keep top 15.</p></li><li><p>Re-rank with an LLM ranker to get top 5.</p></li></ol><p>The top 5 go out by email through Resend.</p><p>Stars from the email write back to <code>ground_truth_papers.csv</code>, which is both the knowledge base and the eval set.</p><p>She started from 26 papers she had rated herself. After tuning a 0.6 semantic / 0.4 LLM blend, NDCG@5 went from 0.871 to 0.923. A Streamlit dashboard shows timestamps, average score, and ratings over time. The hardest part was making the ranker get better each morning.</p><p><a href="https://www.linkedin.com/in/hana-ben-ali-357b1a18b">Hana&#8217;s LinkedIn</a></p><p>The other submissions weren&#8217;t presented live. Here they are.</p><h2><strong>6) AI Diet Coach by Thet Su Win</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JxS6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JxS6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 424w, https://substackcdn.com/image/fetch/$s_!JxS6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 848w, https://substackcdn.com/image/fetch/$s_!JxS6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 1272w, https://substackcdn.com/image/fetch/$s_!JxS6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JxS6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;AI Diet Coach weekly meal plan showing Southeast Asian dishes, meal tracking, and nutrition totals&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="AI Diet Coach weekly meal plan showing Southeast Asian dishes, meal tracking, and nutrition totals" title="AI Diet Coach weekly meal plan showing Southeast Asian dishes, meal tracking, and nutrition totals" srcset="https://substackcdn.com/image/fetch/$s_!JxS6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 424w, https://substackcdn.com/image/fetch/$s_!JxS6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 848w, https://substackcdn.com/image/fetch/$s_!JxS6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 1272w, https://substackcdn.com/image/fetch/$s_!JxS6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0604f074-a5b9-46f0-8842-cf757199f811_1536x864.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">AI Diet Coach weekly plan with Southeast Asian meals, tracking state, and nutrition totals</figcaption></figure></div><p><a href="https://github.com/thetsuwin66/ai-diet-coach-agent">AI Diet Coach</a> is a meal planner for Southeast Asian users. Most Western diet apps ignore this group, that&#8217;s why Thet decided to create an app.</p><p>You set goals, restrictions, cuisines, and location. The agent builds a 7-day plan from 256 recipes: 201 from TheMealDB and 55 Asian dishes added by the author. It tracks meals and weight, swaps meals, and builds a shopping list.</p><p>It also looks up USDA nutrition data and suggests nearby restaurants through Google Maps. The app is live on <a href="https://ai-diet-coach-agent-htbjcoo2pylrfhdyv2cjbm.streamlit.app/">Streamlit</a>.</p><p>A judge went through four versions on 88 labeled answers, and accuracy moved from 46.6% to 73.9%.</p><p><a href="https://sg.linkedin.com/in/thet-su-win-169221172">Thet Su Win&#8217;s LinkedIn</a></p><h2><strong>7) AI Learning OS by Wesley Tan</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8lDG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8lDG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 424w, https://substackcdn.com/image/fetch/$s_!8lDG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 848w, https://substackcdn.com/image/fetch/$s_!8lDG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 1272w, https://substackcdn.com/image/fetch/$s_!8lDG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8lDG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png" width="1440" height="810" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:810,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;AI Learning OS queue showing staged learning resources and knowledge-base progress counts&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="AI Learning OS queue showing staged learning resources and knowledge-base progress counts" title="AI Learning OS queue showing staged learning resources and knowledge-base progress counts" srcset="https://substackcdn.com/image/fetch/$s_!8lDG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 424w, https://substackcdn.com/image/fetch/$s_!8lDG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 848w, https://substackcdn.com/image/fetch/$s_!8lDG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 1272w, https://substackcdn.com/image/fetch/$s_!8lDG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F100ac2ca-028d-4c5f-b9c8-1868efc266ad_1440x810.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">AI Learning OS queue of staged resources, with knowledge-base and progress counts in the sidebar</figcaption></figure></div><p>Wesley wanted to solve the watch-later graveyard problem. Self-learners save dozens of videos and articles across YouTube playlists, browser bookmarks, and note-taking apps, but most of this material is never used.</p><p>They do finish some content, but it rarely becomes structured knowledge that&#8217;s possible to use later.</p><p>To solve that, Wesley created <a href="https://github.com/wesleytanjiale/ai-learning-os">AI Learning OS</a>.</p><p>It separates saved content from learned content. You queue YouTube videos, articles, markdown, and images. After a learning walkthrough, the system writes only what you actually studied into a knowledge base. Chat search cites that base and reports progress.</p><p>The agent has six tools</p><ul><li><p>Add to queue</p></li><li><p>Browse queue</p></li><li><p>Load for learning</p></li><li><p>Consolidate to the knowledge base</p></li><li><p>Search</p></li><li><p>Get progress</p></li></ul><p>Retrieval is hybrid BM25 plus dense vectors with reciprocal rank fusion. A 400-word chunk size with 5 results scored 15/15 on the tuning set. 600-word chunks with 7 results scored 12/15. An empty-knowledge-base test checks that the agent doesn&#8217;t invent content.</p><p><a href="https://sg.linkedin.com/in/wesley-tan-jia-le-b31871219">Wesley&#8217;s LinkedIn</a></p><h2><strong>8) Applied ML Teaching Copilot by Marco Teran</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zLgB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zLgB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 424w, https://substackcdn.com/image/fetch/$s_!zLgB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 848w, https://substackcdn.com/image/fetch/$s_!zLgB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 1272w, https://substackcdn.com/image/fetch/$s_!zLgB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zLgB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png" width="1140" height="650" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:650,&quot;width&quot;:1140,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Applied ML Teaching Copilot answer citing course material aml-001 and showing the tool-call panel&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Applied ML Teaching Copilot answer citing course material aml-001 and showing the tool-call panel" title="Applied ML Teaching Copilot answer citing course material aml-001 and showing the tool-call panel" srcset="https://substackcdn.com/image/fetch/$s_!zLgB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 424w, https://substackcdn.com/image/fetch/$s_!zLgB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 848w, https://substackcdn.com/image/fetch/$s_!zLgB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 1272w, https://substackcdn.com/image/fetch/$s_!zLgB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca583c6f-31ea-4fda-93e1-8f8c63976eb6_1140x650.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Applied ML Teaching Copilot answers questions based on Applied ML course materials</figcaption></figure></div><p>Marco&#8217;s <a href="https://github.com/marcoteran/applied-ml-teaching-copilot">Applied ML Teaching Copilot</a> answers questions based on the Applied ML course materials. You can search, fetch a record by id, and get citations.</p><p>The stack is Streamlit, OpenAI, minsearch, PydanticAI, and Logfire. There&#8217;s a <a href="https://applied-ml-teaching-copilot.streamlit.app/">Streamlit demo</a>.</p><p>The first judge scored 0.727 accuracy. The calibrated judge reached 1.0 on 33 labeled cases.</p><p><a href="https://www.linkedin.com/in/marcoteran">Marco&#8217;s LinkedIn</a></p><h2><strong>9) Datawarehouse Agent by Lars van Asseldonk</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5dqt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5dqt!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 424w, https://substackcdn.com/image/fetch/$s_!5dqt!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 848w, https://substackcdn.com/image/fetch/$s_!5dqt!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 1272w, https://substackcdn.com/image/fetch/$s_!5dqt!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5dqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png" width="1440" height="875" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:875,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Datawarehouse Agent answer showing an incident count, success flags, and the read-only SQL query used&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Datawarehouse Agent answer showing an incident count, success flags, and the read-only SQL query used" title="Datawarehouse Agent answer showing an incident count, success flags, and the read-only SQL query used" srcset="https://substackcdn.com/image/fetch/$s_!5dqt!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 424w, https://substackcdn.com/image/fetch/$s_!5dqt!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 848w, https://substackcdn.com/image/fetch/$s_!5dqt!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 1272w, https://substackcdn.com/image/fetch/$s_!5dqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9f188dc-b070-4be6-9843-144a9ff2d1ec_1440x875.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Datawarehouse Agent answer with success flags and the read-only SQL used for the incident count</figcaption></figure></div><p>Lars&#8217;s <a href="https://github.com/larsvasseldonk/datawarehouse_agent">Datawarehouse Agent</a> is a natural-language-to-SQL assistant over Dutch Railways (NS) station-safety incidents. Non-technical staff ask a question and get a plain-language answer plus the SQL.</p><p>Lars used PydanticAI, OpenAI, DuckDB, and Streamlit. He also used Logfire, minsearch, and Plotly, and he recorded a <a href="https://github.com/user-attachments/assets/9534b98f-0aab-442b-97e7-491813ba3e81">demo video</a>.</p><h2><strong>10) WorkerChronicle by Miki Foster</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!o831!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!o831!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 424w, https://substackcdn.com/image/fetch/$s_!o831!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 848w, https://substackcdn.com/image/fetch/$s_!o831!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 1272w, https://substackcdn.com/image/fetch/$s_!o831!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!o831!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png" width="1456" height="860" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:860,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;WorkerChronicle project overview describing its worker's experience library and cover-letter generator&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="WorkerChronicle project overview describing its worker's experience library and cover-letter generator" title="WorkerChronicle project overview describing its worker's experience library and cover-letter generator" srcset="https://substackcdn.com/image/fetch/$s_!o831!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 424w, https://substackcdn.com/image/fetch/$s_!o831!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 848w, https://substackcdn.com/image/fetch/$s_!o831!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 1272w, https://substackcdn.com/image/fetch/$s_!o831!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38128697-e9b0-4746-aee5-ebc097921be2_1904x1124.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">WorkerChronicle&#8217;s experience library and cover-letter generator</figcaption></figure></div><p>When applying for jobs, people often struggle to remember concrete examples from their past work and adapt them to each role. Miki built <a href="https://github.com/MikiYamFos/work-chronicle">WorkerChronicle</a> to help with this.</p><p>You use can use it when applying for jobs:</p><ul><li><p>Before applying, you import your resume, previous cover letters, and raw notes to build a library of projects and accomplishments.</p></li><li><p>When you find a job, you give it the job description. The app finds relevant examples from your library, shows where evidence is missing, and creates an editable cover-letter outline.</p></li><li><p>Before an interview, it uses the same material to prepare likely questions and remind you which examples to discuss.</p></li></ul><p><a href="https://www.linkedin.com/in/mikifoster">Miki&#8217;s LinkedIn</a></p><h2><strong>11) Multi-Agent Software Development Platform by Nitesh Mishra</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!OUF-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OUF-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 424w, https://substackcdn.com/image/fetch/$s_!OUF-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 848w, https://substackcdn.com/image/fetch/$s_!OUF-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 1272w, https://substackcdn.com/image/fetch/$s_!OUF-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!OUF-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png" width="1400" height="1000" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1000,&quot;width&quot;:1400,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Multi-Agent Software Builder showing PM agent user stories, acceptance criteria, and a review step&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Multi-Agent Software Builder showing PM agent user stories, acceptance criteria, and a review step" title="Multi-Agent Software Builder showing PM agent user stories, acceptance criteria, and a review step" srcset="https://substackcdn.com/image/fetch/$s_!OUF-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 424w, https://substackcdn.com/image/fetch/$s_!OUF-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 848w, https://substackcdn.com/image/fetch/$s_!OUF-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 1272w, https://substackcdn.com/image/fetch/$s_!OUF-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f76a323-3877-4649-8b16-1a5e8896a1d8_1400x1000.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Multi-Agent Software Builder after the PM stage, with user stories and acceptance criteria awaiting review</figcaption></figure></div><p>Nitesh&#8217;s <a href="https://github.com/mishranitesh/AI_Engineering_Buildcamp_From_RAG_to_Agents/tree/main/capstone/multi-agent-dev-platform">Multi-Agent Software Development Platform</a> turns a plain-English requirement into user stories, architecture, and backend code.</p><p>It also writes tests, reviews the generated code, and packages the result. Optionally, it can create a JIRA epic with stories and tasks, or open a draft GitHub pull request.</p><p>Each requirement goes through the same steps:</p><ul><li><p>PM agent creates user stories and acceptance criteria.</p></li><li><p>Human confirmation approves the plan before code generation.</p></li><li><p>Architect agent designs the system and creates a Mermaid diagram.</p></li><li><p>Developer agent uses RAG to generate the backend code.</p></li><li><p>QA agent writes the tests.</p></li><li><p>Review agent uses RAG to find problems in the code.</p></li><li><p>Artifact generation packages the code, tests, architecture, and README into a ZIP file.</p></li></ul><p><a href="https://www.linkedin.com/in/mishra-nitesh">Nitesh&#8217;s LinkedIn</a></p><h2><strong>12) LearnMate AI by Dianne Bronola</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hxmN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hxmN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 424w, https://substackcdn.com/image/fetch/$s_!hxmN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 848w, https://substackcdn.com/image/fetch/$s_!hxmN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 1272w, https://substackcdn.com/image/fetch/$s_!hxmN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hxmN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png" width="1456" height="1166" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1166,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;LearnMate active review session showing a question, a student answer, and evaluated feedback&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="LearnMate active review session showing a question, a student answer, and evaluated feedback" title="LearnMate active review session showing a question, a student answer, and evaluated feedback" srcset="https://substackcdn.com/image/fetch/$s_!hxmN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 424w, https://substackcdn.com/image/fetch/$s_!hxmN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 848w, https://substackcdn.com/image/fetch/$s_!hxmN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 1272w, https://substackcdn.com/image/fetch/$s_!hxmN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8932363a-69d1-461e-adc0-334b365cf8e5_1504x1204.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">LearnMate active review session with a generated question, student answer, and evaluated feedback</figcaption></figure></div><p>Dianne built <a href="https://github.com/dinobronx/learnmate-ai">LearnMate AI</a> to help her learn better from YouTube videos.</p><p>Many of us have this problem: you watch a tutorial, understand it at the time, and forget most of it a week later. You can rewatch the video to refresh that knowledge, but it&#8217;s slow and ineffective. LearnMate helps with that problem.</p><p>You give it a video, it extracts concepts from it, connects them into a knowledge map. When the map is ready, it asks you questions about each concept, gives hints when you get stuck, and generates coding exercises if you need.</p><p>The stack includes FastAPI, React, PydanticAI, SQLite, ChromaDB, and OpenAI.</p><p></p><h2><strong>13) GapFinder by Katja Weber</strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Q2Of!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Q2Of!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 424w, https://substackcdn.com/image/fetch/$s_!Q2Of!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 848w, https://substackcdn.com/image/fetch/$s_!Q2Of!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 1272w, https://substackcdn.com/image/fetch/$s_!Q2Of!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Q2Of!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png" width="1456" height="785" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:785,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;GapFinder report separating understood concepts from missed concepts and listing video timestamps to revisit&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="GapFinder report separating understood concepts from missed concepts and listing video timestamps to revisit" title="GapFinder report separating understood concepts from missed concepts and listing video timestamps to revisit" srcset="https://substackcdn.com/image/fetch/$s_!Q2Of!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 424w, https://substackcdn.com/image/fetch/$s_!Q2Of!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 848w, https://substackcdn.com/image/fetch/$s_!Q2Of!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 1272w, https://substackcdn.com/image/fetch/$s_!Q2Of!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f8bb0c-083a-49fe-8ee2-c1647b53434d_1800x970.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">GapFinder report separating understood concepts from missed concepts and listing timestamps to revisit</figcaption></figure></div><p>Katja&#8217;s <a href="https://github.com/katjaweb/gapfinder">GapFinder</a> is another tool to help with studying. You give it a link to a YouTube tutorial, it fetches the transcript, chunks it, and asks diagnostic questions.</p><p>Then it grades your answers against the video and reports:</p><ul><li><p>what you understood vs what you missed</p></li><li><p>which sections you should rewatch</p></li></ul><p>The stack is PydanticAI, OpenAI, Streamlit, Logfire, and minsearch.</p><p><a href="https://www.linkedin.com/in/katjaweber/">Katja&#8217;s LinkedIn</a></p><p></p><h2><strong>New Cohort</strong></h2><p>I&#8217;m happy to see all these projects. At the same time, I want more participants to make it to the finish line and graduate with a completed project.</p><p>This is one of my main priorities for the next cohort. I&#8217;m actively researching what helps learners stay on track and finish with ambitious projects. I&#8217;ll apply what I learn to the course. You&#8217;ll see the results in the upcoming cohort.</p><p>The next cohort of the <a href="https://maven.com/alexey-grigorev/from-rag-to-agents">AI Engineering Bootcamp</a> starts on September 21.</p><p>If you want to build a system like these, join us. Use code &#8220;SUBSTACK&#8221; to get 20% off.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://maven.com/alexey-grigorev/from-rag-to-agents&quot;,&quot;text&quot;:&quot;AI Engineering Buildcamp&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://maven.com/alexey-grigorev/from-rag-to-agents"><span>AI Engineering Buildcamp</span></a></p><p></p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[AI Dev Tools Zoomcamp 2026 Starts Today]]></title><description><![CDATA[From Idea to Production with AI Coding Agents]]></description><link>https://aishippingblog.com/p/ai-dev-tools-zoomcamp-2026-starts</link><guid isPermaLink="false">https://aishippingblog.com/p/ai-dev-tools-zoomcamp-2026-starts</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Mon, 31 Aug 2026 12:54:09 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/4c572dda-473c-4302-8758-5142bd8782b9_1536x1024.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The 2026 cohort of <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a> starts today (August 31). The course is free, and registration is open for the next few weeks.</p><p>In this course, we use AI coding tools across the full software-development lifecycle.</p><ul><li><p>Part 1: <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a></p></li><li><p>Part 2: <a href="https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with">Build and Ship a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 3: <a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Deploy a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 4: <a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">DevOps and Observability for an AI-Built App</a></p></li><li><p>Part 5: <a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a></p></li></ul><p>In this course we give you a tool-agnostic way to work with any coding agent and increate your productivity as a developer.</p><p>Watch the launch stream:</p><div id="youtube2-YEsrF3EX4Hw" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;YEsrF3EX4Hw&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/YEsrF3EX4Hw?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h2><strong>Start Here</strong></h2><p>Use these links to join the course and find the materials:</p><ul><li><p><a href="https://courses.datatalks.club/register/ai-dev-tools/">Register for the course</a>. Registration is free.</p></li><li><p><a href="https://courses.datatalks.club/ai-dev-tools-2026/">Open the course platform</a> to see deadlines, submit homework and projects.</p></li><li><p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">Star the course repository</a>. It contains the module materials, recordings, homework, and final-project requirements.</p></li><li><p><a href="https://datatalks.club/docs/courses/ai-dev-tools-zoomcamp/">Read the course documentation</a> for prerequisites, setup, logistics, and detailed guidance.</p></li><li><p><a href="https://datatalks.club/slack.html">Join the DataTalks.Club Slack</a> for talking to your peer course participants and join the <code>#course-ai-dev-tools-zoomcamp</code> channel.</p></li><li><p><a href="https://t.me/aidevtoolszoomcamp">Join the Telegram channel</a> for announcements and deadline updates.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Q63d!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Q63d!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 424w, https://substackcdn.com/image/fetch/$s_!Q63d!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 848w, https://substackcdn.com/image/fetch/$s_!Q63d!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 1272w, https://substackcdn.com/image/fetch/$s_!Q63d!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Q63d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png" width="1456" height="454" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:454,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The AI Dev Tools Zoomcamp GitHub repository shows folders for the five course modules and a description of the course&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The AI Dev Tools Zoomcamp GitHub repository shows folders for the five course modules and a description of the course" title="The AI Dev Tools Zoomcamp GitHub repository shows folders for the five course modules and a description of the course" srcset="https://substackcdn.com/image/fetch/$s_!Q63d!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 424w, https://substackcdn.com/image/fetch/$s_!Q63d!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 848w, https://substackcdn.com/image/fetch/$s_!Q63d!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 1272w, https://substackcdn.com/image/fetch/$s_!Q63d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a5cf474-4936-4bbd-a235-741d82635dda_1540x480.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The repository is the source for module materials, homework, recordings, and project requirements</figcaption></figure></div><p></p><h2><strong>Course Curriculum</strong></h2><p>In this course, we use AI coding tools across the full software-development lifecycle.</p><ul><li><p>Part 1: <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a></p></li><li><p>Part 2: <a href="https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with">Build and Ship a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 3: <a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Deploy a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 4: <a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">DevOps and Observability for an AI-Built App</a></p></li><li><p>Part 5: <a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a></p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!lWGI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lWGI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 424w, https://substackcdn.com/image/fetch/$s_!lWGI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 848w, https://substackcdn.com/image/fetch/$s_!lWGI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 1272w, https://substackcdn.com/image/fetch/$s_!lWGI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!lWGI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png" width="1456" height="1543" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1543,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The curriculum page lists five modules covering AI-native workflow, full-stack development, deployment, DevOps and observability, and coding agent capabilities&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The curriculum page lists five modules covering AI-native workflow, full-stack development, deployment, DevOps and observability, and coding agent capabilities" title="The curriculum page lists five modules covering AI-native workflow, full-stack development, deployment, DevOps and observability, and coding agent capabilities" srcset="https://substackcdn.com/image/fetch/$s_!lWGI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 424w, https://substackcdn.com/image/fetch/$s_!lWGI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 848w, https://substackcdn.com/image/fetch/$s_!lWGI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 1272w, https://substackcdn.com/image/fetch/$s_!lWGI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd27b052d-5137-4f36-897e-9e4d8e21e64d_1536x1628.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The five modules take one application from specification through deployment, operations, and agent extensions</figcaption></figure></div><h2><strong>Module 1: AI-Native Developer Workflow</strong></h2><p>Giving a coding agent a vague request often produces a plausible application that doesn&#8217;t solve the problem you had in mind. We begin by improving the instructions and context we give it.</p><p>You turn a product idea into a specification and a backlog of small tasks. You create durable project context, then separate product management, implementation, and QA into focused sessions. You also learn how agent loops and multi-agent workflows can help with larger backlogs without removing your responsibility for review.</p><p>By the end of the module, you have a repeatable way to direct an agent and verify its work independently.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;4df66fbc-3af0-4ffb-9ce7-f6ae15bf813e&quot;,&quot;caption&quot;:&quot;This is the first article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;AI-Native Development: Specifications, Loop and Graph Engineering&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-07-22T06:59:57.781Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/youtube/w_728,c_limit/VUJxJGpaDEs&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/ai-native-development-specifications&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:208017204,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:69,&quot;comment_count&quot;:1,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><ul><li><p><a href="https://aishippingblog.com/p/ai-native-development-specifications">Companion article</a></p></li><li><p><a href="https://www.youtube.com/watch?v=VUJxJGpaDEs">Recording</a></p></li><li><p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/01-ai-native-workflow">Module 1 materials</a></p></li><li><p>Homework: <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/blob/main/cohorts/2026/homework/01-ai-native-workflow/homework.md">Instructions</a> &#183; <a href="https://courses.datatalks.club/ai-dev-tools-2026/homework/hw1">Submit</a> &#8212; due 2026-09-07</p></li></ul><h2><strong>Module 2: Build and Ship an AI-Assisted Full-Stack App</strong></h2><p>Next, you use that way of working to create a complete application. You write a product specification and build a frontend. Then you describe the API with OpenAPI, implement the backend, add SQLite persistence, and test the main behavior.</p><p>We use AI tools to produce the first version of each part, but we don&#8217;t accept generated code because it looks reasonable. We look at the code, run it, compare it with the specification, and write tests for the behavior we expect.</p><p>You finish with a tested full-stack application that runs locally and follows a documented OpenAPI specification.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;2ffdcffe-a767-46d3-9b5a-fc378f6927aa&quot;,&quot;caption&quot;:&quot;This is the second article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Build and Ship a Full-Stack App with AI Coding Assistants&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-08-03T12:59:20.309Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!I96k!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:209595653,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:17,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><ul><li><p><a href="https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with">Companion article</a></p></li><li><p><a href="https://www.youtube.com/watch?v=x9dq5nBpDg8">Recording</a></p></li><li><p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/02-development">Module 2 materials</a></p></li><li><p>Homework: <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/blob/main/cohorts/2026/homework/02-development/homework.md">Instructions</a> &#183; <a href="https://courses.datatalks.club/ai-dev-tools-2026/homework/hw2">Submit</a> &#8212; due 2026-09-14</p></li></ul><h2><strong>Module 3: Test, Containerize, and Deploy</strong></h2><p>A working local application is only the beginning. In this module, you test the seams between the frontend, backend, and database. You add integration tests, move from SQLite to Postgres, package the application with Docker, and run the checks automatically in CI.</p><p>Then you deploy the application to a public URL and connect deployment to the CI pipeline. When you merge a change that passes the required checks, the pipeline ships it.</p><p>You finish with a containerized application that other people can use and a documented way to test, deploy, and roll it back.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;c3eed8f7-e594-49fa-b7af-24df15f9b76e&quot;,&quot;caption&quot;:&quot;This is the third article in a series for AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Deploy a Full-Stack App with AI Coding Assistants&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-08-10T08:32:29.598Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!4sUJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:210571363,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:18,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><ul><li><p><a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Companion article</a></p></li><li><p><a href="https://www.youtube.com/watch?v=gxt5ZDVnBMM">Recording</a></p></li><li><p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/03-deployment">Module 3 materials</a></p></li><li><p>Homework: <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/blob/main/cohorts/2026/homework/03-deployment/homework.md">Instructions</a> &#183; <a href="https://courses.datatalks.club/ai-dev-tools-2026/homework/hw3">Submit</a> &#8212; due 2026-09-21</p></li></ul><h2><strong>Module 4: DevOps and Observability for AI-Built Apps</strong></h2><p>When the deployment command exits successfully, you know only that the command finished. You still need to check whether people can use the application.</p><p>You instrument an important request with OpenTelemetry and connect metrics, logs, and traces in Grafana. You write an alert for sustained user impact, collect a bounded evidence packet, and give a coding agent read-only access to that evidence. The agent can investigate and propose an action, while code outside the model enforces which actions are allowed.</p><p>You also run recurring security audits and record enough information to reconstruct an incident from the first alert to recovery.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;f69db83d-059f-4d1d-b3e7-def72298536b&quot;,&quot;caption&quot;:&quot;This is the fourth article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;DevOps and Observability for an AI-Built App&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-08-18T11:54:47.884Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!i4Rw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/devops-and-observability-for-an-ai&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:211674506,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:14,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><ul><li><p><a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">Companion article</a></p></li><li><p><a href="https://www.youtube.com/watch?v=YkxLo_FRoQw">Recording</a></p></li><li><p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/04-devops">Module 4 materials</a></p></li><li><p>Homework: <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/blob/main/cohorts/2026/homework/04-devops/homework.md">Instructions</a> &#183; <a href="https://courses.datatalks.club/ai-dev-tools-2026/homework/hw4">Submit</a> &#8212; due 2026-09-28</p></li></ul><h2><strong>Module 5: Coding Agent Capabilities</strong></h2><p>In the final module, you look inside the coding-agent workflow and extend it around your project.</p><p>You work with project instructions, MCP, and reusable skills. You also explore hooks, specialized subagents, plugins, and small custom agents. We focus on each capability rather than one product&#8217;s name for it, so you can apply the ideas across modern coding agents.</p><p>You finish by packaging reusable instructions and a repeatable way to review changes in your application. You also include a specialist agent, an MCP tool or server, guardrails, and documented permissions.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;a9a61812-ae80-4a8c-8648-a0978cae3e4a&quot;,&quot;caption&quot;:&quot;This is the fifth article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Coding Agent Building Blocks: Reusable Skills and Specialized Subagents&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-08-31T12:20:13.047Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!LxDz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/coding-agent-building-blocks-reusable&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:213529654,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:2,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><ul><li><p><a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Companion article</a></p></li><li><p><a href="https://www.youtube.com/watch?v=t8OrAjNO2Zs">Recording</a></p></li><li><p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/05-agent-capabilities">Module 5 materials</a></p></li><li><p>No homework</p></li></ul><p></p><h2><strong>Office Hours + AI Shipping Labs Membership</strong></h2><p>As a part of the course, we will also have office hours. During these office hours you can ask questions about the course (and beyond).</p><p>There will be 4 sessions on Tuesdays at 18:30 Berlin time (Sep 08, Sep 15, Sep 22 and Sep 29).</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://aishippinglabs.com/events/series/4/ai-dev-tools-office-hours&quot;,&quot;text&quot;:&quot;Join Office Hours&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://aishippinglabs.com/events/series/4/ai-dev-tools-office-hours"><span>Join Office Hours</span></a></p><p>Joining office hours requires a <a href="https://aishippinglabs.com/membership">paid membership on AI Shipping Labs</a>.</p><p>This course is completely free and there are no sponsors for this course, so I&#8217;m not earning anything by running it. If you want to support me, and also get more out of it, consider becoming an AI Shipping Labs member and joining office hours.</p><p>You will also get a personalized onboarding: we will discuss your current situation, your goals, and build a development plan for you.</p><p></p><h2><strong>Cohort Format</strong></h2><p>The lectures are pre-recorded, so you don&#8217;t need to attend a class at a particular time. In a live cohort, we move through the course together with shared deadlines, graded homework, and community support. You can also join the leaderboard, review peer projects, and work toward a certificate.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HWBw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HWBw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 424w, https://substackcdn.com/image/fetch/$s_!HWBw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 848w, https://substackcdn.com/image/fetch/$s_!HWBw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 1272w, https://substackcdn.com/image/fetch/$s_!HWBw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HWBw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png" width="1456" height="980" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:980,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:179319,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/213541554?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HWBw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 424w, https://substackcdn.com/image/fetch/$s_!HWBw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 848w, https://substackcdn.com/image/fetch/$s_!HWBw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 1272w, https://substackcdn.com/image/fetch/$s_!HWBw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63304cc6-bc4c-48e4-b0c7-5dd12d911188_1910x1286.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Course management platform: <a href="https://courses.datatalks.club/ai-dev-tools-2026/">https://courses.datatalks.club/ai-dev-tools-2026/</a></figcaption></figure></div><p>For each module:</p><ol><li><p>Open the module folder in the course repository.</p></li><li><p>Watch the recording and follow the materials.</p></li><li><p>Complete the homework in your own public repository.</p></li><li><p>Submit your answers through the course platform before the deadline.</p></li><li><p>Ask questions and share what you learn in Slack.</p></li></ol><p>Homework is optional for the certificate, but I recommend doing it. It gives you a smaller task for each module, helps you keep pace with the cohort, and prepares you for the final project. On-time homework and learning-in-public contributions also give you leaderboard points.</p><p>You can follow all the materials without joining the live cohort. If you study at your own pace, you can watch the same recordings, complete the homework for practice, and build a portfolio project. We offer graded submissions, peer review, and certificates during the live cohort.</p><p>For more information, read the guides to <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/course-work/">course work</a>, <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/homework/">homework</a>, and <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/joining-mid-cohort/">joining a cohort after it starts</a>.</p><h2><strong>Leaderboard</strong></h2><p>The leaderboard is optional: it doesn&#8217;t affect certification. It gives you a way to track your progress alongside the cohort and recognizes contributions beyond homework. You can earn points for submitting homework on time, learning in public, contributing to the FAQ, completing projects, and reviewing peer projects.</p><p>The screenshot below shows the <a href="https://courses.datatalks.club/ai-dev-tools-2025/leaderboard">AI Dev Tools Zoomcamp 2025 leaderboard</a>. Your leaderboard profile is anonymous by default, but you can make your name, bio, and links public from your enrollment profile.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!j-MV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!j-MV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 424w, https://substackcdn.com/image/fetch/$s_!j-MV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 848w, https://substackcdn.com/image/fetch/$s_!j-MV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 1272w, https://substackcdn.com/image/fetch/$s_!j-MV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!j-MV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png" width="896" height="745" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:745,&quot;width&quot;:896,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The 2025 AI Dev Tools Zoomcamp leaderboard lists participant names, positions, and scores&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The 2025 AI Dev Tools Zoomcamp leaderboard lists participant names, positions, and scores" title="The 2025 AI Dev Tools Zoomcamp leaderboard lists participant names, positions, and scores" srcset="https://substackcdn.com/image/fetch/$s_!j-MV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 424w, https://substackcdn.com/image/fetch/$s_!j-MV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 848w, https://substackcdn.com/image/fetch/$s_!j-MV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 1272w, https://substackcdn.com/image/fetch/$s_!j-MV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34994d34-7a3c-4129-b861-e8632c326fb9_896x745.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Last year&#8217;s leaderboard recognized steady course progress and community contributions</figcaption></figure></div><p>For more information, read the <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/leaderboard/">leaderboard documentation</a> for the point categories, timing, and profile options.</p><h2><strong>Learn in Public</strong></h2><p>As you work through the course, share what you build and learn. A short post about a useful idea, a problem you solved, or a screenshot of your project reinforces the material and creates a public record of your progress. It can also bring feedback, new connections, and opportunities.</p><p>After publishing a post, submit its link through the course platform with your homework, project, or peer review. Each accepted link earns a leaderboard point.</p><p>For more information, read the <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/learning-in-public/">learning-in-public guide</a> for the submission process, post ideas, and examples.</p><h2><strong>FAQ and Questions</strong></h2><p>Start with the <a href="https://datatalks.club/faq/ai-dev-tools-zoomcamp.html">AI Dev Tools Zoomcamp FAQ</a> when you run into a course-specific or technical problem. It collects answers from previous cohorts, so you may find the exact error or question already covered.</p><p>If the FAQ doesn&#8217;t answer your question, ask in the <code>#course-ai-dev-tools-zoomcamp</code> Slack channel. Include what you tried, the result you expected, the result you got, and enough context for someone else to reproduce the problem. When you find a useful answer that is missing from the FAQ, send it through the <a href="https://github.com/DataTalksClub/faq/issues/new?template=faq-proposal.yml">FAQ proposal form</a>. Accepted FAQ contributions earn leaderboard points.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!lDTS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lDTS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 424w, https://substackcdn.com/image/fetch/$s_!lDTS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 848w, https://substackcdn.com/image/fetch/$s_!lDTS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 1272w, https://substackcdn.com/image/fetch/$s_!lDTS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!lDTS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png" width="1456" height="694" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:694,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:164689,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/213541554?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!lDTS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 424w, https://substackcdn.com/image/fetch/$s_!lDTS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 848w, https://substackcdn.com/image/fetch/$s_!lDTS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 1272w, https://substackcdn.com/image/fetch/$s_!lDTS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15495d79-3b24-4b89-9959-1bafaac0bdd6_1942x926.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">FAQ available at <a href="https://datatalks.club/faq/ai-dev-tools-zoomcamp.html">https://datatalks.club/faq/ai-dev-tools-zoomcamp.html</a></figcaption></figure></div><p>For more information, read the <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/asking-questions/">guide to asking questions</a> for examples and troubleshooting guidance.</p><h2><strong>Final Project and Certificate</strong></h2><p>For the <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/project">final project</a>, you choose your own idea and build a complete application with AI assistance.</p><p>Your project should include:</p><ul><li><p>a frontend and backend</p></li><li><p>an OpenAPI specification</p></li><li><p>persistent storage</p></li><li><p>automated tests</p></li><li><p>containers and a public deployment</p></li><li><p>clear instructions for running the application</p></li></ul><p>You also document how you used AI. Show how you described tasks and gave the agent context. Explain how you reviewed generated code, tested the result, and handled permissions. In the later modules, you add operations, security, and agent-extension work around the application.</p><p>Homework isn&#8217;t required for the certificate. To earn the certificate, you need to submit a passing final project during the live cohort and complete the required peer reviews on time. Check the <a href="https://courses.datatalks.club/ai-dev-tools-2026/">course platform</a> for the exact project and review deadlines.</p><ul><li><p><a href="https://courses.datatalks.club/ai-dev-tools-2026/project/project1">Project 1</a>: submit by 2026-10-19 and complete peer reviews by 2026-10-26.</p></li><li><p><a href="https://courses.datatalks.club/ai-dev-tools-2026/project/project2">Project 2</a>: submit by 2026-11-09 and complete peer reviews by 2026-11-16.</p></li></ul><p>For more information, read the <a href="https://datatalks.club/docs/courses/ai-dev-tools-zoomcamp/project/">course-specific project requirements</a>, the <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/peer-review/">peer-review guide</a>, and the <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/certification/">certification guide</a>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DCnI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DCnI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 424w, https://substackcdn.com/image/fetch/$s_!DCnI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 848w, https://substackcdn.com/image/fetch/$s_!DCnI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 1272w, https://substackcdn.com/image/fetch/$s_!DCnI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DCnI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A 2025 AI Dev Tools Zoomcamp statement of accomplishment awarded to Rick Astley&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A 2025 AI Dev Tools Zoomcamp statement of accomplishment awarded to Rick Astley" title="A 2025 AI Dev Tools Zoomcamp statement of accomplishment awarded to Rick Astley" srcset="https://substackcdn.com/image/fetch/$s_!DCnI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 424w, https://substackcdn.com/image/fetch/$s_!DCnI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 848w, https://substackcdn.com/image/fetch/$s_!DCnI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 1272w, https://substackcdn.com/image/fetch/$s_!DCnI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78f3fed7-64b8-40da-b6e1-4c8c1e868887_2000x1125.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Participants who complete a passing project and the required peer reviews earn a certificate</figcaption></figure></div><p></p><h2><strong>Course Links</strong></h2><p>You don&#8217;t need to remember every link, but it helps to know where to go for each part of the course.</p><ul><li><p>Use the <a href="https://courses.datatalks.club/ai-dev-tools-2026/">course platform</a> for the schedule, deadlines, submission forms, scores, project, and certificate.</p></li><li><p>Use the <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">GitHub repository</a> for module materials, recordings, code, homework, and project requirements.</p></li><li><p>Use the <a href="https://datatalks.club/docs/courses/ai-dev-tools-zoomcamp/">course docs</a> for setup and detailed guidance, and the <a href="https://datatalks.club/docs/courses/zoomcamp-logistics/">Zoomcamp logistics docs</a> for the shared rules.</p></li><li><p>Use the <a href="https://app.slack.com/client/T01ATQK62F8/C09HWT76L95">course Slack channel</a> for questions and discussion.</p></li><li><p>Use <a href="https://t.me/aidevtoolszoomcamp">Telegram</a> for announcements.</p></li><li><p>Use the <a href="https://www.youtube.com/playlist?list=PL3MmuxUbc_hLuyafXPyhTdbF4s_uNhc43">YouTube playlist</a> for recordings.</p></li></ul><h2><strong>Start the Course</strong></h2><p>Open the <a href="https://courses.datatalks.club/ai-dev-tools-2026/">course platform</a>, star the <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">course repository</a>, and join <a href="https://datatalks.club/slack.html">Slack</a> and <a href="https://t.me/aidevtoolszoomcamp">Telegram</a>. Then begin with <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/01-ai-native-workflow">Module 1</a>.</p><p>You can still join after August 31. Check the platform for the deadlines that remain, and start working through the materials.</p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[Coding Agent Building Blocks: Reusable Skills and Specialized Subagents]]></title><description><![CDATA[This is the fifth article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.]]></description><link>https://aishippingblog.com/p/coding-agent-building-blocks-reusable</link><guid isPermaLink="false">https://aishippingblog.com/p/coding-agent-building-blocks-reusable</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Mon, 31 Aug 2026 12:20:13 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!LxDz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>This is the fifth article in a series based on <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a>, the free course we run at DataTalks.Club.</p><p>All articles in the series:</p><ul><li><p>Part 1: <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a></p></li><li><p>Part 2: <a href="https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with">Build and Ship a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 3: <a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Deploy a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 4: <a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">DevOps and Observability for an AI-Built App</a></p></li><li><p>Part 5: <a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a> (this article)</p></li></ul><p>In this article, I want to cover the two most important aspects of working with coding assistants: skills and subagents.</p><ul><li><p>A skill describes how to perform a repeatable task.</p></li><li><p>A subagent is an agent that runs in a separate context with clear task instructions.</p></li></ul><p>In addition, I&#8217;ll cover my process for creating skills and show how to run multiple agents in parallel and turn your coding agent into a task orchestrator.</p><p>Getting these concepts is enough to work with coding agents productively, and you most likely won&#8217;t need anything else.</p><p>This article is loosely based on a live workshop I did for the course. In the workshop, I had to improvise, so the content doesn&#8217;t directly map to this article. But it&#8217;s still going to be useful, so check it out.</p><div id="youtube2-t8OrAjNO2Zs" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;t8OrAjNO2Zs&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/t8OrAjNO2Zs?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h2><strong>From Markdown Files to Skills and Subagents</strong></h2><p>In Part 1 of the course, <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a>, we touched on both skills and subagents, but I didn&#8217;t define them explicitly.</p><p>Specifically, we created a few documents:</p><ul><li><p><code>_docs/process.md</code> and the other project documents contained reusable instructions (see <a href="https://github.com/alexeygrigorev/retroloop/tree/main/_docs">here</a>)</p></li><li><p><code>_docs/team/pm.md</code>, <code>_docs/team/software-engineer.md</code>, and <code>_docs/team/qa-engineer.md</code> described specialized roles for subagents (see <a href="https://github.com/alexeygrigorev/retroloop/tree/main/_docs/team">here</a>).</p></li></ul><p>I wanted to keep the discussion high-level and not focus on the mechanics of any specific implementation.</p><p>We didn&#8217;t explicitly define them because we can tell a coding assistant:</p><pre><code><code>Implement feature #101 following _docs/process.md</code></code></pre><p>It will read the document and follow the steps there.</p><p>Or you can also specify the role of the agent at the beginning of a session:</p><pre><code><code>You're a QA engineer. Your role is defined in `_docs/team/qa-engineer.md`. Review work for task #101.</code></code></pre><p>Even better, you can ask it to start a subagent with this role:</p><pre><code><code>Start a QA engineer subagent. The role is defined in `_docs/team/qa-engineer.md`. Review work for task #101.</code></code></pre><p>It will work. The coding assistant can just go read the file and do the task.</p><p>But if you define these things as skills and subagents explicitly, you&#8217;ll be more effective in using them. In this article, I&#8217;ll show you how.</p><p>Let&#8217;s start with skills.</p><h2><strong>Building Block 1: Skills</strong></h2><p>A skill is a structured set of instructions for a repeatable procedure.</p><p>If you need to follow a specific sequence of actions for completing a task, you can document the steps in a skill. Then later you can ask the agent to use this skill to perform a task. It will read the file and follow the steps from there.</p><h3><strong>My Skills</strong></h3><p>I use skills a lot.</p><p>For example, one of my skills is for releasing a new version of a Python package. I maintain a lot of Python libraries that I need to regularly update and publish. I describe how I do it in <a href="https://aishippingblog.com/p/my-pypi-release-pipeline-for-python">My PyPI Release Pipeline for Python Libraries</a>.</p><p>It&#8217;s a process with multiple steps:</p><ul><li><p>Run the tests</p></li><li><p>Update the version in pyproject.toml</p></li><li><p>Push the code to GitHub as a tag</p></li><li><p>Verify that CI publishes the package</p></li></ul><p>If I want to release a new version of <a href="https://aishippingblog.com/p/minsearch-the-small-search-library">minsearch</a>, I don&#8217;t want to describe these steps each time I do it. Instead, I document this in a <a href="https://github.com/alexeygrigorev/.agents/blob/main/skills/release/SKILL.md">release skill</a> and just say:</p><pre><code><code>release a new version</code></code></pre><p>The agent understands what I need, reads the skill, and follows the steps there.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SeJc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SeJc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 424w, https://substackcdn.com/image/fetch/$s_!SeJc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 848w, https://substackcdn.com/image/fetch/$s_!SeJc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 1272w, https://substackcdn.com/image/fetch/$s_!SeJc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SeJc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png" width="1456" height="370" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:370,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A request to release minsearch prompts the coding agent to load the release skill and prepare version 0.2.0&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A request to release minsearch prompts the coding agent to load the release skill and prepare version 0.2.0" title="A request to release minsearch prompts the coding agent to load the release skill and prepare version 0.2.0" srcset="https://substackcdn.com/image/fetch/$s_!SeJc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 424w, https://substackcdn.com/image/fetch/$s_!SeJc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 848w, https://substackcdn.com/image/fetch/$s_!SeJc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 1272w, https://substackcdn.com/image/fetch/$s_!SeJc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F209fb2c7-cfa3-49fa-bcc9-69b5115ac01e_1851x470.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>The agent recognizes the release request and loads the relevant skill without an explicit skill command</span></p><p>Internally, a skill is a markdown file that looks like this:</p><pre><code><span>---</span>
<span>name</span>: <span>release</span>
<span>description</span>: <span>Release a new version of the current project</span>
<span>---</span>

Release

<span>1.</span> Inspect the project and determine its package registry.
<span>2.</span> Run the tests and other required checks.
<span>3.</span> Bump the version in the source of truth.
<span>4.</span> Commit the change and push a version tag.
<span>5.</span> Verify that the CI publishing job succeeds.</code></pre><p>The main difference between a plain markdown file that we used previously and a skill is that the skill is automatically discoverable.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DMY9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DMY9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 424w, https://substackcdn.com/image/fetch/$s_!DMY9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 848w, https://substackcdn.com/image/fetch/$s_!DMY9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 1272w, https://substackcdn.com/image/fetch/$s_!DMY9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DMY9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png" width="1456" height="948" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:948,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:266209,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/213529654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DMY9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 424w, https://substackcdn.com/image/fetch/$s_!DMY9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 848w, https://substackcdn.com/image/fetch/$s_!DMY9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 1272w, https://substackcdn.com/image/fetch/$s_!DMY9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef44b32b-59a5-4b3c-9281-8ee96e7fc89d_2024x1318.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">My actual <a href="https://github.com/alexeygrigorev/.agents/blob/main/skills/release/SKILL.md">release skill</a></figcaption></figure></div><p>I can say:</p><pre><code><code>Make a release. The process is described in _docs/release.md</code></code></pre><p>And that will absolutely work. The agent will read the file and follow the steps there.</p><p>However, with skills I don&#8217;t need to say which file to use. The coding agent loads the list of skills into its context at startup. When I need to release a library, it can see that there&#8217;s a skill for that, so it goes ahead and uses it.</p><p><span>The agent can discover the right skill from the request, while an explicit command remains available when you want direct control</span></p><p>A skill is a markdown document, but in order to make it discoverable, we need to follow a convention:</p><ul><li><p>It must have a frontmatter section with a name and description</p></li><li><p>It&#8217;s saved in <code>.agents/skills/&lt;NAME&gt;/SKILL.md</code></p></li></ul><p>And that&#8217;s it - the rest is up to you. You can also add other markdown documents or scripts together with the <code>SKILL.md</code> file in the same directory and describe how the agent should use them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JV7u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JV7u!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 424w, https://substackcdn.com/image/fetch/$s_!JV7u!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 848w, https://substackcdn.com/image/fetch/$s_!JV7u!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 1272w, https://substackcdn.com/image/fetch/$s_!JV7u!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JV7u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png" width="1456" height="760" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:760,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:374431,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/213529654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!JV7u!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 424w, https://substackcdn.com/image/fetch/$s_!JV7u!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 848w, https://substackcdn.com/image/fetch/$s_!JV7u!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 1272w, https://substackcdn.com/image/fetch/$s_!JV7u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522ed877-f944-4e86-b344-59ddc6382eb6_2812x1468.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Most coding agents support the Agent Skills convention (except Claude)</figcaption></figure></div><p>The <a href="https://agentskills.io/specification">Agent Skills specification</a> describes this structure and the format in more detail.</p><p>If you want to better understand how skills work, the best way is to build a small coding agent with skills. In the AI Shipping Labs workshop <a href="https://aishippinglabs.com/workshops/coding-agent-v2">Build a Coding Agent with Tools, Skills and PydanticAI</a> I show you how. We first learn about agents and function calls, and then see how to implement a special function for loading skills.</p><h3><strong>Global Skills</strong></h3><p>There are some skills that all your projects need. But some are only needed for local work.</p><p>Examples of global skills:</p><ul><li><p>I use <a href="https://github.com/alexeygrigorev/.agents/tree/main/skills/release">release</a> for releasing a new version of a library, <a href="https://github.com/alexeygrigorev/.agents/tree/main/skills/init-library">init-library</a> to scaffold a new library or <a href="https://github.com/alexeygrigorev/.agents/tree/main/skills/create-github-repo">create-github-repo</a> to create a GitHub repository</p></li><li><p>I also <a href="https://github.com/alexeygrigorev/.agents/tree/main/skills/fetch-youtube">fetch-youtube</a>, <a href="https://github.com/alexeygrigorev/.agents/tree/main/skills/fetch-loom">fetch-loom</a>, and <a href="https://github.com/alexeygrigorev/.agents/tree/main/skills/fetch-zoom">fetch-zoom</a>, to fetch recordings or transcripts for the agent to analyze.</p></li><li><p>For creating diagrams, I have a <a href="https://github.com/alexeygrigorev/.agents/tree/main/skills/diagram-creator">diagram-creator</a> skill.</p></li></ul><p>I use these skills across multiple projects, so it&#8217;s better to make them global, so any agent on my computer can see and use them.</p><p>To create global skills, you need to put them in</p><ul><li><p><code>~/.codex/skills</code> for Codex</p></li><li><p><code>~/.claude/skills</code> for Claude</p></li><li><p>other coding assistants have a similar structure</p></li></ul><p>I use multiple assistants, so I want all of them to use the same skills. For that, I maintain one canonical copy of the skills in my <a href="https://github.com/alexeygrigorev/.agents">.agents repository</a>, and then create symlinks from there to <code>~/.codex/skills</code>, <code>~/.claude/skills</code> and other places.</p><h3><strong>Project-Specific Skills</strong></h3><p>Most skills are not useful globally though - they are project-specific.</p><p>For example, in my projects:</p><ul><li><p><a href="https://github.com/DataTalksClub/course-management-platform/tree/main/.agents/skills">Course Management Platform</a> has an <a href="https://github.com/DataTalksClub/course-management-platform/tree/main/.agents/skills/cmp-incident-response">incident-response</a> skill. Every time there&#8217;s an alert, it knows how to investigate and fix the problem, similar to what we covered in <a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">Article 4</a>.</p></li><li><p><a href="https://github.com/DataTalksClub/faq/tree/main/.agents/skills">DataTalks.Club FAQ</a> has skills for fetching course questions from Slack and turning recurring questions into FAQ entries.</p></li></ul><p>They work in the same way as global skills, but you place them in your project root:</p><ul><li><p><code>./.agents/skills/</code> for all the assistants except Claude</p></li><li><p><code>./.claude/skills/</code> for Claude Code</p></li></ul><p>It&#8217;s annoying that Claude has its own format - just like with <code>CLAUDE.md</code>.</p><p>I solve this problem by creating a symlink from <code>.claude/skills/</code> to <code>.agents/skills/</code>.</p><p>I hope Anthropic starts listening to its users and adopts the <code>.agents</code> convention too, so I can finally delete all these symlinks.</p><h3><strong>Creating Skills</strong></h3><p>You don&#8217;t need to create these skills manually. Coding assistants know the format, so you can ask them to create these skills.</p><p>However, I don&#8217;t just start a session and say &#8220;create a skill for releasing a new version of this library&#8221;. Usually, I do the task with the agent first and see where I need to correct it.</p><p>We work together until the task is done, and then at the end I say:</p><pre><code><code>Save this as a skill. Cover the workflow we followed and keep all the corrections I made in mind, so the next agent can do the task correctly from the start.

This skill is specific to this project, so don't create a global skill.</code></code></pre><p>If I don&#8217;t include the last line, Codex will create a global skill, so I always have to add it. I didn&#8217;t observe this behavior from other agents.</p><p>Improving an existing skill works in the same way. I trigger a skill, and if I see that something requires my intervention, I correct the agent. At the end of the session, I ask to update the process in the skill, so I don&#8217;t have to do it next time.</p><h2><strong>Building Block 2: Subagents</strong></h2><p>Most coding assistants allow you to start one session within another. We call the agents created this way &#8220;subagents&#8221;.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LxDz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LxDz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 424w, https://substackcdn.com/image/fetch/$s_!LxDz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 848w, https://substackcdn.com/image/fetch/$s_!LxDz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!LxDz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LxDz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A meme reads, 'Yo dawg I heard you like agents, so I put an agent inside your agents, so you can prompt while you prompt'&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A meme reads, 'Yo dawg I heard you like agents, so I put an agent inside your agents, so you can prompt while you prompt'" title="A meme reads, 'Yo dawg I heard you like agents, so I put an agent inside your agents, so you can prompt while you prompt'" srcset="https://substackcdn.com/image/fetch/$s_!LxDz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 424w, https://substackcdn.com/image/fetch/$s_!LxDz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 848w, https://substackcdn.com/image/fetch/$s_!LxDz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!LxDz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5507188-f9e1-4108-94b3-6ae943a75425_1536x1024.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Subagents let one coding agent prompt another agent inside its own session</span></p><h3><strong>Session Context</strong></h3><p>The context for a session is all the information the agent has about the current task and all the actions it has taken.</p><p>It includes:</p><ul><li><p>AGENTS.md file</p></li><li><p>Skills and tools definitions</p></li><li><p>All the files it read</p></li><li><p>All the code it read and wrote</p></li><li><p>All the text you sent it and all the text it output</p></li></ul><p>That&#8217;s a lot of information. As the session gets longer, the agent starts &#8220;forgetting&#8221; things from the beginning. This is called &#8220;context rot&#8221;.</p><p>That&#8217;s why I start a new focused session for each task. In the article <a href="https://aishippingblog.com/p/from-idea-to-production">From Idea to Production in 28 Prompts</a>, I show the steps you need to take your idea to production and recommend starting a new session for each step.</p><p>Also, the current context will influence the future actions of the agent. If the agent implemented a feature, you shouldn&#8217;t ask the agent to test it in the same window. The implementation history will bias the agent, and it can skip some checks. Instead, it&#8217;s better to start a fresh session and ask the agent there to check the work.</p><p></p><h3><strong>Agentic Team</strong></h3><p>This is what we did in Part 1 of the course, <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a>. We defined multiple roles for our agentic team:</p><ul><li><p>Product manager (PM) - turns raw intake into specifications</p></li><li><p>Software engineer (SWE) - implements the task based on the specifications</p></li><li><p>QA engineer (QA) - validates that the task is implemented correctly</p></li></ul><p>Each of these agents lives in a separate session. The context from SWE never gets into the QA session, so the QA engineer can test the system independently.</p><p>If I have an issue #101 that&#8217;s already groomed and I want to implement it, I can start a new session and say:</p><pre><code><code>You're a software engineer agent. Your role is described in `_docs/team/swe.md`. Work on issue #101.</code></code></pre><p>But instead, I can also launch it as a subagent:</p><pre><code><code>Launch a subagent to implement #101. Follow the process.
</code></code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zjS4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zjS4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 424w, https://substackcdn.com/image/fetch/$s_!zjS4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 848w, https://substackcdn.com/image/fetch/$s_!zjS4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 1272w, https://substackcdn.com/image/fetch/$s_!zjS4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zjS4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png" width="1456" height="596" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:596,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A coding assistant shows a main session with a backgrounded general-purpose subagent implementing issue 101&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A coding assistant shows a main session with a backgrounded general-purpose subagent implementing issue 101" title="A coding assistant shows a main session with a backgrounded general-purpose subagent implementing issue 101" srcset="https://substackcdn.com/image/fetch/$s_!zjS4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 424w, https://substackcdn.com/image/fetch/$s_!zjS4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 848w, https://substackcdn.com/image/fetch/$s_!zjS4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 1272w, https://substackcdn.com/image/fetch/$s_!zjS4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9515cd0f-3696-4b41-95f7-226afe471441_1910x782.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>The main session can keep working while a subagent handles a focused task in the background</span></p><p>It will create a new agent with an empty context (plus AGENTS.md) and automatically prompt it with something like this:</p><pre><code><code>You're a software engineer. Your role is defined in `_docs/team/swe.md`.
Your task is to work on issue #101. Follow the process in `_docs/process.md`.</code></code></pre><p>In some coding assistants, you&#8217;ll also be able to look inside the subagent sessions. In others, you&#8217;ll only know that the subagent is doing something, but you won&#8217;t know what exactly.</p><p></p><h3><strong>Defining Subagents</strong></h3><p>The coding assistant can read the agent definition from a markdown document (like <code>_docs/team/swe.md</code>) and launch a subagent.</p><p>Just like with skills, you can also create a reusable and discoverable subagent definition.</p><p>For example, for the software engineer, the definition could be:</p><pre><code><span>---</span>
<span>name</span>: <span>software-engineer</span>
<span>description</span>: <span>Use this agent to execute a well-scoped coding assignment.</span>
<span>---</span>

Implement the assigned task, add or update tests, run the project checks, and
report the files changed and any remaining risks. Do not expand the scope.</code></pre><p>Or even this:</p><pre><code><span>---</span>
<span>name</span>: <span>software-engineer</span>
<span>description</span>: <span>Use this agent to execute a well-scoped coding assignment.</span>
<span>---</span>

Read your role definition from <span>_</span>docs/team/swe.md</code></pre><p>You put these definitions in:</p><ul><li><p><code>.agents/agents/&lt;NAME&gt;.md</code> for the majority of assistants</p></li><li><p><code>.claude/agents/&lt;NAME&gt;.md</code> for Claude</p></li></ul><p>You can also define subagents globally, but I never needed it. My subagents are always project-specific.</p><p>Now you can ask:</p><pre><code><code>Launch software engineer for #101</code></code></pre><p>And it will start the software engineer agent.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!U88l!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U88l!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 424w, https://substackcdn.com/image/fetch/$s_!U88l!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 848w, https://substackcdn.com/image/fetch/$s_!U88l!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 1272w, https://substackcdn.com/image/fetch/$s_!U88l!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!U88l!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png" width="1456" height="647" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:647,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A coding assistant starts a software-engineer agent to implement issue 101 in the background&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A coding assistant starts a software-engineer agent to implement issue 101 in the background" title="A coding assistant starts a software-engineer agent to implement issue 101 in the background" srcset="https://substackcdn.com/image/fetch/$s_!U88l!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 424w, https://substackcdn.com/image/fetch/$s_!U88l!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 848w, https://substackcdn.com/image/fetch/$s_!U88l!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 1272w, https://substackcdn.com/image/fetch/$s_!U88l!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19b68093-eed1-4bc4-a0f3-f10eda1123fc_1890x840.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>A subagent can take on a specialized role, such as software engineering</span></p><p>Note that for some coding agents like Claude, you&#8217;ll have to restart the session to discover the agent you just defined.</p><h3><strong>Orchestration Session</strong></h3><p>When you start using subagents, your main session becomes the orchestrator.</p><p>If we have a process specified in <code>process.md</code>:</p><ul><li><p>PM grooms the issue.</p></li><li><p>SWE implements it.</p></li><li><p>QA checks it.</p></li><li><p>If the issue passes the check, QA outputs <code>PASS</code>; otherwise, it outputs <code>FAIL</code>.</p></li><li><p>If it outputs <code>FAIL</code>, SWE needs to reimplement it until QA accepts it.</p></li></ul><p>This way, we create a graph.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RpIZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 424w, https://substackcdn.com/image/fetch/$s_!RpIZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 848w, https://substackcdn.com/image/fetch/$s_!RpIZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 1272w, https://substackcdn.com/image/fetch/$s_!RpIZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RpIZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png" width="1179" height="191" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/109fdaee-084a-4510-9871-334fd00265fc_1179x191.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:191,&quot;width&quot;:1179,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The PM grooms an issue, the engineer implements it, and QA either completes the task or sends a failed check back to the engineer&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The PM grooms an issue, the engineer implements it, and QA either completes the task or sends a failed check back to the engineer" title="The PM grooms an issue, the engineer implements it, and QA either completes the task or sends a failed check back to the engineer" srcset="https://substackcdn.com/image/fetch/$s_!RpIZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 424w, https://substackcdn.com/image/fetch/$s_!RpIZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 848w, https://substackcdn.com/image/fetch/$s_!RpIZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 1272w, https://substackcdn.com/image/fetch/$s_!RpIZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F109fdaee-084a-4510-9871-334fd00265fc_1179x191.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p><span>The workflow from Part 1 is a graph: PM hands the issue to engineering, QA closes passing work, and failures loop back for reimplementation</span></p><p>This sequence needs to be enforced by something. Typically, we use the main coding session for that.</p><p>If we describe the process in our <code>process.md</code> document and link it in <code>AGENTS.md</code>, then we can ask:</p><pre><code><code>Implement #101</code></code></pre><p>It will automatically run #101 through the PM -&gt; SWE -&gt; QA sequence, each step will run in its own fresh context, and the orchestrator will make sure that on <code>FAIL</code> the task goes to SWE who will fix the problems.</p><h2><strong>Parallel Execution</strong></h2><p>In the first article, I only showed how to run this flow sequentially, and we can&#8217;t run it any other way: we always have to follow the PM -&gt; SWE -&gt; QA sequence.</p><p>But we can take care of multiple independent issues in parallel.</p><p>Our main agent is already the orchestrator, so it can run multiple subagents in parallel. But we need to make sure that the agents aren&#8217;t stepping on each other&#8217;s toes.</p><p>For that, we use git worktrees:</p><ul><li><p>For each new task, create a new git worktree</p></li><li><p>The agents work independently in these worktrees</p></li><li><p>You go through the entire PM -&gt; SWE -&gt; QA cycle in that worktree</p></li><li><p>When the cycle is over and the task is done, the orchestrator merges it back into main (alternatively, you can have a conflict resolver agent)</p></li><li><p>The worktree is deleted</p></li></ul><p>The orchestrator&#8217;s role here becomes more than just launching subagents in order.</p><p>It also needs to:</p><ul><li><p>Keep track of the status of each task</p></li><li><p>Schedule the agents</p></li><li><p>Create and merge worktrees</p></li><li><p>Prevent logical conflicts by picking tasks that don&#8217;t need to work on the same files</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ljue!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ljue!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 424w, https://substackcdn.com/image/fetch/$s_!Ljue!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 848w, https://substackcdn.com/image/fetch/$s_!Ljue!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Ljue!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ljue!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png" width="970" height="600" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:600,&quot;width&quot;:970,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The orchestrator assigns independent issues to as many as five isolated worktrees, then merges approved branches into main one at a time&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The orchestrator assigns independent issues to as many as five isolated worktrees, then merges approved branches into main one at a time" title="The orchestrator assigns independent issues to as many as five isolated worktrees, then merges approved branches into main one at a time" srcset="https://substackcdn.com/image/fetch/$s_!Ljue!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 424w, https://substackcdn.com/image/fetch/$s_!Ljue!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 848w, https://substackcdn.com/image/fetch/$s_!Ljue!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Ljue!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c28769f-eb75-42d4-bb71-04f213b96f16_970x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Agents work concurrently in isolated worktrees while the orchestrator controls review and merge order</span></p><p>To set it up, I put something like this in the <code>process.md</code> file:</p><pre><code><code>Work in isolated git worktrees and run up to five agents in parallel.

You are the orchestrator. Your job is to:

- Split the backlog into independent issue tracks.
- Give each agent a separate worktree and clear file ownership.
- Keep the agent slots full while eligible work remains.
- Send completed work to a fresh reviewer.
- Return review findings to the correct implementer.
- Merge approved work into the main branch one issue at a time.
- Coordinate the work. Never implement the issues yourself.</code></code></pre><p>This is the approach I use for the majority of my projects.</p><h2><strong>Summary</strong></h2><ul><li><p>Use a skill when you repeat the same sequence of actions.</p></li><li><p>Use a subagent when a task needs a separate context or a specialized role.</p></li><li><p>Use the main session as an orchestrator when several agents need to follow a process.</p></li><li><p>Use git worktrees when independent tasks need to run in parallel.</p></li></ul><p>If you want to learn more about using AI for development see the <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a> course.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[From Idea to Production in 28 Prompts]]></title><description><![CDATA[Build, deploy and operate your app with AI coding agents]]></description><link>https://aishippingblog.com/p/from-idea-to-production</link><guid isPermaLink="false">https://aishippingblog.com/p/from-idea-to-production</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 28 Aug 2026 14:21:53 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Yzr9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a> is the free course we run at DataTalks.Club.</p><p>Across the series we build an application with AI coding agents from the idea to making it production-ready.</p><p>I showed how to do it in 3 stages:</p><ul><li><p><a href="https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with">Stage 1: Build</a> - coming up with an idea, creating frontend and backend, and running it locally</p></li><li><p><a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Stage 2: Deploy</a> - making the application testable and deployable, and setting up CI/CD</p></li><li><p><a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">Stage 3: Operate</a> - adding dev/prod environments, collecting metrics, logs, and traces, and having AI agents react to incidents</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Yzr9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Yzr9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 424w, https://substackcdn.com/image/fetch/$s_!Yzr9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 848w, https://substackcdn.com/image/fetch/$s_!Yzr9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 1272w, https://substackcdn.com/image/fetch/$s_!Yzr9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Yzr9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png" width="1456" height="756" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:756,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:972637,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://aishippingblog.com/i/213149828?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Yzr9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 424w, https://substackcdn.com/image/fetch/$s_!Yzr9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 848w, https://substackcdn.com/image/fetch/$s_!Yzr9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 1272w, https://substackcdn.com/image/fetch/$s_!Yzr9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9542a620-6599-4814-b08d-233e11b6e18d_1740x904.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In this article I&#8217;ll put everything together, so you can use it as a reference for developing your projects. I&#8217;ll keep it tool-agnostic: I don&#8217;t assume any specific coding agent, programming language, framework, cloud, or observability tool.</p><p>It doesn&#8217;t really matter which assistant you use. You can choose Codex, Claude Code, OpenCode, Grok Build, Cursor, OpenCode, or anything else you like.</p><p>I show a sequence of steps you can follow, one after another, to take your product from a vague idea to production.</p><p>Each step is a prompt that you can put directly into your coding assistant. In most cases, it helps to start a new session for each step: this way, only the important things are in your agent&#8217;s context so it can finish the work faster and better.</p><p>Let&#8217;s go!</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h2><strong>Stage 1: Build</strong></h2><p>This is the first stage of our process.</p><p>In the build stage, we:</p><ul><li><p>Turn the raw idea into a concrete specification</p></li><li><p>Create frontend and backend, and connect them</p></li></ul><h3><strong>1. Specification</strong></h3><p>If you give a coding agent a vague idea, it&#8217;ll fill in the gaps with its own assumptions, and you&#8217;ll get an app you didn&#8217;t want.</p><p>It&#8217;s better to be very clear about what you want and capture this in a document called &#8220;specification&#8221;.</p><p>I usually use ChatGPT for that: I open it in dictation mode and do a brain dump with my idea, and then we iterate until it&#8217;s clear what it could look like.</p><p>You can start your discussion with your AI assistant using this prompt:</p><pre><code><code>I want to build a tool for forming study groups.

Help me set the scope for this project precisely. I want to brainstorm with you and understand how the tool should work. Give me options.

Ask me one question at a time and keep your output short.</code></code></pre><p>At the end, ask the assistant to summarize the discussion into a markdown document.</p><p>Create an empty folder for your project, and save the specs into <code>_docs/specs.md</code>.</p><p>Use this document to build the frontend.</p><p></p><h3><strong>2. Frontend</strong></h3><p>I build the frontend before the backend so I can test the idea quickly. It&#8217;s the least expensive way: you create a clickable interface and you test it with some mock data. If it looks the way you want, you can continue.</p><p>Put this into your coding assistant (or tools like Lovable, v0 or Bolt.new):</p><pre><code><code>Create a system design interview application.

[Paste the ChatGPT-generated specification here.]

Centralize every backend call in one services layer, and create a mock implementation of it so the whole app runs without a real backend.

Add tests.</code></code></pre><p>Save the frontend code in the <code>frontend/</code> directory.</p><p>After you have iterated on the frontend and like it, it&#8217;s time to define how it will interact with the backend.</p><p></p><h3><strong>3. Manual test</strong></h3><p>Come up with a test you can use to validate that your application works. This should be a sequence of actions you perform to ensure the app works the way you want.</p><p>For an app that helps you find study partners, it could be something like:</p><ol><li><p>Open the home page, sign up</p></li><li><p>Create a group for studying</p></li><li><p>Create an event there</p></li><li><p>Share the event</p></li><li><p>Sign up to the event from another browser session</p></li><li><p>Verify that the sign up is visible in the first session</p></li></ol><p>You can use AI to come up with this scenario, but remember it - and maybe save it to <code>_docs</code> too.</p><p></p><h3><strong>4. AGENTS.md</strong></h3><p>AGENTS.md is loaded into the agent&#8217;s context every time you start a new session. You can add anything that you think is important for your app. For now let&#8217;s add one line there:</p><pre><code><code>Make focused, regular commits, and explain your decisions in the commit messages so we can use them as logs</code></code></pre><p>Later you can add more things here.</p><p>If you use Claude, create <code>CLAUDE.md</code> with a single line:</p><pre><code><code>@AGENTS.md</code></code></pre><p></p><h3><strong>5. OpenAPI schema</strong></h3><p>We previously asked the assistant to centralize all backend calls in one place. Now let&#8217;s use this to create the API specification. We will use this specification to define frontend-backend interaction:</p><pre><code><code>Read the frontend's API client in frontend/

Create openapi.yaml at the repository root that defines every endpoint, method, path, request body, response body, and which endpoints need authentication.</code></code></pre><p>Now we have the specification for our backend so we can build it.</p><p></p><h3><strong>6. Choose the backend stack</strong></h3><p>Before actually building it, you need to decide which technology you want to use for this application.</p><p>Ask the coding assistant to help:</p><pre><code><code>Based on openapi.yaml, propose 2-3 backend stack options. Describe pros and cons for each. Recommend one.</code></code></pre><p>You can challenge your AI assistant and ask it some questions, or just accept the option it suggests.</p><p>At the end, save the results to <code>_docs/stack.md</code>.</p><p></p><h3><strong>7. Backend</strong></h3><p>We have selected the technology, so let&#8217;s use it to build the backend.</p><p>Just like we created the frontend with mocked backend calls, we can create the backend with a mocked database. First, focus on making sure the frontend-backend integration works, and then connect a proper database.</p><pre><code><code>Build backend in backend/ based on `_docs/stack.md` that implements the openapi.yaml spec.
Use an in-memory store and seed it with data so the frontend has something to show.
Add authentication with hashed passwords and bearer tokens for the endpoints that need it.
Write tests.</code></code></pre><p></p><h3><strong>8. Makefile</strong></h3><p>I always have problems remembering how to start the backend, how to run the frontend, install the dependencies, and do other things. For that, I usually use Makefiles, so I can just type:</p><pre><code><code>make backend</code></code></pre><p>And the backend runs. Let&#8217;s define a Makefile:</p><pre><code><code>Create a Makefile so I can easily run the application.</code></code></pre><p></p><h3><strong>9. Connect frontend and backend</strong></h3><p>We created a backend but the frontend still serves the mock data. Let&#8217;s connect them:</p><pre><code><code>Switch the frontend to use the real backend client.</code></code></pre><p>Use the test scenario from before to make sure it works.</p><p>You may do a few iterations here. Test the application and ask the agent to fix the problems you discover until it works correctly.</p><p></p><h3><strong>10. Database</strong></h3><p>We don&#8217;t have a real database yet. Let&#8217;s add it. I usually use SQLite for local development, so I&#8217;ll use it here too.</p><pre><code><code>Replace the in-memory store with a database. Use SQLite with ORM, so later we can easily switch to a different database. </code></code></pre><p>Instead of SQLite, your application may need something else, so you can chat with your coding assistant about it before you implement it.</p><p>After it&#8217;s done, use the test scenario again and iterate until it fully works.</p><p></p><h2><strong>Stage 2: Deploy</strong></h2><p>Our app works locally. Great.</p><p>Let&#8217;s deploy it and make it available for others to use:</p><ul><li><p>Containerize it</p></li><li><p>Deploy it to the cloud</p></li><li><p>Automate the deployment with CI/CD</p></li></ul><p></p><h3><strong>11. Containerization</strong></h3><p>We need to put our application in a container, so we can later use any container management system for deploying it.</p><p>In our local environment, we have two processes: one for the frontend, one for the backend. We need to run them separately. In practice, you usually compile the frontend into static files and serve them with the backend.</p><p>So, instead of creating separate containers for the frontend and backend, we&#8217;ll create one that serves both:</p><pre><code><code>Create a Dockerfile that builds the frontend with Node, then builds the backend. Backend should serve the frontend.</code></code></pre><p>Build and run the image, then verify the flow with the test scenario and iterate until it works.</p><p></p><h3><strong>12. Production database</strong></h3><p>In the previous stage I used SQLite. It helps with keeping the local setup lightweight, but it&#8217;s not a database we can use in production. Typically we use something like Postgres.</p><p>You can ask your agent to select the best database for your case, but it&#8217;s very likely it&#8217;ll be Postgres.</p><p>So let&#8217;s add it:</p><pre><code><code>Add Postgres support to the backend.</code></code></pre><p>Use the test scenario to make sure the app works.</p><p></p><h3><strong>13. Docker Compose</strong></h3><p>To make it easier to manage the local environment with all the services, we use Docker Compose.</p><p>Let&#8217;s ask the agent to set it up:</p><pre><code><code>Create docker-compose.yaml with two services: database and the app.</code></code></pre><p>Run it with <code>docker compose up</code> and test it.</p><p></p><h3><strong>14. Integration</strong></h3><p>It&#8217;s possible that by now you already have tests. AI coding assistants are instructed to create them. But maybe in your project there are still none or too few.</p><p>Let&#8217;s make it explicit and ask the assistant to create them.</p><p>We also want to have a specific category of tests: integration tests. These tests are used to make sure the interaction between backend and the database is correct.</p><p>You can also ask the assistant to help you with finding the right test scenarios to include in the test suite:</p><pre><code><code>Create integration tests that run against docker-compose.yaml. What scenarios should we test?</code></code></pre><p></p><h3><strong>15. End-to-end test</strong></h3><p>Integration tests verify that the backend can connect to the database.</p><p>End-to-end tests verify the entire user-backend flow: from the browser to the frontend, from the frontend to the backend, and from the backend to the database.</p><p>So far, we have been doing these tests ourselves manually. Let&#8217;s now automate it.</p><pre><code><code>Add an end-to-end test that runs against docker-compose.yaml.

Use Playwright to:

1. &lt;Describe the first step&gt;
2. &lt;Describe the second step&gt;
3. ...

Put the tests in the e2e/ folder in the repository root.</code></code></pre><p>Now we don&#8217;t need to repeat this test manually every time we make a change.</p><p></p><h3><strong>16. Deployment stack</strong></h3><p>Now the application works locally and is properly tested. It&#8217;s time to deploy it. But where? There are many options.</p><p>Let&#8217;s choose what works best for us. Ask the coding agent:</p><pre><code><code>Propose 2-3 ways to deploy the application. Compare cost, complexity and other factors.

&lt;List any other constraints that you have&gt;</code></code></pre><p>It&#8217;s also helpful to use an infrastructure as code (IaC) tool to manage your cloud resources:</p><pre><code><code>I want to use an infrastructure-as-code tool for managing my infra. What are my options?</code></code></pre><p>It&#8217;s possible that for the deployment option you chose, there are no infrastructure-as-code options. In this case, just move on.</p><p></p><h3><strong>17. Deployment</strong></h3><p>Now that we have selected the stack, let&#8217;s deploy.</p><p>Ask the agent:</p><pre><code><code>Deploy this application to &lt;Selected option&gt;.</code></code></pre><p>If applicable, add:</p><pre><code><code>use &lt;IaC tool&gt; for managing the infrastructure</code></code></pre><p></p><h3><strong>18. CI/CD pipeline</strong></h3><p>We can deploy a few times manually, but we want to automate it. It&#8217;s done with CI/CD:</p><ul><li><p>CI is continuous integration - running tests every time you push to git</p></li><li><p>CD is continuous deployment - automatically deploying when the tests pass</p></li></ul><pre><code><code>Create a CI/CD pipeline that:

- runs backend and frontend tests in parallel
- builds the Docker Compose stack and runs integration and end-to-end tests against it
- deploys the application
- validates that the deploy is successful by checking the health endpoint</code></code></pre><p>You probably use GitHub for hosting code, so it&#8217;s likely that you&#8217;ll use GitHub Actions. If you don&#8217;t like it, chat with AI to select a different CI/CD tool.</p><p>We successfully deployed our application, but we&#8217;re not done yet. We need to make it more production-ready.</p><p></p><h2><strong>Stage 3: Operate</strong></h2><p>To make it more production-ready, we need to add a few things:</p><ul><li><p>Create dev/prod environments</p></li><li><p>Capture metrics, logs, and traces</p></li><li><p>Set up alerts</p></li><li><p>Have an AI agent taking care of the alerts</p></li></ul><p>Let&#8217;s do that.</p><p></p><h3><strong>19. Second environment</strong></h3><p>Now each push brings the updates directly to the deployment. But what if it has a bug that we missed?</p><p>To avoid that, we have two environments:</p><ul><li><p>Dev: the environment for testing. We deploy here on every push</p></li><li><p>Prod: the environment that real users use. We promote a release from dev to prod only when we confirm that it works correctly</p></li></ul><p>If we use infrastructure as code, it&#8217;s trivial to set up. If not, let the agent handle it.</p><p>Either way, it&#8217;s enough to just ask the agent to do it and it&#8217;ll figure the rest out:</p><pre><code><code>Create a second, independent copy of our infrastructure. We will use the copy as production, and the existing infrastructure as a dev environment.</code></code></pre><p></p><h3><strong>20. Manual promotion workflow</strong></h3><p>Now we automatically release the code to development on every push. Let&#8217;s set manual release promotion from dev to prod:</p><pre><code><code>Create a manual GitHub Actions workflow that promotes the dev version to production.</code></code></pre><p></p><h3><strong>21. Split build/deploy</strong></h3><p>It&#8217;s possible that in your CI/CD the build and deploy happen in one step (it happened to me when I was deploying to AWS EC2).</p><p>This is an anti-pattern. We want to build the image only once, first deploy it to development, and use it for production too if it works fine.</p><p>If for you these steps are not separate, we need to split them. Ask AI if you&#8217;re not sure.</p><pre><code><code>Currently we build the image during the deploy stage.

Split it into two stages:

- Build: build the image and push it to a container registry
- Deploy: pull the image from the registry and serve it

The manual prod promotion CI/CD workflow pulls the currently deployed dev image to prod.

Tag each image using the YYYYMMDD-HHMMSS-shortsha pattern (e.g. "20260818-163457-83242da")</code></code></pre><p>Depending on where you deploy your application, you may have to use different options for the container registry. Talk to AI if you&#8217;re not sure what to select and how to do it.</p><p></p><h3><strong>22. Observability and OpenTelemetry</strong></h3><p>Our application is deployed and serving real users in production. But we don&#8217;t know how reliable it is.</p><p>We need data for that. That&#8217;s called telemetry, and we will collect metrics, logs, and traces. The industry standard is OpenTelemetry, so you&#8217;ll probably use it.</p><pre><code><code>Instrument the backend with OpenTelemetry.

Include in the telemetry:

- service name
- environment
- deployed version</code></code></pre><p>To &#8220;instrument&#8221; the backend means to include the code for capturing the telemetry in our application. But we also need to save it somewhere.</p><p></p><h3><strong>23. Choose the observability backend</strong></h3><p>There are many options for storing the telemetry and displaying the dashboards. Some are open-source but you need to self-host them. Some are fully managed, but you&#8217;ll need to pay for using them. Some of the self-hosted options give a generous free tier.</p><p>Ask the agent to help us select:</p><pre><code><code>Propose 2-3 options for storing the telemetry (metrics, logs, traces) via OTel and for displaying them on a dashboard.

&lt;If you have any preferences or restrictions add them here&gt;</code></code></pre><p></p><h3><strong>24. OTel Collector stack</strong></h3><p>Now we&#8217;ve selected the stack. If you decide to go with open-source and self-hosting, you can also deploy it locally:</p><pre><code><code>Add an OpenTelemetry Collector. Create "observability/" directory with Docker Compose for: &lt;list selected technologies&gt;. Keep this as a separate Compose project from the application stack.</code></code></pre><p>Then deploy it:</p><pre><code><code>Deploy the observability stack. It should be separate from the application stack. Connect both development and production to it.</code></code></pre><p>If you decide to go with a managed service, connect it:</p><pre><code><code>Add an OpenTelemetry Collector for &lt;selected technology&gt;. Help me configure it</code></code></pre><p></p><h3><strong>25. Application metrics</strong></h3><p>We configured OTel. Let&#8217;s collect important metrics.</p><p>They are application-dependent and depend on what&#8217;s important for your product.</p><p>You can also ask the agent to come up with the most important metrics:</p><pre><code><code>What are the most important product metrics I need to track in my application? Give me top 5</code></code></pre><p>Start tracking them:</p><pre><code><code>Track the following metrics with OTel

- &lt;List them&gt;</code></code></pre><p></p><h3><strong>26. Dashboards</strong></h3><p>We&#8217;re tracking the metrics but not displaying them yet.</p><p>Let&#8217;s fix that:</p><pre><code><code>Display the metrics we export in a dashboard. Make it possible to filter by environment and deployed version.</code></code></pre><p></p><h3><strong>27. Alerts</strong></h3><p>We can&#8217;t babysit the metrics in a dashboard. Instead, we can configure an alert that will fire every time a metric goes too low or too high.</p><pre><code><code>Add an actionable alert for &lt;most important metric&gt;. Include the service, environment, deployed version, owner, and dashboard URL in the alert.</code></code></pre><p>It&#8217;s also useful to track 5xx error codes and for each error send an alert (or configure a threshold that makes sense for your application). Talk to your AI assistant to set it up.</p><p></p><h3><strong>28. On-call worker</strong></h3><p>Now when alerts fire, we get an email or a Slack message. But we still need to act on them.</p><p>We can have AI watch the alerts and fix the errors when they happen.</p><p>Let&#8217;s configure it.</p><pre><code><code>Add an on-call-engineer/ directory with a script that connects to our alerts API.

When an alert fires, the script

- wakes up an on-call agent that runs in headless mode
- passes all the alert details

The on-call agent should be able to resolve the issue autonomously and commit the fix.</code></code></pre><p>Your on-call engineer may have instructions that look like this:</p><pre><code><code>You are the on-call engineer for this repository. An alert just fired.

Investigate the root cause. Read the code and reproduce the failure. If you find a real bug, make the smallest correction, run the backend tests, and commit the fix with a clear message.
If the alert is a false positive, explain why and do not change the code.</code></code></pre><p></p><h2><strong>Stage 4: Going Further</strong></h2><p>In this article, I gave you 28 concrete steps to bring your idea to production. If you follow them, you&#8217;ll have a working application that can serve real users in less than a week.</p><p>But there are still many things you need to think about:</p><ul><li><p>Do you use a container manager? If not, consider using one.</p></li><li><p>Do you use a managed database service? If not, consider using one.</p></li><li><p>Regularly back up your database and test that the backups actually work.</p></li><li><p>Learn about VPCs and see if you need one.</p></li><li><p>When you start getting traffic, learn about scaling and load balancing.</p></li></ul><p>Also, periodically ask Fable or GPT-5.6-Sol Max (preferably both) to audit your code for security vulnerabilities.</p><p>I&#8217;m only scratching the surface of a &#8220;real production system&#8221; in this article, but the stages and steps in this tutorial are more than enough to get started.</p><p>With this setup, you can use AI to build a product that runs reliably. The rest you can learn when you actually need it.</p><p>Have fun building!</p><p>And check our <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a> if you want to learn more about using AI to increase your development productivity.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[DevOps and Observability for an AI-Built App]]></title><description><![CDATA[Part 4 of AI Dev Tools Zoomcamp]]></description><link>https://aishippingblog.com/p/devops-and-observability-for-an-ai</link><guid isPermaLink="false">https://aishippingblog.com/p/devops-and-observability-for-an-ai</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Tue, 18 Aug 2026 11:54:47 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!i4Rw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>This is the fourth article in a series based on <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a>, the free course we run at DataTalks.Club.</p><p>All articles in the series:</p><ul><li><p>Part 1: <a href="https://aishippingblog.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a></p></li><li><p>Part 2: <a href="https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with">Build and Ship a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 3: <a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Deploy a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 4: <a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">DevOps and Observability for an AI-Built App</a> (this article)</p></li><li><p>Part 5: <a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a></p></li></ul><p>In part 2, we developed an application for conducting system design interviews. In part 3, we deployed it to a cloud environment and configured CI/CD.</p><p>In this article, we&#8217;ll make this deployment production-ready:</p><ul><li><p>Separate development and production environments</p></li><li><p>Create a manual CI/CD trigger for promoting the app from development to production</p></li><li><p>Use OpenTelemetry to collect metrics, traces, and logs and create</p></li><li><p>Define application-specific metrics and create a dashboard that displays them</p></li><li><p>Create an alert when the metrics change</p></li><li><p>Use an AI agent as the on-call responder for the alerts</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!O5gB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!O5gB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 424w, https://substackcdn.com/image/fetch/$s_!O5gB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 848w, https://substackcdn.com/image/fetch/$s_!O5gB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 1272w, https://substackcdn.com/image/fetch/$s_!O5gB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!O5gB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png" width="1080" height="720" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/18adc488-7028-4b99-8f96-a3511706623b_1080x720.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:720,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A user pushes a change through CI/CD to development; a release owner promotes the tested release to production, where telemetry flows through observability and alerting to an AI agent&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A user pushes a change through CI/CD to development; a release owner promotes the tested release to production, where telemetry flows through observability and alerting to an AI agent" title="A user pushes a change through CI/CD to development; a release owner promotes the tested release to production, where telemetry flows through observability and alerting to an AI agent" srcset="https://substackcdn.com/image/fetch/$s_!O5gB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 424w, https://substackcdn.com/image/fetch/$s_!O5gB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 848w, https://substackcdn.com/image/fetch/$s_!O5gB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 1272w, https://substackcdn.com/image/fetch/$s_!O5gB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adc488-7028-4b99-8f96-a3511706623b_1080x720.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Deploy changes to development first, promote a tested release to production, and respond to observed failures</figcaption></figure></div><p>We will continue using AWS and CloudFormation, but the principles we show in this article are tool-agnostic and will work for any environment.</p><div id="youtube2-YkxLo_FRoQw" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;YkxLo_FRoQw&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/YkxLo_FRoQw?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p></p><h2><strong>Recap</strong></h2><p>We started building the <a href="https://github.com/alexeygrigorev/interview-canvas-share">Interview Canvas project</a> in <a href="https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with">Part 2: Build and Ship a Full-Stack App with AI Coding Assistants</a>:</p><ul><li><p>We brainstormed with an AI assistant to come up with the specification</p></li><li><p>Based on that we created a React-based frontend with no backend</p></li><li><p>Next, we defined the frontend-backend OpenAPI schema</p></li><li><p>Using the API contract we created the backend</p></li><li><p>We added database support with SQLite and SQLAlchemy</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!lNCj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lNCj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 424w, https://substackcdn.com/image/fetch/$s_!lNCj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 848w, https://substackcdn.com/image/fetch/$s_!lNCj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 1272w, https://substackcdn.com/image/fetch/$s_!lNCj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!lNCj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png" width="900" height="320" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f80a35a6-2854-4151-8437-eb949182ba6e_900x320.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:320,&quot;width&quot;:900,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A user interacts with the frontend, which calls the FastAPI backend backed by SQLite&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A user interacts with the frontend, which calls the FastAPI backend backed by SQLite" title="A user interacts with the frontend, which calls the FastAPI backend backed by SQLite" srcset="https://substackcdn.com/image/fetch/$s_!lNCj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 424w, https://substackcdn.com/image/fetch/$s_!lNCj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 848w, https://substackcdn.com/image/fetch/$s_!lNCj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 1272w, https://substackcdn.com/image/fetch/$s_!lNCj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff80a35a6-2854-4151-8437-eb949182ba6e_900x320.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The React frontend calls the FastAPI backend, which stores application data in SQLite</figcaption></figure></div><p>The app was running locally, and then we deployed it in <a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Part 3: Deploy a Full-Stack App with AI Coding Assistants</a>:</p><ul><li><p>We containerized the application</p></li><li><p>Then added Postgres support</p></li><li><p>To make sure the system works reliably, we created integration and end-to-end tests</p></li><li><p>We deployed it to AWS</p></li><li><p>In the end, we set up CI/CD via GitHub Actions, so every push runs the tests and deploys the app to development</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7nZv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7nZv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 424w, https://substackcdn.com/image/fetch/$s_!7nZv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 848w, https://substackcdn.com/image/fetch/$s_!7nZv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 1272w, https://substackcdn.com/image/fetch/$s_!7nZv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7nZv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png" width="1456" height="618" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:618,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A successful CI/CD run that tests the application and deploys it to development&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A successful CI/CD run that tests the application and deploys it to development" title="A successful CI/CD run that tests the application and deploys it to development" srcset="https://substackcdn.com/image/fetch/$s_!7nZv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 424w, https://substackcdn.com/image/fetch/$s_!7nZv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 848w, https://substackcdn.com/image/fetch/$s_!7nZv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 1272w, https://substackcdn.com/image/fetch/$s_!7nZv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8957d80-a8ea-415c-a41a-778543f5ae22_1962x833.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">When we push code to main, it&#8217;s automatically deployed</figcaption></figure></div><p>Now we take the application that&#8217;s already deployed and make it follow production best practices.</p><h2><strong>Dev and prod environments</strong></h2><p>When we push the code, we automatically deploy the changes, and they go live immediately. It&#8217;s okay when we are just starting to work on our project. But when we have real users, we want to be more careful and check that our changes didn&#8217;t introduce any regressions.</p><p>To avoid having this problem, we usually have two copies of the same environment:</p><ul><li><p>Dev (development): the environment we use internally for checking that everything works. Typically we run the latest version of our project there, and every time we push, the changes are automatically deployed there.</p></li><li><p>Prod (production): this is what our users use. We don&#8217;t want to deploy every single change there automatically and we want to have more control over the process.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gKA7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gKA7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 424w, https://substackcdn.com/image/fetch/$s_!gKA7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 848w, https://substackcdn.com/image/fetch/$s_!gKA7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 1272w, https://substackcdn.com/image/fetch/$s_!gKA7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gKA7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png" width="940" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/df6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:940,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A push automatically deploys to development, while a release owner manually promotes a tested release to production&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A push automatically deploys to development, while a release owner manually promotes a tested release to production" title="A push automatically deploys to development, while a release owner manually promotes a tested release to production" srcset="https://substackcdn.com/image/fetch/$s_!gKA7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 424w, https://substackcdn.com/image/fetch/$s_!gKA7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 848w, https://substackcdn.com/image/fetch/$s_!gKA7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 1272w, https://substackcdn.com/image/fetch/$s_!gKA7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf6c9e80-6f2a-4444-aa53-1487c214b4f8_940x440.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">On push to main we automatically deploy to dev. Promoting from dev to prod happens only after a manual action.</figcaption></figure></div><p>Because we previously defined our infrastructure as code, we can reuse most of it. We will most likely need to change a few things, like the size of the machine where the application is running, but the majority of the resources will stay the same.</p><p>Let&#8217;s ask our coding assistant to create a copy of the existing environment and call it &#8220;production&#8221;:</p><pre><code><code>Create a second, independent copy of our infrastructure. We will use the copy as production, and the existing infrastructure as a dev environment.</code></code></pre><p>Once we have the prod environment, let&#8217;s update our CI/CD. We will only deploy to dev on every push. For production, we will have a manual action that will take whatever development has and promote (apply) it to production.</p><pre><code><code>Create a manual GitHub Actions workflow that promotes the dev version to production.</code></code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kARi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kARi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 424w, https://substackcdn.com/image/fetch/$s_!kARi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 848w, https://substackcdn.com/image/fetch/$s_!kARi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 1272w, https://substackcdn.com/image/fetch/$s_!kARi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kARi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png" width="1456" height="935" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:935,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A manual production deployment workflow with a confirmation checkbox and an optional release tag&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A manual production deployment workflow with a confirmation checkbox and an optional release tag" title="A manual production deployment workflow with a confirmation checkbox and an optional release tag" srcset="https://substackcdn.com/image/fetch/$s_!kARi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 424w, https://substackcdn.com/image/fetch/$s_!kARi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 848w, https://substackcdn.com/image/fetch/$s_!kARi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 1272w, https://substackcdn.com/image/fetch/$s_!kARi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a1a0877-76f5-4283-af4b-dfa297ba411d_1997x1283.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Promotion to prod happens only after a manual approval</figcaption></figure></div><p>Now we have two environments.</p><h2><strong>Container repository</strong></h2><p>In the pipeline we have so far, the Docker image is built during the deploy stage. I deploy to EC2 by executing the build script on the machine and then running the image in Docker.</p><p>It&#8217;s an anti-pattern.</p><p>There are multiple problems with this approach:</p><ol><li><p>The deploy stage is actually two things: build and deploy. If build fails, there should be no deploy, so it&#8217;s better to have these steps separately.</p></li><li><p>When we promote the dev version to production, we have to build again. During this time, many things could have changed, so the build will not be identical to dev, and it can cause problems.</p></li></ol><p>So we split the deploy step into two separate steps:</p><ul><li><p>Build the image and upload it to a container registry</p></li><li><p>Pull this image from the registry during the deploy</p></li></ul><p>When promoting to prod, we simply pull the same image to prod.</p><p>For AWS, we can use <a href="https://aws.amazon.com/ecr/">Amazon ECR</a> as the registry. You can also push your images to Docker Hub or another container registry if you&#8217;re running outside of AWS and your cloud doesn&#8217;t have a special service for that.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!75xn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!75xn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 424w, https://substackcdn.com/image/fetch/$s_!75xn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 848w, https://substackcdn.com/image/fetch/$s_!75xn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 1272w, https://substackcdn.com/image/fetch/$s_!75xn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!75xn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png" width="1200" height="700" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:700,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A user pushes a change, then CI/CD runs separate Build and Deploy steps; the version tag goes to Deploy and a manually triggered Prod release, which updates Production&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A user pushes a change, then CI/CD runs separate Build and Deploy steps; the version tag goes to Deploy and a manually triggered Prod release, which updates Production" title="A user pushes a change, then CI/CD runs separate Build and Deploy steps; the version tag goes to Deploy and a manually triggered Prod release, which updates Production" srcset="https://substackcdn.com/image/fetch/$s_!75xn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 424w, https://substackcdn.com/image/fetch/$s_!75xn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 848w, https://substackcdn.com/image/fetch/$s_!75xn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 1272w, https://substackcdn.com/image/fetch/$s_!75xn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e9bde1d-77fd-4fc3-92db-fcf212c1d06c_1200x700.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">We split build and deploy into two separate stages inside CI/CD. The build step builds a docker image, tags it and uploads to the registry. The deploy step takes the tag and pushes it to dev. The production release step takes the tag currently deployed in dev and promotes it to prod.</figcaption></figure></div><p>Let&#8217;s implement that:</p><pre><code><code>Currently we build the image during the deploy stage.

Split it into two stages:

- Build: build the image and push it to a container registry (ECR)
- Deploy: pull the image from the registry and serve it

The manual prod promotion CI/CD workflow pulls the currently deployed dev image to prod.

Tag each image using the YYYYMMDD-HHMMSS-shortsha pattern (e.g. "20260818-163457-83242da")</code></code></pre><p>Now the build step produces a tagged image, and the deploy step deploys it to dev. We can test the dev application, and when we later promote it to production, we will be certain that it&#8217;s exactly the same image.</p><h2><strong>Observability</strong></h2><p>Having two environments helps us avoid accidentally pushing buggy code to production. But accidents will still happen, and we need to make sure we detect them and react as fast as possible.</p><p>For that, we need to have observability. &#8220;Observability&#8221; means collecting information about the application so we can understand its behavior. With it, when something breaks, we can quickly find the problem.</p><p>We achieve observability by adding monitoring to our applications. At the minimum, we need to collect basic performance metrics like CPU and memory utilization and requests per second (RPS).</p><p>If we see that CPU and memory utilization are growing and RPS is dropping, something may be off.</p><p>This information alone is not enough. It doesn&#8217;t explain what happened inside a request and why it&#8217;s causing errors or degraded performance. For that, we need to collect more.</p><h2><strong>OpenTelemetry</strong></h2><p><a href="https://opentelemetry.io/docs/">OpenTelemetry</a> (often abbreviated as &#8220;OTel&#8221;) is the industry standard for telemetry.</p><p>Telemetry is all the information that the application produces:</p><ul><li><p>Metrics - requests per second, response latency, and number of errors</p></li><li><p>Logs - timestamped records of individual events, like an error message or a failed database query</p></li><li><p>Traces - all the steps (called &#8220;spans&#8221;) that a single request makes through the entire application</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8Yyo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8Yyo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 424w, https://substackcdn.com/image/fetch/$s_!8Yyo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 848w, https://substackcdn.com/image/fetch/$s_!8Yyo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 1272w, https://substackcdn.com/image/fetch/$s_!8Yyo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8Yyo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png" width="780" height="620" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:620,&quot;width&quot;:780,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An application branches to three telemetry examples: a metrics time series, a timestamped log record and a trace waterfall&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An application branches to three telemetry examples: a metrics time series, a timestamped log record and a trace waterfall" title="An application branches to three telemetry examples: a metrics time series, a timestamped log record and a trace waterfall" srcset="https://substackcdn.com/image/fetch/$s_!8Yyo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 424w, https://substackcdn.com/image/fetch/$s_!8Yyo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 848w, https://substackcdn.com/image/fetch/$s_!8Yyo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 1272w, https://substackcdn.com/image/fetch/$s_!8Yyo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb499e88e-e816-4abc-9c2c-f03947d2d549_780x620.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Metrics give us concrete numbers, logs give details, and traces show the path of a request with each step.</figcaption></figure></div><p>OTLP is the protocol that applications use to send this telemetry.</p><p>For many popular libraries, we can start capturing telemetry with just a few lines of code. This process is called &#8220;instrumenting&#8221; - adding telemetry collection to an application. Auto-instrumentation can instrument libraries like FastAPI without changing their code.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LjEv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LjEv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 424w, https://substackcdn.com/image/fetch/$s_!LjEv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 848w, https://substackcdn.com/image/fetch/$s_!LjEv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 1272w, https://substackcdn.com/image/fetch/$s_!LjEv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LjEv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png" width="1456" height="1045" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1045,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Python code configuring OpenTelemetry resource metadata, a tracer provider and an OTLP exporter for a FastAPI service&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Python code configuring OpenTelemetry resource metadata, a tracer provider and an OTLP exporter for a FastAPI service" title="Python code configuring OpenTelemetry resource metadata, a tracer provider and an OTLP exporter for a FastAPI service" srcset="https://substackcdn.com/image/fetch/$s_!LjEv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 424w, https://substackcdn.com/image/fetch/$s_!LjEv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 848w, https://substackcdn.com/image/fetch/$s_!LjEv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 1272w, https://substackcdn.com/image/fetch/$s_!LjEv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0c0401f-ab92-4e85-8691-3057519664d0_1742x1250.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Instrumenting FastAPI is just a few lines of code</figcaption></figure></div><p>Let&#8217;s do it. Ask the coding agent:</p><pre><code><code>Instrument the backend with OpenTelemetry.

Include in the telemetry:

- service name
- environment
- deployed version</code></code></pre><p>The application now produces telemetry and exports it over OTLP. We still need to decide where to send and store it, and how to view it. Let&#8217;s do that now.</p><h2><strong>OTel Collectors</strong></h2><p>We export telemetry over OTLP, but it&#8217;s not saved anywhere. For that, we define an <a href="https://opentelemetry.io/docs/collector/">OpenTelemetry Collector</a> between the application and the storage systems.</p><p>There are many services you can use for that. Ask your AI assistant and select the option that works best for your application.</p><p>In our case, I&#8217;ll use:</p><ul><li><p>Prometheus for metrics</p></li><li><p>Loki for logs</p></li><li><p>Tempo for traces</p></li><li><p>Grafana for dashboards</p></li></ul><p>I had never used Loki or Tempo before, but when I told my coding agent that I wanted to use Prometheus and Grafana, it suggested including Loki and Tempo too.</p><p>So let&#8217;s implement it:</p><pre><code><code>Add an OpenTelemetry Collector.

Create "observability/" directory with Docker Compose for:

- OpenTelemetry Collector
- Prometheus
- Loki
- Tempo
- Grafana

Keep this as a separate Compose project from the application stack</code></code></pre><p>Now we have the infrastructure for observability, so we&#8217;re ready to build a dashboard.</p><h2><strong>Metrics</strong></h2><p>Metrics are numerical measurements collected over time. They can show a current value, such as the number of active participants, or count events during an interval, such as rooms created or errors in the last five minutes.</p><p>For System Design Canvas, useful measurements may include:</p><ul><li><p>Total number of interview rooms</p></li><li><p>Number of active rooms and number of active participants</p></li><li><p>Number of elements on the canvas across the application</p></li><li><p>Change propagation delay: when I add an element to the canvas, how long does it take for you to see it</p></li><li><p>Number of errors</p></li></ul><p>Let&#8217;s start collecting some of them:</p><pre><code><code>Track these application metrics:

- interview rooms created
- active interview participants
- canvas elements created
- failures in component creation

Include the environment and deployed version</code></code></pre><p>Now we have the data and can display it in a dashboard:</p><pre><code><code>Add a Grafana panel with these metrics. Make it possible to filter by environment and deployed version.</code></code></pre><p>Run it locally and test it by opening our application, creating a room, adding a canvas element, and checking that it appears in Grafana.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DYr-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DYr-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!DYr-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!DYr-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!DYr-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DYr-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A dark Grafana-style dashboard with five-minute event counts for interview rooms, canvas elements and component creation failures, plus active participants, filtered to production and a deployed version&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A dark Grafana-style dashboard with five-minute event counts for interview rooms, canvas elements and component creation failures, plus active participants, filtered to production and a deployed version" title="A dark Grafana-style dashboard with five-minute event counts for interview rooms, canvas elements and component creation failures, plus active participants, filtered to production and a deployed version" srcset="https://substackcdn.com/image/fetch/$s_!DYr-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!DYr-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!DYr-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!DYr-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F091a0f56-6f8e-498b-ac5e-effc59aa0fb7_1672x941.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Grafana dashboard with the metrics</figcaption></figure></div><h2><strong>Deploy the observability stack</strong></h2><p>If you use the same stack as me (Prometheus, Grafana and others), we need to deploy and manage them ourselves.</p><p>If you use a managed OTel-compatible service such as CloudWatch, Grafana Cloud, Datadog, or Sentry, you can skip this step.</p><p>So if you&#8217;re following my steps, let&#8217;s deploy our stack:</p><pre><code><code>Deploy the observability stack. It should be separate from the application stack.
Connect both development and production to it.</code></code></pre><p>After it finishes, you can open Grafana and perform the same test that we did locally.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!i4Rw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!i4Rw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 424w, https://substackcdn.com/image/fetch/$s_!i4Rw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 848w, https://substackcdn.com/image/fetch/$s_!i4Rw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 1272w, https://substackcdn.com/image/fetch/$s_!i4Rw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!i4Rw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png" width="1456" height="605" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:605,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The build-once deployment workflow extended with OTel attached to development and production; a collector sends metrics to Prometheus, logs to Loki and traces to Tempo for Grafana dashboards&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The build-once deployment workflow extended with OTel attached to development and production; a collector sends metrics to Prometheus, logs to Loki and traces to Tempo for Grafana dashboards" title="The build-once deployment workflow extended with OTel attached to development and production; a collector sends metrics to Prometheus, logs to Loki and traces to Tempo for Grafana dashboards" srcset="https://substackcdn.com/image/fetch/$s_!i4Rw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 424w, https://substackcdn.com/image/fetch/$s_!i4Rw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 848w, https://substackcdn.com/image/fetch/$s_!i4Rw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 1272w, https://substackcdn.com/image/fetch/$s_!i4Rw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Development and production export telemetry through OTel. The collector sends metrics to Prometheus, logs to Loki and traces to Tempo. Grafana reads from them and displays this information on dashboards.</figcaption></figure></div><p>Note: here we deploy Grafana and the other services openly on the Internet, so everyone can access them. In practice, you put sensitive resources like observability services and databases on private networks, restrict network access, and require authentication so only authorized users can access them.</p><h2><strong>Alerting</strong></h2><p>We have the dashboards, but we can&#8217;t watch them all the time to see if some metrics drop or grow too large. If this happens, we need a system that triggers an alert saying that something happened.</p><pre><code><code>Add an actionable alert for repeated canvas component-creation failures.

Use a threshold and duration that represent real user impact. Include the
service, environment, deployed version, owner, and dashboard URL in the alert.</code></code></pre><h2><strong>On-Call Engineer</strong></h2><p>Now that we have alerts, we need to react to them. In companies, there&#8217;s usually an on-call engineer. That&#8217;s the engineer who receives the alert when something happens. They need to figure out what&#8217;s happening and find a quick solution to stop the problem.</p><p>When a metric crosses its threshold for long enough, the alert fires and the on-call engineer receives it.</p><p>Our AI assistant can be the on-call engineer, and if something happens, it can try to fix it. In real scenarios, AI on-call agents also need to understand if the issue is serious enough to escalate the alert to a human on-call engineer.</p><p>In our case, we won&#8217;t do it. We will implement a system that checks the observability system for alerts, and if something is happening, an agent session will start and try to fix the problem.</p><pre><code><code>Add an on-call-engineer/ directory with a script that polls the observability
alert API every minute.

When an alert fires, pass the alert details to a headless coding agent.</code></code></pre><p>As a result, we have a worker that polls for alerts and starts an agent session when one fires.</p><p>In my case, the agent had this prompt:</p><pre><code><code>You are the on-call engineer for this repository. An alert just fired.

Investigate the root cause. Read the code and reproduce the failure.
If you find a real bug, make the smallest correction, run the backend tests,
and commit the fix with a clear message.

If the alert is a false positive, explain why and do not change the code.</code></code></pre><p>This is a small proof-of-concept script. In reality, you will probably have a system that looks like this:</p><ul><li><p>An alert is triggered and sent via SNS or a similar service.</p></li><li><p>A Lambda reacts to that alert and starts a container job.</p></li><li><p>The job launches Codex or Claude in headless mode.</p></li><li><p>Once the session is over, the logs are saved and the compute is terminated.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JrQ5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JrQ5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 424w, https://substackcdn.com/image/fetch/$s_!JrQ5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 848w, https://substackcdn.com/image/fetch/$s_!JrQ5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 1272w, https://substackcdn.com/image/fetch/$s_!JrQ5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JrQ5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png" width="1080" height="500" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An alert passes through SNS and Lambda to an isolated agent container with access to code, logs and metrics; the session log is saved after the run&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An alert passes through SNS and Lambda to an isolated agent container with access to code, logs and metrics; the session log is saved after the run" title="An alert passes through SNS and Lambda to an isolated agent container with access to code, logs and metrics; the session log is saved after the run" srcset="https://substackcdn.com/image/fetch/$s_!JrQ5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 424w, https://substackcdn.com/image/fetch/$s_!JrQ5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 848w, https://substackcdn.com/image/fetch/$s_!JrQ5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 1272w, https://substackcdn.com/image/fetch/$s_!JrQ5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffa95f1d0-e94a-40e3-a8c0-c981612b68a8_1080x500.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The alert starts an agent running inside a container job. When the job is done, the agent saves the log and the machine is teminated.</figcaption></figure></div><p>Let&#8217;s test our idea by introducing a bug.</p><h2><strong>Introducing a bug</strong></h2><p>We want to test that this system works, so let&#8217;s start a coding agent and ask it to add a bug or find an existing one.</p><pre><code><code>Introduce a realistic bug in canvas component creation.

For some requests, creating a component should fail even though the existing tests pass. Keep the failure reproducible so we can test that the bug causes an alert and the on-call response.</code></code></pre><p>Here, the goal is to debug the process and make sure our on-call engineer actually wakes up and solves the problem.</p><p>After a few iterations, we have an on-call engineer that wakes up when there&#8217;s an alert and tries to fix the problem.</p><h2><strong>Clean up</strong></h2><p>After you finish everything for the module, I recommend cleaning up everything. We created these things for learning, so we shouldn&#8217;t forget to clean them up. If you don&#8217;t, expect a bigger bill at the end of the month.</p><p>Ask your agent to clean the infra:</p><pre><code><code>Delete the CloudFormation stacks.</code></code></pre><p>Check yourself that everything is indeed gone. You may want to double-check everything in the AWS console or ask another agent to scan the running resources in your account.</p><h2><strong>Next steps after this module</strong></h2><p>We started with an idea, and now our setup is much closer to production. But here I could only briefly touch on the most important concepts.</p><p>There are many more things that you should consider for a real production application:</p><ul><li><p>Here I deploy to EC2 by running a shell script on the instance. It&#8217;s okay for a proof of concept but very problematic for production. Use a container management system like ECS or an alternative.</p></li><li><p>Sometimes you will need to roll back a change you promoted to prod. Ask your agent to make it easy to do.</p></li><li><p>Use a managed database service.</p></li><li><p>Regularly back up your database. It&#8217;s best if your backups live outside of your infrastructure-as-code stack and you have multiple independent copies.</p></li><li><p>Test that you can actually use these backups.</p></li><li><p>Put internal services and resources on private subnets inside a VPC, and restrict network access to them.</p></li><li><p>If your application needs to handle a lot of traffic, learn about scaling and load balancing. Container management systems make this easier to manage.</p></li><li><p>Ask Fable or GPT-5.6-Sol Max (better both) to audit your code for security vulnerabilities. Do it multiple times.</p></li></ul><h2><strong>Next in the series</strong></h2><p>With this article, we finish developing our end-to-end application. We took it from a raw idea and turned it into a system that&#8217;s much closer to production.</p><p>In the last article, we will look into coding agent capabilities:</p><ul><li><p>MCP</p></li><li><p>skills</p></li><li><p>commands</p></li><li><p>plugins</p></li><li><p>specialized agents</p></li></ul><p><a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>You can find the course materials and the next cohort in <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a>. It&#8217;s free.</p>]]></content:encoded></item><item><title><![CDATA[How to Do Evals in 2026]]></title><description><![CDATA[A tool-agnostic framework for evaluating AI agents]]></description><link>https://aishippingblog.com/p/how-to-do-evals-in-2026</link><guid isPermaLink="false">https://aishippingblog.com/p/how-to-do-evals-in-2026</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Fri, 14 Aug 2026 18:47:37 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!bCza!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Across all 4,894 AI engineering job descriptions that I collected for the <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide">AI Engineering Field Guide</a>, evaluation consistently comes as the number one skill that AI Engineers must have.</p><p>It also appears in interview questions. In interviews, you&#8217;ll definitely hear something like &#8220;how do you evaluate a RAG pipeline?&#8221; or &#8220;how do you measure agent performance?&#8221;. If you don&#8217;t include evals in your home assignment, it&#8217;s a red flag.</p><p>Why is it such an important skill? Because it&#8217;s hard, and it matters.</p><p>It&#8217;s easy to build an agent these days. I show how to do it in my <a href="https://aishippinglabs.com/courses/aihero">&#8220;AI Hero: 7-Day AI Agents Crash-Course&#8221;</a>. You need to get an OpenAI key, take any agentic framework, define the instructions and the tools, and that&#8217;s it. You have an agent.</p><p>But this agent will break in so many ways. It will:</p><ul><li><p>Give a wrong answer</p></li><li><p>Confidently hallucinate a plausible answer</p></li><li><p>Finish without giving any answer</p></li><li><p>Get into a loop and call the same tool over and over again</p></li><li><p>Exhaust the context</p></li><li><p>Call a tool with invalid arguments</p></li></ul><p>That&#8217;s just the tip of the iceberg. To make sure this agent works reliably, you need to test and evaluate it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bCza!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bCza!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 424w, https://substackcdn.com/image/fetch/$s_!bCza!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 848w, https://substackcdn.com/image/fetch/$s_!bCza!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 1272w, https://substackcdn.com/image/fetch/$s_!bCza!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!bCza!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png" width="642" height="380.8971820258949" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:779,&quot;width&quot;:1313,&quot;resizeWidth&quot;:642,&quot;bytes&quot;:976390,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/211217209?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bCza!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 424w, https://substackcdn.com/image/fetch/$s_!bCza!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 848w, https://substackcdn.com/image/fetch/$s_!bCza!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 1272w, https://substackcdn.com/image/fetch/$s_!bCza!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd560ed5d-f374-48db-b81d-8d3f487ec359_1313x779.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Also, without evaluations, you&#8217;re blind. You don&#8217;t know how the agent is performing. You also can&#8217;t really change anything: every change you introduce may break the system in many ways, including the areas where you least expect it.</p><p>Agent evaluation is not simple. It takes a lot of time to understand the problem, get input from real users, and gather the right data. It&#8217;s not something you can just delegate to Claude and forget about it.</p><p>That&#8217;s what makes evals such an important part of AI Engineering.</p><p>In this article, I will tell you about my approach to evaluations:</p><ul><li><p>Start manually by &#8220;vibe-checking&#8221; the system, but also collect logs as early as possible</p></li><li><p>Create a tool for labelling the logs and put together a gold standard dataset</p></li><li><p>Create a judge that&#8217;s aligned with our judgement</p></li><li><p>Break the agent like a QA engineer</p></li><li><p>Get more data by generating it synthetically</p></li><li><p>Start collecting data from real users</p></li><li><p>Monitor your system with online evaluation</p></li><li><p>Always refine your gold standard dataset</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!RDZh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RDZh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 424w, https://substackcdn.com/image/fetch/$s_!RDZh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 848w, https://substackcdn.com/image/fetch/$s_!RDZh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 1272w, https://substackcdn.com/image/fetch/$s_!RDZh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RDZh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png" width="1456" height="747" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:747,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A descending staircase of six stages - vibe-check and log, label good or bad, align the judge, break the agent, synthetic data, real users&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A descending staircase of six stages - vibe-check and log, label good or bad, align the judge, break the agent, synthetic data, real users" title="A descending staircase of six stages - vibe-check and log, label good or bad, align the judge, break the agent, synthetic data, real users" srcset="https://substackcdn.com/image/fetch/$s_!RDZh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 424w, https://substackcdn.com/image/fetch/$s_!RDZh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 848w, https://substackcdn.com/image/fetch/$s_!RDZh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 1272w, https://substackcdn.com/image/fetch/$s_!RDZh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8d571b4-c691-4595-bef4-57e360e5d9c3_1560x800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The stages, in the order I go through them. Each stage adds new cases to the gold standard dataset.</figcaption></figure></div><p>Let&#8217;s get into details.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h2><strong>Collecting the Gold Standard Dataset</strong></h2><p>You have an agent. What&#8217;s next?</p><p>There&#8217;s nothing wrong with &#8220;vibe-checking&#8221; it: you poke it, ask questions you want it to answer, and look at the results. If they aren&#8217;t good, you figure out how to fix it.</p><p>But already at this step you can be a bit more organized and start logging what the agent is doing. When you ask a question and see the answer, make sure they are saved somewhere. You do a few vibe-checking sessions, and you already have 10-15 logged records.</p><p>Now you can systematically look at these logs and classify each record into</p><ul><li><p>&#8220;good&#8221;: the answer is what you expect</p></li><li><p>&#8220;bad&#8221;: the answer is not what you wanted to see</p></li></ul><p>To help me do it, I usually vibe-code a small labelling tool.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xRpV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xRpV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 424w, https://substackcdn.com/image/fetch/$s_!xRpV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 848w, https://substackcdn.com/image/fetch/$s_!xRpV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 1272w, https://substackcdn.com/image/fetch/$s_!xRpV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xRpV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png" width="626" height="496.7173913043478" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:803,&quot;width&quot;:1012,&quot;resizeWidth&quot;:626,&quot;bytes&quot;:945110,&quot;alt&quot;:&quot;Labelling tool showing a logged record with the user question, the assistant answer, and retrieved FAQ entries, plus Good, Bad, and Skip buttons and a progress bar&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Labelling tool showing a logged record with the user question, the assistant answer, and retrieved FAQ entries, plus Good, Bad, and Skip buttons and a progress bar" title="Labelling tool showing a logged record with the user question, the assistant answer, and retrieved FAQ entries, plus Good, Bad, and Skip buttons and a progress bar" srcset="https://substackcdn.com/image/fetch/$s_!xRpV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 424w, https://substackcdn.com/image/fetch/$s_!xRpV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 848w, https://substackcdn.com/image/fetch/$s_!xRpV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 1272w, https://substackcdn.com/image/fetch/$s_!xRpV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc1b87f7-e242-4c0e-8ed2-a366896d1b89_1012x803.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The labelling tool for the &#8220;version 0&#8221; of the gold standard dataset</figcaption></figure></div><p>This gives you a &#8220;version 0&#8221; of the gold standard dataset - the dataset you will use for evaluating your system (sometimes it&#8217;s also called the &#8220;ground truth&#8221; dataset).</p><p>Don&#8217;t delegate this step to coding assistants. It&#8217;s only a few examples, so you can go through them relatively quickly. But this way, you will get a lot of insights into how your agent is behaving, what&#8217;s working and what isn&#8217;t.</p><h2><strong>Judge alignment</strong></h2><p>Once you manually labelled 10-15 examples, it&#8217;s time to start automating it.</p><p>I use coding assistants for that:</p><ul><li><p>Ask the assistant to analyze the logs and classify the examples into good or bad.</p></li><li><p>See if the results match yours. If they don&#8217;t, ask the agent to explain its decision - maybe you missed something, or the agent didn&#8217;t have enough context.</p></li><li><p>Iterate until you both agree.</p></li><li><p>At the end of the session, ask the agent to create a judge.md file that describes the classification rules.</p></li><li><p>Read this file carefully to see that the rules are generic and not specific to the examples you have in the dataset.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yFJA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yFJA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 424w, https://substackcdn.com/image/fetch/$s_!yFJA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 848w, https://substackcdn.com/image/fetch/$s_!yFJA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 1272w, https://substackcdn.com/image/fetch/$s_!yFJA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yFJA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png" width="596" height="386.4175824175824" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:944,&quot;width&quot;:1456,&quot;resizeWidth&quot;:596,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Example judge.md file with GOOD and BAD criteria for an FAQ assistant, ending with the rule when uncertain, prefer BAD&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Example judge.md file with GOOD and BAD criteria for an FAQ assistant, ending with the rule when uncertain, prefer BAD" title="Example judge.md file with GOOD and BAD criteria for an FAQ assistant, ending with the rule when uncertain, prefer BAD" srcset="https://substackcdn.com/image/fetch/$s_!yFJA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 424w, https://substackcdn.com/image/fetch/$s_!yFJA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 848w, https://substackcdn.com/image/fetch/$s_!yFJA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 1272w, https://substackcdn.com/image/fetch/$s_!yFJA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F750ce6b3-f1de-4527-a94b-d8b33d9063c7_1475x956.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">An example of a judge.md file with generic classification rules</figcaption></figure></div><p>We just created a judge - a system that evaluates our agent.</p><p>Now we need to test that using judge.md alone is sufficient:</p><ul><li><p>Start a new session</p></li><li><p>Ask to start a subagent which reads judge.md and classifies the records in your dataset</p></li><li><p>If something is off, ask the agent to change judge.md</p></li><li><p>Iterate until you both align on what&#8217;s good and bad</p></li></ul><p>This process is called &#8220;alignment&#8221; - we align the judge with what we think is good or bad.</p><p>It&#8217;s the same as training a binary classifier that predicts your decisions. But instead of weights, you have a judge.md file. (And it&#8217;s also helpful to have a train-test split here too! But later, when you have more data.)</p><p>At this point, you also get a metric - the fraction of good examples. Now if you change something in your agent, you can re-run the judge and make sure this metric doesn&#8217;t go down.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!eG3Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!eG3Z!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 424w, https://substackcdn.com/image/fetch/$s_!eG3Z!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 848w, https://substackcdn.com/image/fetch/$s_!eG3Z!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 1272w, https://substackcdn.com/image/fetch/$s_!eG3Z!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!eG3Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png" width="1456" height="825" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:825,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Terminal running Codex, which executed the agent against the dataset, judged 15 records with judge.md, reported 12 good and 3 bad results with an 80% score, and asks whether to inspect the bad cases&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Terminal running Codex, which executed the agent against the dataset, judged 15 records with judge.md, reported 12 good and 3 bad results with an 80% score, and asks whether to inspect the bad cases" title="Terminal running Codex, which executed the agent against the dataset, judged 15 records with judge.md, reported 12 good and 3 bad results with an 80% score, and asks whether to inspect the bad cases" srcset="https://substackcdn.com/image/fetch/$s_!eG3Z!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 424w, https://substackcdn.com/image/fetch/$s_!eG3Z!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 848w, https://substackcdn.com/image/fetch/$s_!eG3Z!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 1272w, https://substackcdn.com/image/fetch/$s_!eG3Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F539961c7-5034-4d34-af11-122c2c3245b5_1586x899.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Codex running the evaluation: 12 good, 3 bad, 80% score</figcaption></figure></div><h2><strong>Breaking the agent</strong></h2><p>We have the initial version of our gold standard dataset. But it&#8217;s not enough. Now we want to find cases that break our system.</p><p>QA engineers have been breaking software for many decades. For coming up with cases that should derail our agent, we can borrow some concepts from their frameworks.</p><h3><strong>1. Equivalence partitions</strong></h3><p>The first step is <a href="https://www.ministryoftesting.com/software-testing-glossary/equivalence-partitioning">equivalence partitioning</a>. You divide the input space into groups where inputs within each group should produce similar behavior.</p><p>If we have a RAG agent that answers questions about our docs, it could be:</p><ul><li><p>Questions about topics in the documentation,</p></li><li><p>Questions about relevant topics but not in the documentation,</p></li><li><p>Questions about irrelevant topics,</p></li><li><p>Ambiguous questions,</p></li><li><p>Questions in different languages</p></li></ul><p>Once you come up with a list of groups, come up with 2-3 questions from each group. You can ask AI to help you with the entire process.</p><h3><strong>2. Boundary testing</strong></h3><p>The next concept is <a href="https://www.ministryoftesting.com/software-testing-glossary/boundary-testing">boundary testing</a>.</p><p>We already have some questions from the previous step, but we can be even more aggressive in our testing.</p><p>After defining the partitions, QA engineers would test the boundaries of each partition. We can do the same.</p><p>For a RAG agent, come up with queries that:</p><ul><li><p>Exactly match a document that we have in our database (easy case),</p></li><li><p>Are semantically related but use different words (typical case),</p></li><li><p>Are semantically related to multiple documents (ambiguous case),</p></li><li><p>Are completely off-topic (boundary case).</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gLJk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gLJk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 424w, https://substackcdn.com/image/fetch/$s_!gLJk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 848w, https://substackcdn.com/image/fetch/$s_!gLJk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 1272w, https://substackcdn.com/image/fetch/$s_!gLJk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gLJk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png" width="1456" height="784" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:784,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Spreadsheet with test questions for a course FAQ agent, with columns for id, course, question, kind, and expected outcome, covering kinds like in_docs, not_in_docs, ambiguous, off_topic, different_language, exact_match, paraphrase, and short_query&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Spreadsheet with test questions for a course FAQ agent, with columns for id, course, question, kind, and expected outcome, covering kinds like in_docs, not_in_docs, ambiguous, off_topic, different_language, exact_match, paraphrase, and short_query" title="Spreadsheet with test questions for a course FAQ agent, with columns for id, course, question, kind, and expected outcome, covering kinds like in_docs, not_in_docs, ambiguous, off_topic, different_language, exact_match, paraphrase, and short_query" srcset="https://substackcdn.com/image/fetch/$s_!gLJk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 424w, https://substackcdn.com/image/fetch/$s_!gLJk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 848w, https://substackcdn.com/image/fetch/$s_!gLJk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 1272w, https://substackcdn.com/image/fetch/$s_!gLJk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42c05f23-2845-48da-98f3-25c13b4a60a3_1589x856.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Test questions from equivalence partitions and boundary testing, tagged by kind and expected outcome</figcaption></figure></div><p></p><h2><strong>Further alignment</strong></h2><p>This way you can easily add 30-40 more questions to your gold standard dataset.</p><p>Let&#8217;s take all the data we have, and run it against our agent. Then use the judge to classify the output.</p><p>We already have a tool that we created earlier for manual evaluation. At this point, I adjust this tool to also support checking the output of the judge. For each record, I just decide whether I &#8220;agree&#8221; or &#8220;disagree&#8221; with the judge. I find it more convenient than labelling the new logs as &#8220;good&#8221; or &#8220;bad&#8221;.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Fw8r!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Fw8r!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 424w, https://substackcdn.com/image/fetch/$s_!Fw8r!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 848w, https://substackcdn.com/image/fetch/$s_!Fw8r!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 1272w, https://substackcdn.com/image/fetch/$s_!Fw8r!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Fw8r!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png" width="650" height="441.9630156472262" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:956,&quot;width&quot;:1406,&quot;resizeWidth&quot;:650,&quot;bytes&quot;:1526441,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/211217209?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb5f16480-b5de-45c6-8bc2-f2c821adef27_1448x1086.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Fw8r!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 424w, https://substackcdn.com/image/fetch/$s_!Fw8r!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 848w, https://substackcdn.com/image/fetch/$s_!Fw8r!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 1272w, https://substackcdn.com/image/fetch/$s_!Fw8r!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7cac8a1b-426a-464e-94fb-0180247dd022_1406x956.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>If we have many &#8220;disagree&#8221; cases, tune the judge following the same process as before. We will never have 100% alignment, so we don&#8217;t need to be too strict about it, but the judge should be correct most of the time.</p><h2><strong>Using the Judge to improve your agent</strong></h2><p>Now we have a fully automated process that:</p><ol><li><p>Runs the agent against our data</p></li><li><p>Runs the judge against the output of step 1</p></li><li><p>Outputs a metric - the fraction of &#8220;good&#8221; results</p></li></ol><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3Xg2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3Xg2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 424w, https://substackcdn.com/image/fetch/$s_!3Xg2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 848w, https://substackcdn.com/image/fetch/$s_!3Xg2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 1272w, https://substackcdn.com/image/fetch/$s_!3Xg2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3Xg2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png" width="1310" height="370" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:370,&quot;width&quot;:1310,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Pipeline diagram: the gold standard dataset feeds the agent, the agent output goes to the judge which reads judge.md, and the judge produces a metric, with a feedback arrow going back to the agent&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Pipeline diagram: the gold standard dataset feeds the agent, the agent output goes to the judge which reads judge.md, and the judge produces a metric, with a feedback arrow going back to the agent" title="Pipeline diagram: the gold standard dataset feeds the agent, the agent output goes to the judge which reads judge.md, and the judge produces a metric, with a feedback arrow going back to the agent" srcset="https://substackcdn.com/image/fetch/$s_!3Xg2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 424w, https://substackcdn.com/image/fetch/$s_!3Xg2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 848w, https://substackcdn.com/image/fetch/$s_!3Xg2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 1272w, https://substackcdn.com/image/fetch/$s_!3Xg2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F713ee840-c63d-4d6a-9694-5ea51c7b7c84_1310x370.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The automated evaluation pipeline: fix and re-run</figcaption></figure></div><p>Because it&#8217;s automated, we can use it to improve our agent. You can ask the coding assistant to do it.</p><p>I recommend following the <a href="https://alexeyondata.substack.com/p/ai-native-development-specifications">implementer-tester pattern</a> for that:</p><ul><li><p>Define the implementer subagent that modifies the agent</p></li><li><p>Define the judge subagent that uses judge.md to evaluate the output</p></li><li><p>Iterate until your &#8220;good&#8221; ratio is acceptable</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4E8p!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4E8p!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 424w, https://substackcdn.com/image/fetch/$s_!4E8p!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 848w, https://substackcdn.com/image/fetch/$s_!4E8p!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 1272w, https://substackcdn.com/image/fetch/$s_!4E8p!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4E8p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png" width="680" height="410.52197802197804" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:879,&quot;width&quot;:1456,&quot;resizeWidth&quot;:680,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two terminal sessions side by side: the implementer agent modifying the agent code, and the tester running the evaluation and reporting the results&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two terminal sessions side by side: the implementer agent modifying the agent code, and the tester running the evaluation and reporting the results" title="Two terminal sessions side by side: the implementer agent modifying the agent code, and the tester running the evaluation and reporting the results" srcset="https://substackcdn.com/image/fetch/$s_!4E8p!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 424w, https://substackcdn.com/image/fetch/$s_!4E8p!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 848w, https://substackcdn.com/image/fetch/$s_!4E8p!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 1272w, https://substackcdn.com/image/fetch/$s_!4E8p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8729ec6e-cc59-4544-b194-d5d4eb8bfebb_2234x1349.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The implementer-tester loop: one session changes the agent, the other one evaluates it</figcaption></figure></div><h2><strong>Synthetic data</strong></h2><p>So far we focused on the quality of our gold standard dataset. In this section, I want to talk about the quantity.</p><p>Quantity is also important because agents are non-deterministic. Even if the agent works properly on our small dataset, it will break when we get more traffic - just because of probability and statistics. When you expose your agent to a bigger number of inputs, things will break: with more repetitions, you increase the chances of something going wrong. That&#8217;s why you want quantity here.</p><p>We can generate a lot more data using AI. Open your AI assistant, ask it to analyze your code, describe to it your target users, and then tell it to generate questions that these users are likely to ask.</p><p>For a RAG system, you can:</p><ul><li><p>Take a document from your knowledge base</p></li><li><p>Ask AI to come up with 5 questions based on this document</p></li><li><p>Do it for all documents</p></li></ul><p>You will end up with a lot of questions. You can keep all of them, or take a sample.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_gXJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_gXJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 424w, https://substackcdn.com/image/fetch/$s_!_gXJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 848w, https://substackcdn.com/image/fetch/$s_!_gXJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 1272w, https://substackcdn.com/image/fetch/$s_!_gXJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_gXJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png" width="714" height="523.7307692307693" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1068,&quot;width&quot;:1456,&quot;resizeWidth&quot;:714,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Terminal session where an AI assistant generated five user questions from a course document, covering homework deadlines, certificates, project submission, and technical issues&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Terminal session where an AI assistant generated five user questions from a course document, covering homework deadlines, certificates, project submission, and technical issues" title="Terminal session where an AI assistant generated five user questions from a course document, covering homework deadlines, certificates, project submission, and technical issues" srcset="https://substackcdn.com/image/fetch/$s_!_gXJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 424w, https://substackcdn.com/image/fetch/$s_!_gXJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 848w, https://substackcdn.com/image/fetch/$s_!_gXJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 1272w, https://substackcdn.com/image/fetch/$s_!_gXJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa8be12d7-0c41-421e-8f8c-8157abdf2595_1568x1150.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Generating synthetic questions from a document in the knowledge base</figcaption></figure></div><p>Then follow the same process: run your agent against this data and then use the judge to evaluate the results.</p><p>This way you will probably run into many different problems:</p><ul><li><p>The agent will loop without stopping and exhaust the context</p></li><li><p>It won&#8217;t follow the instructions and do something that it shouldn&#8217;t</p></li><li><p>A tool will break and your agent won&#8217;t finish the task</p></li></ul><p>And many others. You don&#8217;t know in advance what you&#8217;ll get, which is why you want to expose the agent to a wide variety of inputs.</p><p>Once you identify the scenarios where your agent breaks or doesn&#8217;t perform well, add them to your gold standard dataset.</p><p>Then you can fix the agent, and re-run the whole evaluation set to make sure you don&#8217;t introduce any regressions.</p><h2><strong>Start testing on real users as soon as possible</strong></h2><p>We have evaluated our agent on many cases, so we&#8217;re confident that it&#8217;s working okay. But we always need more data.</p><p>Now it&#8217;s time to put it in front of real users:</p><ul><li><p>Invite people for user testing sessions to check your system</p></li><li><p>If you plan to integrate it into an existing product, include it in a small subset of the traffic</p></li><li><p>If it&#8217;s a small personal project, ask your friends and family to test it.</p></li><li><p>If it&#8217;s a course project, ask your classmates to play with it.</p></li><li><p>Share it on social media.</p></li></ul><p>Make sure you&#8217;re collecting the logs (be direct with the user about what you&#8217;re collecting) and use this for adding new cases into the gold standard dataset.</p><p>Add +1/-1 buttons. This way, the users can give you explicit feedback, and if somebody dislikes the answer, you should inspect it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5B7m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5B7m!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 424w, https://substackcdn.com/image/fetch/$s_!5B7m!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 848w, https://substackcdn.com/image/fetch/$s_!5B7m!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 1272w, https://substackcdn.com/image/fetch/$s_!5B7m!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5B7m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png" width="1448" height="945" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:945,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1431256,&quot;alt&quot;:&quot;Deployed FAQ assistant answering a user question in Slack, with thumbs up and thumbs down buttons for collecting feedback on the answer&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Deployed FAQ assistant answering a user question in Slack, with thumbs up and thumbs down buttons for collecting feedback on the answer" title="Deployed FAQ assistant answering a user question in Slack, with thumbs up and thumbs down buttons for collecting feedback on the answer" srcset="https://substackcdn.com/image/fetch/$s_!5B7m!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 424w, https://substackcdn.com/image/fetch/$s_!5B7m!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 848w, https://substackcdn.com/image/fetch/$s_!5B7m!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 1272w, https://substackcdn.com/image/fetch/$s_!5B7m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6cbce2c-a920-4eaa-8855-6710133b7c1b_1448x945.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The deployed agent in front of real users, with feedback buttons under every answer</figcaption></figure></div><h2><strong>Online evaluation</strong></h2><p>We have a system for offline evaluations: we run the agent against our dataset and we use the judge to score the output to see how well it behaves.</p><p>It&#8217;s okay, but it&#8217;s not scalable. Instead, we want to run it continuously: every time somebody uses our agent, we react in real time - save the logs, evaluate them, and display the results on the dashboard.</p><p>We can&#8217;t really continue using our coding agent for that, so we&#8217;ll need to create a stand-alone judge.</p><p>It will give you a real-time view of the agent&#8217;s quality: the performance on live traffic, and new cases that you can add to the gold standard dataset as they appear.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PHhl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PHhl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 424w, https://substackcdn.com/image/fetch/$s_!PHhl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 848w, https://substackcdn.com/image/fetch/$s_!PHhl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 1272w, https://substackcdn.com/image/fetch/$s_!PHhl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PHhl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png" width="1491" height="803" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:803,&quot;width&quot;:1491,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1664069,&quot;alt&quot;:&quot;Grafana dashboard for online evaluation with a good vs bad time series over 24 hours, a pie chart of judge verdicts, and a live traffic table showing each question with its verdict, score, and reason&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Grafana dashboard for online evaluation with a good vs bad time series over 24 hours, a pie chart of judge verdicts, and a live traffic table showing each question with its verdict, score, and reason" title="Grafana dashboard for online evaluation with a good vs bad time series over 24 hours, a pie chart of judge verdicts, and a live traffic table showing each question with its verdict, score, and reason" srcset="https://substackcdn.com/image/fetch/$s_!PHhl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 424w, https://substackcdn.com/image/fetch/$s_!PHhl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 848w, https://substackcdn.com/image/fetch/$s_!PHhl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 1272w, https://substackcdn.com/image/fetch/$s_!PHhl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0efa17bd-ebef-451a-b90e-30f97e7690b6_1491x803.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Online evaluation dashboard: judge evaluates the live traffic</figcaption></figure></div><p>Periodically inspect bad results, and if you see something new, add these cases to your gold standard dataset. Also sample the good results to make sure they are actually good.</p><p>For inspecting this live traffic I also often use coding assistants. I take a sample of the data, save it locally and ask them to go through these cases with me.</p><h2><strong>Multiple judges</strong></h2><p>We started with a good/bad evaluator, but I didn&#8217;t really explain what it means. And it anyways depends on your problem.</p><p>Over time, we will need to have multiple judges, each evaluating the agent from a different angle:</p><ul><li><p>Task completion: the agent actually solved the user&#8217;s problem</p></li><li><p>Relevance/Correctness: the answer is relevant/correct</p></li><li><p>Groundedness: the claims in the answer are supported by the documents in the knowledge base</p></li><li><p>Completeness: the answer includes all the required information</p></li><li><p>Following instructions: the agent follows the instructions correctly</p></li><li><p>Trajectory optimality: the sequence of tool calls the agent used to complete the task is an optimal one</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!unlv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!unlv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 424w, https://substackcdn.com/image/fetch/$s_!unlv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 848w, https://substackcdn.com/image/fetch/$s_!unlv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 1272w, https://substackcdn.com/image/fetch/$s_!unlv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!unlv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png" width="1120" height="760" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:760,&quot;width&quot;:1120,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;One agent run fanning out to six judges - task completion, correctness, groundedness, completeness, instructions, trajectory - which all feed into a single scorecard&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="One agent run fanning out to six judges - task completion, correctness, groundedness, completeness, instructions, trajectory - which all feed into a single scorecard" title="One agent run fanning out to six judges - task completion, correctness, groundedness, completeness, instructions, trajectory - which all feed into a single scorecard" srcset="https://substackcdn.com/image/fetch/$s_!unlv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 424w, https://substackcdn.com/image/fetch/$s_!unlv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 848w, https://substackcdn.com/image/fetch/$s_!unlv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 1272w, https://substackcdn.com/image/fetch/$s_!unlv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33683d03-f316-4572-8a12-5cac4bd19756_1120x760.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">One agent run, several judges, one metric per dimension</figcaption></figure></div><p>There are other criteria too that you can consider for your application.</p><p>But I usually recommend starting with one judge and then adding others as your system matures.</p><h2><strong>The agent evaluation checklist</strong></h2><p>As a summary for this article, I wanted to give you a checklist that you can use next time you work on a new agent.</p><pre><code><code>Gold standard dataset:

- [ ] Vibe-check the agent and log 10-15 interactions
- [ ] Label results as "good" or "bad" yourself
- [ ] Define equivalence partitions and test cases within each
- [ ] Include boundary cases (exact match, paraphrase, multiple matching documents, off-topic)
- [ ] Generate synthetic questions

The judge:

- [ ] judge.md describes the classification rules in generic terms, not specific to your examples
- [ ] The judge runs as a subagent that reads only judge.md
- [ ] Check the judge's answers with "agree" or "disagree"
- [ ] Use the fraction of "good" results as your metric
- [ ] Add more judges as your system matures

Improving the agent:

- [ ] The evaluation pipeline is automated: run the agent, run the judge, get the metric
- [ ] Each time you make a change, run the full evaluation set to catch regressions

Online evaluation:

- [ ] Put the agent in front of real users
- [ ] Collect user feedback
- [ ] Score online traffic with a stand-alone judge, show results on a dashboard
- [ ] Inspect bad results periodically, add new failing cases to the eval dataset</code></code></pre><p></p><p>If you want to learn more about building AI Agents, I teach that in my course <a href="https://maven.com/alexey-grigorev/from-rag-to-agents">AI Engineering Buildcamp: From RAG to Agents</a>. In the course, I also cover evaluation in a lot of detail with real code, real tools, and real agents.</p><p>The next cohort starts September 21. You can use code &#8220;SUBSTACK&#8221; to get 20% off. See you there!</p>]]></content:encoded></item><item><title><![CDATA[Deploy a Full-Stack App with AI Coding Assistants]]></title><description><![CDATA[Part 3 of AI Dev Tools Zoomcamp]]></description><link>https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding</link><guid isPermaLink="false">https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Mon, 10 Aug 2026 08:32:29 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!4sUJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>This is the third article in a series for <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a>, the free course we run at DataTalks.Club.</p><p>All articles in the series:</p><ul><li><p>Part 1: <a href="https://alexeyondata.substack.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a></p></li><li><p>Part 2: <a href="https://alexeyondata.substack.com/p/build-and-ship-a-full-stack-app-with">Build and Ship a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 3: <a href="https://alexeyondata.substack.com/p/deploy-a-full-stack-app-with-ai-coding">Deploy a Full-Stack App with AI Coding Assistants</a> (this article)</p></li><li><p><span>Part 4: </span><a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">DevOps and Observability for an AI-Built App</a></p></li><li><p>Part 5: <a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a></p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div id="youtube2-gxt5ZDVnBMM" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;gxt5ZDVnBMM&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/gxt5ZDVnBMM?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p>In this article, I want to take a web application and make it accessible for everyone on the internet.</p><p>We will:</p><ul><li><p>Take the application that&#8217;s already working locally</p></li><li><p>Containerize the application</p></li><li><p>Switch from SQLite to Postgres</p></li><li><p>Add integration tests</p></li><li><p>Deploy to AWS</p></li><li><p>Set up CI/CD via GitHub Actions</p></li></ul><p>It&#8217;s based on the second half of the full-day workshop I did for AI Shipping Labs: <a href="https://aishippinglabs.com/workshops/full-stack-vibe-coding">Build and Deploy a Full-Stack App with AI Coding Assistants</a>.</p><h2><strong>Recap</strong></h2><p>In the <a href="https://alexeyondata.substack.com/p/build-and-ship-a-full-stack-app-with">previous article</a>, we started building an application for system-design interviews. An interviewer creates a session and shares a link with a candidate. When the candidate updates something on the canvas, the interviewer sees the updates in real time.</p><ul><li><p>First, we created the frontend only with React</p></li><li><p>Then we created an OpenAPI specification for defining the frontend-backend API</p></li><li><p>Next, we created the backend from this specification</p></li><li><p>We added database support with SQLite and SQLAlchemy</p></li></ul><p>You can find the code in the <a href="https://github.com/alexeygrigorev/interview-canvas-share">interview-canvas-share</a> repository.</p><h2><strong>Containerization</strong></h2><p>When we run the application locally, we need to execute two commands: one for the frontend and one for the backend.</p><p>Let&#8217;s run them in two separate terminals:</p><pre><code><span># first terminal</span>
<span>cd</span> frontend
npm run dev

<span># second terminal</span>
make run</code></pre><p>So, we have two services, and we need to deploy them to production. We may think that we need two containers: one for the frontend and one for the backend.</p><p>During development, we run the frontend as a separate service because we use Vite. Vite watches the frontend code and refreshes the page when we make changes. It&#8217;s very convenient because we can see our changes immediately.</p><p>In production, the frontend code doesn&#8217;t change while the application is running. We build it once and get a set of static HTML, CSS, and JavaScript files.</p><p>This means we don&#8217;t need a separate container for the frontend: the backend can serve these files. So, we need only one container.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_QEl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_QEl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 424w, https://substackcdn.com/image/fetch/$s_!_QEl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 848w, https://substackcdn.com/image/fetch/$s_!_QEl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 1272w, https://substackcdn.com/image/fetch/$s_!_QEl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_QEl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png" width="1110" height="640" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:640,&quot;width&quot;:1110,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Three stages show the Vite frontend and FastAPI backend running separately in development, the frontend compiling into static files, and one application container serving the frontend and backend in production&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Three stages show the Vite frontend and FastAPI backend running separately in development, the frontend compiling into static files, and one application container serving the frontend and backend in production" title="Three stages show the Vite frontend and FastAPI backend running separately in development, the frontend compiling into static files, and one application container serving the frontend and backend in production" srcset="https://substackcdn.com/image/fetch/$s_!_QEl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 424w, https://substackcdn.com/image/fetch/$s_!_QEl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 848w, https://substackcdn.com/image/fetch/$s_!_QEl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 1272w, https://substackcdn.com/image/fetch/$s_!_QEl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7c551f-c669-4852-af03-4d2fa94916c2_1110x640.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">In development (1), the frontend and backend run separately. In production (2, 3), FastAPI serves the built frontend from one container</figcaption></figure></div><p>Ask the coding assistant to create it:</p><pre><code><code>Create a Dockerfile that builds the frontend with Node, then builds a Python image with the backend and the frontend static files.

Backend should serve the frontend.</code></code></pre><p>We get a two-stage Docker build:</p><ul><li><p>First, we use a Node.js image to compile the frontend</p></li><li><p>Then we build the backend and copy only the frontend files without the Node.js dependencies</p></li></ul><p>You can see the <a href="https://github.com/alexeygrigorev/interview-canvas-share/blob/main/Dockerfile">Dockerfile here</a>.</p><p>Build and run the image from the repository root:</p><pre><code>docker build -t sdip:latest <span>.</span>

docker run --rm -p 8000:8000 \
  -v sdip-data:/data \
  -e SDIP_DATABASE_URL=sqlite:////data/sdip.db \
  --name sdip sdip:latest</code></pre><p>Here we specify a named Docker volume <code>sdip-data</code> that will keep the SQLite database between container runs.</p><p>Open the application at <a href="http://localhost:8000/">localhost:8000</a> and test it:</p><ul><li><p>Create an interview session</p></li><li><p>Open the join link in a different browser window</p></li><li><p>Move an element in the candidate window</p></li><li><p>Check that the interviewer sees the change</p></li></ul><p>We&#8217;ll repeat this test again. I&#8217;ll refer to it as the two-session test. </p><p></p><h2><strong>Switch from SQLite to Postgres</strong></h2><p>SQLite is very convenient for local development. It keeps the data in a single file and doesn&#8217;t need a separate database server.</p><p>But for production, we typically use Postgres or a similar database.</p><p>When we set up the foundation <a href="https://alexeyondata.substack.com/p/build-and-ship-a-full-stack-app-with">in the previous article</a>, we asked the coding agent to use SQLAlchemy.</p><p>I did this on purpose because I knew that later I&#8217;d switch to Postgres.</p><p>Start Postgres locally:</p><pre><code>docker run -d \
  --name interview-canvas-db \
  -e POSTGRES_USER=sdip \
  -e POSTGRES_PASSWORD=sdip \
  -e POSTGRES_DB=sdip \
  -p 5432:5432 \
  -v interview-canvas-pgdata:/var/lib/postgresql/data \
  postgres:16-alpine</code></pre><p>Now we can ask the assistant to use it:</p><pre><code><code>Add Postgres support to the backend.</code></code></pre><p>Run the backend against Postgres from the repository root:</p><pre><code><span>export</span> SDIP_DATABASE_URL=postgresql://sdip:sdip@localhost:5432/sdip
make run</code></pre><p>When it&#8217;s done, repeat the two-session test.</p><h2><strong>Docker Compose</strong></h2><p>Previously, I started a Postgres container with a separate command. But now let&#8217;s put all the services our application needs inside one Docker Compose file.</p><p>With this file, we can run our entire application with a single command <code>docker compose up</code>.</p><p>Ask the assistant to implement it:</p><pre><code><code>Create docker-compose.yaml with two services: Postgres and the app.</code></code></pre><p>The file defines the database and adds a health check, so our application waits until Postgres is ready to accept connections.</p><p>Start it:</p><pre><code>docker compose up --build</code></pre><p>In our case, it runs the application at <a href="http://localhost:8100/">localhost:8100</a>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!e6eo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!e6eo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 424w, https://substackcdn.com/image/fetch/$s_!e6eo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 848w, https://substackcdn.com/image/fetch/$s_!e6eo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 1272w, https://substackcdn.com/image/fetch/$s_!e6eo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!e6eo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png" width="1110" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:1110,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The application moves from a container using a local SQLite file to Docker Compose with separate application and Postgres services backed by a data volume&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The application moves from a container using a local SQLite file to Docker Compose with separate application and Postgres services backed by a data volume" title="The application moves from a container using a local SQLite file to Docker Compose with separate application and Postgres services backed by a data volume" srcset="https://substackcdn.com/image/fetch/$s_!e6eo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 424w, https://substackcdn.com/image/fetch/$s_!e6eo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 848w, https://substackcdn.com/image/fetch/$s_!e6eo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 1272w, https://substackcdn.com/image/fetch/$s_!e6eo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49e64777-96c9-41ed-b3ab-a1d6cedfde49_1110x440.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">We run the application with Postgres using Docker Compose</figcaption></figure></div><h2><strong>Integration and end-to-end tests</strong></h2><p>The AI assistant might have created some backend tests.</p><p>If it didn&#8217;t, ask it to create integration tests:</p><pre><code><code>Create integration tests that run against docker-compose.yaml. What scenarios should we test?</code></code></pre><p>We added two things that can potentially break, so let&#8217;s test them too. We&#8217;ll verify that:</p><ul><li><p>The frontend compiles correctly</p></li><li><p>The backend can communicate with Postgres</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Th6Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Th6Z!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 424w, https://substackcdn.com/image/fetch/$s_!Th6Z!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 848w, https://substackcdn.com/image/fetch/$s_!Th6Z!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 1272w, https://substackcdn.com/image/fetch/$s_!Th6Z!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Th6Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png" width="1100" height="650" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:650,&quot;width&quot;:1100,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Playwright controls separate interviewer and candidate browser sessions connected through a shared WebSocket room to FastAPI and Postgres&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Playwright controls separate interviewer and candidate browser sessions connected through a shared WebSocket room to FastAPI and Postgres" title="Playwright controls separate interviewer and candidate browser sessions connected through a shared WebSocket room to FastAPI and Postgres" srcset="https://substackcdn.com/image/fetch/$s_!Th6Z!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 424w, https://substackcdn.com/image/fetch/$s_!Th6Z!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 848w, https://substackcdn.com/image/fetch/$s_!Th6Z!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 1272w, https://substackcdn.com/image/fetch/$s_!Th6Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f5aa3d7-caf7-48b2-a00c-68245dcd739a_1100x650.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Playwright can run the two-session test for us</figcaption></figure></div><p>Ask the AI assistant to implement a test:</p><pre><code><code>Add an end-to-end test that runs against docker-compose.yaml.

Use Playwright to:

1. Log in as the interviewer (session 1).
2. Create an interview session.
3. Share the join link.
4. Join from a separate client as the candidate (session 2).
5. Change the canvas as the candidate (session 2).
6. Verify that the interviewer sees the change (session 1).

Put the tests in the e2e/ folder in the repository root.</code></code></pre><p>After it finishes, we can run the tests with a single make command:</p><pre><code><code>make e2e</code></code></pre><h2><strong>Deploy to AWS</strong></h2><p>We&#8217;re now certain that the application works well. We can deploy it.</p><p>Our application runs in a container and only needs Postgres, so we have a lot of options for deploying it. We can use Render, Railway, Fly.io, or any other managed container system.</p><p><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp/tree/main/cohorts/2025/02-end-to-end">Last year we deployed to Render</a>, but this year I want to deploy to AWS. You don&#8217;t have to use AWS, and instead you can ask the coding assistant to recommend an environment for your application.</p><p>Ask your assistant to deploy it:</p><pre><code><code>Deploy this application to AWS. Use AWS CloudFormation.</code></code></pre><p>For that to work, you need to have an AWS user. I typically create a temporary user with admin permissions and watch every step of what the coding agents are doing.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4sUJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4sUJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 424w, https://substackcdn.com/image/fetch/$s_!4sUJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 848w, https://substackcdn.com/image/fetch/$s_!4sUJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 1272w, https://substackcdn.com/image/fetch/$s_!4sUJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4sUJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png" width="1180" height="510" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:510,&quot;width&quot;:1180,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A public browser connects to Caddy over HTTPS and WebSockets; Caddy, the application, Postgres, and its pgdata volume run inside one EC2 boundary managed by CloudFormation&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A public browser connects to Caddy over HTTPS and WebSockets; Caddy, the application, Postgres, and its pgdata volume run inside one EC2 boundary managed by CloudFormation" title="A public browser connects to Caddy over HTTPS and WebSockets; Caddy, the application, Postgres, and its pgdata volume run inside one EC2 boundary managed by CloudFormation" srcset="https://substackcdn.com/image/fetch/$s_!4sUJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 424w, https://substackcdn.com/image/fetch/$s_!4sUJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 848w, https://substackcdn.com/image/fetch/$s_!4sUJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 1272w, https://substackcdn.com/image/fetch/$s_!4sUJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbf2a4c2-b165-4212-815d-9c5e66751e99_1180x510.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">One EC2 instance runs Caddy, the app, and Postgres. We manage it through CloudFormation.</figcaption></figure></div><p>You can see what I got <a href="https://github.com/alexeygrigorev/interview-canvas-share/blob/main/deploy/aws/sdip-stack.yaml">here</a>. It runs the app, Postgres, and <a href="https://caddyserver.com/">Caddy</a> (adds HTTPS and WSS support for our app) on one EC2 instance.</p><p>It&#8217;s fine for a proof-of-concept, but using managed database services (such as RDS) is better. We will not do it here.</p><h2><strong>CI/CD with GitHub Actions</strong></h2><p>We used a user with the admin permissions to deploy the application. It&#8217;s okay for the first deployment, but only when we carefully watch it. The next step is to configure CI/CD and remove that access.</p><ul><li><p>Continuous integration (CI) means that every time we make a change and push it to GitHub, we automatically run all the tests to make sure we didn&#8217;t break anything.</p></li><li><p>Continuous deployment (CD) is about deploying this change automatically.</p></li></ul><p>In GitHub, we use GitHub Actions for that.</p><p>Let&#8217;s configure it. Every time we make a push to main, we want to:</p><ul><li><p>run frontend and backend tests</p></li><li><p>build the containers</p></li><li><p>run the integration tests</p></li><li><p>run the end-to-end tests</p></li><li><p>if all the tests pass, deploy the new version</p></li></ul><p>For the last step, the runner (the process that will deploy the application) will need to be able to access our AWS infrastructure. We will use OpenID Connect (OIDC) for this: the runner will assume a role with the necessary permissions and update the application.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!toEY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!toEY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 424w, https://substackcdn.com/image/fetch/$s_!toEY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 848w, https://substackcdn.com/image/fetch/$s_!toEY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 1272w, https://substackcdn.com/image/fetch/$s_!toEY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!toEY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png" width="1440" height="500" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A push to main starts frontend and backend tests in parallel, then builds the Docker Compose stack, runs integration and end-to-end tests, and deploys to AWS through OIDC&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A push to main starts frontend and backend tests in parallel, then builds the Docker Compose stack, runs integration and end-to-end tests, and deploys to AWS through OIDC" title="A push to main starts frontend and backend tests in parallel, then builds the Docker Compose stack, runs integration and end-to-end tests, and deploys to AWS through OIDC" srcset="https://substackcdn.com/image/fetch/$s_!toEY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 424w, https://substackcdn.com/image/fetch/$s_!toEY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 848w, https://substackcdn.com/image/fetch/$s_!toEY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 1272w, https://substackcdn.com/image/fetch/$s_!toEY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbce4dae7-81ce-4916-b27f-3d0fa1d6ef7c_1440x500.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">GitHub Actions runs frontend and backend tests in parallel, checks the full stack, and then deploys to AWS</figcaption></figure></div><p>Create the role and the workflow:</p><pre><code><code>Create a CI/CD pipeline that:

- runs backend and frontend tests in parallel
- builds the Docker Compose stack and runs integration and end-to-end tests against it
- deploys to AWS using a GitHub OIDC role
- validates that the deploy is successful by checking the health endpoint</code></code></pre><p>The <a href="https://github.com/alexeygrigorev/interview-canvas-share/blob/main/.github/workflows/ci.yml">finished workflow</a> uses a restricted AWS role for deployment.</p><p>Change something in the application, commit, and push to see it go live.</p><h2><strong>Clean up</strong></h2><p>When we&#8217;re done, we need to delete all the resources created by CloudFormation:</p><pre><code>aws cloudformation delete-stack --stack-name sdip
aws cloudformation <span>wait</span> stack-delete-complete --stack-name sdip</code></pre><p></p><h2><strong>The next step after deployment</strong></h2><p>Let&#8217;s recap what we have done so far:</p><ul><li><p>We created the frontend application with React</p></li><li><p>Then we defined the frontend-backend API with OpenAPI specs</p></li><li><p>Based on the specs, we created the backend</p></li><li><p>Next, we added database support with SQLite and SQLAlchemy to the backend</p></li><li><p>To make it easier to deploy the application, we put both frontend and backend inside one container. The backend serves the frontend.</p></li><li><p>To go to production, we replaced SQLite with Postgres.</p></li><li><p>Next, we simplified running everything locally with Docker Compose</p></li><li><p>Once everything was in Compose, we created an end-to-end test</p></li><li><p>We took the application that worked locally and deployed it to AWS using CloudFormation</p></li><li><p>Finally, we created a CI/CD deployment pipeline to deploy every change automatically.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oFTO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oFTO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 424w, https://substackcdn.com/image/fetch/$s_!oFTO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 848w, https://substackcdn.com/image/fetch/$s_!oFTO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 1272w, https://substackcdn.com/image/fetch/$s_!oFTO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oFTO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png" width="1180" height="650" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:650,&quot;width&quot;:1180,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Separate interviewer and candidate browser sessions connect through Caddy to the React and FastAPI application, which stores sessions and canvas state in Postgres and its pgdata volume inside one EC2 instance&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Separate interviewer and candidate browser sessions connect through Caddy to the React and FastAPI application, which stores sessions and canvas state in Postgres and its pgdata volume inside one EC2 instance" title="Separate interviewer and candidate browser sessions connect through Caddy to the React and FastAPI application, which stores sessions and canvas state in Postgres and its pgdata volume inside one EC2 instance" srcset="https://substackcdn.com/image/fetch/$s_!oFTO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 424w, https://substackcdn.com/image/fetch/$s_!oFTO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 848w, https://substackcdn.com/image/fetch/$s_!oFTO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 1272w, https://substackcdn.com/image/fetch/$s_!oFTO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37eae46e-c552-4478-a3ee-f347bab387bd_1180x650.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The end-state architecture: two browser sessions connect through Caddy to the app and persistent Postgres data</figcaption></figure></div><p>But we still need to do a lot more for the app to run reliably:</p><ul><li><p>Dev and prod environments</p></li><li><p>Observability: logs, metrics, alerts</p></li><li><p>Using AI as the first responder when the application stops working</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!54DZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!54DZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 424w, https://substackcdn.com/image/fetch/$s_!54DZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 848w, https://substackcdn.com/image/fetch/$s_!54DZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 1272w, https://substackcdn.com/image/fetch/$s_!54DZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!54DZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png" width="1456" height="766" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:766,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A user pushes code to main and manually promotes a release from development to production; the Observe group collects telemetry and raises alerts, and the Respond group uses an AI agent to create a fix&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A user pushes code to main and manually promotes a release from development to production; the Observe group collects telemetry and raises alerts, and the Respond group uses an AI agent to create a fix" title="A user pushes code to main and manually promotes a release from development to production; the Observe group collects telemetry and raises alerts, and the Respond group uses an AI agent to create a fix" srcset="https://substackcdn.com/image/fetch/$s_!54DZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 424w, https://substackcdn.com/image/fetch/$s_!54DZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 848w, https://substackcdn.com/image/fetch/$s_!54DZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 1272w, https://substackcdn.com/image/fetch/$s_!54DZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F739cee86-2195-4336-8ebd-1e424a30e6e1_1520x800.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The user promotes releases while separate observe and respond loops detect production problems and create fixes</figcaption></figure></div><p>We&#8217;ll cover it in the next lesson.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;5c023f9c-ee7f-4a0c-9c58-7372abcf5aed&quot;,&quot;caption&quot;:&quot;This is the fourth article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;DevOps and Observability for an AI-Built App&quot;,&quot;publishedBylines&quot;:[{&quot;id&quot;:2259961,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;,&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null}],&quot;post_date&quot;:&quot;2026-08-18T11:54:47.884Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!i4Rw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd1811ce-b404-4c3c-ac2a-f338909c95c0_1830x760.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://aishippingblog.com/p/devops-and-observability-for-an-ai&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:211674506,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:12,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[Getting an AI Engineering Job]]></title><description><![CDATA[The 5-step framework that will land you a job]]></description><link>https://aishippingblog.com/p/getting-an-ai-engineering-job</link><guid isPermaLink="false">https://aishippingblog.com/p/getting-an-ai-engineering-job</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Sat, 08 Aug 2026 18:25:44 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/1b652b6c-b174-4918-82f2-4046e84098e2_1200x630.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><span>One of the first videos I ever published on YouTube was about getting a data science job. I presented the algorithm that you can follow to get hired in data science. Many things have changed since then, but the algorithm that I described still works.</span></p><div id="youtube2-jYYR1fH8k7o" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;jYYR1fH8k7o&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/jYYR1fH8k7o?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p>In this article I take that algorithm and update it for AI Engineering. The updates are minimal and you can use this algorithm for any other engineering job. </p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h1><span>The Job Search Algorithm</span></h1><p><span>The algorithm is straightforward:</span></p><ol><li><p><span>Network: Connect with people</span></p></li><li><p><span>Apply: Apply for jobs even if you don&#8217;t qualify</span></p></li><li><p><span>Build: Create projects and share them online</span></p></li><li><p><span>Interview: Record what you were asked</span></p></li><li><p><span>Reflect: Review and improve</span></p></li><li><p><span>Repeat</span></p></li></ol><p><span>This is a while loop. Continue networking, applying, building, and interviewing until you receive an offer you like.</span></p><div class="image-gallery-embed" data-attrs="{&quot;gallery&quot;:{&quot;images&quot;:[{&quot;type&quot;:&quot;image/svg+xml&quot;,&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f4fda882-960c-4d75-8053-baaa525618b9_812x770.svg&quot;}],&quot;caption&quot;:&quot;The framework is a while loop&quot;,&quot;alt&quot;:&quot;&quot;,&quot;staticGalleryImage&quot;:{&quot;type&quot;:&quot;image/svg+xml&quot;,&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f4fda882-960c-4d75-8053-baaa525618b9_812x770.svg&quot;}},&quot;isEditorNode&quot;:true}"></div><h1><span>Step 1: Network</span></h1><p>Networking helps you understand the market, find opportunities, and learn what companies are actually looking for. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8kFy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8kFy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 424w, https://substackcdn.com/image/fetch/$s_!8kFy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 848w, https://substackcdn.com/image/fetch/$s_!8kFy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 1272w, https://substackcdn.com/image/fetch/$s_!8kFy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8kFy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png" width="599" height="331.17788461538464" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/df7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:805,&quot;width&quot;:1456,&quot;resizeWidth&quot;:599,&quot;bytes&quot;:390646,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/210233610?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8kFy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 424w, https://substackcdn.com/image/fetch/$s_!8kFy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 848w, https://substackcdn.com/image/fetch/$s_!8kFy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 1272w, https://substackcdn.com/image/fetch/$s_!8kFy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf7ece07-35df-460b-b34c-ff365bcf7313_1906x1054.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Screenshot from my talk</figcaption></figure></div><p><span>Use LinkedIn and meetups for that.</span></p><p><span>Talk to people working in your field. Talk to people who work as AI engineers. Ask about the projects they work on, and the technologies they use. And ask if they are hiring and ask for a referral.</span></p><p>If you are a junior, look for other juniors who have recently got a job. Ask what helped them get hired, which companies they targeted, what the interview process involved, and what they would do differently.</p><p>A conversation doesn&#8217;t have to <span>end with a referral. By talking to people you discover companies that hire, understand which skills to develop, improve your search strategy, and meet interesting people.</span></p><h1><span>Step 2: Apply</span></h1><p>Start applying for jobs right now. Many people postpone it because &#8220;they are not ready". You are ready. Apply even if you don&#8217;t qualify.</p><p>Don&#8217;t reject yourself because you are unsure if you&#8217;re qualified. Apply and let the hiring team evaluate your application. If you apply, you have a chance to move forward. If you don&#8217;t, you never will.</p><p><span>When you look at the job requirements, don&#8217;t treat them all as mandatory. The job description always describes the ideal candidate and often these candidates don&#8217;t exist. </span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!i1Nx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!i1Nx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 424w, https://substackcdn.com/image/fetch/$s_!i1Nx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 848w, https://substackcdn.com/image/fetch/$s_!i1Nx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 1272w, https://substackcdn.com/image/fetch/$s_!i1Nx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!i1Nx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png" width="1382" height="1090" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1090,&quot;width&quot;:1382,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:217492,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/210233610?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F027dd047-22ee-4568-8fc6-d28df4005589_1615x1092.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!i1Nx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 424w, https://substackcdn.com/image/fetch/$s_!i1Nx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 848w, https://substackcdn.com/image/fetch/$s_!i1Nx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 1272w, https://substackcdn.com/image/fetch/$s_!i1Nx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff99fe152-6fd2-446d-a5a8-1f41aaf55add_1382x1090.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Job descriptions often describe the ideal candidate who doesn&#8217;t exist.</figcaption></figure></div><p>There are, of course, cases where you shouldn&#8217;t apply. If you&#8217;re a junior, but the position is senior, then don&#8217;t apply. If you match only 20% of the requirements, it&#8217;s also a bad sign. If you&#8217;re in doubt, apply.   </p><p><span>If you apply for relevant roles but don&#8217;t hear back, review your CV. Make sure it shows the experience, skills, and projects that are relevant to the positions you apply. </span></p><p><span>Recently I published a workshop at AI Shipping Labs about </span><a href="https://aishippinglabs.com/workshops/tailor-cv-ai-engineering"><span>tailoring your CV for AI engineering roles</span></a><span> that you may find helpful (heads up: it requires a paid subscription). </span></p><h1><span>Step 3: Build</span></h1><p><span>You learn by doing. When you apply, note which skills are required for the positions you apply for, and build projects using these technologies.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_EE_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_EE_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!_EE_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!_EE_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!_EE_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_EE_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:851143,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/210233610?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_EE_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!_EE_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!_EE_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!_EE_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65d23ac0-8491-4123-807c-f50b32632c7c_1672x941.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Another screenshot from my talk</figcaption></figure></div><p>Always timebox these projects. Set the time limit to 10-20 hours and then move on to the next project. At this stage, your goal is to have an awareness of different technologies and an understanding when to use them. You can go deeper when you&#8217;re already working.</p><p><span>I wrote a full guide on building projects with six steps you can follow: </span></p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;9b899937-4a75-42ea-a81f-5867f21ddeb7&quot;,&quot;caption&quot;:&quot;When preparing personal plans for AI Shipping Labs members, I often get the question, &#8220;What should I build for my portfolio?&#8221; There are too many options, and no clear way to pick one.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Choosing a Portfolio Project. The Definite Guide&quot;,&quot;publishedBylines&quot;:[{&quot;bestseller_tier&quot;:null,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;id&quot;:2259961,&quot;is_guest&quot;:false,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;}],&quot;post_date&quot;:&quot;2026-07-03T15:17:33.164Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!qTlw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cece23b-8531-40f2-83b8-a3fdca57ad00_2314x1636.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://alexeyondata.substack.com/p/choosing-a-portfolio-project-the&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:204912260,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:20,&quot;comment_count&quot;:5,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p><span>A complete project should include working code in a public GitHub repository and a clear README. The README is the most important document of your project, so make sure you put enough effort into polishing it before you add a project into your portfolio. </span></p><p><span>A few weeks ago I wrote an article about that:</span></p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;cbd8e701-1778-44c3-830c-a5f30135622c&quot;,&quot;caption&quot;:&quot;Your README is the first file people read in your project, and sometimes the only one.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;How to Write a Good README&quot;,&quot;publishedBylines&quot;:[{&quot;bestseller_tier&quot;:null,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;id&quot;:2259961,&quot;is_guest&quot;:false,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;}],&quot;post_date&quot;:&quot;2026-07-16T19:40:57.336Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!Lao2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F956ba912-9f3c-4017-9ef9-523280df5d31_1768x970.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://alexeyondata.substack.com/p/how-to-write-a-good-readme&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:207284212,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:13,&quot;comment_count&quot;:2,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p>Don&#8217;t build these projects silently. Share what you build and what you learned online.</p><p>Here&#8217;s a suggested sequence of posts for LinkedIn and/or X:</p><ol><li><p>Describe the project idea and the technologies</p></li><li><p>For each milestone, post a screenshot or a code snippet with what you built </p></li><li><p>When the project is ready, announce it and publish the link</p></li><li><p>Optionally, write a blog post </p></li></ol><p>And yes, there&#8217;s another article I wrote about it:</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;3eba0eb0-a23a-430f-8bb6-793a5cee3e2e&quot;,&quot;caption&quot;:&quot;One Idea I Want to Share this Week&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Benefits of Learning in Public and Why It Works&quot;,&quot;publishedBylines&quot;:[{&quot;bestseller_tier&quot;:null,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;id&quot;:2259961,&quot;is_guest&quot;:false,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;}],&quot;post_date&quot;:&quot;2026-01-02T07:01:30.583Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!FXiq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6ad71eb-5e9a-44c2-8ea0-b0b862aef333_930x735.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://alexeyondata.substack.com/p/benefits-of-learning-in-public-and&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:182086832,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:26,&quot;comment_count&quot;:4,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p></p><h1><span>Step 4: Interview</span></h1><p><span>Once your CV gets past the screening stage, you get invited to an interview. </span></p><p>Don&#8217;t treat interviews as exams. You&#8217;re not in school. It&#8217;s a two-way process: they evaluate you, but you also evaluate them. </p><p>Interviewing is a also skill. Your first interview may not be great, but it improves with practice. Each subsequent interview helps you identify gaps in your preparation and see what you should do differently next time.</p><p>Some interviews will lead to rejections. It&#8217;s normal and may happen for many reasons outside of your control: the company chose someone with more suitable experience, hired an internal candidate, changed the role, or froze hiring. Don&#8217;t take the rejections personally, learn from them and move on.</p><div class="image-gallery-embed" data-attrs="{&quot;gallery&quot;:{&quot;images&quot;:[{&quot;type&quot;:&quot;image/jpeg&quot;,&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bff5a82e-53ee-42dd-ad2a-b0e8882f4903_1670x783.jpeg&quot;}],&quot;caption&quot;:&quot;The interview process&quot;,&quot;alt&quot;:&quot;&quot;,&quot;staticGalleryImage&quot;:{&quot;type&quot;:&quot;image/jpeg&quot;,&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bff5a82e-53ee-42dd-ad2a-b0e8882f4903_1670x783.jpeg&quot;}},&quot;isEditorNode&quot;:true}"></div><p>The hiring process varies between companies, but typically you will have some of these interviews in the pipeline:</p><ol><li><p>Recruiter interview: a short conversation about your background, motivation, availability, salary expectations, and basic fit</p></li><li><p>Theoretical interview: questions about LLMs, RAG, agents</p></li><li><p>Coding interview: coding, Python, SQL, code review, or practical engineering tasks</p></li><li><p>Project deep dive: a deeper discussion of your experience and past projects</p></li><li><p>System design: defining the architecture of a project</p></li><li><p>Behavioral interview: examples of how you dealt with different situations at work (and outside)</p></li><li><p>Take-home assignment: a small project you complete at home</p></li></ol><p>You probably won&#8217;t get all of them: some stages may be bundled in one round, and other stages may be absent.</p><p>Now let&#8217;s go through each step of the hiring pipeline. In this article, I mostly used the materials from <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide">AI Engineering Field Guide</a>. You will find a lot more relevant information there if you&#8217;re preparing for AI Engineering interviews.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://github.com/alexeygrigorev/ai-engineering-field-guide&quot;,&quot;text&quot;:&quot;&#11088; Star the Field Guide repo&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://github.com/alexeygrigorev/ai-engineering-field-guide"><span>&#11088; Star the Field Guide repo</span></a></p><p> </p><h2><span>1. Recruiter interview</span></h2><p><span>The recruiter is the first person you speak with during the interview process.</span></p><p>It&#8217;s not a technical interview. They will tell you about the company, the role and ask some basic questions.</p><p><span>You should be ready to answer:</span></p><ul><li><p>&#8220;What do you know about our company?&#8221; Make sure you do some basic research. If you can&#8217;t answer it, the interview process is likely to stop there (it happened to me!).</p></li><li><p>&#8220;What are your salary expectations?&#8221; Research the market and be ready to provide a realistic range. You can dodge this question and delay it to other stages, but I don&#8217;t think it&#8217;s helpful.</p></li><li><p>&#8220;Tell me about yourself&#8221;. Prepare a 3-minutes story. It will be useful at later stages too. </p></li></ul><p>Use this interview to make sure you understand what the interview process looks like and what the expectations are at each stage of the process. Ask for tips how to prepare for these stages. </p><h2><span>2. Theoretical interview</span></h2><p>There will be one or multiple rounds where the company checks if you <span>understand how LLM applications work. They may appear in a dedicated technical interview or as part of the project deep dive or AI system design sessions.</span></p><p><span>For AI Engineering, you will be asked about:</span></p><ul><li><p><span>LLM behavior, prompting, context windows, and structured outputs</span></p></li><li><p><span>RAG, retrieval, reranking, and common ways RAG fails</span></p></li><li><p><span>Agents, tool use, tool design</span></p></li><li><p><span>Evaluation and debugging</span></p></li><li><p><span>Monitoring, cost, latency, and scalability</span></p></li></ul><p><span>Fine-tuning, model training, and transformer internals can appear too, but this only happens for roles that require these skills.</span></p><p>You can see theoretical questions in <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide/blob/main/interview/questions/01-theory.md">the &#8220;Theory&#8221; part of AI Engineering Field Guide interview preparation section</a>.</p><h2><span>3. Technical interview</span></h2><p><span>Coding rounds generally fall into two categories: implementation and algorithm rounds.</span></p><p><span>Traditionally, for implementation rounds, you&#8217;re given a task like &#8220;implement a web crawler&#8221;. You&#8217;re not expected to deliver a fully working solution, but rather demonstrate your approach to coding. You may need to design (in the &#8220;object-oriented design&#8221; sense of the word) the interaction between different components, define proper abstractions, and then implement only some parts of it. </span></p><p>However, more and more often companies give a vague task and ask you to use AI to implement it. Y<span>ou are expected to produce a reasonably complete working solution. The interviewer evaluates how you interact with the coding agent, how you review the code, test it, correct errors, and refine the implementation.</span></p><p>The algorithm rounds usually involve easy and medium Leetcode-style problems. Hard problems are rare. There is already a lot of information on the Internet on how to prepare for them, so I won&#8217;t spend time on it. Joining Leetcode and solving some problems there definitely helps. </p><p>You can see technical questions in <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide/blob/main/interview/questions/02-coding.md">the &#8220;Coding&#8221; part of AI Engineering Field Guide interview preparation section</a>.</p><h2>4. Project deep dive</h2><p>Project deep dive interviews assess your level of seniority and check if your CV matches the reality.</p><p>At my previous jobs I regularly conducted these interviews.</p><p>I start by asking the candidate to introduce themselves and then ask what they were doing recently. Our discussion naturally goes into projects they worked on and then I start going deeper with questions: </p><ul><li><p>What problem does this project solve?</p></li><li><p>How does it work?</p></li><li><p>Which part did you build?</p></li><li><p>Who made the technical decisions? How?</p></li><li><p>What went wrong?</p></li><li><p>What would you change if you were implementing this project today?</p></li></ul><p>This is by no means an exhaustive list, but it should give you some ideas of what to expect. </p><p>I recently had a workshop about it at AI Shipping Labs. Based on it, I wrote an article: <a href="https://aishippinglabs.com/blog/ai-engineering-hiring-manager-interview">AI Engineering Hiring Manager Interview</a> (it&#8217;s free).</p><p>You can see deep-dive questions in <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide/blob/main/interview/questions/03-project-deep-dive.md">the &#8220;Project Deep-Dive&#8221; part of AI Engineering Field Guide interview preparation section</a>.</p><h2><span>5. System design </span></h2><p>If you&#8217;re applying for more senior roles, you&#8217;ll have a system design round. They are very uncommon for junior roles.</p><p>There are two types of system design interviews: &#8220;classical&#8221; system design and AI system design. </p><p>AI engineers are software engineers who specialize in AI, so they are expected to design software systems too. That&#8217;s why one of the interview rounds may be classical system design.</p><p>In these interviews you will need to design a system related to what the company and the prospective team do.</p><p>A food delivery company may ask you to design an order-processing or delivery-tracking system. When I interviewed at Yandex, they asked me to design search. When I interviewed with Meta, they asked me to design a geo-aware recommender system. When I worked at OLX and we were hiring people to join the recommender platform, we were asking to design a recommender system for an online marketplace. </p><p>But for AI engineers there&#8217;s also an AI system design round. <span>Common tasks include designing a document Q&amp;A system, a coding assistant, an AI search engine, or an agent. The specific system will depend on what the team is doing. </span></p><p><span>For these interviews, clarify the requirements, ask which data is available and what&#8217;s the expected scale. After that, sketch the main components of the system.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ygvD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ygvD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!ygvD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!ygvD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!ygvD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ygvD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:914668,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/210233610?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ygvD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!ygvD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!ygvD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!ygvD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f795536-ef95-441f-a072-ce5bf3772092_1672x941.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>You can see AI system design questions in <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide/blob/main/interview/questions/04-ai-system-design.md">the &#8220;AI System Design&#8221; part of AI Engineering Field Guide interview preparation section</a>.</p><h2><span>6. Behavioral interview</span></h2><p><span>The behavioral interview checks how you collaborate, make decisions, take responsibility, and handle setbacks. &#8220;Tell me about a time you needed to help a colleague&#8221;. There&#8217;s nothing specific in these interviews for AI engineers. </span></p><p>The best way to prepare for these interviews is to research the company&#8217;s values. You can ask the recruiter about them too. For each value, prepare examples of how you demonstrated them. As a general reference, I recommend <a href="https://www.amazon.jobs/content/en/our-workplace/leadership-principles"><span>Amazon&#8217;s Leadership Principles</span></a><span>.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!maIn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!maIn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 424w, https://substackcdn.com/image/fetch/$s_!maIn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 848w, https://substackcdn.com/image/fetch/$s_!maIn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 1272w, https://substackcdn.com/image/fetch/$s_!maIn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!maIn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png" width="1456" height="716" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:716,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:349898,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/210233610?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!maIn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 424w, https://substackcdn.com/image/fetch/$s_!maIn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 848w, https://substackcdn.com/image/fetch/$s_!maIn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 1272w, https://substackcdn.com/image/fetch/$s_!maIn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d102dd-872e-40f6-813a-a5b8ff964387_1456x716.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Amazon leadership principles</figcaption></figure></div><p>You can see behavioral questions in <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide/blob/main/interview/questions/05-behavioral.md">the &#8220;Behavioral&#8221; part of AI Engineering Field Guide interview preparation section</a>.</p><p></p><h2><span>7. Take-home assignment</span></h2><p>In take-home <span>assignments you need to build a small AI application, submit the code, a README, and a working demo. It&#8217;s also typical to have a defense session where you need to present the solution and answer questions about it.</span></p><div class="image-gallery-embed" data-attrs="{&quot;gallery&quot;:{&quot;alt&quot;:&quot;&quot;,&quot;caption&quot;:&quot;A take-home assignment for an AI engineer position&quot;,&quot;images&quot;:[{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54894fa5-d68a-4870-b223-555fee1530c9_1080x600.png&quot;,&quot;type&quot;:&quot;image/png&quot;}],&quot;staticGalleryImage&quot;:{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54894fa5-d68a-4870-b223-555fee1530c9_1080x600.png&quot;,&quot;type&quot;:&quot;image/png&quot;}},&quot;isEditorNode&quot;:true}"></div><p><span>Before starting, clarify the requirements, ask if you can use AI tools, and what the evaluation criteria are.</span></p><p><span>Before finishing, </span>make sure you have tests, an evaluation set, and a good README that describes the system well.</p><p>Take-home assignments are time-consuming. They are a good addition to your project portfolio, but if you interview with multiple companies, and all of them give you homework, it can add up.</p><p>If you&#8217;re short on time, it&#8217;s okay to decline them. When you do, make sure you have projects in your portfolio that demonstrate the qualities they want to assess. Then you can say <span>&#8220;I unfortunately don&#8217;t have time to work on it, but I have similar projects on GitHub and can walk you through one&#8221;. Don&#8217;t expect it to work with everyone, but some companies may agree to this.</span></p><p><span>When getting a home task, watch out for warning signs like these:</span></p><ul><li><p><span>You get an automated take-home assignment right after applying, even before speaking to anyone.</span></p></li><li><p><span>The company states the task takes two hours, but the requirements suggest it could take several days.</span></p></li><li><p><span>The company won&#8217;t clarify how your submission will be assessed or utilized.</span></p></li></ul><p>In these cases you may spend time working on the test and never hear back from the company after submitting your project. </p><p>You can see examples of tasks in <a href="https://github.com/alexeygrigorev/ai-engineering-field-guide/blob/main/interview/questions/06-home-assignments.md">the &#8220;Home Assignments&#8221; part of AI Engineering Field Guide interview preparation section</a>.</p><p></p><h1><span>Step 5: Reflect (after the interview)</span></h1><p><span>After each interview, write down all the questions you were asked and tasks you were given. What was difficult? Which answers could have been better?</span></p><p><span>You can also talk to AI, describe in detail how the interview went, and ask for feedback. </span></p><p><span>Write that all down in your interview journal.</span></p><p><span>This will help you identify areas where you can improve, and give better answers in the next interviews.</span></p><p><span>If you&#8217;re rejected, ask the company for feedback. Many companies will decline: there&#8217;s a non-zero chance that you may sue them and use this feedback against them, so they won&#8217;t risk it. But some will share it - most likely over a phone call. Also, don&#8217;t take it personally. Rejections are part of the process.</span></p><p>Continue networking, applying for more jobs, keep building and sharing what you learn. </p><p><span>If you get an offer - congratulations! But don&#8217;t rush into accepting it. Evaluate if the package is good and if you will like working there.</span></p><p><span>If you&#8217;re interviewing with other companies, finish the process with them too. It&#8217;s okay to be explicit about it with the company that made the offer. If they are pushing you to accept it as soon as possible, this is a red flag, so consider this carefully before agreeing. You got one offer, so you&#8217;ll definitely get others. </span></p><p><span>Good luck and have fun!</span></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://alexeyondata.substack.com/?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share&quot;,&quot;text&quot;:&quot;Share Alexey On Data&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://alexeyondata.substack.com/?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share"><span>Share Alexey On Data</span></a></p><p></p><h1><span>What I&#8217;ve been working on this week</span></h1><h2><span>1) Building and deploying a website with ChatGPT Work</span></h2><div class="image-gallery-embed" data-attrs="{&quot;gallery&quot;:{&quot;alt&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;images&quot;:[{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/afe3605e-6994-4bc3-9b8c-516219c06ab9_922x2048.jpeg&quot;,&quot;type&quot;:&quot;image/jpeg&quot;},{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0f36030f-fe2f-4616-94af-dc8eba04313c_922x2048.jpeg&quot;,&quot;type&quot;:&quot;image/jpeg&quot;}],&quot;staticGalleryImage&quot;:{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7f3bb4c9-7c8d-4632-bae0-b1edfd9b7e0a_1456x720.png&quot;,&quot;type&quot;:&quot;image/png&quot;}},&quot;isEditorNode&quot;:true}"></div><p><span>This week, I </span><a href="https://x.com/Al_Grigor/status/2084662790438998468"><span>experimented</span></a><span> with building a small website using ChatGPT Work.</span></p><p><span>I&#8217;m really impressed. I sent it photos of a music festival brochure (it was Brutal Assault) and asked it to turn them into a website. It extracted the schedule, built a festival planner, and deployed it to a </span><a href="https://brutal-assault-planner.alexey-s-grigoriev.chatgpt.site/"><span>public URL</span></a><span> that I could share with others. </span></p><p><span>I didn&#8217;t even need to run Codex. Everything happened from my phone from the ChatGPT app. It also has built-in authentication (via ChatGPT) and a built-in database.</span></p><p></p><h2>2) FAQ Assistant in Practice</h2><div id="youtube2-CyH61xiYSnk" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;CyH61xiYSnk&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/CyH61xiYSnk?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p>Last week I wrote an article about the FAQ assistant that we use at DataTalks.Club. Then this week I did a webinar about it where I gave a more detailed overview of the system.</p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;5765df06-f658-4162-81b2-e6b7f82e3d4a&quot;,&quot;caption&quot;:&quot;FAQ assistant is the system that we use in DataTalks.Club to help thousands of our students find the answers to their questions faster.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;(Re)Building a FAQ System for DataTalks.Club&quot;,&quot;publishedBylines&quot;:[{&quot;bestseller_tier&quot;:null,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;id&quot;:2259961,&quot;is_guest&quot;:false,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;}],&quot;post_date&quot;:&quot;2026-08-01T22:59:53.228Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!k3pR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://alexeyondata.substack.com/p/rebuilding-a-faq-system-for-datatalksclub&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:209424783,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:8,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p></p><h2><span>3) Building a full-stack application with AI</span></h2><div id="youtube2-x9dq5nBpDg8" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;x9dq5nBpDg8&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/x9dq5nBpDg8?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p><span>I hosted the second workshop for </span><a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp"><span>AI Dev Tools Zoomcamp</span></a><span>. In the first, we talked about </span><a href="https://alexeyondata.substack.com/p/ai-native-development-specifications"><span>specification-driven development, loop and graph engineering</span></a><span>. In the second, I showed how to build an end-to-end application with frontend, backend, and a database. As an example, we built an application for system design interviews.</span></p><p><span>I&#8217;m hosting the notes on Substack, so you can read them here:</span></p><div class="digest-post-embed" data-attrs="{&quot;nodeId&quot;:&quot;9ab340ee-0e64-437e-86cd-6a18dfd1f2ae&quot;,&quot;caption&quot;:&quot;This is the second article in a series based on AI Dev Tools Zoomcamp, the free course we run at DataTalks.Club.&quot;,&quot;cta&quot;:null,&quot;showBylines&quot;:true,&quot;showDescription&quot;:true,&quot;showImage&quot;:true,&quot;size&quot;:&quot;sm&quot;,&quot;isEditorNode&quot;:true,&quot;title&quot;:&quot;Build and Ship a Full-Stack App with AI Coding Assistants&quot;,&quot;publishedBylines&quot;:[{&quot;bestseller_tier&quot;:null,&quot;bio&quot;:&quot;15 years of experience. Teaching AI and Data to 100k+ students. Founder of DataTalks.Club &#183; Creator of the Zoomcamp Series &#183; ML Engineer &amp; Author&quot;,&quot;id&quot;:2259961,&quot;is_guest&quot;:false,&quot;name&quot;:&quot;Alexey Grigorev&quot;,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54a0ec21-251b-49e2-aba6-93c13f3cd8cb_800x800.jpeg&quot;}],&quot;post_date&quot;:&quot;2026-08-03T12:59:20.309Z&quot;,&quot;cover_image&quot;:&quot;https://substackcdn.com/image/fetch/$s_!I96k!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png&quot;,&quot;cover_image_alt&quot;:null,&quot;cover_image_crop&quot;:null,&quot;canonical_url&quot;:&quot;https://alexeyondata.substack.com/p/build-and-ship-a-full-stack-app-with&quot;,&quot;section_name&quot;:null,&quot;video_upload_id&quot;:null,&quot;id&quot;:209595653,&quot;type&quot;:&quot;newsletter&quot;,&quot;reaction_count&quot;:11,&quot;comment_count&quot;:0,&quot;publication_id&quot;:7154940,&quot;publication_name&quot;:&quot;Alexey On Data&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!otON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe2d46bd-51fb-428c-923c-aa76536f9595_800x800.png&quot;,&quot;belowTheFold&quot;:true,&quot;youtube_url&quot;:null,&quot;show_links&quot;:null,&quot;feed_url&quot;:null}"></div><p><span>There will be more workshops and events for AI Dev Tools Zoomcamp:</span></p><ul><li><p><span>August 10: </span><a href="https://luma.com/v5tjy562">Test, Containerize, and Deploy an AI-Assisted App</a></p></li><li><p><span>August 18: </span><a href="https://luma.com/ycsfxigi">DevOps and Observability for AI-Built Apps</a></p></li><li><p><span>August 24: </span><a href="https://luma.com/a8qa5s2s"><span>Pre-Course Live Q&amp;A</span></a></p></li><li><p><span>August 31: </span><a href="https://luma.com/tsiusx8s"><span>Course launch</span></a></p></li></ul><p>I will publish the materials from all these workshops as Substack articles too.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Subscribe to receive new posts</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h2><span>4) Inference Engineering - AI Shipping Labs book club</span></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zRm8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zRm8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 424w, https://substackcdn.com/image/fetch/$s_!zRm8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 848w, https://substackcdn.com/image/fetch/$s_!zRm8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 1272w, https://substackcdn.com/image/fetch/$s_!zRm8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zRm8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png" width="1456" height="659" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:659,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:549573,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/210233610?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zRm8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 424w, https://substackcdn.com/image/fetch/$s_!zRm8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 848w, https://substackcdn.com/image/fetch/$s_!zRm8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 1272w, https://substackcdn.com/image/fetch/$s_!zRm8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1793ba4-2dc7-44da-976e-83b935ff3bf3_2107x954.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>We are also starting a book club at AI Shipping Labs.</span></p><p><span>Our first book is Inference Engineering by Philip Kiely. It covers model architecture, hardware, serving software, optimization, cost, reliability, and running inference systems in production. The digital edition is available for free.</span></p><p><span>The kickoff is on August 10 (Monday) at 5:00 PM CET. We will review the book, discuss the reading cadence, and decide together how the sessions and discussions should work.</span></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://aishippinglabs.com/events/44/inference-engineering-book-club-kickoff&quot;,&quot;text&quot;:&quot;Join the kickoff&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://aishippinglabs.com/events/44/inference-engineering-book-club-kickoff"><span>Join the kickoff</span></a></p><p></p><h2><span>5) Stock Markets Analytics Zoomcamp</span></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pgeC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pgeC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 424w, https://substackcdn.com/image/fetch/$s_!pgeC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 848w, https://substackcdn.com/image/fetch/$s_!pgeC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 1272w, https://substackcdn.com/image/fetch/$s_!pgeC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pgeC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png" width="1456" height="594" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:594,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:639712,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/210233610?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!pgeC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 424w, https://substackcdn.com/image/fetch/$s_!pgeC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 848w, https://substackcdn.com/image/fetch/$s_!pgeC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 1272w, https://substackcdn.com/image/fetch/$s_!pgeC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F581fe658-a25e-4cb6-8777-abeb375a027c_1987x811.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p><span>In DataTalks.Club, we&#8217;re preparing the next cohort of Stock Markets Analytics Zoomcamp. It starts </span>on August 24<span>.</span></p><p><span>The course provides a gentle introduction to stock market trading, Python programming in Google Colab, analytics, and data visualization.</span></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://pythoninvest.com/course&quot;,&quot;text&quot;:&quot;Register for the course&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://pythoninvest.com/course"><span>Register for the course</span></a></p><div><hr></div><p>Edited <span>by </span><a href="https://www.linkedin.com/in/valeriia-kuka/">Valeriia Kuka</a></p>]]></content:encoded></item><item><title><![CDATA[Build and Ship a Full-Stack App with AI Coding Assistants]]></title><description><![CDATA[Part 2 of the AI Dev Tools Zoomcamp series]]></description><link>https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with</link><guid isPermaLink="false">https://aishippingblog.com/p/build-and-ship-a-full-stack-app-with</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Mon, 03 Aug 2026 12:59:20 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!I96k!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>This is the second article in a series based on <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">AI Dev Tools Zoomcamp</a>, the free course we run at DataTalks.Club.</p><p>In the <a href="https://alexeyondata.substack.com/p/ai-native-development-specifications">first article</a>, I wrote about turning an idea into a specification. The specification is only the beginning. In this article, we will create an end-to-end application from scratch. We will cover both frontend and backend, and add a database.</p><p>All articles in the series:</p><ul><li><p>Part 1: <a href="https://alexeyondata.substack.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a></p></li><li><p>Part 2: <a href="https://alexeyondata.substack.com/p/build-and-ship-a-full-stack-app-with">Build and Ship a Full-Stack App with AI Coding Assistants</a> (this article)</p></li><li><p>Part 3: <a href="https://alexeyondata.substack.com/p/deploy-a-full-stack-app-with-ai-coding">Deploy a Full-Stack App with AI Coding Assistants</a></p></li><li><p>Part 4: <a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">DevOps and Observability for an AI-Built App</a></p></li><li><p>Part 5: <a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a></p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div id="youtube2-x9dq5nBpDg8" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;x9dq5nBpDg8&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/x9dq5nBpDg8?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><p>We will create an application for system-design interviews.</p><p>As an interviewer, you create a session and share the link with the interviewee. They can create diagrams in the app, and you see the changes in real time.</p><p>Both frontends connect to the same room through WebSockets, so the changes appear in both sessions simultaneously. The FastAPI backend handles those events and saves them to SQLite.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uq-e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uq-e!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 424w, https://substackcdn.com/image/fetch/$s_!uq-e!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 848w, https://substackcdn.com/image/fetch/$s_!uq-e!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 1272w, https://substackcdn.com/image/fetch/$s_!uq-e!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uq-e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png" width="1050" height="600" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:600,&quot;width&quot;:1050,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An interviewer and candidate use separate browser sessions connected through a shared WebSocket room to a FastAPI backend backed by SQLite&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An interviewer and candidate use separate browser sessions connected through a shared WebSocket room to a FastAPI backend backed by SQLite" title="An interviewer and candidate use separate browser sessions connected through a shared WebSocket room to a FastAPI backend backed by SQLite" srcset="https://substackcdn.com/image/fetch/$s_!uq-e!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 424w, https://substackcdn.com/image/fetch/$s_!uq-e!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 848w, https://substackcdn.com/image/fetch/$s_!uq-e!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 1272w, https://substackcdn.com/image/fetch/$s_!uq-e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78808130-42cf-48ca-aa02-f3c4c3611a5d_1050x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Two browser sessions collaborate through one WebSocket-backed interview room</figcaption></figure></div><p></p><p>You can find the code in the <a href="https://github.com/alexeygrigorev/interview-canvas-share">interview-canvas-share</a> repository.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!I96k!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!I96k!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 424w, https://substackcdn.com/image/fetch/$s_!I96k!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 848w, https://substackcdn.com/image/fetch/$s_!I96k!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 1272w, https://substackcdn.com/image/fetch/$s_!I96k!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!I96k!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png" width="1456" height="809" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:809,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:309800,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/209595653?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!I96k!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 424w, https://substackcdn.com/image/fetch/$s_!I96k!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 848w, https://substackcdn.com/image/fetch/$s_!I96k!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 1272w, https://substackcdn.com/image/fetch/$s_!I96k!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65018649-b9b5-40f4-9fef-b877197c4621_2880x1600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">AI System Design Canvas: the application we will develop.</figcaption></figure></div><p>It&#8217;s based on my full-day workshop I did for AI Shipping Labs: <a href="https://aishippinglabs.com/workshops/full-stack-vibe-coding">Build and Deploy a Full-Stack App with AI Coding Assistants</a>.</p><h2><strong>Overview</strong></h2><p>Like in the <a href="https://alexeyondata.substack.com/p/ai-native-development-specifications">previous article</a>, we start with an idea. As this is only an idea, we turn it into a specification using ChatGPT.</p><p>From there, we start building:</p><ol><li><p>Generate a frontend that uses mocked backend calls.</p></li><li><p>Establish backend-frontend contract by creating OpenAPI specifications from the frontend service layer.</p></li><li><p>Implement the FastAPI backend, connect it to the frontend, and test the application.</p></li><li><p>Add persistence with SQLite and SQLAlchemy.</p></li></ol><p>At each stage we get something concrete that we can test:</p><ul><li><p>First we define the specification and make sure it reflects what we want to build.</p></li><li><p>After that, we use the specs to create a frontend prototype that we can interact with. We mock backend calls so we can test the idea.</p></li><li><p>Then we connect the frontend to the backend. We start with an in-memory store to make sure the frontend-backend connection works.</p></li><li><p>Finally, we add a database so the application keeps its state after a restart.</p></li></ul><p>At the end, we have a working local application that is ready for deployment.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qJIy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qJIy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 424w, https://substackcdn.com/image/fetch/$s_!qJIy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 848w, https://substackcdn.com/image/fetch/$s_!qJIy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 1272w, https://substackcdn.com/image/fetch/$s_!qJIy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qJIy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png" width="1110" height="520" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:520,&quot;width&quot;:1110,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Three architectural snapshots show a mock service replaced by FastAPI and an in-memory store replaced by SQLite&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Three architectural snapshots show a mock service replaced by FastAPI and an in-memory store replaced by SQLite" title="Three architectural snapshots show a mock service replaced by FastAPI and an in-memory store replaced by SQLite" srcset="https://substackcdn.com/image/fetch/$s_!qJIy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 424w, https://substackcdn.com/image/fetch/$s_!qJIy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 848w, https://substackcdn.com/image/fetch/$s_!qJIy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 1272w, https://substackcdn.com/image/fetch/$s_!qJIy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f825fa0-c9e2-43aa-85ed-c55e9be77a05_1110x520.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The interfaces stay stable while temporary components are replaced one at a time</figcaption></figure></div><p></p><h2><strong>Start with a specification</strong></h2><p>Before building the frontend, we need to describe the application precisely. If we don&#8217;t do it, we will get something that works but we don&#8217;t need.</p><p>For our application, we need to specify:</p><ul><li><p>who creates an interview session</p></li><li><p>how a candidate joins it</p></li><li><p>which components they can place on the canvas</p></li><li><p>how both people see changes in real time</p></li></ul><p>This is called &#8220;Specification-Driven Development&#8221;. We don&#8217;t spend a lot of time here because it was the focus of the previous article <a href="https://alexeyondata.substack.com/p/ai-native-development-specifications">AI-Native Development: Specifications, Loop and Graph Engineering</a>.</p><p>For creating the specification, I always use ChatGPT in dictation mode. Give the assistant as much information as possible at this stage.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Hzmv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Hzmv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 424w, https://substackcdn.com/image/fetch/$s_!Hzmv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 848w, https://substackcdn.com/image/fetch/$s_!Hzmv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 1272w, https://substackcdn.com/image/fetch/$s_!Hzmv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Hzmv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png" width="1456" height="780" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:780,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Using ChatGPT in dictation mode to describe the system-design interview application&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Using ChatGPT in dictation mode to describe the system-design interview application" title="Using ChatGPT in dictation mode to describe the system-design interview application" srcset="https://substackcdn.com/image/fetch/$s_!Hzmv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 424w, https://substackcdn.com/image/fetch/$s_!Hzmv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 848w, https://substackcdn.com/image/fetch/$s_!Hzmv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 1272w, https://substackcdn.com/image/fetch/$s_!Hzmv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb83bc8a-17fc-42af-b742-32feeb503ffb_2181x1168.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Dictating the initial application idea to ChatGPT</figcaption></figure></div><p>You can see the result <a href="https://github.com/alexeygrigorev/interview-canvas-share/blob/main/docs/spec.md">here</a>.</p><h2><strong>Frontend First</strong></h2><p>Now we have the specification, but it&#8217;s only text. There are multiple options of what you can do next:</p><ul><li><p>Focus on the database layer, define the entities, and work all the way up through backend to frontend</p></li><li><p>Alternatively, you can start with specifying OpenAPI and define how backend and frontend interact and build both independently from there</p></li><li><p>Or you can focus on the frontend first and then build the rest</p></li></ul><p>All these approaches make sense and have their pros and cons.</p><p>For simple projects that you&#8217;re building alone, I recommend starting with frontend. You can quickly judge if you&#8217;re moving in the right direction, and if it solves your problem or not.</p><p>Treat it as a way to test your idea and your specification, and do it before you build the rest of the application.</p><p>For this step I like using <a href="https://lovable.dev/">Lovable</a>, which generates a React app from one prompt. Claude Design, Replit, v0, and Bolt are similar tools that you can use instead.</p><p>Alternatively, you can start with coding assistants directly and ask them to create a React app (or whatever technology you want).</p><p>I use Lovable because it creates really nice designs. Also, I&#8217;m not a frontend engineer and I don&#8217;t know much about the frontend world. Lovable makes technology choices for me that I know will make sense, while a coding agent can select something randomly.</p><p>Now open any tool of your choice and give it this prompt:</p><pre><code><code>Create a system design interview application. 

[Paste the ChatGPT-generated specification here.]

Centralize every backend call in one services layer, and create a mock
implementation of it so the whole app runs without a real backend.

Add tests.
</code></code></pre><p>This part is quite important:</p><blockquote><p>Centralize every backend call in one services layer, and create a mock implementation of it so the whole app runs without a real backend.</p></blockquote><p>Without it, the agent can do something arbitrary, but here we explicitly say that we want a mock service. Later, it will become the single point of integration of our frontend with backend. And because we mock it, it will work from the beginning.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hqDM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 424w, https://substackcdn.com/image/fetch/$s_!hqDM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 848w, https://substackcdn.com/image/fetch/$s_!hqDM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 1272w, https://substackcdn.com/image/fetch/$s_!hqDM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hqDM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png" width="620" height="156" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/79236268-03d9-40bd-bead-38fc6884d272_620x156.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:156,&quot;width&quot;:620,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:13388,&quot;alt&quot;:&quot;A user interacts with a real frontend whose backend calls are handled by a local mock service&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A user interacts with a real frontend whose backend calls are handled by a local mock service" title="A user interacts with a real frontend whose backend calls are handled by a local mock service" srcset="https://substackcdn.com/image/fetch/$s_!hqDM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 424w, https://substackcdn.com/image/fetch/$s_!hqDM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 848w, https://substackcdn.com/image/fetch/$s_!hqDM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 1272w, https://substackcdn.com/image/fetch/$s_!hqDM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79236268-03d9-40bd-bead-38fc6884d272_620x156.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">The frontend is real and interactive; only its backend service is mocked</figcaption></figure></div><p>Lovable creates a React app in TypeScript. We can interact with it in the Lovable interface.</p><p>Next, save it to GitHub. If you use Lovable:</p><ul><li><p>Select the plus icon in the bottom-left corner</p></li><li><p>Connect your GitHub account.</p></li><li><p>Lovable creates a private repository. I usually change it to public.</p></li></ul><h2><strong>Move the Frontend into the Project</strong></h2><p>Next, we can clone the repository locally.</p><p>For the rest of the project, I want this setup:</p><pre><code><code>/backend     # backend application and its tests
/docs        # supporting documentation
/frontend    # frontend application
AGENTS.md    # instructions for coding agents
openapi.yaml # API agreement</code></code></pre><p>So let&#8217;s create these folders and move all the frontend stuff to &#8220;frontend&#8221;, and the specification we created to &#8220;/docs/spec.md&#8221;.</p><p>After we re-arranged the files, commit the changes.</p><p>At this point, you should also be able to run the application locally and test that things work the way you want. If they don&#8217;t, use a coding agent to fix it.</p><p>To run a project you exported from Lovable:</p><pre><code><span>cd</span> frontend
npm i
npm run dev</code></pre><h2><strong>AGENTS.md</strong></h2><p>We already discussed the importance of <code>AGENTS.md</code> in the first article.</p><p>Let&#8217;s create one for this project too. Place it in the repository root:</p><pre><code><code>for backend, use uv for dependency management. a few useful commands:

uv sync
uv add &lt;PACKAGE-NAME&gt;
uv run python &lt;PYTHON-FILE&gt;

regularly commit code to git</code> </code></pre><p>This is only the starting point and it will change as your project grows.</p><h2><strong>OpenAPI Specifications</strong></h2><p>When creating frontend, we asked the AI assistant to put everything in a centralized service layer. Later we will replace it with actual calls to backend.</p><p>But now we should define the specification - the agreement between frontend and backend. We will use <a href="https://learn.openapis.org/introduction.html">OpenAPI</a> for that.</p><p>This specification gives explicit information about the endpoints, paths, request bodies, response bodies, and authentication rules.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rUI-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 424w, https://substackcdn.com/image/fetch/$s_!rUI-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 848w, https://substackcdn.com/image/fetch/$s_!rUI-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 1272w, https://substackcdn.com/image/fetch/$s_!rUI-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rUI-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png" width="960" height="182" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:182,&quot;width&quot;:960,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:24784,&quot;alt&quot;:&quot;The frontend and backend both connect to a shared OpenAPI contract&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The frontend and backend both connect to a shared OpenAPI contract" title="The frontend and backend both connect to a shared OpenAPI contract" srcset="https://substackcdn.com/image/fetch/$s_!rUI-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 424w, https://substackcdn.com/image/fetch/$s_!rUI-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 848w, https://substackcdn.com/image/fetch/$s_!rUI-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 1272w, https://substackcdn.com/image/fetch/$s_!rUI-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12ab5a0e-253f-48db-acc4-79ccb4c78bde_960x182.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">OpenAPI is the explicit contract shared by the frontend and backend</figcaption></figure></div><p></p><p>Let&#8217;s create it:</p><pre><code><code>Read the frontend's API client in frontend/

Create openapi.yaml at the repository root.

Specify the backend this frontend expects: every endpoint, method, path, request body, response body, and which endpoints need authentication.</code></code></pre><p>You can skip this step. But I wouldn&#8217;t recommend it.</p><p>It takes a few minutes, but has many benefits. The backend gets a precise target instead of being inferred from frontend code. Not only we save tokens this way, but also get a clear picture of what exactly the backend needs.</p><p>You can see the result in <a href="https://github.com/alexeygrigorev/interview-canvas-share/blob/main/openapi.yaml">openapi.yaml</a>.</p><h2><strong>The Backend</strong></h2><p>Now we have the OpenAPI specs and we can use this file to create the backend. We will use Python and FastAPI for that. But you can choose any technology you want. </p><p>When I start with frontend, I use a mocked backend and then replace it with a real one. In the same way, for FastAPI backends, I start with a mocked database, and then later change it.</p><p>I do that because I want to make sure the frontend-backend connection works, and until it&#8217;s smooth, I don&#8217;t worry about the persistence.</p><p>Let&#8217;s ask the coding assistant to implement it:</p><pre><code><code>Build a FastAPI backend in backend/ that implements the openapi.yaml spec.

Use an in-memory store and seed it with data so the frontend has something to show. Add authentication with hashed passwords and bearer tokens for the endpoints that need it.

Split the code into modules - routers, models, store, auth

Write tests</code></code></pre><p>After 5-10 minutes, you will have the backend ready.</p><p></p><h2><strong>Makefile</strong></h2><p>When it&#8217;s ready, you can run the application. Normally, for FastAPI, the command is something like that:</p><pre><code><span>cd</span> backend
uv run uvicorn backend.main:app --reload --port 8091</code></pre><p>I&#8217;ll use port 8091 but you can replace it with any other port you like.</p><p>But for me it&#8217;s always hard to remember these commands, so I ask the coding assistant to create a Makefile:</p><pre><code><code>Create a Makefile so I can easily run it.</code></code></pre><p>Then running it is as simple as</p><pre><code>make run</code></pre><p>When it&#8217;s running, open <a href="http://localhost:8091/docs">http://localhost:8091/docs</a> in the browser. You will see the OpenAPI specification of the implemented backend.</p><p>We can compare it with the actual specs. From this point we no longer need the original <code>openapi.yaml</code> - the one that&#8217;s generated by FastAPI is enough.</p><h2><strong>Connecting Frontend and Backend</strong></h2><p>The backend is runnable now, so we can connect it to the frontend:</p><pre><code><code>Switch the frontend to use the real backend client. </code></code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_p8i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_p8i!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 424w, https://substackcdn.com/image/fetch/$s_!_p8i!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 848w, https://substackcdn.com/image/fetch/$s_!_p8i!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 1272w, https://substackcdn.com/image/fetch/$s_!_p8i!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_p8i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png" width="900" height="320" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/badb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:320,&quot;width&quot;:900,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A user interacts with the frontend, which calls the FastAPI backend backed by a temporary in-memory store&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A user interacts with the frontend, which calls the FastAPI backend backed by a temporary in-memory store" title="A user interacts with the frontend, which calls the FastAPI backend backed by a temporary in-memory store" srcset="https://substackcdn.com/image/fetch/$s_!_p8i!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 424w, https://substackcdn.com/image/fetch/$s_!_p8i!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 848w, https://substackcdn.com/image/fetch/$s_!_p8i!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 1272w, https://substackcdn.com/image/fetch/$s_!_p8i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbadb6dc7-05a4-444e-8941-2b6c8c15eae4_900x320.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Connect the real frontend and backend while keeping persistence mocked</figcaption></figure></div><p>It most likely won&#8217;t work from the first try. You will hist CORS errors and probably some others.</p><p>It&#8217;s also time to test the application thoroughly. Open two browser windows:</p><ul><li><p>Create a session in one and get the join link</p></li><li><p>Use the join link in the second window</p></li><li><p>Interact with the system in the second window and see the changes being propagated to the first one.</p></li></ul><p>If something doesn&#8217;t work, ask the agent to fix it.</p><h2><strong>Database</strong></h2><p>At this point we have a working application with frontend and backend. They can connect to each other.</p><p>But because we don&#8217;t have a real database yet, when the backend is restarted, all the data is lost.</p><p>Let&#8217;s fix it:</p><pre><code><code>Replace the in-memory store with a database. Use SQLite and SQLAlchemy.
Use an environment variable to configure which DB the server should connect to.
Make it database-agnostic - later we will add support for other databases (e.g. Postgres).</code></code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!B3gj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!B3gj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 424w, https://substackcdn.com/image/fetch/$s_!B3gj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 848w, https://substackcdn.com/image/fetch/$s_!B3gj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 1272w, https://substackcdn.com/image/fetch/$s_!B3gj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!B3gj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png" width="900" height="320" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:320,&quot;width&quot;:900,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A user interacts with the frontend, which calls the FastAPI backend backed by SQLite&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A user interacts with the frontend, which calls the FastAPI backend backed by SQLite" title="A user interacts with the frontend, which calls the FastAPI backend backed by SQLite" srcset="https://substackcdn.com/image/fetch/$s_!B3gj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 424w, https://substackcdn.com/image/fetch/$s_!B3gj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 848w, https://substackcdn.com/image/fetch/$s_!B3gj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 1272w, https://substackcdn.com/image/fetch/$s_!B3gj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9394ee4b-f375-4325-a840-75d58f09dad7_900x320.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">SQLite replaces the in-memory store without changing the rest of the application flow</figcaption></figure></div><p></p><p>There are a few important parts in this prompt:</p><ul><li><p>First, I ask it to use SQLite. It&#8217;s a very nice and lightweight database for testing the application locally.</p></li><li><p>At some point, you&#8217;ll want to switch to a different database for production, e.g. Postgres. Using SQLAlchemy makes this process very simple.</p></li><li><p>But we also mention that explicitly so the agent doesn&#8217;t use any SQLite specific features in the first version.</p></li></ul><p>After 5-10 minutes you will have a fully working end-to-end application that&#8217;s running locally. Now you can test it: create multiple sessions and see that changes in one are reflected in another. Stop your backend server, start again, and check that all the changes are still there.</p><p>If something doesn&#8217;t work, ask the coding assistant to fix it.</p><h2><strong>Ready for Deployment</strong></h2><p>Now we have a working application:</p><ul><li><p>we turned an idea into a specification</p></li><li><p>created frontend</p></li><li><p>defined the API contract</p></li><li><p>implemented the backend from the contract</p></li><li><p>connected them</p></li><li><p>added SQLite for persistence</p></li></ul><p>But this application is only working locally. Next, we need to deploy it. That&#8217;s something we will do in the next article. We will take what we developed here and turn it into a deployed service.</p><p>We will:</p><ul><li><p>containerize the frontend and backend</p></li><li><p>add integration tests for the full application flow</p></li><li><p>set up CI to run the checks automatically</p></li><li><p>run Postgres with Docker Compose</p></li><li><p>add database migrations</p></li><li><p>deploy the application to a public environment</p></li><li><p>set up CD to deploy changes after the checks pass</p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h2><strong>Step-by-Step Approach</strong></h2><p>In this workshop, we created a working application using the step-by-step approach. At the end of each step, we had something functioning:</p><ul><li><p>Clear speciation</p></li><li><p>Working frontend prototype with mock backend</p></li><li><p>Properly integrated backend with mock database</p></li><li><p>Persistence with SQLite</p></li></ul><p>For each step, we started from a new session and drove the agent to produce a working version.</p><p>But we did it one prompt at a time. After the first version is working, it&#8217;s time to introduce the development process. We talked about it in the <a href="https://alexeyondata.substack.com/p/ai-native-development-specifications">first article</a>. This process will help you application continue working properly as it grows bigger.</p><h2><strong>Next in the Series</strong></h2><p>In the next modules we continue working with this app:</p><ul><li><p><a href="https://aishippingblog.com/p/deploy-a-full-stack-app-with-ai-coding">Deployment: deploy the application to the cloud and set up CI/CD</a></p></li><li><p><a href="https://aishippingblog.com/p/devops-and-observability-for-an-ai">DevOps and Observability: add observability to the application and use AI as the first line of support when the application breaks</a></p></li><li><p><a href="https://aishippingblog.com/p/coding-agent-building-blocks-reusable">Coding Agent Building Blocks: Reusable Skills and Specialized Subagents</a></p></li></ul><p>You can find the entire course in <a href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp">the course repository</a>. It&#8217;s free.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://github.com/DataTalksClub/ai-dev-tools-zoomcamp&quot;,&quot;text&quot;:&quot;&#11088; Star the course repo&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://github.com/DataTalksClub/ai-dev-tools-zoomcamp"><span>&#11088; Star the course repo</span></a></p><p></p><p>This article is based on the the workshop I did for AI Shipping Labs: <a href="https://aishippinglabs.com/workshops/full-stack-vibe-coding">Build and Deploy a Full-Stack App with AI Coding Assistants</a>. In that workshop I created the snake game, deployed it with AWS and configured CI/CD to make sure tests run on every push. Check it out!</p>]]></content:encoded></item><item><title><![CDATA[(Re)Building a FAQ System for DataTalks.Club]]></title><description><![CDATA[Serving 100k+ Learners with a Serverless FAQ Assistant]]></description><link>https://aishippingblog.com/p/rebuilding-a-faq-system-for-datatalksclub</link><guid isPermaLink="false">https://aishippingblog.com/p/rebuilding-a-faq-system-for-datatalksclub</guid><dc:creator><![CDATA[Alexey Grigorev]]></dc:creator><pubDate>Sat, 01 Aug 2026 22:59:53 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!k3pR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>FAQ assistant is the system that we use in DataTalks.Club to help thousands of our students find the answers to their questions faster.</p><p>There are two components:</p><ul><li><p>FAQ curation: building the FAQ dataset from student contributions, Slack discussions, and YouTube transcripts. It lives in <a href="https://github.com/DataTalksClub/faq">DataTalksClub/faq</a>.</p></li><li><p>FAQ retrieval: the Slack bot that answers students&#8217; questions using this dataset. It lives in <a href="https://github.com/DataTalksClub/faq-assistant">DataTalksClub/faq-assistant</a> and runs on AWS Lambda.</p></li></ul><p>In this article, I want to describe these components and the entire project in more detail:</p><ul><li><p>the data sources and how they become the FAQ dataset</p></li><li><p>the architecture and how data flows through it</p></li><li><p>the decisions I made and why I made them</p></li><li><p>how I evaluate each part of the system</p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2><strong>ZoomcampQABot</strong></h2><p>Yes, I already wrote about the FAQ assistant in <a href="https://alexeyondata.substack.com/p/from-google-docs-to-an-automated">From Google Docs to an Automated FAQ System for DataTalks.Club Courses</a>.</p><p>In that article, I described the <code>ZoomcampQABot</code> bot developed by Alex Litvinov. When it occasionally breaks down, I have to pull in Alex and ask him to fix the issue. Sometimes it would be handling an expired key, or sometimes it&#8217;s fixing a small bug.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1T5W!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1T5W!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 424w, https://substackcdn.com/image/fetch/$s_!1T5W!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 848w, https://substackcdn.com/image/fetch/$s_!1T5W!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 1272w, https://substackcdn.com/image/fetch/$s_!1T5W!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1T5W!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png" width="1384" height="436" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:436,&quot;width&quot;:1384,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:130151,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/209424783?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1T5W!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 424w, https://substackcdn.com/image/fetch/$s_!1T5W!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 848w, https://substackcdn.com/image/fetch/$s_!1T5W!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 1272w, https://substackcdn.com/image/fetch/$s_!1T5W!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe34c7820-9cc7-4934-aa23-a6d73821c44d_1384x436.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><code>ZoomcampQABot</code> in action</figcaption></figure></div><p>Two months ago his OpenAI account ran out of money, so the bot stopped working. I had to ping Alex again. I always felt bad that he uses his own money to pay for the project, so I offered to take it over and run it in the DataTalks.Club infra. Typically he&#8217;d decline it, but this time he agreed.</p><p>This is the setup <code>ZoomcampQABot</code> uses:</p><ul><li><p>OpenAI for generating the answers</p></li><li><p>Fly.io for hosting</p></li><li><p>HuggingFace for embeddings</p></li><li><p>Milvus for vector search, running on Zilliz Cloud in production, with four collections and query engines</p></li><li><p>Cohere for reranking</p></li><li><p>Upstash Redis as an embeddings cache</p></li><li><p>LangSmith for feedback logging</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!TXb9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TXb9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 424w, https://substackcdn.com/image/fetch/$s_!TXb9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 848w, https://substackcdn.com/image/fetch/$s_!TXb9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 1272w, https://substackcdn.com/image/fetch/$s_!TXb9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!TXb9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png" width="1438" height="705" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/506ad431-9850-415b-961a-355e8bb0b133_1438x705.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:705,&quot;width&quot;:1438,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:122659,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!TXb9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 424w, https://substackcdn.com/image/fetch/$s_!TXb9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 848w, https://substackcdn.com/image/fetch/$s_!TXb9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 1272w, https://substackcdn.com/image/fetch/$s_!TXb9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506ad431-9850-415b-961a-355e8bb0b133_1438x705.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Image from <a href="https://github.com/aaalexlit#llm-powered-question-answering-slack-bot">Alex Litvinov&#8217;s GitHub repository</a></figcaption></figure></div><p>The data comes from multiple sources:</p><ul><li><p>the FAQ dataset</p></li><li><p>Slack history</p></li><li><p>Course GitHub repository data</p></li><li><p>YouTube transcripts</p></li></ul><p>Each course has a separate ingest entrypoint on its own cron job in GitHub Actions. YouTube videos are pulled in manually.</p><p>This setup makes a lot of sense, but I couldn&#8217;t port it easily to the DataTalks.Club infra. I wanted to use it in combination with the <a href="https://github.com/DataTalksClub/au-tomator-lambda">Au-Tomator Slack Bot</a> that I described in <a href="https://alexeyondata.substack.com/p/building-and-maintaining-a-slack">Building and Maintaining a Slack Moderation Bot for an 88k-Member Community</a>.</p><p>The Au-Tomator bot runs on Serverless - on AWS Lambda. It&#8217;s been running for years now and I never had to pay for it: it was always under the free tier for AWS Lambda usage. And when I have to pay for it, I expect it to be minimal. So I wanted to run the FAQ bot on Serverless too.</p><p>So I had to redesign the Slack bot. I&#8217;ll describe the result later in the article, but now I&#8217;ll cover the main data source: the FAQ dataset. This dataset influenced many architectural decisions in the assistant, so I want to talk about it first.</p><h2><strong>Part 1: The FAQ Dataset</strong></h2><p>The FAQ dataset is <a href="https://datatalks.club/faq">open for everyone</a>. We host the website via GitHub pages and the data is in the <a href="https://github.com/DataTalksClub/faq">DataTalksClub/faq</a> repository.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zyK0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zyK0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 424w, https://substackcdn.com/image/fetch/$s_!zyK0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 848w, https://substackcdn.com/image/fetch/$s_!zyK0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 1272w, https://substackcdn.com/image/fetch/$s_!zyK0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zyK0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png" width="646" height="365.5934065934066" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:824,&quot;width&quot;:1456,&quot;resizeWidth&quot;:646,&quot;bytes&quot;:175415,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/209424783?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zyK0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 424w, https://substackcdn.com/image/fetch/$s_!zyK0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 848w, https://substackcdn.com/image/fetch/$s_!zyK0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 1272w, https://substackcdn.com/image/fetch/$s_!zyK0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75b94972-4f46-4511-81f2-0bbf4d607698_2800x1584.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The FAQ website at <a href="https://datatalks.club/faq">https://datatalks.club/faq</a> </figcaption></figure></div><p>Previously it lived in a bunch of Google Docs. It was convenient, but this approach had a few problems:</p><ul><li><p>It was frequently vandalized, so I had to manually roll the documents back.</p></li><li><p>I wanted to automate the curation with AI, but using AI in Google Docs is not trivial.</p></li></ul><p>Eventually I moved the data to a bunch of markdown files. I described the migration process in <a href="https://alexeyondata.substack.com/p/from-google-docs-to-an-automated">From Google Docs to an Automated FAQ System for DataTalks.Club Courses</a>.</p><p>Now with AI assistants I spend a lot less time maintaining this dataset, and I can keep it clean and up-to-date.</p><p>There are multiple sources of questions for the FAQ dataset:</p><ul><li><p>The questions that people contributed via GitHub</p></li><li><p>Discussions in Slack</p></li><li><p>Q&amp;A videos on YouTube</p></li></ul><h2><strong>FAQ Automation: User-Contributed FAQ Records</strong></h2><p>Anyone can contribute to the FAQ dataset:</p><ul><li><p>You <a href="https://github.com/DataTalksClub/faq/blob/main/CONTRIBUTING.md">submit an issue</a>, specifying the question, the course and your answer.</p></li><li><p>A GitHub Actions workflow indexes the entire dataset with minsearch.</p></li><li><p>It then searches twice - on the question alone, and on the question and answer together - and combines the two results with reciprocal rank fusion.</p></li><li><p>It sends the results to OpenAI, which returns a structured decision: <code>NEW</code>, <code>UPDATE</code>, <code>DUPLICATE</code> or <code>WRONG_COURSE</code>.</p></li><li><p>For <code>NEW</code> or <code>UPDATE</code>, it opens a pull request.</p></li><li><p>For <code>DUPLICATE</code> or <code>WRONG_COURSE</code>, it closes the issue.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DWMb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DWMb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 424w, https://substackcdn.com/image/fetch/$s_!DWMb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 848w, https://substackcdn.com/image/fetch/$s_!DWMb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 1272w, https://substackcdn.com/image/fetch/$s_!DWMb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DWMb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png" width="1440" height="410" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:410,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;FAQ contribution workflow: an issue triggers dataset indexing, question-only and full-record searches, reciprocal rank fusion, an LLM decision, and either a pull request or issue closure&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="FAQ contribution workflow: an issue triggers dataset indexing, question-only and full-record searches, reciprocal rank fusion, an LLM decision, and either a pull request or issue closure" title="FAQ contribution workflow: an issue triggers dataset indexing, question-only and full-record searches, reciprocal rank fusion, an LLM decision, and either a pull request or issue closure" srcset="https://substackcdn.com/image/fetch/$s_!DWMb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 424w, https://substackcdn.com/image/fetch/$s_!DWMb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 848w, https://substackcdn.com/image/fetch/$s_!DWMb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 1272w, https://substackcdn.com/image/fetch/$s_!DWMb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d7ec029-5466-4e58-be37-08489a7d498b_1440x410.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">How a user-contributed FAQ issue is classified and handled</figcaption></figure></div><h2><strong>FAQ Automation Evaluations</strong></h2><p>When the first version of the script was created, we didn&#8217;t have any evaluation set. I mostly relied on my gut feeling. But as more people started using it, the number of incorrect decisions increased. I needed to improve it. Naturally, I didn&#8217;t want to fix it blindly and just hope for the best: I needed a proper evaluation framework.</p><p>For creating it, I used those incorrect decisions. I analyzed the issues where I needed to manually adjust the result, and focused on the cases that were sufficiently different from each other.</p><p>Not all decisions made by the script are equal. If the agent makes a mistake in a <code>NEW</code> or <code>UPDATE</code> decision, it&#8217;s easy to correct. I use AI assistants to help me with that.</p><p>But if the decision is <code>DUPLICATE</code> or <code>WRONG_COURSE</code>, the PR will never be created, and the issue will be closed. In this case, I will not even have a chance to review them. Thus, a mistake in this case is way more expensive. Which means, I had to give these cases more importance in the eval set to make sure that these cases don&#8217;t happen often.</p><p>There&#8217;s also a problem with collecting the evals data from historical decisions. The FAQ data is constantly changing, so we have to use a &#8220;leave-one-out&#8221; setup.</p><p>When the script decides that something is <code>NEW</code> and we later use it in the evals, it&#8217;s no longer <code>NEW</code>: the item was already merged into the dataset. So if we run it with the same issue again, it will say <code>DUPLICATE</code> instead of <code>NEW</code>.</p><p>In order to properly test the <code>NEW</code> cases, we therefore need to remove the record from our dataset. So if we have 200 records, and the record <code>D</code> is the one we added, we remove the <code>D</code> and test this case against the remaining 199 records.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Lol7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Lol7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 424w, https://substackcdn.com/image/fetch/$s_!Lol7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 848w, https://substackcdn.com/image/fetch/$s_!Lol7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 1272w, https://substackcdn.com/image/fetch/$s_!Lol7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Lol7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png" width="1140" height="405" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:405,&quot;width&quot;:1140,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Leave-one-out evaluation: remove record D from the current 200-record FAQ dataset, then run the held-out issue D through the FAQ workflow using the remaining 199 records to recover the expected NEW decision&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Leave-one-out evaluation: remove record D from the current 200-record FAQ dataset, then run the held-out issue D through the FAQ workflow using the remaining 199 records to recover the expected NEW decision" title="Leave-one-out evaluation: remove record D from the current 200-record FAQ dataset, then run the held-out issue D through the FAQ workflow using the remaining 199 records to recover the expected NEW decision" srcset="https://substackcdn.com/image/fetch/$s_!Lol7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 424w, https://substackcdn.com/image/fetch/$s_!Lol7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 848w, https://substackcdn.com/image/fetch/$s_!Lol7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 1272w, https://substackcdn.com/image/fetch/$s_!Lol7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fff9f47-f08a-4b75-8528-010a71d49511_1140x405.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><span>Leave-one-out evaluation for a historical NEW decision</span></figcaption></figure></div><p>When I analyzed all my past corrections, the most common error turned out to be incorrect section placement. For example, instead of placing a record about projects in the &#8220;Project&#8221; section, it would place it in &#8220;General&#8221;. My eval set also tests for that.</p><p>Right now I have 61 cases:</p><ul><li><p>38 expecting <code>NEW</code></p></li><li><p>10 expecting <code>DUPLICATE</code></p></li><li><p>7 expecting <code>WRONG_COURSE</code></p></li><li><p>4 not expecting <code>WRONG_COURSE</code></p></li><li><p>2 expecting <code>UPDATE</code></p></li></ul><p>In addition to testing the whole flow, I have a retrieval-only evaluation set. I use this part to test that duplicate detection works correctly. This part is less interesting, so I&#8217;ll skip it. You can read more about it in the <a href="https://github.com/DataTalksClub/faq/#retrieval">project&#8217;s README</a>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!l9wt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!l9wt!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 424w, https://substackcdn.com/image/fetch/$s_!l9wt!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 848w, https://substackcdn.com/image/fetch/$s_!l9wt!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 1272w, https://substackcdn.com/image/fetch/$s_!l9wt!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!l9wt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png" width="1456" height="619" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:619,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Comparison of the two FAQ evaluation suites: a 25-case retrieval suite running in about two seconds with recall at five of 0.840, and a 61-case generation suite running in about two minutes with 42 passing cases on gpt-5.4-nano&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Comparison of the two FAQ evaluation suites: a 25-case retrieval suite running in about two seconds with recall at five of 0.840, and a 61-case generation suite running in about two minutes with 42 passing cases on gpt-5.4-nano" title="Comparison of the two FAQ evaluation suites: a 25-case retrieval suite running in about two seconds with recall at five of 0.840, and a 61-case generation suite running in about two minutes with 42 passing cases on gpt-5.4-nano" srcset="https://substackcdn.com/image/fetch/$s_!l9wt!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 424w, https://substackcdn.com/image/fetch/$s_!l9wt!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 848w, https://substackcdn.com/image/fetch/$s_!l9wt!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 1272w, https://substackcdn.com/image/fetch/$s_!l9wt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5251fa64-607f-44ac-a20c-5a4d9f4b1469_1832x779.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The retrieval and generation evaluation suites</figcaption></figure></div><p></p><h2><strong>Bulk Reviewing</strong></h2><p>I usually let the PRs accumulate and process them every two weeks.</p><p>For that, I use AI assistants. I have a <a href="https://github.com/DataTalksClub/faq/blob/main/.claude/skills/clear-backlog/SKILL.md">custom skill</a> that lets me go through each PR and merge it as is, or fix it if it&#8217;s not correct.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!UUvf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UUvf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 424w, https://substackcdn.com/image/fetch/$s_!UUvf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 848w, https://substackcdn.com/image/fetch/$s_!UUvf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 1272w, https://substackcdn.com/image/fetch/$s_!UUvf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UUvf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png" width="1456" height="444" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b212a448-5a52-402b-b01c-4aa950960448_1946x594.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:444,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Claude Code loading the clear-backlog skill, reading the repository conventions, and listing the open pull request backlog&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Claude Code loading the clear-backlog skill, reading the repository conventions, and listing the open pull request backlog" title="Claude Code loading the clear-backlog skill, reading the repository conventions, and listing the open pull request backlog" srcset="https://substackcdn.com/image/fetch/$s_!UUvf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 424w, https://substackcdn.com/image/fetch/$s_!UUvf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 848w, https://substackcdn.com/image/fetch/$s_!UUvf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 1272w, https://substackcdn.com/image/fetch/$s_!UUvf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb212a448-5a52-402b-b01c-4aa950960448_1946x594.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Starting a bulk FAQ review with the clear-backlog skill</figcaption></figure></div><p>If I come across an interesting error, I ask the assistant to add it to our evals set. I don&#8217;t try to fix it immediately. I wait till we have a few more cases, then I ask the assistant to see if there are simple ways to fix them.</p><p>If some cases can&#8217;t be fixed easily, I don&#8217;t sweat over it. I just let it sit in the evals with a FAIL status. I don&#8217;t want the system prompt to grow too large and handle all the corner cases. But I do want to know that these corner cases exist.</p><h2><strong>Other Sources: Slack and YouTube</strong></h2><p>Our Slack is very active. Course participants are asking questions and helping each other. In many cases, these discussions are worth saving in the FAQ dataset.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!f3s8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!f3s8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 424w, https://substackcdn.com/image/fetch/$s_!f3s8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 848w, https://substackcdn.com/image/fetch/$s_!f3s8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 1272w, https://substackcdn.com/image/fetch/$s_!f3s8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!f3s8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png" width="1456" height="788" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:788,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;LLM Zoomcamp Slack channel with several course participants asking about project deadlines, submissions, and the number of required projects&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="LLM Zoomcamp Slack channel with several course participants asking about project deadlines, submissions, and the number of required projects" title="LLM Zoomcamp Slack channel with several course participants asking about project deadlines, submissions, and the number of required projects" srcset="https://substackcdn.com/image/fetch/$s_!f3s8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 424w, https://substackcdn.com/image/fetch/$s_!f3s8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 848w, https://substackcdn.com/image/fetch/$s_!f3s8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 1272w, https://substackcdn.com/image/fetch/$s_!f3s8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36363aff-9ebc-4f42-a21e-fcc4bb87cc57_2004x1084.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Recurring deadline and submission questions in the LLM Zoomcamp Slack</figcaption></figure></div><p>For that I regularly go through all the Slack threads. I ask my AI assistant to go through each thread, and if something is useful, it&#8217;s saved as a new record.</p><p>Also, for each course I run a few live YouTube sessions, for example:</p><ul><li><p>pre-course Q&amp;A session</p></li><li><p>course launch streams</p></li><li><p>occasional office hours</p></li></ul><p>I get the transcript and use AI assistant to extract potential Q&amp;A candidates. If there&#8217;s something new, it also goes to the dataset.</p><h2><strong>Part 1 Overview</strong></h2><p>The main focus of the first part is the dataset curation.</p><p>I review all the PRs that our FAQ automation creates in batches, and use AI to turn Slack discussions and YouTube videos into focused FAQ records.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BOQK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BOQK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 424w, https://substackcdn.com/image/fetch/$s_!BOQK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 848w, https://substackcdn.com/image/fetch/$s_!BOQK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 1272w, https://substackcdn.com/image/fetch/$s_!BOQK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BOQK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png" width="1020" height="420" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:420,&quot;width&quot;:1020,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Three sources feed the curated FAQ dataset: GitHub issues pass through automated screening and batch review, while Slack threads and YouTube sessions pass through AI-assisted extraction&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Three sources feed the curated FAQ dataset: GitHub issues pass through automated screening and batch review, while Slack threads and YouTube sessions pass through AI-assisted extraction" title="Three sources feed the curated FAQ dataset: GitHub issues pass through automated screening and batch review, while Slack threads and YouTube sessions pass through AI-assisted extraction" srcset="https://substackcdn.com/image/fetch/$s_!BOQK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 424w, https://substackcdn.com/image/fetch/$s_!BOQK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 848w, https://substackcdn.com/image/fetch/$s_!BOQK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 1272w, https://substackcdn.com/image/fetch/$s_!BOQK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a911a5e-fb49-4194-b67b-243645e0ed8e_1020x420.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Three sources, two curation paths, and one FAQ dataset</figcaption></figure></div><p></p><p>Now let&#8217;s go back to the retrieval side. This FAQ dataset is used as the main data source for the Slack bot.</p><h2><strong>Part 2: Slack Bot</strong></h2><p>When you mention <code>@Au-Tomator</code> (my bot) or <code>@ZoomcampQABot</code> from Alex, both would perform RAG:</p><ul><li><p>Use search to fetch the candidate FAQ records</p></li><li><p>Pass them to OpenAI</p></li><li><p>Return the answer from the LLM and post it in the thread as a reply</p></li></ul><p><code>ZoomcampQABot</code> has a lot of moving parts though. Most of them are not straightforward to deploy to the serverless environment. So I decided to simplify it.</p><p>The first thing I dropped was indexing Slack threads and YouTube videos. I already extract this information and put it directly to the FAQ dataset, so this part is not needed anymore.</p><p>Second, I removed the vector database. I know that it improves retrieval, but at the cost of having to maintain more infra. I want to keep the setup very lean. If the bot is sometimes not right, it&#8217;s not a huge problem: I and other community members can simply correct it in Slack.</p><h2><strong>Zero-Dependency Serverless Deployment</strong></h2><p>My to-go search library for text search is minsearch. I wrote about it in <a href="https://alexeyondata.substack.com/p/minsearch-the-small-search-library">Minsearch: The Small Search Library Behind My RAG Workshops and Courses</a>.</p><p>However, deploying minsearch to AWS Lambda isn&#8217;t trivial. When I created this library, my focus was on teaching the concepts, not on deploying it to serverless. Internally, it users Scikit-Learn and Pandas, and these libraries in turn pull in numpy and scipy. Normally, everyone working with data already has these libraries in their standard setup. But for Lambda, they are too heavy and take well over the 50 MB limit.</p><p>So I decided to replace minsearch with a zero-dependency search engine written entirely in pure-Python. I called it <a href="https://github.com/alexeygrigorev/zerosearch">zerosearch</a>. It has only one optional dependency - <a href="https://github.com/alexeygrigorev/stemlite">stemlite</a>, which is a stemmer I use across all my small search libraries (zerosearch, minsearch and <a href="https://alexeyondata.substack.com/p/how-i-built-sqlitesearch-a-lightweight">SQLiteSearch</a>).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!T3yw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!T3yw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 424w, https://substackcdn.com/image/fetch/$s_!T3yw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 848w, https://substackcdn.com/image/fetch/$s_!T3yw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 1272w, https://substackcdn.com/image/fetch/$s_!T3yw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!T3yw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png" width="1250" height="330" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:330,&quot;width&quot;:1250,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two-bar chart showing approximately 75 MB for the compressed scikit-learn and pandas dependency stack and 8 MB for the deployed Zerosearch bot, with a dashed reference line at the 50 MB AWS Lambda direct-upload ZIP limit&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two-bar chart showing approximately 75 MB for the compressed scikit-learn and pandas dependency stack and 8 MB for the deployed Zerosearch bot, with a dashed reference line at the 50 MB AWS Lambda direct-upload ZIP limit" title="Two-bar chart showing approximately 75 MB for the compressed scikit-learn and pandas dependency stack and 8 MB for the deployed Zerosearch bot, with a dashed reference line at the 50 MB AWS Lambda direct-upload ZIP limit" srcset="https://substackcdn.com/image/fetch/$s_!T3yw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 424w, https://substackcdn.com/image/fetch/$s_!T3yw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 848w, https://substackcdn.com/image/fetch/$s_!T3yw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 1272w, https://substackcdn.com/image/fetch/$s_!T3yw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38048412-bca6-4b5c-bd0a-36075db972d0_1250x330.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The scientific Python stack exceeds Lambda&#8217;s direct-upload limit; the Zerosearch deployment fits comfortably</figcaption></figure></div><p>Another challenge was Pydantic. We typically use for structured output, and it works fine in usual environments, but it&#8217;s not trivial to get it right for Lambda. It has a compiled Rust core, and you need to make sure the binaries that you send package into your Lambda deployment are compatible. Plus, loading the binaries core and building the model schemas adds to the cold start time.</p><p>So I removed it along with <code>requests</code> and did all the calls to OpenAI using only the standard Python library.</p><h2><strong>Keeping the Index Fresh</strong></h2><p>Then I started thinking about what to do with updating the index. If a record in the FAQ changes, the search needs to reflect it. Alex solved that by pulling in data using a daily cron job. But I wanted to do this as soon as the data in FAQ changes.</p><p>Almost all my projects have a CI/CD workflow: when I push to main, the code change is propagated to a live environment. I usually use it for code changes, but for this project I did the same for data changes too.</p><p>When a push happens, I re-build the whole index, and push it together with the source code in Lambda. It&#8217;s quite small: it&#8217;s 8 MB total.</p><p>Once it&#8217;s deployed, the lambda loads the local updated index, and can serve the fresh data.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!FfcX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!FfcX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 424w, https://substackcdn.com/image/fetch/$s_!FfcX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 848w, https://substackcdn.com/image/fetch/$s_!FfcX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 1272w, https://substackcdn.com/image/fetch/$s_!FfcX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!FfcX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png" width="840" height="330" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:330,&quot;width&quot;:840,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A FAQ and documentation change triggers an index rebuild, and the new index is sent to AWS Lambda while a Slack user exchanges questions and replies with it&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A FAQ and documentation change triggers an index rebuild, and the new index is sent to AWS Lambda while a Slack user exchanges questions and replies with it" title="A FAQ and documentation change triggers an index rebuild, and the new index is sent to AWS Lambda while a Slack user exchanges questions and replies with it" srcset="https://substackcdn.com/image/fetch/$s_!FfcX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 424w, https://substackcdn.com/image/fetch/$s_!FfcX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 848w, https://substackcdn.com/image/fetch/$s_!FfcX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 1272w, https://substackcdn.com/image/fetch/$s_!FfcX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96c6e9cc-3e09-4603-9220-5a78aa43d802_840x330.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Every push rebuilds and deploys the index used by subsequent Slack requests</figcaption></figure></div><h2><strong>FAQ Assistant Retrieval Evaluation</strong></h2><p>I replaced vector search with text search, and used zerosearch to make it fast and work well in Serverless. But it would still lose to vector search in terms of retrieval quality.</p><p>How can I make sure I squeeze the absolute best from text search? For that I needed to have a proper evaluation set.</p><p>I built the ground truth dataset from real Slack threads:</p><ul><li><p>Get a Slack dump with all the course channels</p></li><li><p>Take all the Slack threads (9,900 of them)</p></li><li><p>Filter them down to keep actual questions</p></li></ul><p>At the end, I had a sample of 130 records:</p><ul><li><p>60 for Data Engineering Zoomcamp</p></li><li><p>40 Stock Markets Analytics</p></li><li><p>30 AI Dev Tools</p></li></ul><p>For each of the questions, I found and marked the correct documents in the index. Then I evaluated the hit-rate and MRR at k=1,3,5.</p><p>Once I had this dataset, I could start experimenting with search optimization.</p><p>I tried multiple options:</p><ul><li><p>Taking raw question as is</p></li><li><p>Keyword expansion</p></li><li><p>Different variants of query rewriting</p></li></ul><p>The best option is to take a Slack message and turn it into a bunch of keywords while preserving exact error messages, tool names, commands and filenames.</p><p>For example, it turns this Slack message:</p><blockquote><p>Does anyone know why minsearch fails when I run <code>uv run python index.py</code>? I get <code>KeyError: 'course'</code> after renaming <code>documents.json</code> to <code>faq.json</code>.</p></blockquote><p>into these keywords:</p><pre><code><code>minsearch FAQ indexing course field "KeyError: 'course'" "uv run python index.py" index.py documents.json faq.json</code></code></pre><p>For query rewriting I use <code>gpt-4o-mini</code> because it&#8217;s the fastest and cheapest model, but for the actual generation I rely on <code>gpt-5.4-mini</code>.</p><h2><strong>FAQ Assistant Generation Evaluation</strong></h2><p>In the previous part I explained how I evaluate search. I also evaluate generation.</p><p>For the evaluation dataset, I collect feedback from Slack. When a bot answers a question, and somebody corrects it or add something extra, it means that the bot couldn&#8217;t answer the question properly. So we can include this question in the evals.</p><p>So far it&#8217;s not large and has a few cases like:</p><ul><li><p>the answer from the bot is incomplete</p></li><li><p>the answer is incorrect</p></li><li><p>the search doesn&#8217;t find anything</p></li></ul><p>Most of the time the way to fix these problems is not tuning the prompt, but fixing the FAQ dataset itself. I check why a record wasn&#8217;t retrieved, or why the answer didn&#8217;t contain the correct information, and then see how I should update the FAQ records to make it work next time.</p><p>Then I&#8217;d fix the record, re-run evaluation, and re-deploy the bot.</p><h2><strong>Deploying with Au-Tomator</strong></h2><p>I use <a href="https://alexeyondata.substack.com/p/building-and-maintaining-a-slack">Au-Tomator</a> to help me with Slack management. So I wanted to use the existing setup, not create a new Slack bot.</p><p>I already have two Lambdas for Au-Tomator:</p><ul><li><p>The router: routes and filters Slack events. It also makes sure we acknowledge the request from Slack quickly within 3 seconds (a requirement from Slack) and forward the request to the next lambda.</p></li><li><p>The automator: the bot itself, it handles all the reactions.</p></li></ul><p>And I added the third one:</p><ul><li><p>The FAQ assistant: rewrites the question, searches the index, and generates the answer</p></li></ul><p>Now when you mention the bot, it first goes to Au-Tomator, and then Au-Tomator sends it to the FAQ assistant. The assistant gets in the question, and send back the answers. Posting the response to Slack is handled by Au-Tomator.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zd0M!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 424w, https://substackcdn.com/image/fetch/$s_!zd0M!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 848w, https://substackcdn.com/image/fetch/$s_!zd0M!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 1272w, https://substackcdn.com/image/fetch/$s_!zd0M!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zd0M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png" width="1120" height="230" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:230,&quot;width&quot;:1120,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Slack event flow through the existing Au-Tomator router and automator Lambdas to the new FAQ assistant Lambda, followed by the answer returning through the automator and being posted to Slack&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Slack event flow through the existing Au-Tomator router and automator Lambdas to the new FAQ assistant Lambda, followed by the answer returning through the automator and being posted to Slack" title="Slack event flow through the existing Au-Tomator router and automator Lambdas to the new FAQ assistant Lambda, followed by the answer returning through the automator and being posted to Slack" srcset="https://substackcdn.com/image/fetch/$s_!zd0M!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 424w, https://substackcdn.com/image/fetch/$s_!zd0M!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 848w, https://substackcdn.com/image/fetch/$s_!zd0M!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 1272w, https://substackcdn.com/image/fetch/$s_!zd0M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ea63ef-b5bc-42de-837d-76dc083033e8_1120x230.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Au-Tomator routes Slack events to the FAQ assistant and posts the generated answer back to Slack</figcaption></figure></div><p>As a bonus, now with this setup, I can trigger the FAQ assistant with <code>:faq:</code> reaction. Previously this reaction would only post a message to the Slack thread saying &#8220;Go check the FAQ&#8221;.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dhoC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 424w, https://substackcdn.com/image/fetch/$s_!dhoC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 848w, https://substackcdn.com/image/fetch/$s_!dhoC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 1272w, https://substackcdn.com/image/fetch/$s_!dhoC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dhoC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png" width="591" height="161.6015625" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:1024,&quot;resizeWidth&quot;:591,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Slack screenshot: someone reacts to a question with the :faq: emoji, and the only response is Au-Tomator posting a static 'Please check the FAQ' link&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Slack screenshot: someone reacts to a question with the :faq: emoji, and the only response is Au-Tomator posting a static 'Please check the FAQ' link" title="Slack screenshot: someone reacts to a question with the :faq: emoji, and the only response is Au-Tomator posting a static 'Please check the FAQ' link" srcset="https://substackcdn.com/image/fetch/$s_!dhoC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 424w, https://substackcdn.com/image/fetch/$s_!dhoC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 848w, https://substackcdn.com/image/fetch/$s_!dhoC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 1272w, https://substackcdn.com/image/fetch/$s_!dhoC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa165320b-c226-41f5-bce7-9f7302160a3f_1024x280.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Before the migration, the :faq: reaction just dropped a static &#8216;check the FAQ&#8217; link - no answer was generated</figcaption></figure></div><p>Now it actually checks the FAQ and generates the answer.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!grvZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!grvZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 424w, https://substackcdn.com/image/fetch/$s_!grvZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 848w, https://substackcdn.com/image/fetch/$s_!grvZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!grvZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!grvZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg" width="374" height="518.7518518518518" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1498,&quot;width&quot;:1080,&quot;resizeWidth&quot;:374,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Slack screenshot: a member asks when the next courses start, and Au-Tomator replies with the courses section, the events page, Luma and the Google calendar, citing a docs source&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Slack screenshot: a member asks when the next courses start, and Au-Tomator replies with the courses section, the events page, Luma and the Google calendar, citing a docs source" title="Slack screenshot: a member asks when the next courses start, and Au-Tomator replies with the courses section, the events page, Luma and the Google calendar, citing a docs source" srcset="https://substackcdn.com/image/fetch/$s_!grvZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 424w, https://substackcdn.com/image/fetch/$s_!grvZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 848w, https://substackcdn.com/image/fetch/$s_!grvZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!grvZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c2142a9-0ef9-40e0-99bb-4760f17bb4c7_1080x1498.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The bot processing the :faq: reaction</figcaption></figure></div><h2><strong>Other Retrieval Sources</strong></h2><p>I don&#8217;t only search the FAQ dataset. I also index the <a href="https://datatalks.club/docs/">DataTalks.Club documentation</a> and the Markdown files from each course repository.</p><p>I combine all three sources into one search index and package it with the FAQ assistant Lambda. In Slack, either a mention or a <code>:faq:</code> reaction triggers Au-Tomator. It sends the question to the assistant and posts the answer back to the thread.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZPft!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZPft!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 424w, https://substackcdn.com/image/fetch/$s_!ZPft!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 848w, https://substackcdn.com/image/fetch/$s_!ZPft!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 1272w, https://substackcdn.com/image/fetch/$s_!ZPft!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZPft!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png" width="840" height="455" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:455,&quot;width&quot;:840,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The FAQ dataset, DataTalks.Club documentation, and Markdown files from course repositories feed one search index in the FAQ assistant Lambda, while a Slack mention or FAQ reaction triggers Au-Tomator to exchange the question and answer with the assistant&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The FAQ dataset, DataTalks.Club documentation, and Markdown files from course repositories feed one search index in the FAQ assistant Lambda, while a Slack mention or FAQ reaction triggers Au-Tomator to exchange the question and answer with the assistant" title="The FAQ dataset, DataTalks.Club documentation, and Markdown files from course repositories feed one search index in the FAQ assistant Lambda, while a Slack mention or FAQ reaction triggers Au-Tomator to exchange the question and answer with the assistant" srcset="https://substackcdn.com/image/fetch/$s_!ZPft!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 424w, https://substackcdn.com/image/fetch/$s_!ZPft!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 848w, https://substackcdn.com/image/fetch/$s_!ZPft!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 1272w, https://substackcdn.com/image/fetch/$s_!ZPft!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0a1c704-b640-4236-83da-e724f6ac79b8_840x455.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Three content sources build the index served through Au-Tomator in Slack</figcaption></figure></div><p></p><h2><strong>The Full Picture</strong></h2><p>In this article, we discussed the two parts of the FAQ system:</p><ul><li><p>The FAQ dataset, where we primarily talked about the dataset curation</p></li><li><p>The FAQ assistant, which is a Slack bot deployed via AWS Lambda</p></li></ul><p>The FAQ assistant reads data from the FAQ dataset and helps students find answers to their questions faster.</p><p>This is how the entire system looks like:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!k3pR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!k3pR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 424w, https://substackcdn.com/image/fetch/$s_!k3pR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 848w, https://substackcdn.com/image/fetch/$s_!k3pR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 1272w, https://substackcdn.com/image/fetch/$s_!k3pR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!k3pR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png" width="1120" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:1120,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Full FAQ assistant system: GitHub proposals, Slack discussions, and YouTube sessions become curated FAQ records. The FAQ joins the docs website and course repository Markdown in a ZeroSearch index deployed with the FAQ assistant Lambda. Slack threads use Au-Tomator to exchange questions and answers with the assistant&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Full FAQ assistant system: GitHub proposals, Slack discussions, and YouTube sessions become curated FAQ records. The FAQ joins the docs website and course repository Markdown in a ZeroSearch index deployed with the FAQ assistant Lambda. Slack threads use Au-Tomator to exchange questions and answers with the assistant" title="Full FAQ assistant system: GitHub proposals, Slack discussions, and YouTube sessions become curated FAQ records. The FAQ joins the docs website and course repository Markdown in a ZeroSearch index deployed with the FAQ assistant Lambda. Slack threads use Au-Tomator to exchange questions and answers with the assistant" srcset="https://substackcdn.com/image/fetch/$s_!k3pR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 424w, https://substackcdn.com/image/fetch/$s_!k3pR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 848w, https://substackcdn.com/image/fetch/$s_!k3pR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 1272w, https://substackcdn.com/image/fetch/$s_!k3pR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82829a1f-3a47-4545-a55a-cbf236ed086d_1120x440.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The complete path from community knowledge to a Slack answer</figcaption></figure></div><ul><li><p>I turn GitHub issues, useful Slack discussions, and live-session transcripts into reviewed FAQ records.</p></li><li><p>I combine the FAQ, docs website, and course-repository Markdown into a zerosearch index. The index is rebuilt every time we push to the FAQ repo.</p></li><li><p>A mention or <code>:faq:</code> reaction triggers Au-Tomator that sends the question to the FAQ assistant and publishes the answer in Slack.</p></li><li><p>I use wrong and missing answers as evaluation cases and FAQ updates, so the next index contains the correction.</p></li></ul><h2><strong>Curation over Complexity</strong></h2><p>This year, I re-recorded the entire LLM Zoomcamp course from scratch. There, we use <a href="https://github.com/DataTalksClub/llm-zoomcamp/blob/main/01-agentic-rag/lessons/04-dataset.md">the FAQ dataset</a> as the main running example.</p><p>When I introduce the FAQ dataset, and we do the first RAG example, I mention that the example looks very simple. But the reason it looks so simple is because the complex work has already happened: the dataset was already cleaned and prepared. In reality, in your AI projects, most of the time will go into data cleaning and data curation.</p><p>In this article, I tried to explain how exactly I did it.</p><p>We can always have a better model, use a vector database, or a faster machine, but none of that will help if your data is missing, outdated or unclear.</p><p>When the assistant gives a bad answer, it&#8217;s not because of the tech stack, but because of the data. So the most useful fix to make it better is to improve the dataset.</p><p>In the FAQ case, I collect the feedback from real users, use it to improve the underlying dataset and thus improve the system itself:</p><ol><li><p>Students contribute issues to the FAQ dataset.</p></li><li><p>Useful discussions in Slack become curated FAQ records.</p></li><li><p>The updated dataset is indexed and deployed with the Lambda.</p></li><li><p>Au-Tomator uses it to answer future questions.</p></li><li><p>Incorrect or incomplete answers become evaluation cases and help improve the system.</p></li></ol><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HV8m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HV8m!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 424w, https://substackcdn.com/image/fetch/$s_!HV8m!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 848w, https://substackcdn.com/image/fetch/$s_!HV8m!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 1272w, https://substackcdn.com/image/fetch/$s_!HV8m!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HV8m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png" width="1100" height="550" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:550,&quot;width&quot;:1100,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:82765,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://alexeyondata.substack.com/i/209424783?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HV8m!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 424w, https://substackcdn.com/image/fetch/$s_!HV8m!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 848w, https://substackcdn.com/image/fetch/$s_!HV8m!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 1272w, https://substackcdn.com/image/fetch/$s_!HV8m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c8fa86-4f2a-492d-acb7-7a799f2576de_1100x550.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Because of this loop, I&#8217;m comfortable with a simpler architecture that I described in this article.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://aishippingblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Alexey On Data! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item></channel></rss>