A new framework reveals how agent loops can be optimized through three execution models and their cost implications

A recent open-source course from Decoding AI highlights three distinct ways to run an agent loop, each with different latency profiles and associated costs. The course, titled 'Building a Coding Agent From Scratch,' introduces a Python-based agent called Decode, which separates execution into three modes. The first mode, terminal UI, is latency-bound and suited for low-latency APIs, allowing real-time interaction with a human reading every token. This mode uses a steering queue and priority gate to manage input, enabling actions like plain Enter for in-turn steering, Alt+Enter for queued follow-ups, and Esc for cooperative aborts. The second mode, remote execution, runs the harness on a server via an agent runtime like Kitaru, deployed on GCP with agents executing on Modal. This mode prioritizes throughput per dollar, with sandboxed tools running in Modal Sandboxes or Docker locally. The third mode, background queue, allows a live session to hand off work to a job queue, enabling asynchronous processing and background workflows, similar to Slack-triggered agents or PR reviews. The cost model varies with latency requirements, with interactive work billed per token and offline work per GPU-hour. For example, processing 1,000 documents at 30,000 input tokens each costs around $97 via frontier APIs, but drops to under $13 when batched on a serverless GPU. Modal's pricing analysis also shows serverless can be cheaper than reserved capacity when peak-to-average ratios exceed reservation discounts, with typical discounts of 2–5× for ratios of 5–10×. Industry surveys indicate reservation utilization is often below 30%, sometimes under 10%.