How redwoodjs/agent-ci works
1. What this repo is
- Agent CI is a local execution platform for GitHub Actions workflows.
- The architecture centers on replacing GitHub’s server-side workflow API with a local emulator, while still running the official GitHub Actions runner binary inside Docker or macOS VMs.
2. Top-level shape: a monorepo with three product surfaces
- CLI/runtime for executing workflows locally.
- Website/docs for marketing and user guidance.
- Distribution/packaging for npm packages, native binaries, and release artifacts.
3. Primary execution path: packages/cli
- This is the main TypeScript implementation used by npx @redwoodjs/agent-ci.
- It handles command parsing, workflow discovery, job scheduling, runner lifecycle, local Docker orchestration, output rendering, run-state management, and retry/abort flows.
- It is the user-facing orchestration layer that turns a workflow file into one or more runnable jobs.
4. Workflow understanding and planning
- packages/cli/src/workflow parses GitHub Actions YAML, resolves reusable workflows, computes matrix expansion, classifies jobs by resource type, and creates execution plans.
- packages/cli/src/runner then turns those plans into concrete runtime actions, including job result collection, workspace setup, dirty-sha handling, step wrapping, and job synchronization.
5. Local container execution boundary: Docker
- Most jobs run in Docker containers derived from the official ghcr.io/actions/actions-runner image.
- The CLI connects to Docker through a configurable daemon host, mounts the repo working tree, and binds host caches directly into containers for near-zero-cost reuse.
- Docker is also the boundary for service containers, cache behavior, shutdown, and remote-daemon support.
6. GitHub Actions emulation boundary: DTU / mock server
- The repo emulates the GitHub-side API surface needed by the official runner: workflow/job routing, artifacts, cache, and GitHub-facing endpoints.
- That emulation is implemented in packages/dtu-github-actions, which contains server routes for GitHub, DTU-specific control, cache, artifacts, and action tarball delivery.
- The emulator state is persisted in an internal store and exercised heavily through tests and fixtures.
7. Rust rewrite path: crates/agent-ci-core, crates/agent-ci-runtime, crates/agent-ci
- The Rust side mirrors the TypeScript architecture for parity testing and eventual native execution.
- agent-ci-core contains workflow expressions, matrices, reusable workflow logic, planning, and shared state/event modeling.
- agent-ci-runtime contains execution backends: Docker, DTU, macOS VM orchestration, runner timeline handling, workspace management, and image handling.
- agent-ci is the Rust CLI entry point and command layer.
8. macOS job execution
- Jobs targeting runs-on: macos-* are routed to a separate macOS VM path rather than Docker.
- The runtime uses Tart plus SSH on Apple Silicon hosts, with per-job VM provisioning, runner binary transfer, and teardown after completion.
- This is a distinct execution lane with its own concurrency limit and image mapping.
9. Caching and prewarm behavior
- A major architectural feature is local bind-mounted caching instead of networked GitHub cache round-trips.
- The system can optionally “prewarm through” a selected step to avoid parallel cold installs colliding on shared node_modules.
- This is a workflow-level optimization that affects job scheduling and initial execution order.
10. Pause/retry/abort model
- Failed runs do not immediately destroy the container in pause mode.
- The paused container preserves filesystem state, installed tools, and intermediate artifacts so only the failed step needs retrying.
- That means the lifecycle includes a long-lived paused state, not just “start run / end run”.
11. Inputs and secrets boundary
- Repo-local configuration comes from .env.agent-ci, shell environment variables, and explicit CLI flags.
- Secrets are resolved differently from workflow variables: secrets can come from environment or token flags, while workflow vars come only from explicit --var/--var-file inputs.
- This separation is important for the architecture diagram because it affects how configuration enters the runtime.
12. Packaging and distribution
- packages/agent-ci-* are platform-specific npm wrappers for native binaries.
- install.sh, packaging/homebrew, and the release workflows support cross-platform distribution.
- The README indicates the published npm path currently favors TypeScript, while the Rust binary is available from source and is being prepared for native packaging.
13. Support systems: website, docs, experiments
- apps/website is a separate web app for the public site and docs/marketing content.
- experiments/ contains evaluation harnesses, benchmarks, and proof-of-concept flows used to validate behaviors like retries, cache performance, and agent skill quality.
- These are not part of the runtime path, but they support product validation and release messaging.
14. Testing and compatibility infrastructure
- The repo relies on many workflow fixture files, smoke tests, and GitHub Actions CI workflows to validate compatibility across runner behaviors.
- There are dedicated fixtures for matrix jobs, reusable workflows, service containers, symlink handling, remote workflow fetches, and failure modes.
- For an architecture diagram, these tests represent the validation layer around the core orchestration engine rather than another runtime subsystem.
Main entry points: runner Runner runtime · src Runtime backends · cli.ts CLI entry · server DTU server
Components 18
TypeScript runtime
Primary local workflow execution path in packages/cli and DTU emulation.
- CLI entry · User-facing command parsing, run orchestration, retries, and abort/clean flows.
packages/cli/src/cli.ts - Config · Loads repo-local env, shell environment, flags, vars, and secrets boundaries.
packages/cli/src/config.ts - Workflow plan · Parses workflow YAML, resolves reusable workflows, expands matrices, and classifies jobs.
packages/cli/src/workflow - Job scheduler · Orders jobs and coordinates execution constraints, including prewarm-through behavior.
packages/cli/src/workflow/job-scheduler.ts - Runner runtime · Turns plans into job execution, workspace setup, dirty-sha handling, and synchronization.
packages/cli/src/runner - Docker bridge · Connects to Docker daemon, configures containers, service containers, images, and shutdown.
packages/cli/src/docker - macOS lane · Routes macOS jobs to Tart/SSH-backed VM execution with image mapping and teardown.
packages/cli/src/runner/macos-vm - Run UI · Renders run state, logs, tree views, concurrency, and final job/result output.
packages/cli/src/output - DTU server · Emulates GitHub-side workflow, cache, artifact, and control endpoints for the runner.
packages/dtu-github-actions/src/server - API routes · GitHub, DTU control, cache, artifacts, and action tarball route handlers.
packages/dtu-github-actions/src/server/routes - State store · Internal state used by the mock GitHub Actions server and its tests/fixtures.
packages/dtu-github-actions/src/server/store.ts
Rust rewrite
Native parity path split across core planning, runtime backends, and CLI entrypoint.
- Rust CLI · Native entrypoint and command layer for local run orchestration.
crates/agent-ci/src/main.rs - Core planner · Workflow expressions, matrices, reusable workflows, state, and plan modeling.
crates/agent-ci-core/src - Runtime backends · Execution backends for Docker, DTU, macOS VM orchestration, workspace, and runner timeline.
crates/agent-ci-runtime/src
Site and distribution
Public website, docs, packaging, and release artifacts.
Validation layer
Workflows, fixtures, and experiments used to verify compatibility and behavior.
- Experiments · Benchmarks and proof-of-concept flows for cache, retries, and skill evaluation.
experiments - CI workflows · Smoke and release workflows that exercise runner compatibility across scenarios.
.github/workflows
Connections 21
- CLI entry → Config: reads · Command execution starts by resolving environment, vars, and secrets.
- CLI entry → Workflow plan: plans · Turns a workflow file into an execution plan.
- Workflow plan → Job scheduler: feeds · Expanded jobs and reusable workflows are scheduled for execution.
- Job scheduler → Runner runtime: dispatches · Creates runnable jobs and coordinates execution order.
- Runner runtime → Docker bridge: uses · Most jobs execute in Docker containers with mounts and cache bindings.
- Runner runtime → macOS lane: routes · macOS-targeted jobs follow the VM execution lane.
- Runner runtime → Run UI: reports · Run and step state are rendered back to the user.
- Docker bridge → DTU server: emulates · Docker jobs interact with the local GitHub Actions emulator.
- DTU server → API routes: serves · Server routes expose GitHub, cache, artifact, and DTU endpoints.
- DTU server → State store: persists · Emulator state is stored for runs and fixtures.
- Rust CLI → Core planner: uses · Native CLI delegates planning and workflow semantics to core.
- Rust CLI → Runtime backends: executes · CLI layer drives the native runtime backends.
- Runtime backends → Core planner: consumes · Runtime executes plans and state from the shared core model.
- Runtime backends → Docker bridge: parallels · Rust backend mirrors the TypeScript Docker execution path.
- Runtime backends → macOS lane: parallels · Rust backend mirrors the TypeScript macOS VM lane.
- Website → CLI entry: documents · Site content explains local workflow execution and usage.
- Packaging → Rust CLI: ships · Distribution packages native binaries and platform wrappers.
- Packaging → CLI entry: also ships · Current npm path still favors the TypeScript CLI surface.
- Experiments → Runner runtime: validates · Benchmarks and eval harnesses exercise retry, cache, and agent behavior.
- CI workflows → Runner runtime: smokes · GitHub Actions workflows validate compatibility and failure modes.
- CI workflows → DTU server: exercises · Smoke workflows depend on the mock server and artifact/cache behavior.