# sarthak-shubham/tailrunner architecture

> Architecture diagram of the GitHub repository sarthak-shubham/tailrunner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sarthak-shubham/tailrunner
- Repository: https://github.com/sarthak-shubham/tailrunner

## Overview

#### Purpose and Deployment
TailRunner is a distributed batch system that reduces tail latency through speculative execution: slow jobs are duplicated onto healthy workers, the first result wins, and the competing attempt is cancelled. Docker Compose provisions Redis, one Python scheduler, and horizontally scalable Go workers via `infra/docker-compose.yml`; service images are defined in `scheduler/Dockerfile` and `worker/Dockerfile`.

#### Process Entry Points
The scheduler process starts from `scheduler/main.py`, assembling the FastAPI-facing control plane and scheduling engine. Go worker replicas start at `worker/main.go`, initialize their Redis and HTTP integrations, then run the worker lifecycle. Scheduler HTTP serving/routing is owned by `scheduler/api/server.py` and `scheduler/api/router.py`; equivalent worker endpoints are in `worker/api/server.go` and `worker/api/router.go`.

#### Scheduler Control Plane
The Python scheduler is the authoritative “brain”: it accepts batches, tracks attempts and workers in RAM, dispatches work, evaluates slow jobs, and arbitrates completions. Its orchestration loop resides in `scheduler/engine.py`. In-memory domain state is separated by concern: batch lifecycle in `scheduler/state/batch.py`, job/attempt state in `scheduler/state/jobs.py`, and worker availability/health in `scheduler/state/workers.py`. Redis is explicitly not the durable source of scheduler state.

#### Queue and Worker Execution Boundary
Redis is the ephemeral cross-language work/result transport, abstracted by `scheduler/data/redis_queue.py` for the scheduler and `worker/data/redis_client.go` for workers. Workers remain intentionally simple: `worker/core/task.go` fetches assigned payloads, executes them, and reports outcomes. Execution is pluggable behind `worker/executors/base.go`; the current concrete workload is `worker/executors/word_count.go`. Scheduler-to-worker control interactions use HTTP clients in `scheduler/api/client.py` and `worker/api/client.go`.

#### Speculation and Completion Arbitration
The scheduler derives straggler decisions from observed runtime history. `scheduler/history.py` maintains timing observations; `scheduler/maths/p95.py` calculates the tail threshold; `scheduler/maths/rules.py` decides when a job qualifies for speculative duplication. `scheduler/maths/arbitration.py` resolves races so one successful attempt becomes canonical and losing work is cancelled. This is the central correctness boundary: duplicate execution is permitted, but batch/job completion must be recorded once.

#### Recovery and Failure Model
Scheduler state is RAM-resident, so restart behavior is intentionally “amnesiac” rather than durable replay. `scheduler/recovery/amnesia.py` handles crash/recovery cleanup and reconciliation against ephemeral infrastructure. Redis availability and worker HTTP reachability are material runtime dependencies; failure handling must tolerate stale queued work, duplicate reports, and workers disappearing during execution.

#### Runtime Stack and Observability
The scheduler requires Python 3.11+ and FastAPI; workers require Go 1.22+. Redis is the shared queue/result service. Containers and scale-out are managed through `infra/docker-compose.yml`, including benchmark-scale worker replicas. Structured logging is implemented separately for scheduler and worker processes in `scheduler/utils/logger.py` and `worker/utils/logger.go`.

Main entry points: [`scheduler/engine.py`](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/engine.py) (Scheduling engine), [`scheduler/main.py`](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/main.py) (Scheduler process), [`worker/core/task.go`](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/core/task.go) (Task lifecycle), [`infra/docker-compose.yml`](https://github.com/sarthak-shubham/tailrunner/blob/main/infra/docker-compose.yml) (Docker Compose)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_infra["Deployment"]
  node_compose["Docker Compose<br/>deployment<br/>[docker-compose.yml]"]
end

subgraph group_scheduler["Scheduler control plane"]
  node_scheduler_main["Scheduler process<br/>Python entry point<br/>[main.py]"]
  node_scheduler_api["Scheduler HTTP API<br/>FastAPI server and router<br/>[server.py]"]
  node_engine{{"Scheduling engine<br/>orchestration loop<br/>[engine.py]"}}
  node_scheduler_state[("In-memory domain state<br/>[jobs.py]")]
  node_speculation["Speculation policy<br/>history, p95, and rules<br/>[rules.py]"]
  node_arbitration{{"Completion arbitration<br/>race resolution<br/>[arbitration.py]"}}
  node_recovery["Amnesiac recovery<br/>cleanup and reconciliation<br/>[amnesia.py]"]
  node_scheduler_queue["Scheduler queue adapter<br/>Redis adapter<br/>[redis_queue.py]"]
  node_scheduler_client["Scheduler worker client<br/>HTTP client<br/>[client.py]"]
end

subgraph group_transport["Transport"]
  node_redis["Redis<br/>ephemeral work/result transport"]
end

subgraph group_worker["Worker replicas"]
  node_worker_main["Worker process<br/>Go entry point<br/>[main.go]"]
  node_worker_api["Worker HTTP API<br/>server and router<br/>[server.go]"]
  node_worker_redis["Worker Redis client<br/>Redis adapter<br/>[redis_client.go]"]
  node_task_lifecycle{{"Task lifecycle<br/>fetch, execute, report<br/>[task.go]"}}
  node_executor["Pluggable executor<br/>execution interface<br/>[base.go]"]
  node_word_count["Word count workload<br/>concrete executor<br/>[word_count.go]"]
  node_worker_client["Worker scheduler client<br/>HTTP client<br/>[client.go]"]
end

node_compose -->|"runs"| node_scheduler_main
node_compose -->|"provisions"| node_redis
node_compose -->|"scales"| node_worker_main
node_scheduler_main -->|"serves"| node_scheduler_api
node_scheduler_main -->|"starts"| node_engine
node_scheduler_api -->|"submits batches"| node_engine
node_engine -->|"tracks"| node_scheduler_state
node_engine -->|"evaluates slow jobs"| node_speculation
node_speculation -->|"permits duplicate attempts"| node_arbitration
node_engine -->|"arbitrates results"| node_arbitration
node_engine -->|"dispatches and reads results"| node_scheduler_queue
node_scheduler_queue -->|"uses"| node_redis
node_scheduler_main -->|"recovers after restart"| node_recovery
node_recovery -.->|"reconciles"| node_redis
node_engine -->|"controls workers"| node_scheduler_client
node_scheduler_client -->|"HTTP"| node_worker_api
node_worker_main -->|"serves"| node_worker_api
node_worker_main -->|"runs"| node_task_lifecycle
node_worker_redis -->|"uses"| node_redis
node_task_lifecycle -->|"fetches and reports"| node_worker_redis
node_task_lifecycle -->|"executes through"| node_executor
node_executor -->|"implemented by"| node_word_count
node_task_lifecycle -->|"reports/control"| node_worker_client
node_worker_client -->|"HTTP"| node_scheduler_api
node_arbitration -->|"cancels losers"| node_scheduler_client

click node_compose "https://github.com/sarthak-shubham/tailrunner/blob/main/infra/docker-compose.yml"
click node_scheduler_main "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/main.py"
click node_scheduler_api "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/api/server.py"
click node_engine "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/engine.py"
click node_scheduler_state "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/state/jobs.py"
click node_speculation "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/maths/rules.py"
click node_arbitration "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/maths/arbitration.py"
click node_recovery "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/recovery/amnesia.py"
click node_scheduler_queue "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/data/redis_queue.py"
click node_worker_main "https://github.com/sarthak-shubham/tailrunner/blob/main/worker/main.go"
click node_worker_api "https://github.com/sarthak-shubham/tailrunner/blob/main/worker/api/server.go"
click node_worker_redis "https://github.com/sarthak-shubham/tailrunner/blob/main/worker/data/redis_client.go"
click node_task_lifecycle "https://github.com/sarthak-shubham/tailrunner/blob/main/worker/core/task.go"
click node_executor "https://github.com/sarthak-shubham/tailrunner/blob/main/worker/executors/base.go"
click node_word_count "https://github.com/sarthak-shubham/tailrunner/blob/main/worker/executors/word_count.go"
click node_scheduler_client "https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/api/client.py"
click node_worker_client "https://github.com/sarthak-shubham/tailrunner/blob/main/worker/api/client.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_compose toneBlue
class node_scheduler_main,node_scheduler_api,node_engine,node_scheduler_state,node_speculation,node_arbitration,node_recovery,node_scheduler_queue,node_scheduler_client toneAmber
class node_redis toneMint
class node_worker_main,node_worker_api,node_worker_redis,node_task_lifecycle,node_executor,node_word_count,node_worker_client toneRose
```

## Components

### Deployment

- [Docker Compose](https://github.com/sarthak-shubham/tailrunner/blob/main/infra/docker-compose.yml): deployment · Provisions Redis, one scheduler, and scalable workers. · `infra/docker-compose.yml`

### Scheduler control plane

- [Scheduler process](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/main.py): Python entry point · `scheduler/main.py`
- [Scheduler HTTP API](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/api/server.py): FastAPI server and router · `scheduler/api/server.py`
- [Scheduling engine](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/engine.py): orchestration loop · Authoritative in-RAM scheduling and completion control. · `scheduler/engine.py`
- [In-memory domain state](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/state/jobs.py): batch, job, and worker state · RAM is authoritative; Redis is not durable scheduler state. · `scheduler/state/jobs.py`
- [Speculation policy](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/maths/rules.py): history, p95, and rules · Identifies stragglers eligible for duplicate execution. · `scheduler/maths/rules.py`
- [Completion arbitration](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/maths/arbitration.py): race resolution · Commits one winner and cancels losing attempts. · `scheduler/maths/arbitration.py`
- [Amnesiac recovery](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/recovery/amnesia.py): cleanup and reconciliation · Reconciles RAM loss with ephemeral infrastructure after restart. · `scheduler/recovery/amnesia.py`
- [Scheduler queue adapter](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/data/redis_queue.py): Redis adapter · `scheduler/data/redis_queue.py`
- [Scheduler worker client](https://github.com/sarthak-shubham/tailrunner/blob/main/scheduler/api/client.py): HTTP client · `scheduler/api/client.py`

### Transport

- **Redis**: ephemeral work/result transport · Cross-language queue and result boundary.

### Worker replicas

- [Worker process](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/main.go): Go entry point · `worker/main.go`
- [Worker HTTP API](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/api/server.go): server and router · Receives scheduler control interactions including cancellation. · `worker/api/server.go`
- [Worker Redis client](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/data/redis_client.go): Redis adapter · `worker/data/redis_client.go`
- [Task lifecycle](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/core/task.go): fetch, execute, report · `worker/core/task.go`
- [Pluggable executor](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/executors/base.go): execution interface · `worker/executors/base.go`
- [Word count workload](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/executors/word_count.go): concrete executor · `worker/executors/word_count.go`
- [Worker scheduler client](https://github.com/sarthak-shubham/tailrunner/blob/main/worker/api/client.go): HTTP client · `worker/api/client.go`

## Connections

- Docker Compose → Scheduler process: runs
- Docker Compose → Redis: provisions
- Docker Compose → Worker process: scales
- Scheduler process → Scheduler HTTP API: serves
- Scheduler process → Scheduling engine: starts
- Scheduler HTTP API → Scheduling engine: submits batches
- Scheduling engine → In-memory domain state: tracks
- Scheduling engine → Speculation policy: evaluates slow jobs
- Speculation policy → Completion arbitration: permits duplicate attempts
- Scheduling engine → Completion arbitration: arbitrates results
- Scheduling engine → Scheduler queue adapter: dispatches and reads results
- Scheduler queue adapter → Redis: uses
- Scheduler process → Amnesiac recovery: recovers after restart
- Amnesiac recovery → Redis: reconciles
- Scheduling engine → Scheduler worker client: controls workers
- Scheduler worker client → Worker HTTP API: HTTP
- Worker process → Worker HTTP API: serves
- Worker process → Task lifecycle: runs
- Worker Redis client → Redis: uses
- Task lifecycle → Worker Redis client: fetches and reports
- Task lifecycle → Pluggable executor: executes through
- Pluggable executor → Word count workload: implemented by
- Task lifecycle → Worker scheduler client: reports/control
- Worker scheduler client → Scheduler HTTP API: HTTP
- Completion arbitration → Scheduler worker client: cancels losers