sarthak-shubham/tailrunner

Purpose and Deployment

TailRunner is a distributed batch system that reduces tail latency through speculative execution: slow jobs are duplicated onto healthy workers, the first result wins, and the competing attempt is cancelled. Docker Compose provisions Redis, one Python scheduler, and horizontally scalable Go workers via infra/docker-compose.yml; service images are defined in scheduler/Dockerfile and worker/Dockerfile.

Process Entry Points

The scheduler process starts from scheduler/main.py, assembling the FastAPI-facing control plane and scheduling engine. Go worker replicas start at worker/main.go, initialize their Redis and HTTP integrations, then run the worker lifecycle. Scheduler HTTP serving/routing is owned by scheduler/api/server.py and scheduler/api/router.py; equivalent worker endpoints are in worker/api/server.go and worker/api/router.go.

Scheduler Control Plane

The Python scheduler is the authoritative “brain”: it accepts batches, tracks attempts and workers in RAM, dispatches work, evaluates slow jobs, and arbitrates completions. Its orchestration loop resides in scheduler/engine.py. In-memory domain state is separated by concern: batch lifecycle in scheduler/state/batch.py, job/attempt state in scheduler/state/jobs.py, and worker availability/health in scheduler/state/workers.py. Redis is explicitly not the durable source of scheduler state.

Queue and Worker Execution Boundary

Redis is the ephemeral cross-language work/result transport, abstracted by scheduler/data/redis_queue.py for the scheduler and worker/data/redis_client.go for workers. Workers remain intentionally simple: worker/core/task.go fetches assigned payloads, executes them, and reports outcomes. Execution is pluggable behind worker/executors/base.go; the current concrete workload is worker/executors/word_count.go. Scheduler-to-worker control interactions use HTTP clients in scheduler/api/client.py and worker/api/client.go.

Speculation and Completion Arbitration

The scheduler derives straggler decisions from observed runtime history. scheduler/history.py maintains timing observations; scheduler/maths/p95.py calculates the tail threshold; scheduler/maths/rules.py decides when a job qualifies for speculative duplication. scheduler/maths/arbitration.py resolves races so one successful attempt becomes canonical and losing work is cancelled. This is the central correctness boundary: duplicate execution is permitted, but batch/job completion must be recorded once.

Recovery and Failure Model

Scheduler state is RAM-resident, so restart behavior is intentionally “amnesiac” rather than durable replay. scheduler/recovery/amnesia.py handles crash/recovery cleanup and reconciliation against ephemeral infrastructure. Redis availability and worker HTTP reachability are material runtime dependencies; failure handling must tolerate stale queued work, duplicate reports, and workers disappearing during execution.

Runtime Stack and Observability

The scheduler requires Python 3.11+ and FastAPI; workers require Go 1.22+. Redis is the shared queue/result service. Containers and scale-out are managed through infra/docker-compose.yml, including benchmark-scale worker replicas. Structured logging is implemented separately for scheduler and worker processes in scheduler/utils/logger.py and worker/utils/logger.go.