How aws-deadline/deadline-cloud-worker-agent works
Project type and purpose
• This is an open-source, Python-based “worker agent” that runs on customer hosts (Linux/Windows) and integrates with AWS Deadline Cloud. It’s not a web or full-stack app but a background service/daemon. Its responsibilities include:
• Bootstrap and install as an OS service (systemd on Linux, Windows Service)
• Load configuration, manage AWS credentials
• Poll Deadline Cloud’s Worker API to claim and run “sessions” (units of work)
• Execute session actions (download/upload attachments, run tasks, sync logs, report progress/metrics)
• Sync logs and telemetry to CloudWatch (and other AWS services)
File structure
• Top-level directories and scripts:
• src/deadline_worker_agent/… → all runtime code, organized by functionality (aws, aws_credentials, boto, config, installer, linux, windows, log_sync, scheduler, sessions, startup, utils, worker)
• scripts/ & pipeline/ → CI/CD and deployment pipelines
• docs/ → architecture and protocol docs (notably docs/dev/worker_api_protocol.md)
• test/ (unit, integ, e2e) → thorough testing layers
• configuration files (pyproject.toml, hatch.toml) and GitHub workflows
README for additional insights
• Confirms the agent implements the AWS Deadline Cloud worker API protocol
• Calls out installation steps, OS service integration, configuration, logging, and telemetry
• References specific AWS services and OpenJD sessions
1. Host OS & Service Manager
– Systemd (Linux) or Windows Service
– Startup/Bootstrap module
– Configuration Loader (CLI args, toml)
– AWS Credentials Manager (sts refresher, boto3 sessions)
– Scheduler & Session Manager
– Session Actions Executor (plugins/scripts)
– Log Sync & Metrics Reporter
3. External Services / Clouds
– AWS Deadline Cloud Worker API (HTTPS)
– S3 or artifact store (for attachments)
– CloudWatch Logs & Metrics
– STS / IAM (for credential refresh)
4. Data Stores / Filesystem
– Local cache, logs, config directories
– Session workspace (downloaded attachments, job outputs)
b. Relationships and interactions
• On startup, Service Manager → invokes Agent’s entrypoint → Bootstrap
• Bootstrap → Configuration Loader → AWS Credentials Manager
• Scheduler cycles: Scheduler → Worker API (poll) → receives Session description
• Session Manager → downloads attachments (S3) → executes actions (local OS calls or scripts)
• During execution → Session Manager → reports status/progress → Worker API
• Log Sync module → tails local logs → pushes to CloudWatch
• Metrics module → collects metrics → publishes to CloudWatch
• AWS Credentials Manager ↔ STS service for temporary creds
c. Architectural patterns & design principles
• Modular component-based design: each folder under src/deadline_worker_agent is a separable component
• Plugin-style actions: sessions/actions directory can grow new action types
• Separation of concerns: configuration, credentials, scheduling, execution, logging are distinct
• Use of well-defined protocols (OpenJD / Worker API)
d. Key technologies & libraries
• Python 3.9+
• boto3 / AWS SDK (via aws_credentials & boto modules)
• systemd or Windows Service manager
• TOML for config, Click or argparse for CLI args
• CloudWatch APIs for logs and metrics
Essential details to capture
• OS service integration (init)
• Credential refresh flow (STS ↔ credentials module ↔ scheduler/session manager)
• Polling and long-poll loops of the Worker API
• Execution of session actions and script plugins
• Log and metric emission paths