# aws-deadline/deadline-cloud-worker-agent architecture

> Architecture diagram of the GitHub repository aws-deadline/deadline-cloud-worker-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-deadline/deadline-cloud-worker-agent
- Repository: https://github.com/aws-deadline/deadline-cloud-worker-agent

## Overview

##### Project type and purpose
- This is an open-source, Python-based “worker agent” that runs on customer hosts (Linux/Windows) and integrates with AWS Deadline Cloud. It’s not a web or full-stack app but a background service/daemon. Its responsibilities include:
- Bootstrap and install as an OS service (systemd on Linux, Windows Service)
- Load configuration, manage AWS credentials
- Poll Deadline Cloud’s Worker API to claim and run “sessions” (units of work)
- Execute session actions (download/upload attachments, run tasks, sync logs, report progress/metrics)
- Sync logs and telemetry to CloudWatch (and other AWS services)

##### File structure
- Top-level directories and scripts:
- src/deadline_worker_agent/… → all runtime code, organized by functionality (aws, aws_credentials, boto, config, installer, linux, windows, log_sync, scheduler, sessions, startup, utils, worker)
- scripts/ & pipeline/ → CI/CD and deployment pipelines
- docs/ → architecture and protocol docs (notably docs/dev/worker_api_protocol.md)
- test/ (unit, integ, e2e) → thorough testing layers
- configuration files (pyproject.toml, hatch.toml) and GitHub workflows

##### README for additional insights
- Confirms the agent implements the AWS Deadline Cloud worker API protocol
- Calls out installation steps, OS service integration, configuration, logging, and telemetry
- References specific AWS services and OpenJD sessions

1. Host OS & Service Manager
– Systemd (Linux) or Windows Service
– Startup/Bootstrap module
– Configuration Loader (CLI args, toml)
– AWS Credentials Manager (sts refre­sher, boto3 sessions)
– Scheduler & Session Manager
– Session Actions Executor (plugins/scripts)
– Log Sync & Metrics Reporter
3. External Services / Clouds
– AWS Deadline Cloud Worker API (HTTPS)
– S3 or artifact store (for attachments)
– CloudWatch Logs & Metrics
– STS / IAM (for credential refresh)
4. Data Stores / Filesystem
– Local cache, logs, config directories
– Session workspace (downloaded attachments, job outputs)

b. Relationships and interactions
- On startup, Service Manager → invokes Agent’s entrypoint → Bootstrap
- Bootstrap → Configuration Loader → AWS Credentials Manager
- Scheduler cycles: Scheduler → Worker API (poll) → receives Session description
- Session Manager → downloads attachments (S3) → executes actions (local OS calls or scripts)
- During execution → Session Manager → reports status/progress → Worker API
- Log Sync module → tails local logs → pushes to CloudWatch
- Metrics module → collects metrics → publishes to CloudWatch
- AWS Credentials Manager ↔ STS service for temporary creds

c. Architectural patterns & design principles
- Modular component-based design: each folder under src/deadline_worker_agent is a separable component
- Plugin-style actions: `sessions/actions` directory can grow new action types
- Separation of concerns: configuration, credentials, scheduling, execution, logging are distinct
- Use of well-defined protocols (OpenJD / Worker API)

d. Key technologies & libraries
- Python 3.9+
- boto3 / AWS SDK (via aws_credentials & boto modules)
- systemd or Windows Service manager
- TOML for config, Click or argparse for CLI args
- CloudWatch APIs for logs and metrics

##### Essential details to capture
- OS service integration (init)
- Credential refresh flow (STS ↔ credentials module ↔ scheduler/session manager)
- Polling and long-poll loops of the Worker API
- Execution of session actions and script plugins
- Log and metric emission paths

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Host Environment
    subgraph "Host Environment"
        direction TB
        ServiceLinux["Systemd (Linux)"]:::host
        ServiceWin["Windows Service"]:::host
        ServiceLinux -->|invokes entrypoint| Entrypoint
        ServiceWin -->|invokes entrypoint| Entrypoint
    end

    %% Agent Process Container
    subgraph "Worker Agent Process" 
        direction TB
        Entrypoint["Entrypoint"]:::host
        Bootstrap["Bootstrap"]:::host
        ConfigLoader["Configuration Loader"]:::host
        CredsMgr["AWS Credentials Manager"]:::host
        Scheduler["Scheduler & Session Manager"]:::host
        SessionCore["Session Definition & Execution Core"]:::host
        ActionsExecutor["Session Actions Executor"]:::host
        LogSync["Log Sync & Logger"]:::host
        Metrics["Metrics Reporter"]:::host

        Entrypoint -->|calls| Bootstrap
        Bootstrap -->|loads config| ConfigLoader
        ConfigLoader -->|provides settings| CredsMgr
        CredsMgr -->|provides creds| Scheduler
        Scheduler -->|pollSession| WorkerAPI
        Scheduler -->|enqueue| SessionCore
        SessionCore -->|execute actions| ActionsExecutor
        ActionsExecutor -->|runs scripts| Scripts
        SessionCore -->|reportStatus| WorkerAPI
        LogSync -->|pushLogs| CloudWatch
        Metrics -->|publishMetrics| CloudWatch
        CredsMgr -->|refresh| STS
    end

    %% Scripts Plugins
    subgraph "Action Scripts Plugins"
        direction TB
        Scripts["Scripts (attachment_download, attachment_upload)"]:::host
    end
    ActionsExecutor --> Scripts

    %% AWS Cloud Services
    subgraph "AWS Services"
        direction TB
        WorkerAPI["AWS Deadline Worker API"]:::aws
        S3["S3 / Artifact Store"]:::aws
        CloudWatch["CloudWatch Logs & Metrics"]:::aws
        STS["STS / IAM"]:::aws
    end

    %% Data Stores
    subgraph "Local Data Stores"
        direction TB
        LocalCache["Local Cache & Workspace"]:::datastore
        LocalLogs["Local Logs & Config"]:::datastore
    end
    ConfigLoader -->|reads/writes| LocalLogs
    SessionCore -->|stores attachments| LocalCache
    SessionCore -->|writes outputs| LocalCache
    LogSync -->|tails| LocalLogs

    %% Component interactions
    Scheduler -->|download attachments| S3
    ActionsExecutor -->|upload results| S3
    Scheduler -->|poll| WorkerAPI
    SessionCore -->|report| WorkerAPI

    %% Click Events
    click ServiceLinux "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/linux/capabilities.py"
    click ServiceWin "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/windows/win_service.py"
    click Entrypoint "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/startup/entrypoint.py"
    click Bootstrap "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/startup/bootstrap.py"
    click ConfigLoader "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/config/cli_args.py"
    click ConfigLoader "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/config/config_file.py"
    click ConfigLoader "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/config/config.py"
    click ConfigLoader "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/config/settings.py"
    click CredsMgr "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/aws_credentials/credentials_refresher.py"
    click CredsMgr "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/aws_credentials/boto3_sessions.py"
    click CredsMgr "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/aws_credentials/temporary_credentials.py"
    click CredsMgr "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/aws_credentials/worker_boto3_session.py"
    click WorkerAPI "https://github.com/aws-deadline/deadline-cloud-worker-agent/tree/mainline/src/deadline_worker_agent/aws/deadline"
    click Scheduler "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/scheduler/scheduler.py"
    click Scheduler "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/scheduler/session_queue.py"
    click Scheduler "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/scheduler/session_cleanup.py"
    click SessionCore "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/sessions/session.py"
    click SessionCore "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/sessions/active_action.py"
    click ActionsExecutor "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/sessions/actions/run_attachment_download.py"
    click ActionsExecutor "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/sessions/actions/run_step_task.py"
    click Scripts "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/sessions/actions/scripts/attachment_download.py"
    click Scripts "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/sessions/actions/scripts/attachment_upload.py"
    click LogSync "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/log_sync/cloudwatch.py"
    click LogSync "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/log_sync/loggers.py"
    click Metrics "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/metrics.py"
    click Installer "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/installer/install.sh"
    click Installer "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/installer/win_installer.py"
    click BotoWrapper "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/boto/config.py"
    click BotoWrapper "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/boto/logger.py"
    click BotoWrapper "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/boto/retries.py"
    click BotoWrapper "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/boto/shim.py"
    click WorkerMain "https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/src/deadline_worker_agent/worker.py"

    %% Styles
    classDef host fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef aws fill:#ffcc80,stroke:#333,stroke-width:1px
    classDef datastore fill:#c8e6c9,stroke:#333,stroke-width:1px
```
