# bytefloater/autopkg-runner architecture

> Architecture diagram of the GitHub repository bytefloater/autopkg-runner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytefloater/autopkg-runner
- Repository: https://github.com/bytefloater/autopkg-runner

## Overview

1. **What this repository is**
   - AutoPkg Runner is a single Django-based application that combines a web UI, REST API, scheduler, background job runner, and AutoPkg pipeline orchestrator for macOS.
   - It is not a distributed service mesh; most subsystems live in one process space and coordinate through the database, filesystem, and internal task execution.

2. **Process entry points and deployment modes**
   - `manage.py` is the main Django entry point for local/admin commands.
   - `run.py` and the `autopkgrunner` package provide the project runtime (`asgi.py`, `wsgi.py`, `settings.py`, `urls.py`).
   - In production it runs under Gunicorn with Uvicorn workers for ASGI, mainly to support live SSE log streaming and concurrent viewers.
   - There is also a macOS launchd system-daemon mode, which is important because the app interacts with macOS permissions and protected filesystem/network locations.

3. **Core application split**
   - The project is divided into three major Django apps:
     - `webapp`: the main UI, models, scheduler, runner, auth backends, middleware, management commands, and broad business logic.
     - `api`: token-authenticated REST endpoints for triggering runs and reading run history/task status.
     - `stages` and `libs`: the AutoPkg execution pipeline and supporting system/OS integration code.
   - This is a modular monolith, with clear separation between presentation, API, orchestration, and OS-facing helpers.

4. **Persistent data boundary**
   - State is stored in the Django database layer, with migrations under `webapp/migrations`.
   - The database holds users, API tokens, configuration values, schedules, notification settings, run history, stage/log data, share tokens, web push subscriptions, and encrypted credentials.
   - The README and code imply SQLite as the default local store, but the architecture is database-backed rather than file-backed for most app state.

5. **Pipeline orchestration flow**
   - The main runtime flow starts from a manual trigger, API call, or scheduler event.
   - `webapp/runner.py` and `stages/` implement the ordered pipeline described in the README: environment check, repo updates, trust verification, repo mount, AutoPkg execution, report generation, garbage collection, and notifications.
   - Each stage is a distinct boundary with its own responsibility and failure semantics, which is useful for diagramming the pipeline as a linear chain with shared run context.

6. **AutoPkg and filesystem integration**
   - The app is tightly coupled to local AutoPkg tooling and the macOS filesystem.
   - `libs/run_command.py`, `libs/hosts.py`, `libs/mdns.py`, `libs/mount_repo.py`, and related helpers handle shelling out to AutoPkg, discovering hosts, mounting repositories, and dealing with network resources.
   - The repository interacts with recipe directories, cache paths, report plists, override files, and mounted shares, so filesystem access is a first-class architectural boundary.

7. **Real-time run visibility**
   - Live run detail pages use server-side event streaming rather than polling-heavy UI refreshes.
   - `webapp/run_broadcaster.py`, the logger interception modules, and the run/detail views work together to fan out stage/log updates to multiple viewers while keeping a single DB-polling source per run.
   - This is the key real-time path: pipeline execution writes logs and stage state, the broadcaster relays updates, and the browser updates the page without reloads.

8. **Web UI structure**
   - The `webapp/views` and `webapp/templates/webapp` tree covers the dashboard, runs, schedule, recipes, configuration, notifications, users, sharing, and account management.
   - Static assets include Tailwind-generated CSS, CodeMirror for XML editing, a service worker, and PWA/mobile-specific templates.
   - The web UI is intentionally split into desktop and mobile experiences, but they share the same backend models and business logic.

9. **REST API boundary**
   - The `api` app exposes a small, focused API surface for automation clients.
   - Its endpoints cover authentication, triggering tasks, checking task status, and retrieving run history/details.
   - The API supports JSON and XML responses, and token authentication is required except for token acquisition itself.

10. **Recipe management subsystem**
    - Recipe management is its own major subsystem under `webapp/recipe_index.py`, `webapp/views/recipes.py`, and the related templates.
    - It handles installed recipe repositories, recipe discovery across local repo search paths, overrides, active run-list toggling, and the community recipe index fetched from GitHub.
    - The subsystem spans both local file editing and remote metadata lookup, so it bridges the filesystem and external GitHub-hosted index.

11. **Notifications and external integrations**
    - `notifiers/` contains provider-specific integrations for Pushover, Discord, Slack, Microsoft Teams, Google Chat, Email/SMTP, WebPush, and local system notifications.
    - Notifications are dispatched at the end of a run and are driven by templated run data such as status, duration, counts, and share links.
    - Additional external dependencies include SMTP servers, webhook endpoints, browser push subscriptions, and macOS network/share services.

12. **Security, auth, and operational constraints**
    - Authentication spans Django user auth, API token auth, custom backends, permissions, and challenge/response mechanisms.
    - Sensitive data is encrypted before being stored, and the project relies on a stable `DJANGO_SECRET_KEY` for decrypting persisted secrets.
    - The architecture is also shaped by macOS security boundaries: Full Disk Access, local network permission, and remote volume access are required for the filesystem and repository operations to work reliably.

13. **Maintenance and housekeeping**
    - The app includes scheduled and ad hoc maintenance paths: repository refresh, garbage collection, DB cleanup, and setup/purge commands.
    - APScheduler-like behavior is embedded in the app runtime rather than delegated to an external job service, so scheduling is part of the application process boundary.
    - Management commands under `webapp/management/commands` provide operational hooks for installation, bootstrapping, and service setup.

14. **Suggested diagram framing**
    - A good architecture diagram should center on the Django application, then branch into:
      - UI/API ingress
      - orchestration/runtime
      - database persistence
      - filesystem/AutoPkg integration
      - external services for notifications, GitHub recipe index, and network shares
      - real-time log/event streaming back to browsers
    - The main distinction is between “control plane” logic in Django and “execution plane” logic in the stage pipeline and OS-level AutoPkg interactions.

Main entry points: [`webapp/views`](https://github.com/bytefloater/autopkg-runner/tree/main/webapp/views) (Views), [`webapp/runner.py`](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/runner.py) (Runner), [`api/views`](https://github.com/bytefloater/autopkg-runner/tree/main/api/views) (API views), [`webapp/models.py`](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/models.py) (Models)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entry points"]
  node_manage_py["manage.py<br/>Django CLI<br/>[manage.py]"]
  node_run_py["run.py<br/>Runtime launcher<br/>[run.py]"]
  node_settings["Settings<br/>Django config<br/>[settings.py]"]
  node_urls["URL routes<br/>Routing<br/>[urls.py]"]
end

subgraph group_webapp["Web app"]
  node_web_views["Views"]
  node_web_models[("Models<br/>Domain data<br/>[models.py]")]
  node_scheduler["Scheduler<br/>Timed jobs<br/>[scheduler.py]"]
  node_runner["Runner<br/>Orchestration<br/>[runner.py]"]
  node_broadcast["Broadcaster<br/>Live updates<br/>[run_broadcaster.py]"]
  node_auth["Auth<br/>Security<br/>[auth_backends.py]"]
  node_db_logger["DB logger<br/>Logging<br/>[db_logger.py]"]
  node_encryption["Encryption<br/>Secrets<br/>[encryption.py]"]
  node_recipe_index["Recipe index<br/>Recipe mgmt<br/>[recipe_index.py]"]
  node_templates["UI templates<br/>Presentation"]
  node_static["Static assets<br/>Frontend assets"]
end

subgraph group_api["API"]
  node_api_views["API views<br/>REST handlers"]
  node_api_auth["API auth<br/>Token auth<br/>[authentication.py]"]
end

subgraph group_pipeline["Pipeline"]
  node_stage_nodes["Stages"]
  node_libs["Libs<br/>OS helpers"]
end

subgraph group_integrations["Integrations"]
  node_notifiers["Notifiers<br/>Integrations"]
end

node_manage_py -->|"boots Django"| node_settings
node_run_py -->|"boots runtime"| node_settings
node_settings -->|"wires routes"| node_urls
node_urls -->|"serves UI"| node_web_views
node_urls -->|"serves API"| node_api_views
node_web_views -->|"reads/writes"| node_web_models
node_web_views -->|"triggers runs"| node_runner
node_scheduler -->|"starts jobs"| node_runner
node_runner -->|"executes pipeline"| node_stage_nodes
node_stage_nodes -->|"uses helpers"| node_libs
node_libs -->|"dispatches alerts"| node_notifiers
node_runner -->|"records logs"| node_db_logger
node_db_logger -->|"streams updates"| node_broadcast
node_broadcast -->|"updates UI"| node_web_views
node_api_views -->|"authenticates"| node_api_auth
node_api_views -->|"triggers runs"| node_runner
node_api_views -->|"reads status"| node_web_models
node_web_views -->|"manages recipes"| node_recipe_index
node_web_views -->|"renders pages"| node_templates
node_web_views -->|"serves assets"| node_static
node_auth -->|"checks access"| node_web_models
node_encryption -->|"protects secrets"| node_web_models

click node_manage_py "https://github.com/bytefloater/autopkg-runner/blob/main/manage.py"
click node_run_py "https://github.com/bytefloater/autopkg-runner/blob/main/run.py"
click node_settings "https://github.com/bytefloater/autopkg-runner/blob/main/autopkgrunner/settings.py"
click node_urls "https://github.com/bytefloater/autopkg-runner/blob/main/autopkgrunner/urls.py"
click node_web_views "https://github.com/bytefloater/autopkg-runner/tree/main/webapp/views"
click node_web_models "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/models.py"
click node_scheduler "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/scheduler.py"
click node_runner "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/runner.py"
click node_broadcast "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/run_broadcaster.py"
click node_auth "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/auth_backends.py"
click node_db_logger "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/db_logger.py"
click node_encryption "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/encryption.py"
click node_recipe_index "https://github.com/bytefloater/autopkg-runner/blob/main/webapp/recipe_index.py"
click node_api_views "https://github.com/bytefloater/autopkg-runner/tree/main/api/views"
click node_api_auth "https://github.com/bytefloater/autopkg-runner/blob/main/api/authentication.py"
click node_stage_nodes "https://github.com/bytefloater/autopkg-runner/tree/main/stages"
click node_libs "https://github.com/bytefloater/autopkg-runner/tree/main/libs"
click node_notifiers "https://github.com/bytefloater/autopkg-runner/tree/main/notifiers"
click node_templates "https://github.com/bytefloater/autopkg-runner/tree/main/webapp/templates/webapp"
click node_static "https://github.com/bytefloater/autopkg-runner/tree/main/webapp/static"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_manage_py,node_run_py,node_settings,node_urls toneBlue
class node_web_views,node_web_models,node_scheduler,node_runner,node_broadcast,node_auth,node_db_logger,node_encryption,node_recipe_index,node_templates,node_static toneAmber
class node_api_views,node_api_auth toneMint
class node_stage_nodes,node_libs toneRose
class node_notifiers toneIndigo
```

## Components

### Entry points

Process startup and Django runtime wiring

- [manage.py](https://github.com/bytefloater/autopkg-runner/blob/main/manage.py): Django CLI · Admin and local management entry point · `manage.py`
- [run.py](https://github.com/bytefloater/autopkg-runner/blob/main/run.py): Runtime launcher · Application startup wrapper for the project runtime · `run.py`
- [Settings](https://github.com/bytefloater/autopkg-runner/blob/main/autopkgrunner/settings.py): Django config · Project settings and environment wiring · `autopkgrunner/settings.py`
- [URL routes](https://github.com/bytefloater/autopkg-runner/blob/main/autopkgrunner/urls.py): Routing · Root URL dispatcher for web and API paths · `autopkgrunner/urls.py`

### Web app

Main Django app for UI, orchestration, and persistence

- [Views](https://github.com/bytefloater/autopkg-runner/tree/main/webapp/views): UI views · Dashboard, runs, recipes, config, users, and sharing pages · `webapp/views`
- [Models](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/models.py): Domain data · Persistent application state and configuration models · `webapp/models.py`
- [Scheduler](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/scheduler.py): Timed jobs · In-process scheduling for maintenance and run triggers · `webapp/scheduler.py`
- [Runner](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/runner.py): Orchestration · Coordinates run execution and shared run context · `webapp/runner.py`
- [Broadcaster](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/run_broadcaster.py): Live updates · Fans out run and log updates to viewers · `webapp/run_broadcaster.py`
- [Auth](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/auth_backends.py): Security · Custom auth backends, permissions, and middleware · `webapp/auth_backends.py`
- [DB logger](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/db_logger.py): Logging · Stores run logs and stage events in the database · `webapp/db_logger.py`
- [Encryption](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/encryption.py): Secrets · Encrypts and decrypts stored credentials · `webapp/encryption.py`
- [Recipe index](https://github.com/bytefloater/autopkg-runner/blob/main/webapp/recipe_index.py): Recipe mgmt · Local recipe discovery and community index lookup · `webapp/recipe_index.py`
- [UI templates](https://github.com/bytefloater/autopkg-runner/tree/main/webapp/templates/webapp): Presentation · Desktop and mobile templates for the web interface · `webapp/templates/webapp`
- [Static assets](https://github.com/bytefloater/autopkg-runner/tree/main/webapp/static): Frontend assets · CSS, code editor assets, service worker, icons, and mobile UI assets · `webapp/static`

### API

Token-authenticated automation endpoints

- [API views](https://github.com/bytefloater/autopkg-runner/tree/main/api/views): REST handlers · Auth, trigger, history, and task status endpoints · `api/views`
- [API auth](https://github.com/bytefloater/autopkg-runner/blob/main/api/authentication.py): Token auth · Token authentication and challenge flows · `api/authentication.py`

### Pipeline

AutoPkg execution stages and OS-facing helpers

- [Stages](https://github.com/bytefloater/autopkg-runner/tree/main/stages): Pipeline stages · Environment check, repo update, trust, mount, run, notify, cleanup · `stages`
- [Libs](https://github.com/bytefloater/autopkg-runner/tree/main/libs): OS helpers · Command execution, host discovery, mDNS, mounts, and stage utilities · `libs`

### Integrations

Notifications and external service boundaries

- [Notifiers](https://github.com/bytefloater/autopkg-runner/tree/main/notifiers): Integrations · Pushover, Slack, Discord, Teams, Chat, email, web push, system alerts · `notifiers`

## Connections

- manage.py → Settings: boots Django · Loads project configuration for management commands
- run.py → Settings: boots runtime · Initializes the app process and deployment mode
- Settings → URL routes: wires routes · Connects the root URL configuration
- URL routes → Views: serves UI · Routes browser requests into the main web app
- URL routes → API views: serves API · Routes automation clients to REST endpoints
- Views → Models: reads/writes · UI actions persist and query application state
- Views → Runner: triggers runs · Manual actions start pipeline execution
- Scheduler → Runner: starts jobs · Scheduled maintenance and run triggers invoke the runner
- Runner → Stages: executes pipeline · Runs the ordered AutoPkg stage chain
- Stages → Libs: uses helpers · Stages rely on OS integration and command helpers
- Libs → Notifiers: dispatches alerts · Run completion and status notifications use provider integrations
- Runner → DB logger: records logs · Execution emits stage and run logs to the database-backed logger
- DB logger → Broadcaster: streams updates · Log and stage updates are fanned out to viewers
- Broadcaster → Views: updates UI · Live run pages receive streamed progress without reloads
- API views → API auth: authenticates · Token and challenge-based access control for automation clients
- API views → Runner: triggers runs · API requests can launch runs and tasks
- API views → Models: reads status · History and task endpoints read persisted run data
- Views → Recipe index: manages recipes · Recipe pages and editors bridge local repos and remote index data
- Views → UI templates: renders pages · Desktop and mobile views are rendered from shared templates
- Views → Static assets: serves assets · UI depends on bundled frontend assets and PWA resources
- Auth → Models: checks access · Permissions and auth backends rely on persisted users and tokens
- Encryption → Models: protects secrets · Encrypted credentials are stored in the database layer