bytefloater/autopkg-runner

1. What this repository is

- AutoPkg Runner is a single Django-based application that combines a web UI, REST API, scheduler, background job runner, and AutoPkg pipeline orchestrator for macOS.

- It is not a distributed service mesh; most subsystems live in one process space and coordinate through the database, filesystem, and internal task execution.

2. Process entry points and deployment modes

- manage.py is the main Django entry point for local/admin commands.

- run.py and the autopkgrunner package provide the project runtime (asgi.py, wsgi.py, settings.py, urls.py).

- In production it runs under Gunicorn with Uvicorn workers for ASGI, mainly to support live SSE log streaming and concurrent viewers.

- There is also a macOS launchd system-daemon mode, which is important because the app interacts with macOS permissions and protected filesystem/network locations.

3. Core application split

- The project is divided into three major Django apps:

- webapp: the main UI, models, scheduler, runner, auth backends, middleware, management commands, and broad business logic.

- api: token-authenticated REST endpoints for triggering runs and reading run history/task status.

- stages and libs: the AutoPkg execution pipeline and supporting system/OS integration code.

- This is a modular monolith, with clear separation between presentation, API, orchestration, and OS-facing helpers.

4. Persistent data boundary

- State is stored in the Django database layer, with migrations under webapp/migrations.

- The database holds users, API tokens, configuration values, schedules, notification settings, run history, stage/log data, share tokens, web push subscriptions, and encrypted credentials.

- The README and code imply SQLite as the default local store, but the architecture is database-backed rather than file-backed for most app state.

5. Pipeline orchestration flow

- The main runtime flow starts from a manual trigger, API call, or scheduler event.

- webapp/runner.py and stages/ implement the ordered pipeline described in the README: environment check, repo updates, trust verification, repo mount, AutoPkg execution, report generation, garbage collection, and notifications.

- Each stage is a distinct boundary with its own responsibility and failure semantics, which is useful for diagramming the pipeline as a linear chain with shared run context.

6. AutoPkg and filesystem integration

- The app is tightly coupled to local AutoPkg tooling and the macOS filesystem.

- libs/run_command.py, libs/hosts.py, libs/mdns.py, libs/mount_repo.py, and related helpers handle shelling out to AutoPkg, discovering hosts, mounting repositories, and dealing with network resources.

- The repository interacts with recipe directories, cache paths, report plists, override files, and mounted shares, so filesystem access is a first-class architectural boundary.

7. Real-time run visibility

- Live run detail pages use server-side event streaming rather than polling-heavy UI refreshes.

- webapp/run_broadcaster.py, the logger interception modules, and the run/detail views work together to fan out stage/log updates to multiple viewers while keeping a single DB-polling source per run.

- This is the key real-time path: pipeline execution writes logs and stage state, the broadcaster relays updates, and the browser updates the page without reloads.

8. Web UI structure

- The webapp/views and webapp/templates/webapp tree covers the dashboard, runs, schedule, recipes, configuration, notifications, users, sharing, and account management.

- Static assets include Tailwind-generated CSS, CodeMirror for XML editing, a service worker, and PWA/mobile-specific templates.

- The web UI is intentionally split into desktop and mobile experiences, but they share the same backend models and business logic.

9. REST API boundary

- The api app exposes a small, focused API surface for automation clients.

- Its endpoints cover authentication, triggering tasks, checking task status, and retrieving run history/details.

- The API supports JSON and XML responses, and token authentication is required except for token acquisition itself.

10. Recipe management subsystem

- Recipe management is its own major subsystem under webapp/recipe_index.py, webapp/views/recipes.py, and the related templates.

- It handles installed recipe repositories, recipe discovery across local repo search paths, overrides, active run-list toggling, and the community recipe index fetched from GitHub.

- The subsystem spans both local file editing and remote metadata lookup, so it bridges the filesystem and external GitHub-hosted index.

11. Notifications and external integrations

- notifiers/ contains provider-specific integrations for Pushover, Discord, Slack, Microsoft Teams, Google Chat, Email/SMTP, WebPush, and local system notifications.

- Notifications are dispatched at the end of a run and are driven by templated run data such as status, duration, counts, and share links.

- Additional external dependencies include SMTP servers, webhook endpoints, browser push subscriptions, and macOS network/share services.

12. Security, auth, and operational constraints

- Authentication spans Django user auth, API token auth, custom backends, permissions, and challenge/response mechanisms.

- Sensitive data is encrypted before being stored, and the project relies on a stable DJANGO_SECRET_KEY for decrypting persisted secrets.

- The architecture is also shaped by macOS security boundaries: Full Disk Access, local network permission, and remote volume access are required for the filesystem and repository operations to work reliably.

13. Maintenance and housekeeping

- The app includes scheduled and ad hoc maintenance paths: repository refresh, garbage collection, DB cleanup, and setup/purge commands.

- APScheduler-like behavior is embedded in the app runtime rather than delegated to an external job service, so scheduling is part of the application process boundary.

- Management commands under webapp/management/commands provide operational hooks for installation, bootstrapping, and service setup.

14. Suggested diagram framing

- A good architecture diagram should center on the Django application, then branch into:

- UI/API ingress

- orchestration/runtime

- database persistence

- filesystem/AutoPkg integration

- external services for notifications, GitHub recipe index, and network shares

- real-time log/event streaming back to browsers

- The main distinction is between “control plane” logic in Django and “execution plane” logic in the stage pipeline and OS-level AutoPkg interactions.