How mishrashardendu22/github-backup-automation-system works
1. This is a monorepo with four distinct runtime surfaces: a root Go backup worker, a Go/PostgreSQL backend API, a Next.js dashboard, and a Python agentic analysis layer. The clean architectural boundary is: backup execution, observability API, user interface, and AI-assisted analysis.
2. The root-level Go application is the primary backup engine. Its job is to traverse GitHub accounts, detect repositories, deduplicate them, verify remote HEAD hashes, clone only changed repos, archive them, and push the resulting artifacts into a centralized backup Git repository.
3. That worker keeps its own local state in SQLite. The root database and model packages indicate persistence for repository hashes, failure logs, schema management, and configuration metadata, so the worker can do incremental backups and avoid reprocessing unchanged repositories.
4. The worker’s internal flow is organized around helper and service layers: Git/GitHub/command helpers perform the actual external interactions, backup and process services orchestrate the phases, and a controller layer appears to coordinate repository-level actions. This is the execution boundary where GitHub, git, and tar are directly used.
5. The backend service under backend/ is a separate Go server focused on observability and operational control. It exposes REST endpoints for analytics, backups, dashboard stats, metrics, and repositories, and also includes a fix path for remediation workflows.
6. The backend has a real-time channel via WebSockets, which is important for live log streaming and status updates. Its database layer is PostgreSQL-backed, with explicit DB files for schema and run-status tracking, so it serves as the system of record for execution history rather than the worker’s local state.
7. The backend also has a middleware boundary for security, which suggests authenticated access to operational data and control endpoints. In architecture diagrams, this should sit between the frontend/agent clients and the handler layer.
8. The frontend is a Next.js application using the app router. It provides separate user-facing areas for AI, analytics, backups, and live monitoring, with page-level routes and a shared layout/sidebar shell.
9. The frontend is structured as a thin client over the backend. Service modules and hooks encapsulate API access, session/auth handling, analytics and backup retrieval, stats loading, and live/streaming behavior. The live page likely consumes the backend WebSocket stream, while charts and dashboards render persisted analytics from the REST API.
10. The agentic-observatory/ package is a Python-based AI layer built around OpenRouter. It contains prompt/state management, tool modules for analytics, backups, logs, repositories, email, and data persistence, plus report templates for executive summaries and failure investigations. Architecturally, it is an analysis and automation sidecar that consumes operational data and produces AI-driven insights or actions.
11. The main external dependencies that shape the system are GitHub, a remote backup Git repository, PostgreSQL, SQLite, OpenRouter, and optional email delivery. The data flow is: GitHub repositories in, worker produces archives and local state, backend stores run history and streams logs, frontend visualizes and controls it, and the agent layer analyzes the same operational data.
12. For an architecture diagram, the most important boundaries are: source control and archive storage on the left, the worker as the execution engine in the middle, backend observability services as the system-of-record layer, the web dashboard as the presentation layer, and the AI observability service as an auxiliary intelligence layer connected to the same operational data.
Main entry points: analytics.py Agent tools · state.py State and prompts · analytics.handler.go API handlers · process.service.go Process flow
Components 26
Execution
Root Go backup worker and local state
- Backup worker · Entry point for the root backup engine
main.go - Backup flow · Coordinates backup phases and repository processing
service/backup.service.go - Process flow · Drives incremental repo traversal and execution steps
service/process.service.go - Git helpers · Git, GitHub, command, and repo helper boundary
service/helper/git.helper.go - Repo controller · Repository-level control surface for worker actions
controller/repo.controller.go - SQLite state · Local database, hashes, schema, and failure logs
database/sqlite.database.go - Config model · Worker configuration and URL metadata models
model/config.model.go
Observability
Go backend API and real-time status
- Backend API · Entry point for observability and control services
backend/main.go - Routes · REST route wiring for backend endpoints
backend/routes/app.routes.go - API handlers · Analytics, backups, dashboard, fix, metrics, and repo endpoints
backend/handlers/analytics.handler.go - WebSocket hub · Live log streaming and status updates
backend/websocket/hub.websocekt.go - Postgres DB · System-of-record persistence for runs and schemas
backend/db/postgres.db.go - Security middleware · Access control for backend operational endpoints
backend/middleware/security.middleware.go
Presentation
Next.js dashboard and UI shell
- Dashboard app · App-router frontend shell and page structure
frontend/src/app/layout.tsx - UI components · Shared layout, charts, live stream, and dashboard panels
frontend/src/components/layout/AppLayout.tsx - Client hooks · Session, auth, stats, backups, analytics, and streaming hooks
frontend/src/hooks/useStats.ts - API services · Thin client wrappers over backend REST and streaming APIs
frontend/src/services/analytics.service.ts
Intelligence
Python analysis and automation sidecar
- AI layer · Entry point for analysis and automation sidecar
agentic-observatory/main.py - Agent tools · Analytics, backup, log, repository, and email tool modules
agentic-observatory/data/tools/analytics.py - State and prompts · Prompt, state, configuration, and persistence layer
agentic-observatory/agent/state.py - Reports · Executive summary and failure investigation report templates
agentic-observatory/reports/templates/executive_summary.html
External systems
GitHub, storage, databases, and AI service
- GitHub · Primary repository source scanned by the worker
- Backup repo · Centralized Git repository for archived backup artifacts
- PostgreSQL · Backend persistent store for operational history
- OpenRouter · External AI provider used by the agent layer
- Email · Optional outbound delivery channel for reports
Connections 25
- Backup worker → Backup flow: starts · Worker entry invokes backup orchestration
- Backup flow → Process flow: coordinates · Backup flow delegates incremental processing
- Process flow → Repo controller: drives · Repository-level control and sequencing
- Repo controller → Git helpers: uses · Execution boundary for Git/GitHub/command interactions
- Process flow → SQLite state: reads/writes · Persists hashes, failures, schema, and config state
- Backup worker → GitHub: scans · Discovers and verifies upstream repositories
- Git helpers → Backup repo: pushes · Publishes archived backup artifacts
- Backend API → Routes: serves · Bootstraps API routing
- Routes → API handlers: dispatches · Maps REST paths to operational handlers
- API handlers → Postgres DB: persists · Stores run history and dashboard data
- API handlers → WebSocket hub: streams · Publishes live logs and status updates
- Security middleware → API handlers: guards · Security boundary in front of handlers
- Postgres DB → PostgreSQL: backed by · Backend database layer uses PostgreSQL
- Dashboard app → UI components: composes · App router shell renders shared UI
- UI components → Client hooks: uses · Views consume client-side data hooks
- Client hooks → API services: calls · Hooks rely on service modules for API access
- API services → Backend API: requests · Frontend talks to backend REST endpoints
- Client hooks → WebSocket hub: subscribes · Live page consumes WebSocket stream
- AI layer → State and prompts: orchestrates · Agent runtime manages prompts, state, and persistence
- State and prompts → Agent tools: invokes · Tools provide access to analytics, logs, repos, and backups
- Agent tools → Postgres DB: consumes data · Reads operational data from system-of-record APIs and stores
- Agent tools → PostgreSQL: queries · Consumes persisted operational history
- State and prompts → OpenRouter: uses · LLM provider for agent reasoning
- Agent tools → Email: sends · Optional report delivery and alerts
- State and prompts → Reports: renders · Produces executive and investigation reports