# mishrashardendu22/github-backup-automation-system architecture

> Architecture diagram of the GitHub repository mishrashardendu22/github-backup-automation-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mishrashardendu22/github-backup-automation-system
- Repository: https://github.com/mishrashardendu22/github-backup-automation-system

## Overview

1. This is a monorepo with four distinct runtime surfaces: a root Go backup worker, a Go/PostgreSQL backend API, a Next.js dashboard, and a Python agentic analysis layer. The clean architectural boundary is: backup execution, observability API, user interface, and AI-assisted analysis.

2. The root-level Go application is the primary backup engine. Its job is to traverse GitHub accounts, detect repositories, deduplicate them, verify remote HEAD hashes, clone only changed repos, archive them, and push the resulting artifacts into a centralized backup Git repository.

3. That worker keeps its own local state in SQLite. The root database and model packages indicate persistence for repository hashes, failure logs, schema management, and configuration metadata, so the worker can do incremental backups and avoid reprocessing unchanged repositories.

4. The worker’s internal flow is organized around helper and service layers: Git/GitHub/command helpers perform the actual external interactions, backup and process services orchestrate the phases, and a controller layer appears to coordinate repository-level actions. This is the execution boundary where GitHub, git, and tar are directly used.

5. The backend service under `backend/` is a separate Go server focused on observability and operational control. It exposes REST endpoints for analytics, backups, dashboard stats, metrics, and repositories, and also includes a fix path for remediation workflows.

6. The backend has a real-time channel via WebSockets, which is important for live log streaming and status updates. Its database layer is PostgreSQL-backed, with explicit DB files for schema and run-status tracking, so it serves as the system of record for execution history rather than the worker’s local state.

7. The backend also has a middleware boundary for security, which suggests authenticated access to operational data and control endpoints. In architecture diagrams, this should sit between the frontend/agent clients and the handler layer.

8. The frontend is a Next.js application using the app router. It provides separate user-facing areas for AI, analytics, backups, and live monitoring, with page-level routes and a shared layout/sidebar shell.

9. The frontend is structured as a thin client over the backend. Service modules and hooks encapsulate API access, session/auth handling, analytics and backup retrieval, stats loading, and live/streaming behavior. The live page likely consumes the backend WebSocket stream, while charts and dashboards render persisted analytics from the REST API.

10. The `agentic-observatory/` package is a Python-based AI layer built around OpenRouter. It contains prompt/state management, tool modules for analytics, backups, logs, repositories, email, and data persistence, plus report templates for executive summaries and failure investigations. Architecturally, it is an analysis and automation sidecar that consumes operational data and produces AI-driven insights or actions.

11. The main external dependencies that shape the system are GitHub, a remote backup Git repository, PostgreSQL, SQLite, OpenRouter, and optional email delivery. The data flow is: GitHub repositories in, worker produces archives and local state, backend stores run history and streams logs, frontend visualizes and controls it, and the agent layer analyzes the same operational data.

12. For an architecture diagram, the most important boundaries are: source control and archive storage on the left, the worker as the execution engine in the middle, backend observability services as the system-of-record layer, the web dashboard as the presentation layer, and the AI observability service as an auxiliary intelligence layer connected to the same operational data.

Main entry points: [`agentic-observatory/data/tools/analytics.py`](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/data/tools/analytics.py) (Agent tools), [`agentic-observatory/agent/state.py`](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/agent/state.py) (State and prompts), [`backend/handlers/analytics.handler.go`](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/handlers/analytics.handler.go) (API handlers), [`service/process.service.go`](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/service/process.service.go) (Process flow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_execution["Execution"]
  node_n_main["Backup worker<br/>Go worker<br/>[main.go]"]
  node_n_backup_service["Backup flow<br/>orchestration<br/>[backup.service.go]"]
  node_n_process_service["Process flow<br/>orchestration<br/>[process.service.go]"]
  node_n_helpers["Git helpers<br/>integration helpers<br/>[git.helper.go]"]
  node_n_controller["Repo controller<br/>coordination<br/>[repo.controller.go]"]
  node_n_sqlite[("SQLite state<br/>local persistence<br/>[sqlite.database.go]")]
  node_n_config_model["Config model<br/>settings model<br/>[config.model.go]"]
end

subgraph group_group_observability["Observability"]
  node_n_backend_main["Backend API<br/>Go server<br/>[main.go]"]
  node_n_routes["Routes<br/>HTTP routing<br/>[app.routes.go]"]
  node_n_handlers["API handlers<br/>REST handlers"]
  node_n_ws(("WebSocket hub<br/>realtime channel<br/>[hub.websocekt.go]"))
  node_n_backend_db[("Postgres DB<br/>server persistence<br/>[postgres.db.go]")]
  node_n_backend_mw["Security middleware<br/>auth boundary"]
end

subgraph group_group_presentation["Presentation"]
  node_n_frontend_app["Dashboard app<br/>Next.js app<br/>[layout.tsx]"]
  node_n_frontend_ui["UI components<br/>React views<br/>[AppLayout.tsx]"]
  node_n_frontend_hooks["Client hooks<br/>data hooks<br/>[useStats.ts]"]
  node_n_frontend_services["API services<br/>client services"]
end

subgraph group_group_intelligence["Intelligence"]
  node_n_agent_main["AI layer<br/>Python agent<br/>[main.py]"]
  node_n_agent_tools["Agent tools<br/>tooling<br/>[analytics.py]"]
  node_n_agent_state["State and prompts<br/>agent runtime<br/>[state.py]"]
  node_n_agent_reports["Reports<br/>templates"]
end

subgraph group_group_external["External systems"]
  node_n_github(("GitHub<br/>source control"))
  node_n_backup_git[("Backup repo<br/>artifact store")]
  node_n_postgres[("PostgreSQL<br/>database")]
  node_n_openrouter(("OpenRouter<br/>LLM service"))
  node_n_email(("Email<br/>notification"))
end

node_n_main -->|"starts"| node_n_backup_service
node_n_backup_service -->|"coordinates"| node_n_process_service
node_n_process_service -->|"drives"| node_n_controller
node_n_controller -->|"uses"| node_n_helpers
node_n_process_service -->|"reads/writes"| node_n_sqlite
node_n_main -->|"scans"| node_n_github
node_n_helpers -->|"pushes"| node_n_backup_git
node_n_backend_main -->|"serves"| node_n_routes
node_n_routes -->|"dispatches"| node_n_handlers
node_n_handlers -->|"persists"| node_n_backend_db
node_n_handlers -->|"streams"| node_n_ws
node_n_backend_mw -->|"guards"| node_n_handlers
node_n_backend_db -->|"backed by"| node_n_postgres
node_n_frontend_app -->|"composes"| node_n_frontend_ui
node_n_frontend_ui -->|"uses"| node_n_frontend_hooks
node_n_frontend_hooks -->|"calls"| node_n_frontend_services
node_n_frontend_services -->|"requests"| node_n_backend_main
node_n_frontend_hooks -->|"subscribes"| node_n_ws
node_n_agent_main -->|"orchestrates"| node_n_agent_state
node_n_agent_state -->|"invokes"| node_n_agent_tools
node_n_agent_tools -->|"consumes data"| node_n_backend_db
node_n_agent_tools -->|"queries"| node_n_postgres
node_n_agent_state -->|"uses"| node_n_openrouter
node_n_agent_tools -->|"sends"| node_n_email
node_n_agent_state -->|"renders"| node_n_agent_reports

click node_n_main "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/main.go"
click node_n_backup_service "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/service/backup.service.go"
click node_n_process_service "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/service/process.service.go"
click node_n_helpers "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/service/helper/git.helper.go"
click node_n_controller "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/controller/repo.controller.go"
click node_n_sqlite "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/database/sqlite.database.go"
click node_n_config_model "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/model/config.model.go"
click node_n_backend_main "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/main.go"
click node_n_routes "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/routes/app.routes.go"
click node_n_handlers "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/handlers/analytics.handler.go"
click node_n_ws "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/websocket/hub.websocekt.go"
click node_n_backend_db "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/db/postgres.db.go"
click node_n_backend_mw "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/middleware/security.middleware.go"
click node_n_frontend_app "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/app/layout.tsx"
click node_n_frontend_ui "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/components/layout/AppLayout.tsx"
click node_n_frontend_hooks "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/hooks/useStats.ts"
click node_n_frontend_services "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/services/analytics.service.ts"
click node_n_agent_main "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/main.py"
click node_n_agent_tools "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/data/tools/analytics.py"
click node_n_agent_state "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/agent/state.py"
click node_n_agent_reports "https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/reports/templates/executive_summary.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_main,node_n_backup_service,node_n_process_service,node_n_helpers,node_n_controller,node_n_sqlite,node_n_config_model toneBlue
class node_n_backend_main,node_n_routes,node_n_handlers,node_n_ws,node_n_backend_db,node_n_backend_mw toneAmber
class node_n_frontend_app,node_n_frontend_ui,node_n_frontend_hooks,node_n_frontend_services toneMint
class node_n_agent_main,node_n_agent_tools,node_n_agent_state,node_n_agent_reports toneRose
class node_n_github,node_n_backup_git,node_n_postgres,node_n_openrouter,node_n_email toneIndigo
```

## Components

### Execution

Root Go backup worker and local state

- [Backup worker](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/main.go): Go worker · Entry point for the root backup engine · `main.go`
- [Backup flow](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/service/backup.service.go): orchestration · Coordinates backup phases and repository processing · `service/backup.service.go`
- [Process flow](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/service/process.service.go): orchestration · Drives incremental repo traversal and execution steps · `service/process.service.go`
- [Git helpers](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/service/helper/git.helper.go): integration helpers · Git, GitHub, command, and repo helper boundary · `service/helper/git.helper.go`
- [Repo controller](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/controller/repo.controller.go): coordination · Repository-level control surface for worker actions · `controller/repo.controller.go`
- [SQLite state](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/database/sqlite.database.go): local persistence · Local database, hashes, schema, and failure logs · `database/sqlite.database.go`
- [Config model](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/model/config.model.go): settings model · Worker configuration and URL metadata models · `model/config.model.go`

### Observability

Go backend API and real-time status

- [Backend API](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/main.go): Go server · Entry point for observability and control services · `backend/main.go`
- [Routes](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/routes/app.routes.go): HTTP routing · REST route wiring for backend endpoints · `backend/routes/app.routes.go`
- [API handlers](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/handlers/analytics.handler.go): REST handlers · Analytics, backups, dashboard, fix, metrics, and repo endpoints · `backend/handlers/analytics.handler.go`
- [WebSocket hub](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/websocket/hub.websocekt.go): realtime channel · Live log streaming and status updates · `backend/websocket/hub.websocekt.go`
- [Postgres DB](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/db/postgres.db.go): server persistence · System-of-record persistence for runs and schemas · `backend/db/postgres.db.go`
- [Security middleware](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/backend/middleware/security.middleware.go): auth boundary · Access control for backend operational endpoints · `backend/middleware/security.middleware.go`

### Presentation

Next.js dashboard and UI shell

- [Dashboard app](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/app/layout.tsx): Next.js app · App-router frontend shell and page structure · `frontend/src/app/layout.tsx`
- [UI components](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/components/layout/AppLayout.tsx): React views · Shared layout, charts, live stream, and dashboard panels · `frontend/src/components/layout/AppLayout.tsx`
- [Client hooks](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/hooks/useStats.ts): data hooks · Session, auth, stats, backups, analytics, and streaming hooks · `frontend/src/hooks/useStats.ts`
- [API services](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/frontend/src/services/analytics.service.ts): client services · Thin client wrappers over backend REST and streaming APIs · `frontend/src/services/analytics.service.ts`

### Intelligence

Python analysis and automation sidecar

- [AI layer](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/main.py): Python agent · Entry point for analysis and automation sidecar · `agentic-observatory/main.py`
- [Agent tools](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/data/tools/analytics.py): tooling · Analytics, backup, log, repository, and email tool modules · `agentic-observatory/data/tools/analytics.py`
- [State and prompts](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/agent/state.py): agent runtime · Prompt, state, configuration, and persistence layer · `agentic-observatory/agent/state.py`
- [Reports](https://github.com/mishrashardendu22/github-backup-automation-system/blob/main/agentic-observatory/reports/templates/executive_summary.html): templates · Executive summary and failure investigation report templates · `agentic-observatory/reports/templates/executive_summary.html`

### External systems

GitHub, storage, databases, and AI service

- **GitHub**: source control · Primary repository source scanned by the worker
- **Backup repo**: artifact store · Centralized Git repository for archived backup artifacts
- **PostgreSQL**: database · Backend persistent store for operational history
- **OpenRouter**: LLM service · External AI provider used by the agent layer
- **Email**: notification · Optional outbound delivery channel for reports

## Connections

- Backup worker → Backup flow: starts · Worker entry invokes backup orchestration
- Backup flow → Process flow: coordinates · Backup flow delegates incremental processing
- Process flow → Repo controller: drives · Repository-level control and sequencing
- Repo controller → Git helpers: uses · Execution boundary for Git/GitHub/command interactions
- Process flow → SQLite state: reads/writes · Persists hashes, failures, schema, and config state
- Backup worker → GitHub: scans · Discovers and verifies upstream repositories
- Git helpers → Backup repo: pushes · Publishes archived backup artifacts
- Backend API → Routes: serves · Bootstraps API routing
- Routes → API handlers: dispatches · Maps REST paths to operational handlers
- API handlers → Postgres DB: persists · Stores run history and dashboard data
- API handlers → WebSocket hub: streams · Publishes live logs and status updates
- Security middleware → API handlers: guards · Security boundary in front of handlers
- Postgres DB → PostgreSQL: backed by · Backend database layer uses PostgreSQL
- Dashboard app → UI components: composes · App router shell renders shared UI
- UI components → Client hooks: uses · Views consume client-side data hooks
- Client hooks → API services: calls · Hooks rely on service modules for API access
- API services → Backend API: requests · Frontend talks to backend REST endpoints
- Client hooks → WebSocket hub: subscribes · Live page consumes WebSocket stream
- AI layer → State and prompts: orchestrates · Agent runtime manages prompts, state, and persistence
- State and prompts → Agent tools: invokes · Tools provide access to analytics, logs, repos, and backups
- Agent tools → Postgres DB: consumes data · Reads operational data from system-of-record APIs and stores
- Agent tools → PostgreSQL: queries · Consumes persisted operational history
- State and prompts → OpenRouter: uses · LLM provider for agent reasoning
- Agent tools → Email: sends · Optional report delivery and alerts
- State and prompts → Reports: renders · Produces executive and investigation reports
