# aravindksk7/atom architecture

> Architecture diagram of the GitHub repository aravindksk7/atom, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aravindksk7/atom
- Repository: https://github.com/aravindksk7/atom

## Overview

1. This repository is a single-process FastAPI application with a static Alpine.js front end. The browser talks to the API over HTTP/JSON, and the API also serves HTML, SVG badges, SSE progress streams, and generated report files.

2. The UI is intentionally thin. `frontend/index.html`, `frontend/app.js`, and `frontend/styles.css` implement a tabbed workspace for Config, Launch, Monitor, History, Adapters, Reports, and Compare. Most state and all business actions are delegated to `/api/*`.

3. The API layer is organized by domain routes in `api/routes/`: configs, jobs, runs, compare, lineage, audit, tokens, notifications, schedules, adapters, and health. `api/middleware/auth.py` enforces bearer-token auth on API routes, while `api/dependencies.py` and `api/schemas.py` provide request wiring and shared models.

4. The persistence boundary is handled through `etl_framework.repository` and the database engine in `etl_framework.db`. The app stores configs, jobs, runs, results, mismatch rows, tokens, notification hooks, schedules, lineage edges, and audit events in a SQLAlchemy-backed database, defaulting to SQLite but configurable via `ETL_DATABASE_URL`.

5. Run execution is centered in `api/services/run_executor.py` and the `etl_framework.runner` package. This is where job ordering, dependency resolution, retries, health checks, and progress/state tracking come together. The README indicates runs execute inside the API process, so there is no separate worker service.

6. The core comparison engine lives in `etl_framework.reconciliation`. It includes a reconciliation engine, a data-quality engine, chunking/normalization utilities, and selectable `pandas` or `polars` backends. This is the main computation boundary for row comparison, schema handling, hash prechecks, and DQ rule evaluation.

7. Job types and external adapters are modeled in the `etl_framework.sap_bo`, `etl_framework.automic`, and dbt-related services. The adapters allow the app to compare SAP BusinessObjects reports, look up Automic jobs/runs, and parse dbt `run_results.json` plus optional `manifest.json` into normal run results.

8. Reporting and observability are separate subsystems. `etl_framework.reporting` generates themed HTML reports and metrics, while `api/services/log_parser.py` and `api/services/artifact_service.py` expose run logs, metrics, and report views back to the UI. These outputs are written to `./reports` and `./logs`.

9. The compare workflows are first-class APIs rather than UI-only features. `api/routes/compare.py` and `api/services/compare_service.py` support BO report comparison, dual-environment run comparison, and file-based reconciliation comparison. The History tab also uses run comparison and baseline pinning to show deltas between stored runs.

10. Operational features are embedded around the core flow: `api/services/scheduler.py` wraps APScheduler for cron-driven runs, `api/services/notifier.py` sends webhook notifications, and `api/services/audit_service.py` records immutable mutation history. These side systems all write back into the same application database.

11. The runtime artifact model is simple and local-first. The app produces a SQLite database by default, plus log files, metrics JSON sidecars, and HTML reports on disk. When configured against live systems, the same API can switch from simulation mode to real SQL Server, SAP BO, and Automic connections without changing the frontend shape.

12. For architecture diagrams, the clearest boundaries are: browser/UI, FastAPI API/auth, execution and reconciliation core, repository/database, external adapters, and local artifacts. The arrows should show the UI driving domain routes, routes invoking services, services calling the core engine/repository/adapters, and the core persisting results and generated files.

Main entry points: [`api/main.py`](https://github.com/aravindksk7/atom/blob/master/api/main.py) (API App), [`etl_framework/repository/repository.py`](https://github.com/aravindksk7/atom/blob/master/etl_framework/repository/repository.py) (Repository), [`api/routes/compare.py`](https://github.com/aravindksk7/atom/blob/master/api/routes/compare.py) (Compare API), [`api/services/compare_service.py`](https://github.com/aravindksk7/atom/blob/master/api/services/compare_service.py) (Compare Svc)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Browser UI"]
  node_browser(("Browser<br/>client"))
  node_frontend_app["App UI<br/>static frontend<br/>[app.js]"]
  node_frontend_html["Shell<br/>static html<br/>[index.html]"]
  node_frontend_css["Styles<br/>static css<br/>[styles.css]"]
end

subgraph group_api["FastAPI API"]
  node_api_main["API App<br/>[main.py]"]
  node_auth_mw["Auth<br/>middleware<br/>[auth.py]"]
  node_routes_compare["Compare API<br/>route<br/>[compare.py]"]
  node_routes_runs["Runs API<br/>route<br/>[runs.py]"]
  node_routes_configs["Configs API<br/>route<br/>[configs.py]"]
  node_routes_adapters["Adapters API<br/>route<br/>[adapters.py]"]
end

subgraph group_core["Execution Core"]
  node_run_executor["Executor<br/>run orchestration<br/>[run_executor.py]"]
  node_compare_service["Compare Svc<br/>comparison service<br/>[compare_service.py]"]
  node_recon_engine["Reconcile<br/>engine<br/>[engine.py]"]
  node_dq_engine["DQ Engine<br/>dq rules<br/>[dq_engine.py]"]
  node_runner_state["Runner State<br/>runtime state<br/>[state.py]"]
  node_reporting["Reports<br/>reporting<br/>[generator.py]"]
end

subgraph group_data["Persistence"]
  node_repo[("Repository<br/>persistence layer<br/>[repository.py]")]
  node_db_engine[("DB Engine<br/>sql engine<br/>[engine.py]")]
end

subgraph group_adapters["Adapters"]
  node_sapbo_client["SAP BO<br/>adapter client<br/>[client.py]"]
  node_automic_client["Automic<br/>adapter client<br/>[client.py]"]
  node_dbt_parser["dbt Parser<br/>artifact parser"]
end

subgraph group_ops["Operations"]
  node_artifact_service["Artifacts<br/>artifact service"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_notifier["Notifier<br/>webhook delivery<br/>[notifier.py]"]
  node_local_artifacts["Local Files<br/>artifact store"]
end

node_external_db[("SQL DB<br/>external system")]
node_external_bo["SAP BO Server<br/>external system"]
node_external_automic["Automic Server<br/>external system"]
node_external_sources["Source Systems<br/>external systems"]

node_browser -->|"loads"| node_frontend_html
node_frontend_html -->|"bootstraps"| node_frontend_app
node_frontend_app -->|"calls /api"| node_api_main
node_api_main -->|"protects"| node_auth_mw
node_api_main -->|"mounts"| node_routes_compare
node_api_main -->|"mounts"| node_routes_runs
node_api_main -->|"mounts"| node_routes_configs
node_api_main -->|"mounts"| node_routes_adapters
node_routes_runs -->|"starts"| node_run_executor
node_routes_compare -->|"orchestrates"| node_compare_service
node_compare_service -->|"uses"| node_recon_engine
node_recon_engine -->|"validates"| node_dq_engine
node_run_executor -->|"tracks"| node_runner_state
node_run_executor -->|"persists"| node_repo
node_repo -->|"writes"| node_db_engine
node_db_engine -->|"connects"| node_external_db
node_compare_service -->|"reads BO"| node_sapbo_client
node_compare_service -->|"parses"| node_dbt_parser
node_sapbo_client -->|"connects"| node_external_bo
node_automic_client -->|"connects"| node_external_automic
node_routes_adapters -->|"exposes"| node_sapbo_client
node_routes_adapters -->|"exposes"| node_automic_client
node_routes_runs -->|"reads history"| node_repo
node_routes_configs -->|"stores config"| node_repo
node_routes_compare -->|"reads baselines"| node_repo
node_scheduler -->|"triggers"| node_run_executor
node_notifier -->|"records"| node_repo
node_artifact_service -->|"serves"| node_reporting
node_reporting -->|"writes"| node_local_artifacts
node_artifact_service -->|"reads"| node_local_artifacts
node_routes_compare -->|"shows output"| node_artifact_service
node_frontend_app -->|"views reports"| node_artifact_service

click node_frontend_app "https://github.com/aravindksk7/atom/blob/master/frontend/app.js"
click node_frontend_html "https://github.com/aravindksk7/atom/blob/master/frontend/index.html"
click node_frontend_css "https://github.com/aravindksk7/atom/blob/master/frontend/styles.css"
click node_api_main "https://github.com/aravindksk7/atom/blob/master/api/main.py"
click node_auth_mw "https://github.com/aravindksk7/atom/blob/master/api/middleware/auth.py"
click node_routes_compare "https://github.com/aravindksk7/atom/blob/master/api/routes/compare.py"
click node_routes_runs "https://github.com/aravindksk7/atom/blob/master/api/routes/runs.py"
click node_routes_configs "https://github.com/aravindksk7/atom/blob/master/api/routes/configs.py"
click node_routes_adapters "https://github.com/aravindksk7/atom/blob/master/api/routes/adapters.py"
click node_run_executor "https://github.com/aravindksk7/atom/blob/master/api/services/run_executor.py"
click node_compare_service "https://github.com/aravindksk7/atom/blob/master/api/services/compare_service.py"
click node_recon_engine "https://github.com/aravindksk7/atom/blob/master/etl_framework/reconciliation/engine.py"
click node_dq_engine "https://github.com/aravindksk7/atom/blob/master/etl_framework/reconciliation/dq_engine.py"
click node_repo "https://github.com/aravindksk7/atom/blob/master/etl_framework/repository/repository.py"
click node_db_engine "https://github.com/aravindksk7/atom/blob/master/etl_framework/db/engine.py"
click node_runner_state "https://github.com/aravindksk7/atom/blob/master/etl_framework/runner/state.py"
click node_reporting "https://github.com/aravindksk7/atom/blob/master/etl_framework/reporting/generator.py"
click node_artifact_service "https://github.com/aravindksk7/atom/blob/master/api/services/artifact_service.py"
click node_scheduler "https://github.com/aravindksk7/atom/blob/master/api/services/scheduler.py"
click node_notifier "https://github.com/aravindksk7/atom/blob/master/api/services/notifier.py"
click node_sapbo_client "https://github.com/aravindksk7/atom/blob/master/etl_framework/sap_bo/client.py"
click node_automic_client "https://github.com/aravindksk7/atom/blob/master/etl_framework/automic/client.py"
click node_dbt_parser "https://github.com/aravindksk7/atom/blob/master/api/services/dbt_artifact_parser.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser,node_frontend_app,node_frontend_html,node_frontend_css toneBlue
class node_api_main,node_auth_mw,node_routes_compare,node_routes_runs,node_routes_configs,node_routes_adapters toneAmber
class node_run_executor,node_compare_service,node_recon_engine,node_dq_engine,node_runner_state,node_reporting toneMint
class node_repo,node_db_engine toneRose
class node_sapbo_client,node_automic_client,node_dbt_parser toneIndigo
class node_artifact_service,node_scheduler,node_notifier,node_local_artifacts toneTeal
class node_external_db,node_external_bo,node_external_automic,node_external_sources toneNeutral
```

## Components

### Browser UI

Static Alpine.js frontend for tabbed workflow and results viewing.

- **Browser**: client · User entry point for the tabbed workspace.
- [App UI](https://github.com/aravindksk7/atom/blob/master/frontend/app.js): static frontend · Alpine.js workspace for config, launch, monitor, history, adapters, reports, and compare. · `frontend/app.js`
- [Shell](https://github.com/aravindksk7/atom/blob/master/frontend/index.html): static html · Top-level page shell and tab layout. · `frontend/index.html`
- [Styles](https://github.com/aravindksk7/atom/blob/master/frontend/styles.css): static css · UI presentation and theming. · `frontend/styles.css`

### FastAPI API

HTTP/JSON routes, auth, request wiring, and domain services.

- [API App](https://github.com/aravindksk7/atom/blob/master/api/main.py): FastAPI app · Application bootstrap and route assembly. · `api/main.py`
- [Auth](https://github.com/aravindksk7/atom/blob/master/api/middleware/auth.py): middleware · Bearer-token enforcement for API routes. · `api/middleware/auth.py`
- [Compare API](https://github.com/aravindksk7/atom/blob/master/api/routes/compare.py): route · Report, run, and file comparison endpoints. · `api/routes/compare.py`
- [Runs API](https://github.com/aravindksk7/atom/blob/master/api/routes/runs.py): route · Run launch, status, and history endpoints. · `api/routes/runs.py`
- [Configs API](https://github.com/aravindksk7/atom/blob/master/api/routes/configs.py): route · Configuration CRUD and wiring endpoints. · `api/routes/configs.py`
- [Adapters API](https://github.com/aravindksk7/atom/blob/master/api/routes/adapters.py): route · Adapter discovery and integration endpoints. · `api/routes/adapters.py`

### Execution Core

Run orchestration, reconciliation, comparison, and reporting engines.

- [Executor](https://github.com/aravindksk7/atom/blob/master/api/services/run_executor.py): run orchestration · Orders jobs, resolves dependencies, retries, and tracks progress in-process. · `api/services/run_executor.py`
- [Compare Svc](https://github.com/aravindksk7/atom/blob/master/api/services/compare_service.py): comparison service · Coordinates BO, run, and file-based comparisons. · `api/services/compare_service.py`
- [Reconcile](https://github.com/aravindksk7/atom/blob/master/etl_framework/reconciliation/engine.py): engine · Core row comparison, schema handling, and hash prechecks. · `etl_framework/reconciliation/engine.py`
- [DQ Engine](https://github.com/aravindksk7/atom/blob/master/etl_framework/reconciliation/dq_engine.py): dq rules · Data-quality rule evaluation and validation. · `etl_framework/reconciliation/dq_engine.py`
- [Runner State](https://github.com/aravindksk7/atom/blob/master/etl_framework/runner/state.py): runtime state · Run status and health tracking for in-process execution. · `etl_framework/runner/state.py`
- [Reports](https://github.com/aravindksk7/atom/blob/master/etl_framework/reporting/generator.py): reporting · HTML report and metrics generation. · `etl_framework/reporting/generator.py`

### Persistence

Repository and database boundary for configs, runs, results, lineage, and audit.

- [Repository](https://github.com/aravindksk7/atom/blob/master/etl_framework/repository/repository.py): persistence layer · Application repository for configs, runs, results, lineage, tokens, schedules, and audit. · `etl_framework/repository/repository.py`
- [DB Engine](https://github.com/aravindksk7/atom/blob/master/etl_framework/db/engine.py): sql engine · Database engine and SQLAlchemy-backed connectivity. · `etl_framework/db/engine.py`

### Adapters

External system integrations and artifact parsing.

- [SAP BO](https://github.com/aravindksk7/atom/blob/master/etl_framework/sap_bo/client.py): adapter client · BusinessObjects report and config access. · `etl_framework/sap_bo/client.py`
- [Automic](https://github.com/aravindksk7/atom/blob/master/etl_framework/automic/client.py): adapter client · Automic job and run lookups. · `etl_framework/automic/client.py`
- [dbt Parser](https://github.com/aravindksk7/atom/blob/master/api/services/dbt_artifact_parser.py): artifact parser · Parses run_results.json and optional manifest.json. · `api/services/dbt_artifact_parser.py`

### Operations

Scheduling, notifications, logs, and artifact views.

- [Artifacts](https://github.com/aravindksk7/atom/blob/master/api/services/artifact_service.py): artifact service · Serves run logs, metrics, reports, and artifact views. · `api/services/artifact_service.py`
- [Scheduler](https://github.com/aravindksk7/atom/blob/master/api/services/scheduler.py): apscheduler wrapper · Cron-driven run scheduling. · `api/services/scheduler.py`
- [Notifier](https://github.com/aravindksk7/atom/blob/master/api/services/notifier.py): webhook delivery · Notification delivery for run events. · `api/services/notifier.py`
- **Local Files**: artifact store · Reports, logs, and metrics sidecars written to disk.

### Other components

- **SQL DB**: external system · Primary configured database backend, often SQLite or SQL Server.
- **SAP BO Server**: external system · Live BusinessObjects source system.
- **Automic Server**: external system · Live Automic source system.
- **Source Systems**: external systems · Compared data sources reached by adapters and reconciliation flows.

## Connections

- Browser → Shell: loads · Browser opens the static shell.
- Shell → App UI: bootstraps · Page shell loads the Alpine.js application.
- App UI → API App: calls /api · UI actions are delegated to HTTP/JSON endpoints.
- API App → Auth: protects · API routes are wrapped with bearer-token auth.
- API App → Compare API: mounts · Registers compare endpoints.
- API App → Runs API: mounts · Registers run and history endpoints.
- API App → Configs API: mounts · Registers config endpoints.
- API App → Adapters API: mounts · Registers adapter endpoints.
- Runs API → Executor: starts · Launch requests invoke in-process execution.
- Compare API → Compare Svc: orchestrates · Compare routes delegate to comparison service.
- Compare Svc → Reconcile: uses · Comparison workflows call the reconciliation engine.
- Reconcile → DQ Engine: validates · DQ checks are part of reconciliation.
- Executor → Runner State: tracks · Execution updates live run state and health.
- Executor → Repository: persists · Run results and state are stored through the repository.
- Repository → DB Engine: writes · Repository is backed by the SQLAlchemy engine.
- DB Engine → SQL DB: connects · Database URL targets the configured backend.
- Compare Svc → SAP BO: reads BO · BO report comparison uses the SAP adapter.
- Compare Svc → dbt Parser: parses · File-based and dbt-based comparison inputs are parsed.
- SAP BO → SAP BO Server: connects · Live BusinessObjects connectivity.
- Automic → Automic Server: connects · Live Automic connectivity.
- Adapters API → SAP BO: exposes · Adapter APIs surface BO integration state.
- Adapters API → Automic: exposes · Adapter APIs surface Automic integration state.
- Runs API → Repository: reads history · Run history and stored state come from persistence.
- Configs API → Repository: stores config · Configurations are persisted through the repository.
- Compare API → Repository: reads baselines · Baseline and stored comparison data are retrieved from the repository.
- Scheduler → Executor: triggers · Scheduled jobs start the same execution path.
- Notifier → Repository: records · Notification settings and events are persisted.
- Artifacts → Reports: serves · Report generation and artifact views depend on reporting outputs.
- Reports → Local Files: writes · HTML reports and metrics are emitted to disk.
- Artifacts → Local Files: reads · Logs, metrics, and reports are read back for the UI.
- Compare API → Artifacts: shows output · Compare results can be surfaced as reports and artifacts.
- App UI → Artifacts: views reports · UI fetches logs, metrics, and generated report files.