aravindksk7/atom

1. This repository is a single-process FastAPI application with a static Alpine.js front end. The browser talks to the API over HTTP/JSON, and the API also serves HTML, SVG badges, SSE progress streams, and generated report files.

2. The UI is intentionally thin. frontend/index.html, frontend/app.js, and frontend/styles.css implement a tabbed workspace for Config, Launch, Monitor, History, Adapters, Reports, and Compare. Most state and all business actions are delegated to /api/*.

3. The API layer is organized by domain routes in api/routes/: configs, jobs, runs, compare, lineage, audit, tokens, notifications, schedules, adapters, and health. api/middleware/auth.py enforces bearer-token auth on API routes, while api/dependencies.py and api/schemas.py provide request wiring and shared models.

4. The persistence boundary is handled through etl_framework.repository and the database engine in etl_framework.db. The app stores configs, jobs, runs, results, mismatch rows, tokens, notification hooks, schedules, lineage edges, and audit events in a SQLAlchemy-backed database, defaulting to SQLite but configurable via ETL_DATABASE_URL.

5. Run execution is centered in api/services/run_executor.py and the etl_framework.runner package. This is where job ordering, dependency resolution, retries, health checks, and progress/state tracking come together. The README indicates runs execute inside the API process, so there is no separate worker service.

6. The core comparison engine lives in etl_framework.reconciliation. It includes a reconciliation engine, a data-quality engine, chunking/normalization utilities, and selectable pandas or polars backends. This is the main computation boundary for row comparison, schema handling, hash prechecks, and DQ rule evaluation.

7. Job types and external adapters are modeled in the etl_framework.sap_bo, etl_framework.automic, and dbt-related services. The adapters allow the app to compare SAP BusinessObjects reports, look up Automic jobs/runs, and parse dbt run_results.json plus optional manifest.json into normal run results.

8. Reporting and observability are separate subsystems. etl_framework.reporting generates themed HTML reports and metrics, while api/services/log_parser.py and api/services/artifact_service.py expose run logs, metrics, and report views back to the UI. These outputs are written to ./reports and ./logs.

9. The compare workflows are first-class APIs rather than UI-only features. api/routes/compare.py and api/services/compare_service.py support BO report comparison, dual-environment run comparison, and file-based reconciliation comparison. The History tab also uses run comparison and baseline pinning to show deltas between stored runs.

10. Operational features are embedded around the core flow: api/services/scheduler.py wraps APScheduler for cron-driven runs, api/services/notifier.py sends webhook notifications, and api/services/audit_service.py records immutable mutation history. These side systems all write back into the same application database.

11. The runtime artifact model is simple and local-first. The app produces a SQLite database by default, plus log files, metrics JSON sidecars, and HTML reports on disk. When configured against live systems, the same API can switch from simulation mode to real SQL Server, SAP BO, and Automic connections without changing the frontend shape.

12. For architecture diagrams, the clearest boundaries are: browser/UI, FastAPI API/auth, execution and reconciliation core, repository/database, external adapters, and local artifacts. The arrows should show the UI driving domain routes, routes invoking services, services calling the core engine/repository/adapters, and the core persisting results and generated files.