# saumyaadani05/ael-ineternship architecture

> Architecture diagram of the GitHub repository saumyaadani05/ael-ineternship, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saumyaadani05/ael-ineternship
- Repository: https://github.com/saumyaadani05/ael-ineternship

## Overview

1. **Overall shape**
This repository is an HR attrition analytics pipeline plus a FastAPI dashboard. The architecture is split into three main layers: offline data/ML processing in `pipeline/`, a lightweight API/UI layer in `app/`, and persisted artifacts in `data/` and serialized model files under `pipeline/`.

2. **Source data and storage boundaries**
The starting point is `data/datasets.csv`. From that source, the project builds two SQLite databases: an OLTP-style HR source database (`data/oltp_hr.db`) and an OLAP warehouse (`data/olap_warehouse.db`). The warehouse is the main analytical boundary used by scoring, history, dashboard queries, and simulation refreshes.

3. **ETL and history management**
`pipeline/seed_db.py` creates and seeds the source database, while `pipeline/etl.py` moves data into the warehouse with SCD Type 2 history tracking. That means employee attributes can be versioned over time, which is important for history views, past risk comparison, and rerunning the pipeline after simulated HR changes.

4. **Feature enrichment and synthetic business context**
The original dataset is augmented with deterministic synthetic signals, most notably project assignment from `pipeline/synthetic_attributes.py`. This enrichment is used to make the similarity network and employee grouping more realistic, even though the raw source does not contain those fields.

5. **Model training and scoring**
`pipeline/production_ml.py` is the production scoring entry point. It trains and/or loads artifacts from the pipeline folder, then writes employee risk scores into the warehouse table `flight_risk_scores`. The modeling stack includes XGBoost survival logic and a Cox PH model, with prebuilt artifacts such as `model_xgb.json`, `model_cph.pkl`, `scaler.pkl`, `loo_encoder.pkl`, and `baseline_survival.pkl`.

6. **Time-based attrition output**
The core model output is not a single classification label; it is a set of time-horizon attrition probabilities for 1, 3, 6, and 12 months. That makes survival modeling the central ML abstraction in the repo. The dashboard and APIs consume these scored horizons rather than raw model logits.

7. **What-if and simulation loop**
`pipeline/simulator_actions.py` drives business action simulation. When a simulated HR action is applied, the system reruns ETL and ML refresh steps so the warehouse, risk scores, and derived views stay consistent. The API also exposes a dedicated what-if endpoint, so simulation is a first-class part of the architecture rather than an offline notebook workflow.

8. **Organization graph and contagion analysis**
The repo includes two graph-related subsystems. First, `pipeline/graph_exporter.py` supports exporting attrition contagion data for Neo4j, with optional local Neo4j connectivity and a graceful fallback if the graph database is unavailable. Second, `pipeline/similarity_network.py` precomputes department-specific employee similarity graphs and writes JSON artifacts in `data/` for the UI.

9. **Dashboard/API layer**
`app/server.py` is the runtime entry point for the user-facing system. It serves the dashboard HTML pages in `app/dashboard.html` and `app/org_network_similarity.html`, and exposes endpoints for employee lists, dashboard stats, employee history, exposure history, similarity network data, what-if simulation, retraining, and manager rating updates. The server is the main integration point between warehouse data, model outputs, and graph views.

10. **Frontend and visualization model**
The UI is intentionally simple: static HTML pages rendered by FastAPI, with data fetched from JSON APIs. One view focuses on risk dashboards and employee drill-downs; the other focuses on department-level similarity networks and organization context. This keeps the presentation layer thin and pushes business logic into the pipeline and API layers.

11. **Reporting and offline artifacts**
`pipeline/report.py`, `pipeline/html_reporter.py`, and `run_pipeline.py` support offline reporting and model evaluation outputs. Generated reports live under `reports/generated/` and `reports/markdown/`, so the repo includes both operational scoring and analyst-facing documentation artifacts.

12. **Operational dependencies**
The architecture depends on SQLite for local persistence, FastAPI for serving, and Neo4j as an optional graph backend for contagion exposure queries. Docker support exists via `docker-compose.yml`, but the system also runs in a local Python environment using the scripts in `pipeline/` and `app/`. The key architectural idea is a local analytics stack where the warehouse, model artifacts, graph files, and web app are all derived from the same pipeline state.

Main entry points: [`app/server.py`](https://github.com/saumyaadani05/ael-ineternship/blob/main/app/server.py) (API server), [`pipeline/etl.py`](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/etl.py) (ETL), [`run_pipeline.py`](https://github.com/saumyaadani05/ael-ineternship/blob/main/run_pipeline.py) (Pipeline run), [`pipeline/production_ml.py`](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/production_ml.py) (Modeling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_pipeline["Pipeline"]
  node_n_seed_db["Seed DB<br/>sqlite seed<br/>[seed_db.py]"]
  node_n_etl["ETL<br/>[etl.py]"]
  node_n_synth["Synthetic attrs<br/>feature enrichment"]
  node_n_similarity["Similarity net<br/>graph builder"]
  node_n_graph_exporter["Graph export<br/>neo4j export<br/>[graph_exporter.py]"]
  node_n_model["Modeling<br/>survival ML<br/>[production_ml.py]"]
  node_n_simulator["Simulator<br/>what-if engine"]
  node_n_run_pipeline["Pipeline run<br/>orchestrator<br/>[run_pipeline.py]"]
end

subgraph group_g_app["App"]
  node_n_server["API server<br/>fastapi runtime<br/>[server.py]"]
  node_n_dashboard_html["Dashboard UI<br/>html view<br/>[dashboard.html]"]
  node_n_org_similarity_html["Similarity UI<br/>html view"]
  node_n_org_network_html["Org network UI<br/>html view<br/>[org_network.html]"]
end

subgraph group_g_data["Data & Artifacts"]
  node_n_data_source["Source data<br/>csv source<br/>[datasets.csv]"]
  node_n_warehouse[("Warehouse")]
  node_n_oltp[("HR source DB<br/>sqlite oltp")]
  node_n_risk_scores[("Risk scores<br/>warehouse table")]
  node_n_similarity_json["Similarity JSON<br/>graph artifact"]
  node_n_org_json["Org graph<br/>hierarchy artifact<br/>[org_hierarchy.json]"]
  node_n_models["Model artifacts<br/>serialized models<br/>[model_xgb.json]"]
end

subgraph group_g_reports["Reports"]
  node_n_reports["Reports<br/>generated output"]
  node_n_report_py["Reporting<br/>report generator<br/>[report.py]"]
  node_n_html_reporter["HTML reporter<br/>report formatter<br/>[html_reporter.py]"]
end

node_n_data_source -->|"seed"| node_n_seed_db
node_n_seed_db -->|"writes"| node_n_oltp
node_n_oltp -->|"extract"| node_n_etl
node_n_synth -->|"enrich"| node_n_etl
node_n_etl -->|"load"| node_n_warehouse
node_n_etl -->|"derive"| node_n_org_json
node_n_similarity -->|"emit"| node_n_similarity_json
node_n_graph_exporter -.->|"query"| node_n_warehouse
node_n_graph_exporter -.->|"export"| node_n_org_json
node_n_model -->|"persist"| node_n_models
node_n_model -->|"score"| node_n_risk_scores
node_n_simulator -->|"refresh"| node_n_etl
node_n_simulator -->|"rescore"| node_n_model
node_n_run_pipeline -->|"orchestrate"| node_n_seed_db
node_n_run_pipeline -->|"orchestrate"| node_n_etl
node_n_run_pipeline -->|"orchestrate"| node_n_model
node_n_run_pipeline -->|"orchestrate"| node_n_similarity
node_n_run_pipeline -->|"orchestrate"| node_n_report_py
node_n_server -->|"serve"| node_n_warehouse
node_n_server -->|"serve"| node_n_risk_scores
node_n_server -->|"serve"| node_n_similarity_json
node_n_server -->|"serve"| node_n_org_json
node_n_server -->|"what-if"| node_n_simulator
node_n_server -.->|"retrain"| node_n_model
node_n_server -->|"render"| node_n_dashboard_html
node_n_server -->|"render"| node_n_org_similarity_html
node_n_server -->|"render"| node_n_org_network_html
node_n_report_py -->|"format"| node_n_html_reporter
node_n_report_py -->|"write"| node_n_reports
node_n_html_reporter -->|"write"| node_n_reports

click node_n_data_source "https://github.com/saumyaadani05/ael-ineternship/blob/main/data/datasets.csv"
click node_n_seed_db "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/seed_db.py"
click node_n_etl "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/etl.py"
click node_n_synth "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/synthetic_attributes.py"
click node_n_similarity "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/similarity_network.py"
click node_n_graph_exporter "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/graph_exporter.py"
click node_n_model "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/production_ml.py"
click node_n_simulator "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/simulator_actions.py"
click node_n_run_pipeline "https://github.com/saumyaadani05/ael-ineternship/blob/main/run_pipeline.py"
click node_n_server "https://github.com/saumyaadani05/ael-ineternship/blob/main/app/server.py"
click node_n_dashboard_html "https://github.com/saumyaadani05/ael-ineternship/blob/main/app/dashboard.html"
click node_n_org_similarity_html "https://github.com/saumyaadani05/ael-ineternship/blob/main/app/org_network_similarity.html"
click node_n_org_network_html "https://github.com/saumyaadani05/ael-ineternship/blob/main/app/org_network.html"
click node_n_similarity_json "https://github.com/saumyaadani05/ael-ineternship/blob/main/data/similarity_network_Human_Resources.json"
click node_n_org_json "https://github.com/saumyaadani05/ael-ineternship/blob/main/data/org_hierarchy.json"
click node_n_models "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/model_xgb.json"
click node_n_report_py "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/report.py"
click node_n_html_reporter "https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/html_reporter.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_seed_db,node_n_etl,node_n_synth,node_n_similarity,node_n_graph_exporter,node_n_model,node_n_simulator,node_n_run_pipeline toneBlue
class node_n_server,node_n_dashboard_html,node_n_org_similarity_html,node_n_org_network_html toneAmber
class node_n_data_source,node_n_warehouse,node_n_oltp,node_n_risk_scores,node_n_similarity_json,node_n_org_json,node_n_models toneMint
class node_n_reports,node_n_report_py,node_n_html_reporter toneRose
```

## Components

### Pipeline

Offline data preparation, modeling, graph generation, and simulation.

- [Seed DB](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/seed_db.py): sqlite seed · Creates and seeds the operational HR database from source data. · `pipeline/seed_db.py`
- [ETL](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/etl.py): warehouse ETL · Loads and version-controls HR data into the analytical warehouse with SCD Type 2 history. · `pipeline/etl.py`
- [Synthetic attrs](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/synthetic_attributes.py): feature enrichment · Adds deterministic synthetic business context such as project assignment and grouping signals. · `pipeline/synthetic_attributes.py`
- [Similarity net](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/similarity_network.py): graph builder · Builds department-level employee similarity graphs for UI and analysis. · `pipeline/similarity_network.py`
- [Graph export](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/graph_exporter.py): neo4j export · Exports attrition contagion data and optionally integrates with Neo4j. · `pipeline/graph_exporter.py`
- [Modeling](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/production_ml.py): survival ML · Loads or trains survival models and artifacts used for risk scoring. · `pipeline/production_ml.py`
- [Simulator](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/simulator_actions.py): what-if engine · Applies business actions and triggers refresh of data and scores. · `pipeline/simulator_actions.py`
- [Pipeline run](https://github.com/saumyaadani05/ael-ineternship/blob/main/run_pipeline.py): orchestrator · Top-level offline pipeline runner coordinating processing and outputs. · `run_pipeline.py`

### App

FastAPI runtime and HTML dashboard views.

- [API server](https://github.com/saumyaadani05/ael-ineternship/blob/main/app/server.py): fastapi runtime · Main runtime entry point serving dashboard pages and JSON endpoints. · `app/server.py`
- [Dashboard UI](https://github.com/saumyaadani05/ael-ineternship/blob/main/app/dashboard.html): html view · Static dashboard page for risk and employee drill-down views. · `app/dashboard.html`
- [Similarity UI](https://github.com/saumyaadani05/ael-ineternship/blob/main/app/org_network_similarity.html): html view · Static page for organization similarity and network exploration. · `app/org_network_similarity.html`
- [Org network UI](https://github.com/saumyaadani05/ael-ineternship/blob/main/app/org_network.html): html view · Organization network visualization page. · `app/org_network.html`

### Data & Artifacts

Source data, derived JSON, and model artifacts.

- [Source data](https://github.com/saumyaadani05/ael-ineternship/blob/main/data/datasets.csv): csv source · Raw employee dataset used as the starting point for the pipeline. · `data/datasets.csv`
- **Warehouse**: sqlite warehouse · Main analytical database used by scoring, history, dashboard queries, and simulations.
- **HR source DB**: sqlite oltp · Operational HR database seeded from the source dataset.
- **Risk scores**: warehouse table · Stores time-horizon attrition probabilities for employee scoring.
- [Similarity JSON](https://github.com/saumyaadani05/ael-ineternship/blob/main/data/similarity_network_Human_Resources.json): graph artifact · Department-specific similarity network artifacts consumed by the UI. · `data/similarity_network_Human_Resources.json`
- [Org graph](https://github.com/saumyaadani05/ael-ineternship/blob/main/data/org_hierarchy.json): hierarchy artifact · Organization hierarchy and edge/node data used for graph views. · `data/org_hierarchy.json`
- [Model artifacts](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/model_xgb.json): serialized models · Persisted survival model and preprocessing artifacts used by scoring. · `pipeline/model_xgb.json`

### Reports

Offline evaluation and generated reporting outputs.

- **Reports**: generated output · Generated markdown and HTML reporting outputs.
- [Reporting](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/report.py): report generator · Offline reporting and evaluation generation. · `pipeline/report.py`
- [HTML reporter](https://github.com/saumyaadani05/ael-ineternship/blob/main/pipeline/html_reporter.py): report formatter · Formats offline reporting output into HTML-friendly artifacts. · `pipeline/html_reporter.py`

## Connections

- Source data → Seed DB: seed · Raw dataset is loaded into the operational HR database.
- Seed DB → HR source DB: writes · Seeder populates the OLTP source database.
- HR source DB → ETL: extract · ETL pulls operational HR records for warehouse loading.
- Synthetic attrs → ETL: enrich · Synthetic attributes are incorporated during warehouse preparation.
- ETL → Warehouse: load · Curated and versioned HR data is written to the analytical warehouse.
- ETL → Org graph: derive · Hierarchy and org-structure artifacts are derived from warehouse-ready data.
- Similarity net → Similarity JSON: emit · Similarity network generation produces JSON artifacts for the UI.
- Graph export → Warehouse: query · Graph export consumes warehouse state to build contagion context.
- Graph export → Org graph: export · Exports organization graph artifacts when the backend is unavailable or optional.
- Modeling → Model artifacts: persist · Trained model and preprocessing artifacts are stored for reuse.
- Modeling → Risk scores: score · Produces time-horizon risk probabilities written into the warehouse.
- Simulator → ETL: refresh · Simulation can trigger ETL reruns after actions are applied.
- Simulator → Modeling: rescore · Simulation refreshes scoring after state changes.
- Pipeline run → Seed DB: orchestrate · Top-level runner coordinates database seeding.
- Pipeline run → ETL: orchestrate · Top-level runner coordinates warehouse ETL.
- Pipeline run → Modeling: orchestrate · Top-level runner coordinates model training and scoring.
- Pipeline run → Similarity net: orchestrate · Top-level runner coordinates similarity network generation.
- Pipeline run → Reporting: orchestrate · Top-level runner coordinates offline report generation.
- API server → Warehouse: serve · API reads dashboard and history data from the warehouse.
- API server → Risk scores: serve · API serves scored attrition probabilities to the dashboard.
- API server → Similarity JSON: serve · API exposes similarity network data to the network view.
- API server → Org graph: serve · API exposes organization graph data and hierarchy views.
- API server → Simulator: what-if · API exposes simulation actions through dedicated endpoints.
- API server → Modeling: retrain · API can trigger retraining or refresh operations.
- API server → Dashboard UI: render · Serves the main dashboard page.
- API server → Similarity UI: render · Serves the similarity network page.
- API server → Org network UI: render · Serves the org network page.
- Reporting → HTML reporter: format · Reporting pipeline formats generated outputs for HTML delivery.
- Reporting → Reports: write · Generated report outputs are stored under reports directories.
- HTML reporter → Reports: write · HTML-oriented report artifacts are emitted to report storage.
