How saumyaadani05/ael-ineternship works
1. Overall shape
This repository is an HR attrition analytics pipeline plus a FastAPI dashboard. The architecture is split into three main layers: offline data/ML processing in pipeline/, a lightweight API/UI layer in app/, and persisted artifacts in data/ and serialized model files under pipeline/.
2. Source data and storage boundaries
The starting point is data/datasets.csv. From that source, the project builds two SQLite databases: an OLTP-style HR source database (data/oltp_hr.db) and an OLAP warehouse (data/olap_warehouse.db). The warehouse is the main analytical boundary used by scoring, history, dashboard queries, and simulation refreshes.
3. ETL and history management
pipeline/seed_db.py creates and seeds the source database, while pipeline/etl.py moves data into the warehouse with SCD Type 2 history tracking. That means employee attributes can be versioned over time, which is important for history views, past risk comparison, and rerunning the pipeline after simulated HR changes.
4. Feature enrichment and synthetic business context
The original dataset is augmented with deterministic synthetic signals, most notably project assignment from pipeline/synthetic_attributes.py. This enrichment is used to make the similarity network and employee grouping more realistic, even though the raw source does not contain those fields.
5. Model training and scoring
pipeline/production_ml.py is the production scoring entry point. It trains and/or loads artifacts from the pipeline folder, then writes employee risk scores into the warehouse table flight_risk_scores. The modeling stack includes XGBoost survival logic and a Cox PH model, with prebuilt artifacts such as model_xgb.json, model_cph.pkl, scaler.pkl, loo_encoder.pkl, and baseline_survival.pkl.
6. Time-based attrition output
The core model output is not a single classification label; it is a set of time-horizon attrition probabilities for 1, 3, 6, and 12 months. That makes survival modeling the central ML abstraction in the repo. The dashboard and APIs consume these scored horizons rather than raw model logits.
7. What-if and simulation loop
pipeline/simulator_actions.py drives business action simulation. When a simulated HR action is applied, the system reruns ETL and ML refresh steps so the warehouse, risk scores, and derived views stay consistent. The API also exposes a dedicated what-if endpoint, so simulation is a first-class part of the architecture rather than an offline notebook workflow.
8. Organization graph and contagion analysis
The repo includes two graph-related subsystems. First, pipeline/graph_exporter.py supports exporting attrition contagion data for Neo4j, with optional local Neo4j connectivity and a graceful fallback if the graph database is unavailable. Second, pipeline/similarity_network.py precomputes department-specific employee similarity graphs and writes JSON artifacts in data/ for the UI.
9. Dashboard/API layer
app/server.py is the runtime entry point for the user-facing system. It serves the dashboard HTML pages in app/dashboard.html and app/org_network_similarity.html, and exposes endpoints for employee lists, dashboard stats, employee history, exposure history, similarity network data, what-if simulation, retraining, and manager rating updates. The server is the main integration point between warehouse data, model outputs, and graph views.
10. Frontend and visualization model
The UI is intentionally simple: static HTML pages rendered by FastAPI, with data fetched from JSON APIs. One view focuses on risk dashboards and employee drill-downs; the other focuses on department-level similarity networks and organization context. This keeps the presentation layer thin and pushes business logic into the pipeline and API layers.
11. Reporting and offline artifacts
pipeline/report.py, pipeline/html_reporter.py, and run_pipeline.py support offline reporting and model evaluation outputs. Generated reports live under reports/generated/ and reports/markdown/, so the repo includes both operational scoring and analyst-facing documentation artifacts.
12. Operational dependencies
The architecture depends on SQLite for local persistence, FastAPI for serving, and Neo4j as an optional graph backend for contagion exposure queries. Docker support exists via docker-compose.yml, but the system also runs in a local Python environment using the scripts in pipeline/ and app/. The key architectural idea is a local analytics stack where the warehouse, model artifacts, graph files, and web app are all derived from the same pipeline state.
Main entry points: server.py API server · etl.py ETL · run_pipeline.py Pipeline run · production_ml.py Modeling
Components 22
Pipeline
Offline data preparation, modeling, graph generation, and simulation.
- Seed DB · Creates and seeds the operational HR database from source data.
pipeline/seed_db.py - ETL · Loads and version-controls HR data into the analytical warehouse with SCD Type 2 history.
pipeline/etl.py - Synthetic attrs · Adds deterministic synthetic business context such as project assignment and grouping signals.
pipeline/synthetic_attributes.py - Similarity net · Builds department-level employee similarity graphs for UI and analysis.
pipeline/similarity_network.py - Graph export · Exports attrition contagion data and optionally integrates with Neo4j.
pipeline/graph_exporter.py - Modeling · Loads or trains survival models and artifacts used for risk scoring.
pipeline/production_ml.py - Simulator · Applies business actions and triggers refresh of data and scores.
pipeline/simulator_actions.py - Pipeline run · Top-level offline pipeline runner coordinating processing and outputs.
run_pipeline.py
App
FastAPI runtime and HTML dashboard views.
- API server · Main runtime entry point serving dashboard pages and JSON endpoints.
app/server.py - Dashboard UI · Static dashboard page for risk and employee drill-down views.
app/dashboard.html - Similarity UI · Static page for organization similarity and network exploration.
app/org_network_similarity.html - Org network UI · Organization network visualization page.
app/org_network.html
Data & Artifacts
Source data, derived JSON, and model artifacts.
- Source data · Raw employee dataset used as the starting point for the pipeline.
data/datasets.csv - Warehouse · Main analytical database used by scoring, history, dashboard queries, and simulations.
- HR source DB · Operational HR database seeded from the source dataset.
- Risk scores · Stores time-horizon attrition probabilities for employee scoring.
- Similarity JSON · Department-specific similarity network artifacts consumed by the UI.
data/similarity_network_Human_Resources.json - Org graph · Organization hierarchy and edge/node data used for graph views.
data/org_hierarchy.json - Model artifacts · Persisted survival model and preprocessing artifacts used by scoring.
pipeline/model_xgb.json
Reports
Offline evaluation and generated reporting outputs.
- Reports · Generated markdown and HTML reporting outputs.
- Reporting · Offline reporting and evaluation generation.
pipeline/report.py - HTML reporter · Formats offline reporting output into HTML-friendly artifacts.
pipeline/html_reporter.py
Connections 30
- Source data → Seed DB: seed · Raw dataset is loaded into the operational HR database.
- Seed DB → HR source DB: writes · Seeder populates the OLTP source database.
- HR source DB → ETL: extract · ETL pulls operational HR records for warehouse loading.
- Synthetic attrs → ETL: enrich · Synthetic attributes are incorporated during warehouse preparation.
- ETL → Warehouse: load · Curated and versioned HR data is written to the analytical warehouse.
- ETL → Org graph: derive · Hierarchy and org-structure artifacts are derived from warehouse-ready data.
- Similarity net → Similarity JSON: emit · Similarity network generation produces JSON artifacts for the UI.
- Graph export → Warehouse: query · Graph export consumes warehouse state to build contagion context.
- Graph export → Org graph: export · Exports organization graph artifacts when the backend is unavailable or optional.
- Modeling → Model artifacts: persist · Trained model and preprocessing artifacts are stored for reuse.
- Modeling → Risk scores: score · Produces time-horizon risk probabilities written into the warehouse.
- Simulator → ETL: refresh · Simulation can trigger ETL reruns after actions are applied.
- Simulator → Modeling: rescore · Simulation refreshes scoring after state changes.
- Pipeline run → Seed DB: orchestrate · Top-level runner coordinates database seeding.
- Pipeline run → ETL: orchestrate · Top-level runner coordinates warehouse ETL.
- Pipeline run → Modeling: orchestrate · Top-level runner coordinates model training and scoring.
- Pipeline run → Similarity net: orchestrate · Top-level runner coordinates similarity network generation.
- Pipeline run → Reporting: orchestrate · Top-level runner coordinates offline report generation.
- API server → Warehouse: serve · API reads dashboard and history data from the warehouse.
- API server → Risk scores: serve · API serves scored attrition probabilities to the dashboard.
- API server → Similarity JSON: serve · API exposes similarity network data to the network view.
- API server → Org graph: serve · API exposes organization graph data and hierarchy views.
- API server → Simulator: what-if · API exposes simulation actions through dedicated endpoints.
- API server → Modeling: retrain · API can trigger retraining or refresh operations.
- API server → Dashboard UI: render · Serves the main dashboard page.
- API server → Similarity UI: render · Serves the similarity network page.
- API server → Org network UI: render · Serves the org network page.
- Reporting → HTML reporter: format · Reporting pipeline formats generated outputs for HTML delivery.
- Reporting → Reports: write · Generated report outputs are stored under reports directories.
- HTML reporter → Reports: write · HTML-oriented report artifacts are emitted to report storage.