# vargezai/ms_capstone architecture

> Architecture diagram of the GitHub repository vargezai/ms_capstone, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vargezai/ms_capstone
- Repository: https://github.com/vargezai/ms_capstone

## Overview

1. This repository is an offline, batch-style research pipeline rather than a service. The core architectural boundary is file-based: raw public datasets are transformed into processed CSVs, then consumed by separate analysis modules that each emit reproducible outputs under `outputs/`.

2. The ingestion layer lives in `src/data_loader.py`. It assembles a state-level monthly/annual panel from public government sources named in the README: EIA, EPA eGRID, NOAA, BEA, and DSIRE. Its main contract is the master dataset in `data/processed/FINAL_MASTER_DATASET_2001_2026.csv`, with a compact variant and summary statistics also written to `data/processed/`.

3. Data quality handling is split into a dedicated preprocessing stage in `src/outlier_treatment.py`. That stage takes the master dataset and produces `FINAL_MASTER_DATASET_CLEAN.csv`, plus an outlier log in `outputs/EDA/`. This is an important boundary because downstream analyses assume the cleaned dataset, not the raw master panel.

4. `src/eda.py` is a diagnostic subsystem, not part of the modeling path. It generates exploratory summaries and correlation/outlier artifacts into `outputs/EDA/`, giving a visibility layer over the dataset before the hypothesis-specific models run.

5. `src/feature_engineering.py` is the feature-construction boundary for predictive models. It derives the engineered feature table `FINAL_MASTER_DATASET_FEATURES.csv`, which is used primarily by BH3 and BH4. This module separates analytical feature preparation from inferential modeling.

6. BH1, implemented in `src/panel_models.py`, is the causal panel-econometrics branch. It runs TWFE and IV models, sensitivity checks, and dynamic panel robustness via Arellano-Bond GMM. Its outputs are tabular results in `outputs/BH1/`, intended for coefficient tables and robustness comparison rather than model deployment.

7. BH2, in `src/did_causal_forest.py`, is the staggered-treatment causal inference branch. It combines event-study DiD with causal forest estimation to produce ATT/CATE-style evidence and heterogeneity by state. The main artifacts are event-study and subgroup tables in `outputs/BH2/`.

8. BH3, in `src/lstm_forecaster.py`, is the time-series forecasting branch. It trains multi-horizon LSTM models for 1-, 3-, and 6-month forecasting, and includes Keras Tuner hyperparameter optimization. Artifacts include saved `.keras` models, tuner state, metrics, and prediction CSVs in `outputs/BH3/`. This branch is the most model-artifact-heavy and depends on TensorFlow/Keras and `keras-tuner`.

9. BH4, in `src/xgboost_classifier.py`, is the supervised classification/interpretability branch. It uses XGBoost plus SHAP to predict “top-decarbonizer” status and explain feature influence, including interaction analysis. Outputs include cross-validation metrics, predictions, feature importance tables, SHAP importance, and SHAP interaction matrices in `outputs/BH4/`.

10. BH5, in `src/regional_analysis.py`, is a subgroup econometrics branch that re-estimates RPS effects by Census region. It extends the causal panel logic to test heterogeneity across regions and writes region-specific interaction/results tables to `outputs/BH5/`.

11. The repository’s reporting layer is separate from the computation layer. `docs/final_report.md`, `synopsis/`, and `docs/qm640_compliance_checklist.md` consolidate results for submission, while the `outputs/` directory stores the machine-generated evidence that feeds those reports.

12. In architectural terms, the project is organized around a single data lineage: raw files in `data/raw/` → integrated master panel → cleaned panel → engineered feature table → five independent analysis branches → persisted outputs and report documents. The main design pattern is reproducible, stage-separated analysis with minimal coupling between hypotheses.

Main entry points: [`docs`](https://github.com/vargezai/ms_capstone/tree/main/docs) (Reports), [`data/processed/FINAL_MASTER_DATASET_CLEAN.csv`](https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_CLEAN.csv) (Cleaned panel), [`src/data_loader.py`](https://github.com/vargezai/ms_capstone/blob/main/src/data_loader.py) (Ingest panel), [`data/processed/FINAL_MASTER_DATASET_2001_2026.csv`](https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_2001_2026.csv) (Master panel)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_data["Data pipeline"]
  node_raw_data[("Raw data<br/>dataset")]
  node_data_loader["Ingest panel<br/>etl<br/>[data_loader.py]"]
  node_master_dataset[("Master panel<br/>dataset")]
  node_compact_dataset[("Compact panel<br/>dataset")]
  node_summary_stats["Summary stats<br/>dataset"]
  node_outlier_treatment["Clean panel<br/>preprocess"]
  node_clean_dataset[("Cleaned panel<br/>dataset")]
  node_eda["EDA checks<br/>diagnostics<br/>[eda.py]"]
  node_eda_outputs["EDA outputs<br/>artifacts"]
  node_feature_engineering["Feature build<br/>features"]
  node_feature_table[("Feature table<br/>dataset")]
end

subgraph group_group_analysis["Analysis branches"]
  node_panel_models["BH1 panel<br/>econometrics<br/>[panel_models.py]"]
  node_did_causal_forest["BH2 causal forest<br/>causal inference"]
  node_lstm_forecaster["BH3 LSTM<br/>forecasting<br/>[lstm_forecaster.py]"]
  node_xgboost_classifier["BH4 XGBoost<br/>classification"]
  node_regional_analysis["BH5 regions<br/>subgroup econometrics"]
  node_bh1_outputs["BH1 outputs<br/>results"]
  node_bh2_outputs["BH2 outputs<br/>results"]
  node_bh3_outputs["BH3 outputs<br/>models"]
  node_bh4_outputs["BH4 outputs<br/>results"]
  node_bh5_outputs["BH5 outputs<br/>results"]
end

subgraph group_group_reporting["Reporting"]
  node_docs["Reports<br/>reporting"]
  node_synopsis["Synopsis<br/>reporting"]
end

node_raw_data -->|"source files"| node_data_loader
node_data_loader -->|"writes"| node_master_dataset
node_data_loader -->|"writes"| node_compact_dataset
node_data_loader -->|"writes"| node_summary_stats
node_master_dataset -->|"input"| node_outlier_treatment
node_outlier_treatment -->|"writes"| node_clean_dataset
node_outlier_treatment -->|"logs"| node_eda_outputs
node_master_dataset -->|"input"| node_eda
node_eda -->|"writes"| node_eda_outputs
node_clean_dataset -->|"input"| node_feature_engineering
node_feature_engineering -->|"writes"| node_feature_table
node_clean_dataset -->|"input"| node_panel_models
node_clean_dataset -->|"input"| node_did_causal_forest
node_feature_table -->|"input"| node_lstm_forecaster
node_feature_table -->|"input"| node_xgboost_classifier
node_clean_dataset -->|"input"| node_regional_analysis
node_panel_models -->|"writes"| node_bh1_outputs
node_did_causal_forest -->|"writes"| node_bh2_outputs
node_lstm_forecaster -->|"writes"| node_bh3_outputs
node_xgboost_classifier -->|"writes"| node_bh4_outputs
node_regional_analysis -->|"writes"| node_bh5_outputs
node_bh1_outputs -->|"feeds"| node_docs
node_bh2_outputs -->|"feeds"| node_docs
node_bh3_outputs -->|"feeds"| node_docs
node_bh4_outputs -->|"feeds"| node_docs
node_bh5_outputs -->|"feeds"| node_docs
node_docs -->|"summarizes"| node_synopsis

click node_raw_data "https://github.com/vargezai/ms_capstone/tree/main/data/raw"
click node_data_loader "https://github.com/vargezai/ms_capstone/blob/main/src/data_loader.py"
click node_master_dataset "https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_2001_2026.csv"
click node_compact_dataset "https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_COMPACT_DATASET_2001_2026.csv"
click node_summary_stats "https://github.com/vargezai/ms_capstone/blob/main/data/processed/SUMMARY_STATISTICS.csv"
click node_outlier_treatment "https://github.com/vargezai/ms_capstone/blob/main/src/outlier_treatment.py"
click node_clean_dataset "https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_CLEAN.csv"
click node_eda "https://github.com/vargezai/ms_capstone/blob/main/src/eda.py"
click node_eda_outputs "https://github.com/vargezai/ms_capstone/tree/main/outputs/EDA"
click node_feature_engineering "https://github.com/vargezai/ms_capstone/blob/main/src/feature_engineering.py"
click node_feature_table "https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_FEATURES.csv"
click node_panel_models "https://github.com/vargezai/ms_capstone/blob/main/src/panel_models.py"
click node_did_causal_forest "https://github.com/vargezai/ms_capstone/blob/main/src/did_causal_forest.py"
click node_lstm_forecaster "https://github.com/vargezai/ms_capstone/blob/main/src/lstm_forecaster.py"
click node_xgboost_classifier "https://github.com/vargezai/ms_capstone/blob/main/src/xgboost_classifier.py"
click node_regional_analysis "https://github.com/vargezai/ms_capstone/blob/main/src/regional_analysis.py"
click node_bh1_outputs "https://github.com/vargezai/ms_capstone/tree/main/outputs/BH1"
click node_bh2_outputs "https://github.com/vargezai/ms_capstone/tree/main/outputs/BH2"
click node_bh3_outputs "https://github.com/vargezai/ms_capstone/tree/main/outputs/BH3"
click node_bh4_outputs "https://github.com/vargezai/ms_capstone/tree/main/outputs/BH4"
click node_bh5_outputs "https://github.com/vargezai/ms_capstone/tree/main/outputs/BH5"
click node_docs "https://github.com/vargezai/ms_capstone/tree/main/docs"
click node_synopsis "https://github.com/vargezai/ms_capstone/tree/main/synopsis"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_data,node_data_loader,node_master_dataset,node_compact_dataset,node_summary_stats,node_outlier_treatment,node_clean_dataset,node_eda,node_eda_outputs,node_feature_engineering,node_feature_table toneBlue
class node_panel_models,node_did_causal_forest,node_lstm_forecaster,node_xgboost_classifier,node_regional_analysis,node_bh1_outputs,node_bh2_outputs,node_bh3_outputs,node_bh4_outputs,node_bh5_outputs toneAmber
class node_docs,node_synopsis toneMint
```

## Components

### Data pipeline

Raw inputs transformed into master, cleaned, and feature tables.

- [Raw data](https://github.com/vargezai/ms_capstone/tree/main/data/raw): dataset · Public source files and incoming raw datasets. · `data/raw`
- [Ingest panel](https://github.com/vargezai/ms_capstone/blob/main/src/data_loader.py): etl · Builds the state-level monthly/annual master panel from public sources. · `src/data_loader.py`
- [Master panel](https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_2001_2026.csv): dataset · Integrated master dataset used as the central analytic contract. · `data/processed/FINAL_MASTER_DATASET_2001_2026.csv`
- [Compact panel](https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_COMPACT_DATASET_2001_2026.csv): dataset · Smaller processed variant of the master panel. · `data/processed/FINAL_COMPACT_DATASET_2001_2026.csv`
- [Summary stats](https://github.com/vargezai/ms_capstone/blob/main/data/processed/SUMMARY_STATISTICS.csv): dataset · Processed summary statistics for the master dataset. · `data/processed/SUMMARY_STATISTICS.csv`
- [Clean panel](https://github.com/vargezai/ms_capstone/blob/main/src/outlier_treatment.py): preprocess · Cleans the master dataset and logs outliers for downstream use. · `src/outlier_treatment.py`
- [Cleaned panel](https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_CLEAN.csv): dataset · Downstream-ready dataset after outlier treatment. · `data/processed/FINAL_MASTER_DATASET_CLEAN.csv`
- [EDA checks](https://github.com/vargezai/ms_capstone/blob/main/src/eda.py): diagnostics · Generates exploratory summaries and correlation/outlier artifacts. · `src/eda.py`
- [EDA outputs](https://github.com/vargezai/ms_capstone/tree/main/outputs/EDA): artifacts · Exploratory tables and outlier logs. · `outputs/EDA`
- [Feature build](https://github.com/vargezai/ms_capstone/blob/main/src/feature_engineering.py): features · Derives engineered predictors for forecasting and classification. · `src/feature_engineering.py`
- [Feature table](https://github.com/vargezai/ms_capstone/blob/main/data/processed/FINAL_MASTER_DATASET_FEATURES.csv): dataset · Engineered feature dataset used mainly by BH3 and BH4. · `data/processed/FINAL_MASTER_DATASET_FEATURES.csv`

### Analysis branches

Independent hypothesis and model branches consuming curated datasets.

- [BH1 panel](https://github.com/vargezai/ms_capstone/blob/main/src/panel_models.py): econometrics · TWFE, IV, sensitivity, and dynamic panel robustness analysis. · `src/panel_models.py`
- [BH2 causal forest](https://github.com/vargezai/ms_capstone/blob/main/src/did_causal_forest.py): causal inference · Event-study DiD and causal forest heterogeneity analysis. · `src/did_causal_forest.py`
- [BH3 LSTM](https://github.com/vargezai/ms_capstone/blob/main/src/lstm_forecaster.py): forecasting · Multi-horizon LSTM forecasting with Keras Tuner optimization. · `src/lstm_forecaster.py`
- [BH4 XGBoost](https://github.com/vargezai/ms_capstone/blob/main/src/xgboost_classifier.py): classification · XGBoost classification with SHAP interpretation and interactions. · `src/xgboost_classifier.py`
- [BH5 regions](https://github.com/vargezai/ms_capstone/blob/main/src/regional_analysis.py): subgroup econometrics · Region-level re-estimation of policy effects and heterogeneity. · `src/regional_analysis.py`
- [BH1 outputs](https://github.com/vargezai/ms_capstone/tree/main/outputs/BH1): results · Panel econometrics tables and robustness outputs. · `outputs/BH1`
- [BH2 outputs](https://github.com/vargezai/ms_capstone/tree/main/outputs/BH2): results · Event-study and subgroup causal inference tables. · `outputs/BH2`
- [BH3 outputs](https://github.com/vargezai/ms_capstone/tree/main/outputs/BH3): models · Saved models, tuning state, metrics, and predictions. · `outputs/BH3`
- [BH4 outputs](https://github.com/vargezai/ms_capstone/tree/main/outputs/BH4): results · Cross-validation, predictions, feature importance, and SHAP artifacts. · `outputs/BH4`
- [BH5 outputs](https://github.com/vargezai/ms_capstone/tree/main/outputs/BH5): results · Regional interaction and results tables. · `outputs/BH5`

### Reporting

Narrative deliverables that summarize machine-generated evidence.

- [Reports](https://github.com/vargezai/ms_capstone/tree/main/docs): reporting · Narrative report and compliance documents assembled from outputs. · `docs`
- [Synopsis](https://github.com/vargezai/ms_capstone/tree/main/synopsis): reporting · Executive summary and synopsis documents. · `synopsis`

## Connections

- Raw data → Ingest panel: source files · Public raw datasets are ingested and harmonized.
- Ingest panel → Master panel: writes · Produces the master integrated panel.
- Ingest panel → Compact panel: writes · Produces a compact processed variant.
- Ingest panel → Summary stats: writes · Outputs summary statistics for the processed panel.
- Master panel → Clean panel: input · Cleaning stage consumes the master panel.
- Clean panel → Cleaned panel: writes · Produces the downstream-cleaned dataset.
- Clean panel → EDA outputs: logs · Outlier log is written into the EDA artifact space.
- Master panel → EDA checks: input · EDA inspects the master dataset.
- EDA checks → EDA outputs: writes · Generates exploratory summaries and correlation artifacts.
- Cleaned panel → Feature build: input · Feature construction starts from the cleaned panel.
- Feature build → Feature table: writes · Produces engineered features for modeling branches.
- Cleaned panel → BH1 panel: input · BH1 uses the cleaned panel for econometric estimation.
- Cleaned panel → BH2 causal forest: input · BH2 consumes the cleaned panel for causal inference.
- Feature table → BH3 LSTM: input · BH3 trains forecasting models on engineered features.
- Feature table → BH4 XGBoost: input · BH4 uses engineered features for classification and SHAP.
- Cleaned panel → BH5 regions: input · BH5 re-estimates effects by region from the cleaned panel.
- BH1 panel → BH1 outputs: writes · Exports tables and robustness results.
- BH2 causal forest → BH2 outputs: writes · Exports event-study and CATE/ATT evidence.
- BH3 LSTM → BH3 outputs: writes · Persists models, tuning artifacts, metrics, and predictions.
- BH4 XGBoost → BH4 outputs: writes · Exports performance, predictions, and SHAP artifacts.
- BH5 regions → BH5 outputs: writes · Exports regional interaction and results tables.
- BH1 outputs → Reports: feeds · BH1 evidence contributes to the report layer.
- BH2 outputs → Reports: feeds · BH2 evidence contributes to the report layer.
- BH3 outputs → Reports: feeds · BH3 evidence contributes to the report layer.
- BH4 outputs → Reports: feeds · BH4 evidence contributes to the report layer.
- BH5 outputs → Reports: feeds · BH5 evidence contributes to the report layer.
- Reports → Synopsis: summarizes · Narrative summary and submission materials consolidate results.