vargezai/ms_capstone

1. This repository is an offline, batch-style research pipeline rather than a service. The core architectural boundary is file-based: raw public datasets are transformed into processed CSVs, then consumed by separate analysis modules that each emit reproducible outputs under outputs/.

2. The ingestion layer lives in src/data_loader.py. It assembles a state-level monthly/annual panel from public government sources named in the README: EIA, EPA eGRID, NOAA, BEA, and DSIRE. Its main contract is the master dataset in data/processed/FINAL_MASTER_DATASET_2001_2026.csv, with a compact variant and summary statistics also written to data/processed/.

3. Data quality handling is split into a dedicated preprocessing stage in src/outlier_treatment.py. That stage takes the master dataset and produces FINAL_MASTER_DATASET_CLEAN.csv, plus an outlier log in outputs/EDA/. This is an important boundary because downstream analyses assume the cleaned dataset, not the raw master panel.

4. src/eda.py is a diagnostic subsystem, not part of the modeling path. It generates exploratory summaries and correlation/outlier artifacts into outputs/EDA/, giving a visibility layer over the dataset before the hypothesis-specific models run.

5. src/feature_engineering.py is the feature-construction boundary for predictive models. It derives the engineered feature table FINAL_MASTER_DATASET_FEATURES.csv, which is used primarily by BH3 and BH4. This module separates analytical feature preparation from inferential modeling.

6. BH1, implemented in src/panel_models.py, is the causal panel-econometrics branch. It runs TWFE and IV models, sensitivity checks, and dynamic panel robustness via Arellano-Bond GMM. Its outputs are tabular results in outputs/BH1/, intended for coefficient tables and robustness comparison rather than model deployment.

7. BH2, in src/did_causal_forest.py, is the staggered-treatment causal inference branch. It combines event-study DiD with causal forest estimation to produce ATT/CATE-style evidence and heterogeneity by state. The main artifacts are event-study and subgroup tables in outputs/BH2/.

8. BH3, in src/lstm_forecaster.py, is the time-series forecasting branch. It trains multi-horizon LSTM models for 1-, 3-, and 6-month forecasting, and includes Keras Tuner hyperparameter optimization. Artifacts include saved .keras models, tuner state, metrics, and prediction CSVs in outputs/BH3/. This branch is the most model-artifact-heavy and depends on TensorFlow/Keras and keras-tuner.

9. BH4, in src/xgboost_classifier.py, is the supervised classification/interpretability branch. It uses XGBoost plus SHAP to predict “top-decarbonizer” status and explain feature influence, including interaction analysis. Outputs include cross-validation metrics, predictions, feature importance tables, SHAP importance, and SHAP interaction matrices in outputs/BH4/.

10. BH5, in src/regional_analysis.py, is a subgroup econometrics branch that re-estimates RPS effects by Census region. It extends the causal panel logic to test heterogeneity across regions and writes region-specific interaction/results tables to outputs/BH5/.

11. The repository’s reporting layer is separate from the computation layer. docs/final_report.md, synopsis/, and docs/qm640_compliance_checklist.md consolidate results for submission, while the outputs/ directory stores the machine-generated evidence that feeds those reports.

12. In architectural terms, the project is organized around a single data lineage: raw files in data/raw/ → integrated master panel → cleaned panel → engineered feature table → five independent analysis branches → persisted outputs and report documents. The main design pattern is reproducible, stage-separated analysis with minimal coupling between hypotheses.