# anundathescientist/real-estate-ameshousing-analysis architecture

> Architecture diagram of the GitHub repository anundathescientist/real-estate-ameshousing-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anundathescientist/real-estate-ameshousing-analysis
- Repository: https://github.com/anundathescientist/real-estate-ameshousing-analysis

## Overview

1. **Project shape**
   - This repository is a **Python-based, notebook-driven data science pipeline** for the Ames Housing dataset.
   - The core architecture is sequential: **raw data → cleaning → feature engineering → model training/comparison → dashboard**.

2. **Data layer**
   - `00 Data/Raw data/AmesHousing.csv` is the original source dataset.
   - Derived datasets are persisted as intermediate artifacts:
     - `00 Data/Processed Data/AmesHousing_cleaned.csv`
     - `00 Data/Feature Matrix Data/X_features.csv`
     - `00 Data/Feature Matrix Data/y_target.csv`
     - `00 Data/Feature Matrix Data/ridge_predictions.csv`
     - `00 Data/Feature Matrix Data/scaler.pkl`
   - These files act as handoff boundaries between analysis stages instead of recomputing everything at runtime.

3. **Exploratory analysis stage**
   - `01 EDA/01_eda_ameshousing.ipynb` is the first analytical boundary.
   - Its role is to inspect distributions, missingness, outliers, and relationships before transformation.
   - The paired `Report - EDA` folder suggests this stage produces a documented findings artifact, not just code.

4. **Data cleaning stage**
   - `02 Data Cleaning/02_data_cleaning.ipynb` transforms raw housing records into a cleaned dataset.
   - The output is materialized in `00 Data/Processed Data/AmesHousing_cleaned.csv`.
   - This stage likely owns missing-value handling, inconsistent category fixes, and basic filtering.

5. **Feature engineering stage**
   - `03 Feature Engineering/03_feature_engineering.ipynb` converts cleaned data into model-ready inputs.
   - The outputs `X_features.csv` and `y_target.csv` indicate a clear split between **feature matrix** and **target variable**.
   - `scaler.pkl` shows that feature normalization/standardization is part of the pipeline and is persisted for reuse.

6. **Modeling stage**
   - `04 Regression/04_regression_modeling.ipynb` trains several regression models on the engineered features.
   - Trained model artifacts are stored under `04 Regression/Models/`:
     - `model_linear.pkl`
     - `model_ridge.pkl`
     - `model_lasso.pkl`
     - `model_gb.pkl`
   - This indicates a controlled comparison across linear and regularized/ensemble approaches.

7. **Model selection and outputs**
   - `04 Regression/Best Model/` holds the selected champion model and its outputs:
     - `best_model.pkl`
     - `predictions.csv`
   - `ridge_predictions.csv` in the feature data folder suggests ridge regression was also used for evaluation or downstream analysis.
   - The architecture separates **candidate models** from the **final chosen model**, which is important for diagramming decision flow.

8. **Model comparison stage**
   - `05 Model Comparison/05_model_comparison.ipynb` is the evaluation boundary where model metrics are compared.
   - This stage likely drives the selection of `best_model.pkl` and the final predictions artifact.
   - `Report/` here indicates the comparison results are also packaged as a narrative deliverable.

9. **Dashboard layer**
   - `06 Dashboard/app.py` is the only application runtime entry point in the repo.
   - It likely consumes the saved model/artifacts rather than retraining, which makes the dashboard a **read-only inference layer** over the modeling outputs.
   - For an architecture diagram, treat this as the presentation boundary connected to persisted model/data artifacts.

10. **Operational and environment boundaries**
   - `requirements.txt` defines the Python dependency set for notebooks, training, and dashboard execution.
   - `.devcontainer/devcontainer.json` shows the project is meant to run in a reproducible containerized dev environment.
   - `LICENSE` and the various `Report` folders indicate this repo is also packaged as a deliverable, not only a codebase.

11. **Overall data flow**
   - The main flow is:
     - **Raw CSV ingestion**
     - **Notebook-based cleaning**
     - **Feature generation and scaling**
     - **Regression training**
     - **Model comparison**
     - **Saved artifacts consumed by the dashboard**
   - The key architectural characteristic is **artifact persistence between stages**, with CSVs and pickle files serving as the integration points.

12. **What to draw as boundaries**
   - Draw separate subsystems for:
     - Data storage
     - EDA/cleaning notebooks
     - Feature engineering
     - Model training and selection
     - Dashboard/app runtime
   - The strongest dependency arrows should run from **raw data into notebooks**, and from **saved model/artifact files into the dashboard**.

Main entry points: [`03 Feature Engineering/03_feature_engineering.ipynb`](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/03%20Feature%20Engineering/03_feature_engineering.ipynb) (Feature Notebook), [`05 Model Comparison/05_model_comparison.ipynb`](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/05%20Model%20Comparison/05_model_comparison.ipynb) (Compare Notebook), [`04 Regression/04_regression_modeling.ipynb`](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/04%20Regression/04_regression_modeling.ipynb) (Regression Notebook), [`requirements.txt`](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/requirements.txt) (Requirements)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data Layer"]
  node_raw_csv[("Raw CSV<br/>dataset<br/>[AmesHousing.csv]")]
  node_cleaned_csv[("Cleaned Data<br/>dataset")]
  node_x_features[("Features<br/>feature matrix<br/>[X_features.csv]")]
  node_y_target[("Target<br/>[y_target.csv]")]
  node_scaler["Scaler<br/>preprocessor<br/>[scaler.pkl]"]
end

subgraph group_prep["Preparation"]
  node_eda_nb["EDA Notebook"]
  node_clean_nb["Cleaning Notebook"]
  node_fe_nb["Feature Notebook"]
end

subgraph group_modeling["Modeling"]
  node_reg_nb["Regression Notebook"]
  node_candidate_models["Candidate Models<br/>model artifacts"]
  node_best_model["Best Model<br/>selected model<br/>[best_model.pkl]"]
  node_predictions[("Predictions<br/>output table<br/>[predictions.csv]")]
  node_ridge_preds[("Ridge Predictions<br/>evaluation output")]
  node_compare_nb["Compare Notebook"]
end

subgraph group_presentation["Presentation"]
  node_eda_report["EDA Report"]
  node_clean_report["Cleaning Report"]
  node_fe_report["Feature Report"]
  node_compare_report["Compare Report"]
  node_dashboard["Dashboard App<br/>runtime<br/>[app.py]"]
end

subgraph group_ops["Ops"]
  node_requirements["Requirements<br/>dependency spec<br/>[requirements.txt]"]
  node_devcontainer["Dev Container<br/>environment spec<br/>[devcontainer.json]"]
end

node_raw_csv -->|"inspect"| node_eda_nb
node_raw_csv -->|"clean"| node_clean_nb
node_eda_nb -->|"document"| node_eda_report
node_clean_nb -->|"write"| node_cleaned_csv
node_clean_nb -->|"document"| node_clean_report
node_cleaned_csv -->|"input"| node_fe_nb
node_fe_nb -->|"write"| node_x_features
node_fe_nb -->|"write"| node_y_target
node_fe_nb -->|"persist"| node_scaler
node_fe_nb -->|"document"| node_fe_report
node_x_features -->|"train on"| node_reg_nb
node_y_target -->|"target"| node_reg_nb
node_scaler -->|"transform"| node_reg_nb
node_reg_nb -->|"fit"| node_candidate_models
node_candidate_models -->|"evaluate"| node_compare_nb
node_ridge_preds -->|"compare"| node_compare_nb
node_compare_nb -->|"select"| node_best_model
node_compare_nb -->|"export"| node_predictions
node_compare_nb -->|"document"| node_compare_report
node_best_model -->|"serve"| node_dashboard
node_predictions -->|"display"| node_dashboard
node_requirements -.->|"supports"| node_eda_nb
node_requirements -.->|"supports"| node_reg_nb
node_requirements -.->|"supports"| node_dashboard
node_devcontainer -.->|"pins"| node_requirements

click node_raw_csv "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/00 Data/Raw data/AmesHousing.csv"
click node_cleaned_csv "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/00 Data/Processed Data/AmesHousing_cleaned.csv"
click node_x_features "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/00 Data/Feature Matrix Data/X_features.csv"
click node_y_target "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/00 Data/Feature Matrix Data/y_target.csv"
click node_scaler "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/00 Data/Feature Matrix Data/scaler.pkl"
click node_eda_nb "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/01 EDA/01_eda_ameshousing.ipynb"
click node_eda_report "https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/main/01 EDA/Report - EDA"
click node_clean_nb "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/02 Data Cleaning/02_data_cleaning.ipynb"
click node_clean_report "https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/main/02 Data Cleaning/Report"
click node_fe_nb "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/03 Feature Engineering/03_feature_engineering.ipynb"
click node_fe_report "https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/main/03 Feature Engineering/Report"
click node_reg_nb "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/04 Regression/04_regression_modeling.ipynb"
click node_candidate_models "https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/main/04 Regression/Models"
click node_best_model "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/04 Regression/Best Model/best_model.pkl"
click node_predictions "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/04 Regression/Best Model/predictions.csv"
click node_ridge_preds "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/00 Data/Feature Matrix Data/ridge_predictions.csv"
click node_compare_nb "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/05 Model Comparison/05_model_comparison.ipynb"
click node_compare_report "https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/main/05 Model Comparison/Report"
click node_dashboard "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/06 Dashboard/app.py"
click node_requirements "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/requirements.txt"
click node_devcontainer "https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/.devcontainer/devcontainer.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_csv,node_cleaned_csv,node_x_features,node_y_target,node_scaler toneBlue
class node_eda_nb,node_clean_nb,node_fe_nb toneAmber
class node_reg_nb,node_candidate_models,node_best_model,node_predictions,node_ridge_preds,node_compare_nb toneMint
class node_eda_report,node_clean_report,node_fe_report,node_compare_report,node_dashboard toneRose
class node_requirements,node_devcontainer toneIndigo
```

## Components

### Data Layer

Raw, processed, and feature artifacts passed between notebook stages.

- [Raw CSV](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/00%20Data/Raw%20data/AmesHousing.csv): dataset · Original Ames Housing source data. · `00 Data/Raw data/AmesHousing.csv`
- [Cleaned Data](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/00%20Data/Processed%20Data/AmesHousing_cleaned.csv): dataset · Processed housing records after cleaning and normalization. · `00 Data/Processed Data/AmesHousing_cleaned.csv`
- [Features](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/00%20Data/Feature%20Matrix%20Data/X_features.csv): feature matrix · Model-ready input matrix used for training and inference preparation. · `00 Data/Feature Matrix Data/X_features.csv`
- [Target](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/00%20Data/Feature%20Matrix%20Data/y_target.csv): target vector · Regression target values separated from the feature matrix. · `00 Data/Feature Matrix Data/y_target.csv`
- [Scaler](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/00%20Data/Feature%20Matrix%20Data/scaler.pkl): preprocessor · Persisted normalization/standardization transform used across modeling steps. · `00 Data/Feature Matrix Data/scaler.pkl`

### Preparation

Exploration, cleaning, and feature engineering notebooks.

- [EDA Notebook](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/01%20EDA/01_eda_ameshousing.ipynb): notebook · Explores distributions, missingness, outliers, and relationships. · `01 EDA/01_eda_ameshousing.ipynb`
- [Cleaning Notebook](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/02%20Data%20Cleaning/02_data_cleaning.ipynb): notebook · Transforms raw records into the cleaned dataset. · `02 Data Cleaning/02_data_cleaning.ipynb`
- [Feature Notebook](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/03%20Feature%20Engineering/03_feature_engineering.ipynb): notebook · Builds model-ready features and target splits. · `03 Feature Engineering/03_feature_engineering.ipynb`

### Modeling

Regression training, comparison, and saved model artifacts.

- [Regression Notebook](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/04%20Regression/04_regression_modeling.ipynb): notebook · Trains and evaluates candidate regression models. · `04 Regression/04_regression_modeling.ipynb`
- [Candidate Models](https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/HEAD/04%20Regression/Models): model artifacts · Trained linear, ridge, lasso, and gradient boosting model files. · `04 Regression/Models`
- [Best Model](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/04%20Regression/Best%20Model/best_model.pkl): selected model · Champion model artifact chosen after comparison. · `04 Regression/Best Model/best_model.pkl`
- [Predictions](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/04%20Regression/Best%20Model/predictions.csv): output table · Final prediction output generated from the selected model. · `04 Regression/Best Model/predictions.csv`
- [Ridge Predictions](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/00%20Data/Feature%20Matrix%20Data/ridge_predictions.csv): evaluation output · Auxiliary ridge prediction artifact used in analysis or comparison. · `00 Data/Feature Matrix Data/ridge_predictions.csv`
- [Compare Notebook](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/05%20Model%20Comparison/05_model_comparison.ipynb): notebook · Compares metrics and drives final model selection. · `05 Model Comparison/05_model_comparison.ipynb`

### Presentation

Dashboard runtime and narrative report outputs.

- [EDA Report](https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/HEAD/01%20EDA/Report%20-%20EDA): report · Narrative findings artifact produced from the exploratory analysis stage. · `01 EDA/Report - EDA`
- [Cleaning Report](https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/HEAD/02%20Data%20Cleaning/Report): report · Documented cleaning-stage outputs and notes. · `02 Data Cleaning/Report`
- [Feature Report](https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/HEAD/03%20Feature%20Engineering/Report): report · Narrative outputs for the feature engineering stage. · `03 Feature Engineering/Report`
- [Compare Report](https://github.com/anundathescientist/real-estate-ameshousing-analysis/tree/HEAD/05%20Model%20Comparison/Report): report · Packaged comparison findings and decision rationale. · `05 Model Comparison/Report`
- [Dashboard App](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/HEAD/06%20Dashboard/app.py): runtime · Read-only application layer that consumes saved artifacts for presentation. · `06 Dashboard/app.py`

### Ops

Environment and dependency definitions for reproducible execution.

- [Requirements](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/requirements.txt): dependency spec · Python dependency set for notebooks, training, and dashboard execution. · `requirements.txt`
- [Dev Container](https://github.com/anundathescientist/real-estate-ameshousing-analysis/blob/main/.devcontainer/devcontainer.json): environment spec · Reproducible development environment definition. · `.devcontainer/devcontainer.json`

## Connections

- Raw CSV → EDA Notebook: inspect · EDA starts from the raw housing dataset.
- Raw CSV → Cleaning Notebook: clean · Cleaning notebook ingests the original dataset.
- EDA Notebook → EDA Report: document · Exploratory findings are packaged into a report.
- Cleaning Notebook → Cleaned Data: write · Cleaning stage persists the cleaned dataset.
- Cleaning Notebook → Cleaning Report: document · Cleaning results are summarized in a report.
- Cleaned Data → Feature Notebook: input · Feature engineering consumes cleaned data.
- Feature Notebook → Features: write · Feature matrix is materialized as a CSV artifact.
- Feature Notebook → Target: write · Target values are stored separately from features.
- Feature Notebook → Scaler: persist · Scaling transform is saved for reuse.
- Feature Notebook → Feature Report: document · Feature engineering outputs are summarized.
- Features → Regression Notebook: train on · Regression training uses engineered features.
- Target → Regression Notebook: target · Regression training uses the separated target values.
- Scaler → Regression Notebook: transform · Models rely on the persisted scaling pipeline.
- Regression Notebook → Candidate Models: fit · Notebook produces the set of trained candidate models.
- Candidate Models → Compare Notebook: evaluate · Comparison stage scores the candidate model artifacts.
- Ridge Predictions → Compare Notebook: compare · Auxiliary ridge predictions support evaluation.
- Compare Notebook → Best Model: select · Comparison selects the champion model artifact.
- Compare Notebook → Predictions: export · Final outputs are written from the selected model.
- Compare Notebook → Compare Report: document · Model comparison results are packaged as a report.
- Best Model → Dashboard App: serve · Dashboard consumes the selected model for inference.
- Predictions → Dashboard App: display · Dashboard can present saved prediction outputs.
- Requirements → EDA Notebook: supports · Dependency specification supports notebook execution.
- Requirements → Regression Notebook: supports · Dependency specification supports training and comparison.
- Requirements → Dashboard App: supports · Dependency specification supports application runtime.
- Dev Container → Requirements: pins · Containerized dev environment complements dependency setup.