# darafarm/big_data- architecture

> Architecture diagram of the GitHub repository darafarm/big_data-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darafarm/big_data-
- Repository: https://github.com/darafarm/big_data-

## Overview

This is a reactor-monitoring and 10-minute-ahead viscosity-prediction system. Historian CSVs are watched and uploaded to S3, where the pipeline cleans data into Silver, engineers Gold features, trains and evaluates a Random Forest, and stores predictions for a live Dash dashboard. The README also describes deployment across a Windows PC and AWS EC2. Pipeline stages and dashboard reads are supported by excerpts; watcher and deployment links are documented at a high level, so those relationships are shown as documented rather than inferred from unsampled code.

Main entry points: [`pipeline.py`](https://github.com/darafarm/big_data-/blob/main/pipeline.py) (Model Training), [`dashboard.py`](https://github.com/darafarm/big_data-/blob/main/dashboard.py) (Live Dashboard), [`watch_and_upload.ps1`](https://github.com/darafarm/big_data-/blob/main/watch_and_upload.ps1) (CSV Watcher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data Ingestion"]
  node_watcher["CSV Watcher"]
end

subgraph group_lake["S3 Data Lake"]
  node_s3[("AWS S3")]
  node_bronze["Bronze CSV"]
  node_silver["Silver Parquet"]
  node_gold["Gold Features"]
  node_model["Trained Model"]
  node_predictions["Predictions"]
end

subgraph group_ml["Prediction Pipeline"]
  node_cleaning["Data Cleaning<br/>[pipeline.py]"]
  node_features["Feature Engineering<br/>[pipeline.py]"]
  node_training["Model Training<br/>[pipeline.py]"]
end

subgraph group_ui["Monitoring"]
  node_dashboard["Live Dashboard<br/>[dashboard.py]"]
end

node_historian(("Plant Historian"))
node_operator(("Plant Operator"))
node_ec2["AWS EC2"]

node_historian -.->|"provides CSVs"| node_watcher
node_watcher -.->|"uploads files"| node_s3
node_s3 -.->|"stores raw CSV"| node_bronze
node_bronze -->|"reads raw data"| node_cleaning
node_cleaning -->|"writes clean data"| node_silver
node_silver -->|"engineers features"| node_features
node_features -->|"writes features"| node_gold
node_gold -->|"trains model"| node_training
node_training -->|"saves model"| node_model
node_training -->|"saves predictions"| node_predictions
node_predictions -->|"loads predictions"| node_dashboard
node_operator -.->|"views monitoring"| node_dashboard
node_ec2 -.->|"hosts dashboard"| node_dashboard

click node_watcher "https://github.com/darafarm/big_data-/blob/main/watch_and_upload.ps1"
click node_cleaning "https://github.com/darafarm/big_data-/blob/main/pipeline.py"
click node_features "https://github.com/darafarm/big_data-/blob/main/pipeline.py"
click node_training "https://github.com/darafarm/big_data-/blob/main/pipeline.py"
click node_dashboard "https://github.com/darafarm/big_data-/blob/main/dashboard.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_watcher toneBlue
class node_s3,node_bronze,node_silver,node_gold,node_model,node_predictions toneAmber
class node_cleaning,node_features,node_training toneMint
class node_dashboard toneRose
class node_historian,node_operator,node_ec2 toneIndigo
```

## Components

### Data Ingestion

- [CSV Watcher](https://github.com/darafarm/big_data-/blob/main/watch_and_upload.ps1): `watch_and_upload.ps1`

### S3 Data Lake

- **AWS S3**
- **Bronze CSV**
- **Silver Parquet**
- **Gold Features**
- **Trained Model**
- **Predictions**

### Prediction Pipeline

- [Data Cleaning](https://github.com/darafarm/big_data-/blob/main/pipeline.py): `pipeline.py`
- [Feature Engineering](https://github.com/darafarm/big_data-/blob/main/pipeline.py): `pipeline.py`
- [Model Training](https://github.com/darafarm/big_data-/blob/main/pipeline.py): `pipeline.py`

### Monitoring

- [Live Dashboard](https://github.com/darafarm/big_data-/blob/main/dashboard.py): `dashboard.py`

### Other components

- **Plant Historian**
- **Plant Operator**
- **AWS EC2**

## Connections

- Plant Historian → CSV Watcher: provides CSVs (evidence: `README.md`)
- CSV Watcher → AWS S3: uploads files (evidence: `README.md`)
- AWS S3 → Bronze CSV: stores raw CSV (evidence: `README.md`)
- Bronze CSV → Data Cleaning: reads raw data (evidence: `pipeline.py`)
- Data Cleaning → Silver Parquet: writes clean data (evidence: `pipeline.py`)
- Silver Parquet → Feature Engineering: engineers features (evidence: `pipeline.py`)
- Feature Engineering → Gold Features: writes features (evidence: `pipeline.py`)
- Gold Features → Model Training: trains model (evidence: `pipeline.py`)
- Model Training → Trained Model: saves model (evidence: `pipeline.py`)
- Model Training → Predictions: saves predictions (evidence: `pipeline.py`)
- Predictions → Live Dashboard: loads predictions (evidence: `dashboard.py`)
- Plant Operator → Live Dashboard: views monitoring (evidence: `README.md`)
- AWS EC2 → Live Dashboard: hosts dashboard (evidence: `README.md`)