How darafarm/big_data- works
This is a reactor-monitoring and 10-minute-ahead viscosity-prediction system. Historian CSVs are watched and uploaded to S3, where the pipeline cleans data into Silver, engineers Gold features, trains and evaluates a Random Forest, and stores predictions for a live Dash dashboard. The README also describes deployment across a Windows PC and AWS EC2. Pipeline stages and dashboard reads are supported by excerpts; watcher and deployment links are documented at a high level, so those relationships are shown as documented rather than inferred from unsampled code.
Main entry points: pipeline.py Model Training · dashboard.py Live Dashboard · watch_and_upload.ps1 CSV Watcher
Components 14
Data Ingestion
- CSV Watcher
watch_and_upload.ps1
S3 Data Lake
- AWS S3
- Bronze CSV
- Silver Parquet
- Gold Features
- Trained Model
- Predictions
Prediction Pipeline
- Data Cleaning
pipeline.py - Feature Engineering
pipeline.py - Model Training
pipeline.py
Monitoring
- Live Dashboard
dashboard.py
Other components
- Plant Historian
- Plant Operator
- AWS EC2
Connections 13
- Plant Historian → CSV Watcher: provides CSVs
README.md - CSV Watcher → AWS S3: uploads files
README.md - AWS S3 → Bronze CSV: stores raw CSV
README.md - Bronze CSV → Data Cleaning: reads raw data
pipeline.py - Data Cleaning → Silver Parquet: writes clean data
pipeline.py - Silver Parquet → Feature Engineering: engineers features
pipeline.py - Feature Engineering → Gold Features: writes features
pipeline.py - Gold Features → Model Training: trains model
pipeline.py - Model Training → Trained Model: saves model
pipeline.py - Model Training → Predictions: saves predictions
pipeline.py - Predictions → Live Dashboard: loads predictions
dashboard.py - Plant Operator → Live Dashboard: views monitoring
README.md - AWS EC2 → Live Dashboard: hosts dashboard
README.md