How yuylam/data-engineering works
This is an academic-work portfolio, not a single integrated application. The clearest substantial runtime system in the supplied tree is the palm-oil analytics pipeline: a user starts an orchestrator that runs ingestion, Bronze-to-Silver, Silver-to-Gold, machine-learning, and CSV-export jobs in sequence. The tree supports these stage boundaries and the orchestrator’s dispatch; transformation details, storage locations, and export consumers were not sampled, so their wiring is not expanded here.
Main entry points: run_pipeline.py Pipeline Runner · ingestion Bronze Ingestion · bronze_to_silver Silver Processing · silver_to_gold Gold Modeling
Components 8
Pipeline Control
- Pipeline Runner
year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/run_pipeline.py
Source Ingestion
- Bronze Ingestion
year-3-semester-2/special-topic-in-data-engineering/individual-project/ingestion
Data Transformations
- Silver Processing
year-3-semester-2/special-topic-in-data-engineering/individual-project/bronze_to_silver - Gold Modeling
year-3-semester-2/special-topic-in-data-engineering/individual-project/silver_to_gold
Analytics Outputs
- ML Features And Models
year-3-semester-2/special-topic-in-data-engineering/individual-project/machine_learning - CSV Export
year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/export_csv.py
Other components
- Pipeline Operator
- External Data Sources
Connections 7
- Pipeline Operator → Pipeline Runner: starts
- Pipeline Runner → Bronze Ingestion: runs jobs
- Bronze Ingestion → External Data Sources: fetches data
- Pipeline Runner → Silver Processing: runs jobs
- Pipeline Runner → Gold Modeling: runs jobs
- Pipeline Runner → ML Features And Models: runs jobs
- Pipeline Runner → CSV Export: runs job