# yuylam/data-engineering architecture

> Architecture diagram of the GitHub repository yuylam/data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuylam/data-engineering
- Repository: https://github.com/yuylam/data-engineering

## Overview

This is an academic-work portfolio, not a single integrated application. The clearest substantial runtime system in the supplied tree is the palm-oil analytics pipeline: a user starts an orchestrator that runs ingestion, Bronze-to-Silver, Silver-to-Gold, machine-learning, and CSV-export jobs in sequence. The tree supports these stage boundaries and the orchestrator’s dispatch; transformation details, storage locations, and export consumers were not sampled, so their wiring is not expanded here.

Main entry points: [`year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/run_pipeline.py`](https://github.com/yuylam/data-engineering/blob/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/run_pipeline.py) (Pipeline Runner), [`year-3-semester-2/special-topic-in-data-engineering/individual-project/ingestion`](https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/ingestion) (Bronze Ingestion), [`year-3-semester-2/special-topic-in-data-engineering/individual-project/bronze_to_silver`](https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/bronze_to_silver) (Silver Processing), [`year-3-semester-2/special-topic-in-data-engineering/individual-project/silver_to_gold`](https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/silver_to_gold) (Gold Modeling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Pipeline Control"]
  node_runner["Pipeline Runner<br/>[run_pipeline.py]"]
end

subgraph group_ingest["Source Ingestion"]
  node_bronze_jobs["Bronze Ingestion"]
end

subgraph group_transform["Data Transformations"]
  node_silver_jobs["Silver Processing"]
  node_gold_jobs["Gold Modeling"]
end

subgraph group_analytics["Analytics Outputs"]
  node_ml_jobs["ML Features And Models"]
  node_bi_export["CSV Export<br/>[export_csv.py]"]
end

node_user(("Pipeline Operator"))
node_external_sources(("External Data Sources"))

node_user -->|"starts"| node_runner
node_runner -->|"runs jobs"| node_bronze_jobs
node_bronze_jobs -->|"fetches data"| node_external_sources
node_runner -->|"runs jobs"| node_silver_jobs
node_runner -->|"runs jobs"| node_gold_jobs
node_runner -->|"runs jobs"| node_ml_jobs
node_runner -->|"runs job"| node_bi_export

click node_runner "https://github.com/yuylam/data-engineering/blob/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/run_pipeline.py"
click node_bronze_jobs "https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/ingestion"
click node_silver_jobs "https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/bronze_to_silver"
click node_gold_jobs "https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/silver_to_gold"
click node_ml_jobs "https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/machine_learning"
click node_bi_export "https://github.com/yuylam/data-engineering/blob/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/export_csv.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner toneBlue
class node_bronze_jobs toneAmber
class node_silver_jobs,node_gold_jobs toneMint
class node_ml_jobs,node_bi_export toneRose
class node_user,node_external_sources toneIndigo
```

## Components

### Pipeline Control

- [Pipeline Runner](https://github.com/yuylam/data-engineering/blob/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/run_pipeline.py): `year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/run_pipeline.py`

### Source Ingestion

- [Bronze Ingestion](https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/ingestion): `year-3-semester-2/special-topic-in-data-engineering/individual-project/ingestion`

### Data Transformations

- [Silver Processing](https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/bronze_to_silver): `year-3-semester-2/special-topic-in-data-engineering/individual-project/bronze_to_silver`
- [Gold Modeling](https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/silver_to_gold): `year-3-semester-2/special-topic-in-data-engineering/individual-project/silver_to_gold`

### Analytics Outputs

- [ML Features And Models](https://github.com/yuylam/data-engineering/tree/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/machine_learning): `year-3-semester-2/special-topic-in-data-engineering/individual-project/machine_learning`
- [CSV Export](https://github.com/yuylam/data-engineering/blob/main/year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/export_csv.py): `year-3-semester-2/special-topic-in-data-engineering/individual-project/orchestration/export_csv.py`

### Other components

- **Pipeline Operator**
- **External Data Sources**

## Connections

- Pipeline Operator → Pipeline Runner: starts
- Pipeline Runner → Bronze Ingestion: runs jobs
- Bronze Ingestion → External Data Sources: fetches data
- Pipeline Runner → Silver Processing: runs jobs
- Pipeline Runner → Gold Modeling: runs jobs
- Pipeline Runner → ML Features And Models: runs jobs
- Pipeline Runner → CSV Export: runs job