# ksilogacademy/data-engineer-01-pipeline-api architecture

> Architecture diagram of the GitHub repository ksilogacademy/data-engineer-01-pipeline-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ksilogacademy/data-engineer-01-pipeline-api
- Repository: https://github.com/ksilogacademy/data-engineer-01-pipeline-api

## Overview

This is a monthly NYC TLC Yellow Taxi data pipeline. The user supplies a year and month to `main.py`, which runs extraction and then transformation as separate subprocesses, stopping if either fails. Extraction pages Socrata results, saves resumable raw pages, and checks the extracted total against the API. Transformation loads those pages, cleans and validates the data, imputes missing values, flags fare discrepancies, and writes a CSV and quality report. The graph separates orchestration, extraction, transformation, and shared configuration. The README documents some transformation details whose implementation is only partially excerpted.

Main entry points: [`src/extract.py`](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/extract.py) (Page extraction), [`src/transform.py`](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py) (Transform runner), [`src/main.py`](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/main.py) (Pipeline runner), [`src/config.py`](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/config.py) (Pipeline settings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_main["Pipeline runner<br/>[main.py]"]
end

subgraph group_extraction["Extraction"]
  node_extract["Page extraction<br/>[extract.py]"]
  node_socrata{{"Socrata API"}}
  node_raw_pages["Raw JSON pages"]
  node_count_check["Count validation<br/>[extract.py]"]
end

subgraph group_transformation["Transformation"]
  node_transform["Transform runner<br/>[transform.py]"]
  node_raw_loader["Raw data loader<br/>[transform.py]"]
  node_type_cleanup["Type and deduplicate<br/>[transform.py]"]
  node_quality_rules["Quality filtering<br/>[transform.py]"]
  node_imputation["Missing-value imputation<br/>[transform.py]"]
  node_discrepancy["Fare discrepancy flag<br/>[transform.py]"]
  node_csv_output["Cleaned CSV"]
  node_quality_report["Quality report"]
end

subgraph group_shared["Shared configuration"]
  node_config["Pipeline settings<br/>[config.py]"]
end

node_user(("Pipeline user"))

node_user -->|"provides month"| node_main
node_main -->|"runs first"| node_extract
node_main -->|"runs next"| node_transform
node_extract -->|"reads settings"| node_config
node_transform -->|"reads settings"| node_config
node_extract -->|"requests pages"| node_socrata
node_extract -->|"requests count"| node_socrata
node_extract -->|"writes pages"| node_raw_pages
node_count_check -->|"compares totals"| node_socrata
node_extract -->|"validates total"| node_count_check
node_transform -->|"loads data"| node_raw_loader
node_raw_loader -->|"reads pages"| node_raw_pages
node_raw_loader -->|"passes rows"| node_type_cleanup
node_type_cleanup -->|"passes rows"| node_quality_rules
node_quality_rules -->|"passes rows"| node_imputation
node_imputation -->|"enriches rows"| node_discrepancy
node_transform -->|"saves result"| node_csv_output
node_transform -->|"saves report"| node_quality_report

click node_main "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/main.py"
click node_extract "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/extract.py"
click node_count_check "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/extract.py"
click node_transform "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py"
click node_raw_loader "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py"
click node_type_cleanup "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py"
click node_quality_rules "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py"
click node_imputation "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py"
click node_discrepancy "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py"
click node_quality_report "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/reports/quality_report.json"
click node_config "https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_user toneBlue
class node_extract,node_socrata,node_raw_pages,node_count_check toneAmber
class node_transform,node_raw_loader,node_type_cleanup,node_quality_rules,node_imputation,node_discrepancy,node_csv_output,node_quality_report toneMint
class node_config toneRose
```

## Components

### Orchestration

- [Pipeline runner](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/main.py): `src/main.py`

### Extraction

- [Page extraction](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/extract.py): `src/extract.py`
- **Socrata API**
- **Raw JSON pages**
- [Count validation](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/extract.py): `src/extract.py`

### Transformation

- [Transform runner](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py): `src/transform.py`
- [Raw data loader](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py): `src/transform.py`
- [Type and deduplicate](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py): `src/transform.py`
- [Quality filtering](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py): `src/transform.py`
- [Missing-value imputation](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py): `src/transform.py`
- [Fare discrepancy flag](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/transform.py): `src/transform.py`
- **Cleaned CSV**
- [Quality report](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/reports/quality_report.json): `reports/quality_report.json`

### Shared configuration

- [Pipeline settings](https://github.com/ksilogacademy/data-engineer-01-pipeline-api/blob/main/src/config.py): `src/config.py`

### Other components

- **Pipeline user**

## Connections

- Pipeline user → Pipeline runner: provides month (evidence: `README.md`)
- Pipeline runner → Page extraction: runs first (evidence: `src/main.py`)
- Pipeline runner → Transform runner: runs next (evidence: `src/main.py`)
- Page extraction → Pipeline settings: reads settings (evidence: `src/extract.py`)
- Transform runner → Pipeline settings: reads settings (evidence: `src/transform.py`)
- Page extraction → Socrata API: requests pages (evidence: `src/extract.py`)
- Page extraction → Socrata API: requests count (evidence: `src/extract.py`)
- Page extraction → Raw JSON pages: writes pages (evidence: `src/extract.py`)
- Count validation → Socrata API: compares totals (evidence: `src/extract.py`)
- Page extraction → Count validation: validates total (evidence: `src/extract.py`)
- Transform runner → Raw data loader: loads data (evidence: `src/transform.py`)
- Raw data loader → Raw JSON pages: reads pages (evidence: `src/transform.py`)
- Raw data loader → Type and deduplicate: passes rows (evidence: `README.md`)
- Type and deduplicate → Quality filtering: passes rows (evidence: `README.md`)
- Quality filtering → Missing-value imputation: passes rows (evidence: `README.md`)
- Missing-value imputation → Fare discrepancy flag: enriches rows (evidence: `README.md`)
- Transform runner → Cleaned CSV: saves result (evidence: `README.md`)
- Transform runner → Quality report: saves report (evidence: `src/transform.py`)
