# s-mtzg/high-perf-data-pipeline architecture

> Architecture diagram of the GitHub repository s-mtzg/high-perf-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/s-mtzg/high-perf-data-pipeline
- Repository: https://github.com/s-mtzg/high-perf-data-pipeline

## Overview

##### Project type and purpose
- This is a CLI‐based, high-performance ETL/data-cleaning pipeline written in Python.
- Purpose: Ingest “dirty” CSV data, apply vectorized cleaning, normalization & aggregation via Polars (Rust engine) and emit a clean CSV output in milliseconds.

##### File structure
- Top‐level scripts:
- generate_data.py → dummy data generator (500K rows)
- src/pipeline.py → core ETL logic as pure functions + CLI entrypoint
- Tests: tests/test_pipeline.py → Pytest cases validating cleaning functions
- Requirements.txt → Python + Polars dependencies
- assets/benchmark_result.png → performance proof
- No microservices or frontend—just a single‐process pipeline.

##### README
- Confirms Polars lazy evaluation, SIMD & parallelization, fingerprinting aggregation.
- Describes input/output usage and performance gains.

##### Main components
a. Input (dirty_catalogue.csv)
b. Data Generator (generate_data.py) – optional preprocessing step
c. ETL Pipeline Module (src/pipeline.py)
- Parser/CLI
- Polars LazyFrame builder
- Cleaning sub-pipeline (vectorized string ops & regex)
- Fingerprinting/Aggregation logic
- Sink (write_csv)
d. Polars Engine (Rust backend)
e. Output (clean_catalogue.csv)
f. Tests Suite (Pytest)

##### Relationships & interactions
- CLI script parses flags → passes file path to pipeline
- Pipeline loads CSV into Polars LazyFrame → chain of transformations
- LazyFrame DAG handed off to Polars Engine → optimized execution (SIMD, parallel)
- Engine streams results to output CSV
- Tests invoke individual functions in pipeline module (unit tests)

##### Important architectural patterns & design principles
- Pure functional pipeline stages for testability
- Lazy evaluation for pushdown filtering & memory efficiency
- Vectorized operations replacing Python loops (SIMD)
- Streaming API to keep memory footprint low
- Separation of concerns: CLI, transformation logic, I/O

##### Technologies, frameworks & libraries
- Python 3.10+
- Polars (Rust-powered DataFrame)
- Pytest for tests
- Hyperfine (benchmarking tool; external to code)

##### Detail
- Input Reader
- Cleaning Functions (e.g., normalize case, remove typos)
- Fingerprint Aggregator
- Output Writer

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Generation
    DataGen["Data Generator (generate_data.py)"]:::script
    DirtyCSV["dirty_catalogue.csv"]:::data

    %% ETL Pipeline
    subgraph "ETL Pipeline" 
        Parser["Parser / CLI"]:::script
        LazyFrame["Polars LazyFrame Builder"]:::script
        Cleaning["Cleaning Functions"]:::script
        Aggregation["Fingerprint Aggregator"]:::script
        Sink["Output Writer"]:::script
    end

    %% Polars Engine
    Engine["Polars Engine (Rust)"]:::engine

    %% Output
    CleanCSV["clean_catalogue.csv"]:::data

    %% Tests and Documentation
    Tests["Test Suite (tests/test_pipeline.py)"]:::tests
    Req["Requirements.txt"]:::docs
    Benchmark["Benchmark Asset"]:::docs
    Readme["Documentation (README.md)"]:::docs
    License["License (LICENSE)"]:::docs
    GitIgnore["Git Ignore (.gitignore)"]:::docs

    %% Data Flow
    DataGen -->|500K rows| DirtyCSV
    DirtyCSV -->|input file| Parser
    Parser --> LazyFrame
    LazyFrame --> Cleaning
    Cleaning --> Aggregation
    Aggregation --> Engine
    Engine -->|optimized execution| Sink
    Sink --> CleanCSV

    %% Tests and Docs Relations
    Tests --> Cleaning
    Tests --> Aggregation
    Tests --> Parser

    Req --> Parser
    Readme --> Parser
    Benchmark --> DataGen

    %% Click Events
    click DataGen "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/generate_data.py"
    click Parser "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/src/pipeline.py"
    click LazyFrame "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/src/pipeline.py"
    click Cleaning "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/src/pipeline.py"
    click Aggregation "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/src/pipeline.py"
    click Sink "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/src/pipeline.py"
    click Tests "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/tests/test_pipeline.py"
    click Req "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/requirements.txt"
    click Benchmark "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/assets/benchmark_result.png"
    click Readme "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/README.md"
    click License "https://github.com/s-mtzg/high-perf-data-pipeline/tree/main/LICENSE"
    click GitIgnore "https://github.com/s-mtzg/high-perf-data-pipeline/blob/main/.gitignore"

    %% Styles
    classDef script fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef engine fill:#FFA500,stroke:#333,stroke-width:1px
    classDef data fill:#90EE90,stroke:#333,stroke-width:1px
    classDef tests fill:#FFD700,stroke:#333,stroke-width:1px
    classDef docs fill:#D3D3D3,stroke:#333,stroke-width:1px
```
