How s-mtzg/high-perf-data-pipeline works
Project type and purpose
• This is a CLI‐based, high-performance ETL/data-cleaning pipeline written in Python.
• Purpose: Ingest “dirty” CSV data, apply vectorized cleaning, normalization & aggregation via Polars (Rust engine) and emit a clean CSV output in milliseconds.
File structure
• Top‐level scripts:
• generate_data.py → dummy data generator (500K rows)
• src/pipeline.py → core ETL logic as pure functions + CLI entrypoint
• Tests: tests/test_pipeline.py → Pytest cases validating cleaning functions
• Requirements.txt → Python + Polars dependencies
• assets/benchmark_result.png → performance proof
• No microservices or frontend—just a single‐process pipeline.
README
• Confirms Polars lazy evaluation, SIMD & parallelization, fingerprinting aggregation.
• Describes input/output usage and performance gains.
Main components
a. Input (dirty_catalogue.csv)
b. Data Generator (generate_data.py) – optional preprocessing step
c. ETL Pipeline Module (src/pipeline.py)
• Parser/CLI
• Polars LazyFrame builder
• Cleaning sub-pipeline (vectorized string ops & regex)
• Fingerprinting/Aggregation logic
• Sink (write_csv)
d. Polars Engine (Rust backend)
e. Output (clean_catalogue.csv)
f. Tests Suite (Pytest)
Relationships & interactions
• CLI script parses flags → passes file path to pipeline
• Pipeline loads CSV into Polars LazyFrame → chain of transformations
• LazyFrame DAG handed off to Polars Engine → optimized execution (SIMD, parallel)
• Engine streams results to output CSV
• Tests invoke individual functions in pipeline module (unit tests)
Important architectural patterns & design principles
• Pure functional pipeline stages for testability
• Lazy evaluation for pushdown filtering & memory efficiency
• Vectorized operations replacing Python loops (SIMD)
• Streaming API to keep memory footprint low
• Separation of concerns: CLI, transformation logic, I/O
Technologies, frameworks & libraries
• Python 3.10+
• Polars (Rust-powered DataFrame)
• Pytest for tests
• Hyperfine (benchmarking tool; external to code)
Detail
• Input Reader
• Cleaning Functions (e.g., normalize case, remove typos)
• Fingerprint Aggregator
• Output Writer