How ttt-huong/big-data works
This is a local-first batch ETL lakehouse for image metadata. The principal workflow generates or accepts CSV/Parquet batches, extracts them in chunks, enriches and validates records, routes rejected rows to a DLQ, and idempotently merges clean rows into Delta Lake. The orchestrator selects full, incremental, or backfill behavior and maintains watermark/source state; the loader retries transient write failures. A separate analytics entry point queries the lakehouse and DLQ with DuckDB. The README documents these capabilities; the graph follows sampled source where available.
Main entry points: main_pipeline.py Pipeline Orchestrator · transform.py Enrichment and Routing · query_analytics.py Analytics Queries · data_generator.py Metadata Generator
Components 12
Source and Ingestion
- Pipeline Orchestrator
main_pipeline.py - Metadata Generator
generator/data_generator.py - CSV / Parquet Batches
data - Chunk Extraction
pipeline/extract.py
Transform and Quality
- Enrichment and Routing
pipeline/transform.py - Quality Validation
pipeline/validation.py
Lakehouse and State
- Error Parquet DLQ
data - Retrying Delta Loader
pipeline/load.py - Clean Delta Lake
data - Watermark State
data
Analytics
- Analytics Queries
query_analytics.py
Other components
- Operator
Connections 13
- Operator → Pipeline Orchestrator: runs
- Pipeline Orchestrator → Metadata Generator: generates source
- Metadata Generator → CSV / Parquet Batches: writes batches
- Pipeline Orchestrator → Chunk Extraction: reads chunks
- Chunk Extraction → CSV / Parquet Batches: reads
- Pipeline Orchestrator → Enrichment and Routing: transforms chunks
- Enrichment and Routing → Quality Validation: validates via
- Enrichment and Routing → Error Parquet DLQ: writes rejected rows
- Pipeline Orchestrator → Retrying Delta Loader: loads clean rows
- Retrying Delta Loader → Clean Delta Lake: merges by image_id
- Pipeline Orchestrator → Watermark State: reads and updates
- Analytics Queries → Clean Delta Lake: queries
- Analytics Queries → Error Parquet DLQ: counts errors