How caiopizzol/cnpj-data-pipeline works
CNPJ Data Pipeline downloads Receita Federal monthly archives, normalizes their CSV data, then writes batches to PostgreSQL or Parquet. The main entry point also supports listing months and selecting or forcing a month; PostgreSQL processing tracks completed files, while Parquet export supports typed output and manifests. The README documents resilient WebDAV downloads, configurable loading strategies, and recurring one-shot runs. An opt-in PostgreSQL quality report and SQL recipes are additional product capabilities. Relationships reflect the shown main.py calls and README statements; unsampled internals are not inferred.
Main entry points: main.py Pipeline CLI · database.py PostgreSQL database · downloader.py WebDAV downloader · processor.py CSV processor
Components 13
Archive ingestion
- Pipeline CLI
main.py - Runtime settings
config.py - WebDAV downloader
downloader.py
Data transformation
- File layouts
file_types.py - CSV processor
processor.py
Output sinks
- PostgreSQL database
database.py - Database schema
initial.sql - Parquet writer
parquet_writer.py - Parquet output
Quality and recipes
- Quality report
scripts/data_quality_report.py - SQL recipes
recipes/postgres
Other components
- Operator
- Receita Federal
Connections 13
- Operator → Pipeline CLI: runs
README.md - Pipeline CLI → Runtime settings: loads settings
main.py - Pipeline CLI → WebDAV downloader: downloads archives
main.py - WebDAV downloader → Receita Federal: fetches archives
README.md - Pipeline CLI → File layouts: orders files
main.py - Pipeline CLI → CSV processor: processes CSV
main.py - CSV processor → File layouts: uses layouts
processor.py - Pipeline CLI → PostgreSQL database: loads batches
README.md - PostgreSQL database → Database schema: initializes schema
README.md - Pipeline CLI → Parquet writer: writes batches
main.py - Parquet writer → Parquet output: exports files
README.md - Quality report → PostgreSQL database: reads data
scripts/data_quality_report.py - SQL recipes → PostgreSQL database: queries data