# caiopizzol/cnpj-data-pipeline architecture

> Architecture diagram of the GitHub repository caiopizzol/cnpj-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/caiopizzol/cnpj-data-pipeline
- Repository: https://github.com/caiopizzol/cnpj-data-pipeline

## Overview

CNPJ Data Pipeline downloads Receita Federal monthly archives, normalizes their CSV data, then writes batches to PostgreSQL or Parquet. The main entry point also supports listing months and selecting or forcing a month; PostgreSQL processing tracks completed files, while Parquet export supports typed output and manifests. The README documents resilient WebDAV downloads, configurable loading strategies, and recurring one-shot runs. An opt-in PostgreSQL quality report and SQL recipes are additional product capabilities. Relationships reflect the shown main.py calls and README statements; unsampled internals are not inferred.

Main entry points: [`main.py`](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/main.py) (Pipeline CLI), [`database.py`](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/database.py) (PostgreSQL database), [`downloader.py`](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/downloader.py) (WebDAV downloader), [`processor.py`](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/processor.py) (CSV processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Archive ingestion"]
  node_pipeline["Pipeline CLI<br/>[main.py]"]
  node_settings["Runtime settings<br/>[config.py]"]
  node_archive["WebDAV downloader<br/>[downloader.py]"]
end

subgraph group_transform["Data transformation"]
  node_catalog["File layouts<br/>[file_types.py]"]
  node_csv["CSV processor<br/>[processor.py]"]
end

subgraph group_outputs["Output sinks"]
  node_postgres[("PostgreSQL database<br/>[database.py]")]
  node_schema["Database schema<br/>[initial.sql]"]
  node_parquet["Parquet writer<br/>[parquet_writer.py]"]
  node_parquet_files["Parquet output"]
end

subgraph group_insights["Quality and recipes"]
  node_quality["Quality report"]
  node_recipes["SQL recipes"]
end

node_operator(("Operator"))
node_source["Receita Federal"]

node_operator -->|"runs"| node_pipeline
node_pipeline -->|"loads settings"| node_settings
node_pipeline -->|"downloads archives"| node_archive
node_archive -->|"fetches archives"| node_source
node_pipeline -->|"orders files"| node_catalog
node_pipeline -->|"processes CSV"| node_csv
node_csv -->|"uses layouts"| node_catalog
node_pipeline -->|"loads batches"| node_postgres
node_postgres -->|"initializes schema"| node_schema
node_pipeline -->|"writes batches"| node_parquet
node_parquet -->|"exports files"| node_parquet_files
node_quality -->|"reads data"| node_postgres
node_recipes -.->|"queries data"| node_postgres

click node_pipeline "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/main.py"
click node_settings "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/config.py"
click node_archive "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/downloader.py"
click node_catalog "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/file_types.py"
click node_csv "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/processor.py"
click node_postgres "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/database.py"
click node_schema "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/initial.sql"
click node_parquet "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/parquet_writer.py"
click node_quality "https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/scripts/data_quality_report.py"
click node_recipes "https://github.com/caiopizzol/cnpj-data-pipeline/tree/main/recipes/postgres"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_settings,node_archive toneBlue
class node_catalog,node_csv toneAmber
class node_postgres,node_schema,node_parquet,node_parquet_files toneMint
class node_quality,node_recipes toneRose
class node_operator,node_source toneIndigo
```

## Components

### Archive ingestion

- [Pipeline CLI](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/main.py): `main.py`
- [Runtime settings](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/config.py): `config.py`
- [WebDAV downloader](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/downloader.py): `downloader.py`

### Data transformation

- [File layouts](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/file_types.py): `file_types.py`
- [CSV processor](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/processor.py): `processor.py`

### Output sinks

- [PostgreSQL database](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/database.py): `database.py`
- [Database schema](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/initial.sql): `initial.sql`
- [Parquet writer](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/parquet_writer.py): `parquet_writer.py`
- **Parquet output**

### Quality and recipes

- [Quality report](https://github.com/caiopizzol/cnpj-data-pipeline/blob/main/scripts/data_quality_report.py): `scripts/data_quality_report.py`
- [SQL recipes](https://github.com/caiopizzol/cnpj-data-pipeline/tree/main/recipes/postgres): `recipes/postgres`

### Other components

- **Operator**
- **Receita Federal**

## Connections

- Operator → Pipeline CLI: runs (evidence: `README.md`)
- Pipeline CLI → Runtime settings: loads settings (evidence: `main.py`)
- Pipeline CLI → WebDAV downloader: downloads archives (evidence: `main.py`)
- WebDAV downloader → Receita Federal: fetches archives (evidence: `README.md`)
- Pipeline CLI → File layouts: orders files (evidence: `main.py`)
- Pipeline CLI → CSV processor: processes CSV (evidence: `main.py`)
- CSV processor → File layouts: uses layouts (evidence: `processor.py`)
- Pipeline CLI → PostgreSQL database: loads batches (evidence: `README.md`)
- PostgreSQL database → Database schema: initializes schema (evidence: `README.md`)
- Pipeline CLI → Parquet writer: writes batches (evidence: `main.py`)
- Parquet writer → Parquet output: exports files (evidence: `README.md`)
- Quality report → PostgreSQL database: reads data (evidence: `scripts/data_quality_report.py`)
- SQL recipes → PostgreSQL database: queries data
