# afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda architecture

> Architecture diagram of the GitHub repository afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda
- Repository: https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda

## Overview

##### Project type and purpose
- This is a data-collection and reporting pipeline (an ETL + BI solution) implemented as a Jupyter notebook.
- Purpose: scrape tax-declaration data per Brazilian state/year from the Receita Federal website, consolidate it into an Excel file, then produce a Power BI dashboard for analysis.

##### File structure
- Top-level artifacts:
- coleta_imposto_de_renda.ipynb – main scraping/processing notebook
- data/
– dados_historicos/*.txt – raw per-UF/year text files (historical data)
– dados_consolidados/dados_imposto_de_renda.xlsx – consolidated output
- imgs/ – screenshots of the source website and dashboard
- report/report_declaracoes.pbix – Power BI report file
- requirements/requirements.txt – Python dependencies
- README.md – project overview and instructions
- No separate “src” code folder: the notebook contains both extraction and transformation logic.
- No external orchestration (cron, Airflow) defined—execution is manual via the notebook.

##### README for architecture clues
- Scraping is driven by two control variables: Ano (year) and UF (state).
- Data consolidation into a single Excel accelerates downstream analysis.
- Power BI uses a dimensional model for the dashboard.

##### Main components and their relationships
a. External Data Source
– Receita Federal public statistics endpoint
b. Extraction & Transformation (ETL)
– Python / Jupyter notebook
– Uses libraries from requirements.txt (e.g., requests, BeautifulSoup, pandas)
c. Raw Data Landing Zone
– data/dados_historicos – stores each UF_ANO.txt
d. Data Warehouse / Consolidated Dataset
– data/dados_consolidados/dados_imposto_de_renda.xlsx
e. Reporting Layer
– Power BI desktop file (report/report_declaracoes.pbix)
f. Visualization Assets
– imgs/ (screenshots for documentation)

##### Data flow and interactions
- Step 1: Notebook issues HTTP requests to site → downloads HTML per UF/year
- Step 2: Parser extracts tables → writes raw .txt files in data/dados_historicos
- Step 3: Notebook reads raw files → cleans/transforms → writes to Excel file in data/dados_consolidados
- Step 4: BI tool loads Excel → builds dashboards → output visuals (imgs)

##### Architectural patterns and design principles
- ETL pipeline pattern: clearly separate extraction, staging (raw), transformation, and loading
- Modular separation between raw staging and consolidated store
- Single source of truth for reporting (the Excel file)
- Manual orchestration but could be automated in future

##### Technologies and libraries
- Python 3.x (notebook environment)
- requests / BeautifulSoup or equivalent for web scraping
- pandas for data manipulation and Excel I/O
- Power BI (PBIX) for dashboarding
- Local file system as storage

##### Essential elements
- Notes on data formats (TXT, XLSX)
- Clear boundaries or groupings (e.g., local file system, external web)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External"
        Site["Receita Federal Endpoint"]:::external
    end

    subgraph "Project Repository"
        Notebook["Web Scraper & ETL\n(coleta_imposto_de_renda.ipynb)"]:::process
        Checkpoint["Notebook Checkpoints\n(.ipynb_checkpoints/)"]:::storage
        Req["requirements.txt"]:::config
        Readme["README.md"]:::docs
        Vars["Control Variables\n(Ano, UF)"]:::config
    end

    subgraph "Local File System"
        Raw["Raw Data Landing Zone\n(data/dados_historicos/)"]:::storage
        Consol["Consolidated Excel\n(data/dados_consolidados/dados_imposto_de_renda.xlsx)"]:::storage
        Imgs["Visualization Assets\n(imgs/)"]:::docs
    end

    subgraph "Reporting Layer"
        PowerBI["Power BI Report\n(report/report_declaracoes.pbix)"]:::report
    end

    Site -->|"HTTP GET"| Notebook
    Vars -->|provide parameters| Notebook
    Notebook -->|"writes TXT files"| Raw
    Notebook -->|"reads TXT & writes XLSX"| Consol
    Consol -->|"loaded by"| PowerBI
    PowerBI -->|"screenshots"| Imgs
    Notebook -->|"checkpoint saves"| Checkpoint

    click Notebook "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/blob/main/coleta_imposto_de_renda.ipynb"
    click Raw "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/tree/main/data/dados_historicos/"
    click Consol "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/blob/main/data/dados_consolidados/dados_imposto_de_renda.xlsx"
    click PowerBI "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/blob/main/report/report_declaracoes.pbix"
    click Imgs "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/tree/main/imgs/"
    click Req "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/blob/main/requirements/requirements.txt"
    click Readme "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/blob/main/README.md"
    click Checkpoint "https://github.com/afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda/blob/main/.ipynb_checkpoints/coleta_imposto_de_renda-checkpoint.ipynb"

    classDef external fill:#A8DADC,stroke:#333,stroke-width:1px
    classDef process fill:#457B9D,stroke:#333,stroke-width:1px,color:#fff
    classDef storage fill:#E9C46A,stroke:#333,stroke-width:1px
    classDef report fill:#F4A261,stroke:#333,stroke-width:1px
    classDef docs fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef config fill:#E76F51,stroke:#333,stroke-width:1px,color:#fff
```
