How afonsofeliciano/python-web-scraping-declaracoes-imposto-de-renda works
Project type and purpose
• This is a data-collection and reporting pipeline (an ETL + BI solution) implemented as a Jupyter notebook.
• Purpose: scrape tax-declaration data per Brazilian state/year from the Receita Federal website, consolidate it into an Excel file, then produce a Power BI dashboard for analysis.
File structure
• Top-level artifacts:
• coleta_imposto_de_renda.ipynb – main scraping/processing notebook
• data/
– dados_historicos/*.txt – raw per-UF/year text files (historical data)
– dados_consolidados/dados_imposto_de_renda.xlsx – consolidated output
• imgs/ – screenshots of the source website and dashboard
• report/report_declaracoes.pbix – Power BI report file
• requirements/requirements.txt – Python dependencies
• README.md – project overview and instructions
• No separate “src” code folder: the notebook contains both extraction and transformation logic.
• No external orchestration (cron, Airflow) defined—execution is manual via the notebook.
README for architecture clues
• Scraping is driven by two control variables: Ano (year) and UF (state).
• Data consolidation into a single Excel accelerates downstream analysis.
• Power BI uses a dimensional model for the dashboard.
Main components and their relationships
a. External Data Source
– Receita Federal public statistics endpoint
b. Extraction & Transformation (ETL)
– Python / Jupyter notebook
– Uses libraries from requirements.txt (e.g., requests, BeautifulSoup, pandas)
c. Raw Data Landing Zone
– data/dados_historicos – stores each UF_ANO.txt
d. Data Warehouse / Consolidated Dataset
– data/dados_consolidados/dados_imposto_de_renda.xlsx
e. Reporting Layer
– Power BI desktop file (report/report_declaracoes.pbix)
f. Visualization Assets
– imgs/ (screenshots for documentation)
Data flow and interactions
• Step 1: Notebook issues HTTP requests to site → downloads HTML per UF/year
• Step 2: Parser extracts tables → writes raw .txt files in data/dados_historicos
• Step 3: Notebook reads raw files → cleans/transforms → writes to Excel file in data/dados_consolidados
• Step 4: BI tool loads Excel → builds dashboards → output visuals (imgs)
Architectural patterns and design principles
• ETL pipeline pattern: clearly separate extraction, staging (raw), transformation, and loading
• Modular separation between raw staging and consolidated store
• Single source of truth for reporting (the Excel file)
• Manual orchestration but could be automated in future
Technologies and libraries
• Python 3.x (notebook environment)
• requests / BeautifulSoup or equivalent for web scraping
• pandas for data manipulation and Excel I/O
• Power BI (PBIX) for dashboarding
• Local file system as storage
Essential elements
• Notes on data formats (TXT, XLSX)
• Clear boundaries or groupings (e.g., local file system, external web)