How adeniltonr/python-com-banco-de-dados works
Main components
• Data Extraction Modules: Represent the several Python scripts designed for web scraping and data extraction. This group includes:
• buscar_arrecadacao_atual.py (fetches current fundraising sums directly from the site)
• extrair_dados_top100_doadores.py (extracts data on the top 100 donors)
• copiar_conteudo_html_top_100_doadores.py (copies HTML content related to donor data)
• extrair_dados_mapa_doacao_por_estados.py (extracts donation data by state)
• Data Persistence Module: Use a dedicated component for the database operations handled by bd.py. This module is responsible for creating tables, inserting data, and connecting with the SQLite database (arenacorinthinas.db).
• Main Control/Interface: The main.py file houses the menu-driven interface that orchestrates user interactions and calls the respective modules.
• Data Files: The “dados” folder, including various text and JSON files, serves as intermediate storage for extracted content and acts as a staging area between web scraping and persistence.
• External Libraries/Utilities: Indicate the use of Selenium (with Firefox’s geckodriver) for web scraping, and other native libraries (e.g., sqlite3, json, re) that underpin the application.
Relationships and interactions
• The main.py acts as the central control panel. It directs calls to the extraction modules to gather data from the fundraising website.
• Web scraping components (using Selenium) interact with the external website to collect fundraising and donor information.
• Once data is extracted, it flows either directly to data files in the “dados” folder or is processed and then inserted into the SQLite database via bd.py.
• Some scripts appear to work sequentially: for example, extracting donor data → writing to a file → reading from the file → persisting into the database.
• There is an implicit flow from user-triggered menu options to either update, fetch, or display data.
Architectural patterns and design principles
• Modular Design: The implementation is divided among distinct scripts, making it easier to extend or modify parts of the system (e.g., adding new scraping routines or database updates).
Incorporate the technologies
• Selenium as part of the “Data Extraction” module to showcase its role in interacting with the external website.
• SQLite database (arenacorinthinas.db) within the “Data Persistence” layer.
• Optionally, indicate external visualization tools – such as DBeaver – as clients that users might use to inspect the database.