How puffdapaz/ipeapython works
Main Components
a. Data Storage (Bronze / Silver / Gold) – your “data lake” layers, each holding Parquet files.
b. ETL & Processing Layer – Python scripts (e.g., IPEAv1.py, backend.py, Statistical Analysis scripts) that read, clean, aggregate, and analyze data.
c. Visualization / UI Layer – Streamlit application entry points (_🌐_Início_-_Home.py, pages/_⚽_pt-BR.py, pages/_👑_en-GB.py).
d. External Dependencies – Python libraries (listed in requirements.txt / packages.txt).
e. Users – end users interacting via a web browser.
Relationships & Data Flow
• ETL Layer outputs feed both the Statistical Analysis module (ad hoc reports) and the backend.py data‐serving functions.
• Backend provides processed data to the Streamlit UI.
• Users access the UI; UI calls backend functions; backend reads Gold-layer Parquets.
Architectural Patterns
• “Lambda–style” data pipeline: raw → cleaned → aggregated.
• Modular front end: multilingual pages in a Streamlit multipage app.
• Single‐process Python “backend” serving both ETL orchestration and live data queries.
• Directory segmentation by data maturity (Bronze/Silver/Gold) enforces separation of concerns.
Key Technologies / Frameworks
• Data formats: Parquet (efficient columnar storage).
• Processing: pandas or PyArrow (implied by Parquet use).
• Visualization/UI: Streamlit.
• Orchestration: plain Python scripts (no heavy scheduler indicated).
• Dependency management: requirements.txt / packages.txt.
Final Checklist
• Every component has a clear label.
• Captures the full data lifecycle
• Separates storage, compute, and presentation layers
• Highlights the project’s ETL pattern and multilingual UI
• Remains easy to understand for stakeholders and new engineers alike.