How lucassantosguedes/poupatemploetl works
Project Type and Purpose
• This is a single-page, Python-based web application built on Streamlit for analyzing Excel files in ETL contexts, with a focus on generating recommendations and code snippets (e.g., for Apache NiFi). It is not a multi-service microservices app, but it does have distinct logical layers: UI, data-processing, recommendation engine, and report/export.
File Structure Analysis
• Top-level files/directories:
• .devcontainer/devcontainer.json → environment definition (VS Code remote container)
• PoupaTempoETL.py → main Streamlit application
• requirements.txt, runtime.txt → dependency and runtime constraints
• README.md → documentation
• No separate “src” or “lib” directories; all modules and logic live inside the single Python file, with imported third-party libraries (pandas, Streamlit, ydata-profiling).
README Insights
• Describes features (profiling, ETL issue detection, NiFi recommendations, Groovy scripts, export formats).
• Specifies workflows: file upload → analysis → tabs for profiling, problems, NiFi, Groovy, report, settings.
a. Frontend/UI Layer (Streamlit)
• Handles file upload, navigation among tabs, user configuration.
b. Data Processing Layer (Python logic)
• Invokes pandas and ydata/profiling for statistical analysis
• Scans DataFrame for nulls, inconsistencies, duplicates, special characters, whitespace, date formats, column names
c. Recommendation Engine
• Maps detected problems to NiFi processors & settings
• Stores example dataflows/flow templates
d. Code Generation Module
• Emits Groovy scripts for text cleaning and ETL fixes
e. Reporting & Export Layer
• Aggregates results into Streamlit tables and downloadable Excel, JSON, CSV
f. Environment & Configuration
• requirements.txt and runtime.txt for deployment on Streamlit Cloud or containerized dev environments (.devcontainer)
Relationships & Data Flow
• User uploads Excel → UI Layer
↓
• Data Processing Layer loads into pandas → Profiling & issue detection
↙ ↘
Recommendation Code Generation
Engine Module
↘ ↙
Reporting & Export Layer → UI (results display & downloads)
Architectural Patterns & Design Principles
• Layered architecture: clear separation of UI, business logic (analysis), recommendation logic, and export logic.
• Single-page application: Streamlit drives both UI and orchestrates logic.
• Modular design within one file—consider future refactoring into separate modules/packages if complexity grows.
Relevant Technologies & Frameworks
• Python 3.8+
• Streamlit (UI orchestration)
• pandas & ydata-profiling (data analysis)
• Groovy (code snippet generation)
• Apache NiFi (target ETL platform)
• Docker / devcontainer (optional localized dev environment)
• Streamlit Cloud (deployment target)
Final Advice
• Keep it simple but comprehensive: the goal is clarity for maintainers and new contributors.