# lucassantosguedes/poupatemploetl architecture

> Architecture diagram of the GitHub repository lucassantosguedes/poupatemploetl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucassantosguedes/poupatemploetl
- Repository: https://github.com/lucassantosguedes/poupatemploetl

## Overview

##### Project Type and Purpose
- This is a single-page, Python-based web application built on Streamlit for analyzing Excel files in ETL contexts, with a focus on generating recommendations and code snippets (e.g., for Apache NiFi). It is not a multi-service microservices app, but it does have distinct logical layers: UI, data-processing, recommendation engine, and report/export.

##### File Structure Analysis
- Top-level files/directories:
- .devcontainer/devcontainer.json → environment definition (VS Code remote container)
- PoupaTempoETL.py → main Streamlit application
- requirements.txt, runtime.txt → dependency and runtime constraints
- README.md → documentation
- No separate “src” or “lib” directories; all modules and logic live inside the single Python file, with imported third-party libraries (pandas, Streamlit, ydata-profiling).

##### README Insights
- Describes features (profiling, ETL issue detection, NiFi recommendations, Groovy scripts, export formats).
- Specifies workflows: file upload → analysis → tabs for profiling, problems, NiFi, Groovy, report, settings.

a. Frontend/UI Layer (Streamlit)
- Handles file upload, navigation among tabs, user configuration.
b. Data Processing Layer (Python logic)
- Invokes pandas and ydata/profiling for statistical analysis
- Scans DataFrame for nulls, inconsistencies, duplicates, special characters, whitespace, date formats, column names
c. Recommendation Engine
- Maps detected problems to NiFi processors & settings
- Stores example dataflows/flow templates
d. Code Generation Module
- Emits Groovy scripts for text cleaning and ETL fixes
e. Reporting & Export Layer
- Aggregates results into Streamlit tables and downloadable Excel, JSON, CSV
f. Environment & Configuration
- requirements.txt and runtime.txt for deployment on Streamlit Cloud or containerized dev environments (.devcontainer)

##### Relationships & Data Flow
- User uploads Excel → UI Layer
↓
- Data Processing Layer loads into pandas → Profiling & issue detection
↙                 ↘
Recommendation       Code Generation
Engine               Module
↘                 ↙
Reporting & Export Layer → UI (results display & downloads)

##### Architectural Patterns & Design Principles
- Layered architecture: clear separation of UI, business logic (analysis), recommendation logic, and export logic.
- Single-page application: Streamlit drives both UI and orchestrates logic.
- Modular design within one file—consider future refactoring into separate modules/packages if complexity grows.

##### Relevant Technologies & Frameworks
- Python 3.8+
- Streamlit (UI orchestration)
- pandas & ydata-profiling (data analysis)
- Groovy (code snippet generation)
- Apache NiFi (target ETL platform)
- Docker / devcontainer (optional localized dev environment)
- Streamlit Cloud (deployment target)

##### Final Advice
- Keep it simple but comprehensive: the goal is clarity for maintainers and new contributors.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    User["User"]:::external
    User -->|"Upload Excel File"| UI

    %% Streamlit Application Container
    subgraph "Streamlit Application"
        direction TB
        UI["UI Layer<br/>(File Upload, Tabs, Config)"]:::frontend
        DP["Data Processing Layer<br/>(pandas & profiling)"]:::processing
        RE["Recommendation Engine<br/>(NiFi Mappings)"]:::recommendation
        CG["Code Generation Module<br/>(Groovy Scripts)"]:::recommendation
        RE -->|"Mapped Problems"| Reporting
        CG -->|"Generated Scripts"| Reporting
        DP -->|"DataFrame"| RE
        DP -->|"DataFrame"| CG
        UI -->|"Trigger Analysis"| DP
        Reporting["Reporting & Export Layer<br/>(Excel, JSON, CSV)"]:::processing
        Reporting -->|"Results & Downloads"| UI
    end

    %% Environment & Deployment
    subgraph "Environment & Configuration"
        direction TB
        Devcontainer[".devcontainer/devcontainer.json"]:::env
        Requirements["requirements.txt"]:::env
        Runtime["runtime.txt"]:::env
        Devcontainer --> Requirements
        Requirements --> Runtime
        Runtime -->|"Deploy on Streamlit Cloud"| StreamlitCloud["Streamlit Cloud"]:::external
    end

    %% Click Events
    click UI "https://github.com/lucassantosguedes/poupatemploetl/blob/main/PoupaTempoETL.py"
    click DP "https://github.com/lucassantosguedes/poupatemploetl/blob/main/PoupaTempoETL.py"
    click RE "https://github.com/lucassantosguedes/poupatemploetl/blob/main/PoupaTempoETL.py"
    click CG "https://github.com/lucassantosguedes/poupatemploetl/blob/main/PoupaTempoETL.py"
    click Reporting "https://github.com/lucassantosguedes/poupatemploetl/blob/main/PoupaTempoETL.py"
    click Devcontainer "https://github.com/lucassantosguedes/poupatemploetl/blob/main/.devcontainer/devcontainer.json"
    click Requirements "https://github.com/lucassantosguedes/poupatemploetl/blob/main/requirements.txt"
    click Runtime "https://github.com/lucassantosguedes/poupatemploetl/blob/main/runtime.txt"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#0288D1,stroke-width:1px;
    classDef processing fill:#DFF8E2,stroke:#2E7D32,stroke-width:1px;
    classDef recommendation fill:#FFE0B2,stroke:#EF6C00,stroke-width:1px;
    classDef env fill:#F0F0F0,stroke:#9E9E9E,stroke-width:1px;
    classDef external fill:#E0E0E0,stroke:#616161,stroke-width:1px;
```
