# pedro-fs-garcia/insightflow-backend architecture

> Architecture diagram of the GitHub repository pedro-fs-garcia/insightflow-backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pedro-fs-garcia/insightflow-backend
- Repository: https://github.com/pedro-fs-garcia/insightflow-backend

## Overview

##### Project type and purpose
- This is a full-stack backend component of InsightFlow: a Flask-based REST API that serves both raw and pre-computed statistical/ML insights on Brazilian foreign trade (2014–2024).
- It includes an ETL/data‐pipeline layer to ingest, clean, transform and load data into PostgreSQL; a caching layer (Redis) to store expensive computations; and an API layer to expose endpoints consumed by a frontend or other clients.

##### File structure and infer architecture
- Top-level:
- app/ – core Flask application
- data_pipeline/ – ETL scripts, models and utilities
- database/ – scripts for schema creation and initialization
- docs/ – API and DB documentation
- run.py, init_db.py, tratar_dados.py – orchestrators for running the API, initializing DB, cleaning data
- app/ layered structure:
- config.py – environment/config management
- dao/ – data access objects (PostgreSQL queries)
- database/ – DB connection module
- estatisticas/ – service layer that implements statistics and ML calculations (pandas, statsmodels, scikit-learn)
- models/ – domain models
- routes/ – Flask blueprints mapping URLs → service functions
- utils/ – logging, helper functions
- Docker support: Dockerfile, .dockerignore, docker run examples
- Infrastructure dependencies: PostgreSQL, Redis, optional AWS hosting

##### Key insights from README
- Uses Python, Flask, Pandas, NumPy, Statsmodels, Scikit-learn
- Primary data source: raw tables from MDIC, cleaned via data_pipeline
- Two main workflows:
a) data_pipeline → transform CSV/JSON → generate cleaned tables → load into Postgres
b) Flask API → DAO + estatisticas services → query/Post-compute/caching → JSON responses
- Redis is used to cache heavy statistical endpoints
- Deployable via Docker, environment variables in .env

##### Main components
- Data Sources
- Raw MDIC tables (CSV/JSON)
- ETL/Batch Processing (data_pipeline)
- Pre-processing → cleaning → DataFrame generation → batch load
- Database
- PostgreSQL instance (tables per entidade: pais, estado, ncm, transacao, etc.)
- Cache
- Redis instance caching expensive stats
- Backend API
- Flask application
– Routes (HTTP layer)
– Service/Statistics layer (estatisticas)
– DAO layer (database access)
- Clients
- Frontend dashboard or other REST consumers
- Infrastructure
- Docker containers for API, Redis (and possibly PostgreSQL)
- Environment config (.env)
- (Optional) AWS hosting for containers/DB

##### Relationships and data flows
- Docker containers grouping:
- One container for Flask API
- One container for Redis
- One container for PostgreSQL (or managed externally)

##### Architectural patterns and design principles
- Layered architecture (Presentation → Service → Data Access → Database)
- Separation of concerns: ETL vs API vs Database schema creation
- Stateless API with externalized cache
- Modular code organization (blueprints and packages)
- Use of configuration via environment variables

##### Detailed elements to capture
- Config module reading .env
- DAO utilities connecting via app/database_connection.py
- Repositories/DAOs for each domain (bc, bloco, pais, etc.)
- Estatisticas scripts (crescimento_mensal, regressao_linear, etc.)
- Routes blueprints mapping to DAO+Estatisticas
- Data pipeline run scripts (run.py, run_analise_tabelas.py)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Sources"
        raw["Raw MDIC Tables\n(CSV/JSON)"]:::source
    end

    subgraph "Batch ETL Pipeline"
        run1["run.py"]:::etl
        run2["run_limpar_tabelas.py"]:::etl
        run3["run_analise_tabelas.py"]:::etl
        models_etl["Models"]:::etl
        scripts_etl["Scripts"]:::etl
    end

    subgraph "Database Schema Init"
        dbconfig["config.py"]:::db
        createtables["create_tables_script.py"]:::db
        builddb["build_database.py"]:::db
        initbuild["init_db.py"]:::db
    end

    subgraph "Database"
        postgres["PostgreSQL"]:::db
    end

    subgraph "Cache"
        redis["Redis Cache"]:::cache
    end

    subgraph "Flask API"
        main["run.py\n& app/__init__.py"]:::api
        config["app/config.py\n.env"]:::api
        dotenv["dotenv_config.py"]:::api
        dbconn["database_connection.py"]:::api
        models_api["vidente.py"]:::api
        dao["DAO Layer"]:::api
        service["Statistics Layer"]:::api
        routes["Routes"]:::api
    end

    subgraph "Clients"
        client["Frontend Dashboard\nor REST Consumers"]:::source
    end

    subgraph "Documentation"
        docsapi["docs/api/"]:::doc
        docsdb["docs/database/"]:::doc
    end

    raw --> run1
    run1 --> run2
    run2 --> run3
    run3 --> models_etl
    models_etl --> scripts_etl
    scripts_etl --> postgres

    dbconfig --> createtables --> builddb --> initbuild --> postgres

    postgres --> dao
    dao --> postgres
    service --> redis
    service --> postgres
    routes --> service
    routes --> dao
    routes --> main
    main --> config
    main --> dotenv
    main --> dbconn

    client --> routes
    routes --> client

    docsapi
    docsdb

    classDef etl fill:#f9d5a2,stroke:#333,stroke-width:1px
    classDef api fill:#a2cffe,stroke:#333,stroke-width:1px
    classDef db fill:#a2f9a2,stroke:#333,stroke-width:1px
    classDef cache fill:#f9faa2,stroke:#333,stroke-width:1px
    classDef source fill:#e2e2e2,stroke:#333,stroke-width:1px
    classDef doc fill:#d3d3d3,stroke:#333,stroke-width:1px

    click raw "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/data_pipeline/tabelas_auxiliares/"
    click run1 "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/data_pipeline/run.py"
    click run2 "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/data_pipeline/run_limpar_tabelas.py"
    click run3 "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/data_pipeline/run_analise_tabelas.py"
    click models_etl "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/data_pipeline/models/"
    click scripts_etl "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/data_pipeline/scripts/"
    click dbconfig "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/database/config.py"
    click createtables "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/database/create_tables_script.py"
    click builddb "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/database/build_database.py"
    click initbuild "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/database/init_db.py"
    click main "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/run.py"
    click config "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/app/config.py"
    click dotenv "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/dotenv_config.py"
    click dbconn "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/app/database/database_connection.py"
    click dao "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/app/dao/"
    click service "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/app/estatisticas/"
    click models_api "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/app/models/vidente.py"
    click routes "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/app/routes/"
    click initdb "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/init_db.py"
    click tratar "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/tratar_dados.py"
    click dockerfile "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/Dockerfile"
    click dockerignore "https://github.com/pedro-fs-garcia/insightflow-backend/blob/main/.dockerignore"
    click docsapi "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/docs/api/"
    click docsdb "https://github.com/pedro-fs-garcia/insightflow-backend/tree/main/docs/database/"
```
