How pedro-fs-garcia/insightflow-backend works
Project type and purpose
• This is a full-stack backend component of InsightFlow: a Flask-based REST API that serves both raw and pre-computed statistical/ML insights on Brazilian foreign trade (2014–2024).
• It includes an ETL/data‐pipeline layer to ingest, clean, transform and load data into PostgreSQL; a caching layer (Redis) to store expensive computations; and an API layer to expose endpoints consumed by a frontend or other clients.
File structure and infer architecture
• Top-level:
• app/ – core Flask application
• data_pipeline/ – ETL scripts, models and utilities
• database/ – scripts for schema creation and initialization
• docs/ – API and DB documentation
• run.py, init_db.py, tratar_dados.py – orchestrators for running the API, initializing DB, cleaning data
• app/ layered structure:
• config.py – environment/config management
• dao/ – data access objects (PostgreSQL queries)
• database/ – DB connection module
• estatisticas/ – service layer that implements statistics and ML calculations (pandas, statsmodels, scikit-learn)
• models/ – domain models
• routes/ – Flask blueprints mapping URLs → service functions
• utils/ – logging, helper functions
• Docker support: Dockerfile, .dockerignore, docker run examples
• Infrastructure dependencies: PostgreSQL, Redis, optional AWS hosting
Key insights from README
• Uses Python, Flask, Pandas, NumPy, Statsmodels, Scikit-learn
• Primary data source: raw tables from MDIC, cleaned via data_pipeline
• Two main workflows:
a) data_pipeline → transform CSV/JSON → generate cleaned tables → load into Postgres
b) Flask API → DAO + estatisticas services → query/Post-compute/caching → JSON responses
• Redis is used to cache heavy statistical endpoints
• Deployable via Docker, environment variables in .env
Main components
• Data Sources
• Raw MDIC tables (CSV/JSON)
• ETL/Batch Processing (data_pipeline)
• Pre-processing → cleaning → DataFrame generation → batch load
• Database
• PostgreSQL instance (tables per entidade: pais, estado, ncm, transacao, etc.)
• Cache
• Redis instance caching expensive stats
• Backend API
• Flask application
– Routes (HTTP layer)
– Service/Statistics layer (estatisticas)
– DAO layer (database access)
• Clients
• Frontend dashboard or other REST consumers
• Infrastructure
• Docker containers for API, Redis (and possibly PostgreSQL)
• Environment config (.env)
• (Optional) AWS hosting for containers/DB
Relationships and data flows
• Docker containers grouping:
• One container for Flask API
• One container for Redis
• One container for PostgreSQL (or managed externally)
Architectural patterns and design principles
• Layered architecture (Presentation → Service → Data Access → Database)
• Separation of concerns: ETL vs API vs Database schema creation
• Stateless API with externalized cache
• Modular code organization (blueprints and packages)
• Use of configuration via environment variables
Detailed elements to capture
• Config module reading .env
• DAO utilities connecting via app/database_connection.py
• Repositories/DAOs for each domain (bc, bloco, pais, etc.)
• Estatisticas scripts (crescimento_mensal, regressao_linear, etc.)
• Routes blueprints mapping to DAO+Estatisticas
• Data pipeline run scripts (run.py, run_analise_tabelas.py)