# bmartinezsaiv/forecast-dm architecture

> Architecture diagram of the GitHub repository bmartinezsaiv/forecast-dm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bmartinezsaiv/forecast-dm
- Repository: https://github.com/bmartinezsaiv/forecast-dm

## Overview

1. **Repository shape**
   - This is a monorepo for a forecasting/data-pipeline system, split into reusable Python libraries under `libraries/` and runtime services under `services/`.
   - The code is packaged as multiple installable Python projects (`pyproject.toml` in each library), which suggests the architecture is modular rather than a single app.

2. **Top-level execution model**
   - The system is driven mainly by two services: **Airflow** for scheduling/orchestration and **api_runner** for executing pipeline actions.
   - `services/api_runner/main.py` is the execution entrypoint exposed over HTTP; Airflow DAGs call this API to trigger specific ETLs and forecast jobs.

3. **Orchestration boundary: Airflow**
   - `services/airflow/dags/` contains one DAG per business process, including inflation, FX, elasticity, forecast dataset creation, and exponential forecast uploads.
   - DAGs compute date ranges automatically and run in a daily sequence: external indicators first, then sales data, then elasticity, then forecasts.
   - This is the main scheduling boundary in the system; it coordinates task order and failure handling, but does not contain the business logic itself.

4. **Execution boundary: API runner**
   - `services/api_runner` acts as the operational façade for the data workflows.
   - The README indicates it exposes callable actions with query parameters, and those actions invoke the ETL/model code in the libraries.
   - This means Airflow is not directly implementing transformations; it delegates to the API, which in turn runs library code.

5. **Database layer**
   - `libraries/database` defines a common `Database` abstraction plus implementations for **SQL** and **Postgres**.
   - The SQL implementation is used for the old data warehouse and the `forecast` view; Postgres is used for operational forecast-related tables/views such as `forecast_dataset`, `elasticidad`, FX tables, inflation, and forecast prediction outputs.
   - `database_configurator.py` indicates connection setup is environment-driven, matching the README’s `.env`/Portainer deployment model.

6. **ETL subsystem**
   - `libraries/etls` is the core pipeline layer and is organized by domain: `forecast`, `fx`, `inflation`, `prices`, `elasticity`, `sales_etl`, plus utility ETLs.
   - Each ETL follows a shared interface (`etl_interface.py`) with the classic Extract/Transform/Load lifecycle, and each file includes a local `main` entrypoint.
   - These ETLs are the primary bridge between source systems/web data and the database.

7. **External data acquisition**
   - `libraries/web_scraping` encapsulates scraping/downloading logic for market inputs like FX and inflation indexes.
   - It targets external sources such as INE, BCB, and `dolarboliviahoy.com`, and downloads Excel or historical-data payloads for CPI, PPI, MPI, official FX, parallel FX, and referential FX.
   - This is an important boundary because it separates unstable external web sources from the rest of the data pipeline.

8. **Forecasting and modeling layer**
   - `libraries/forecast` contains the modeling stack: preprocessing, feature engineering, metrics, graphing, and multiple model families.
   - Models include exponential smoothing variants, seasonal models, regression-based elasticity models, and ML models such as LightGBM, XGBoost, CatBoost, and DoubleML.
   - The preprocessing modules (`clean_sales_data`, `features_manager`, `training_preparation`) sit between raw ETL outputs and model training/inference.

9. **Business-specific forecast logic**
   - The repo supports at least two major forecast modes described in the README:
     - plain exponential smoothing based on monthly sales in 9-liter boxes,
     - exponential smoothing adjusted by elasticity and planned price changes.
   - The `forecast/models/regression` and `forecast/models/exponential_smoothing` subpackages reflect that the forecast engine is built from composable model strategies rather than a single monolithic algorithm.

10. **Data analysis / controller layer**
   - `libraries/data_analysis/controller/data_control_center.py` appears to centralize orchestration across ETLs, database access, and modeling components.
   - This layer likely provides a higher-level entry point for running multi-step analyses without exposing each library directly.
   - From an architecture standpoint, it sits above the specialized libraries and below the service layer.

11. **Visualization and reporting**
   - `libraries/custom_grapher` and `libraries/forecast/graphing` provide plotting/graph-generation capabilities.
   - These are supporting subsystems rather than core pipeline components, likely used for interpreting outputs and generating charts from forecasts or model results.

12. **Operational deployment**
   - The repo is containerized with a root `Dockerfile`, `docker-compose.yml`, and a dedicated Airflow Dockerfile.
   - Configuration is environment-based, and the README indicates deployment through Docker/Portainer with versioned images.
   - There is also a `services/dev_workspace` container, which suggests a separate local development/runtime environment from the production API and Airflow services.

13. **End-to-end data flow**
   - External data is scraped or fetched by ETLs, normalized, and loaded into Postgres or the warehouse.
   - Sales and reference datasets are prepared via ETLs, then consumed by the forecasting library for training/inference.
   - Forecast outputs and adjusted predictions are written back to database tables/views, where downstream processes or BI users can consume them.

14. **Architectural takeaway**
   - The repo is structured as a layered data platform: orchestration at the top, API execution in the middle, reusable domain libraries underneath, and databases/external websites as the data sources/sinks.
   - The clearest diagram should show separate boundaries for scheduling, API execution, ETL/data acquisition, model computation, and persistence.

Main entry points: [`libraries/database/src/database/database_interface.py`](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/database/src/database/database_interface.py) (Database), [`libraries/forecast/src/forecast/preprocessing/training_preparation.py`](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/preprocessing/training_preparation.py) (Forecast core), [`services/airflow/dags/upload_forecast_dataset.py`](https://github.com/bmartinezsaiv/forecast-dm/blob/main/services/airflow/dags/upload_forecast_dataset.py) (Airflow), [`libraries/etls/src/etls/etl_interface.py`](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/etl_interface.py) (ETL interface)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_services["Services"]
  node_airflow["Airflow<br/>scheduler"]
  node_api_runner["API runner<br/>execution API<br/>[main.py]"]
  node_dev_workspace["Dev workspace<br/>local runtime<br/>[main.py]"]
end

subgraph group_libraries["Libraries"]
  node_etl_interface["ETL interface<br/>pipeline contract<br/>[etl_interface.py]"]
  node_forecast_etls["Forecast ETLs<br/>domain ETLs"]
  node_market_etls["Market ETLs<br/>domain ETLs<br/>[fx_official_etl.py]"]
  node_elasticity_etl["Elasticity ETL<br/>domain ETL<br/>[elasticity_etl.py]"]
  node_web_scraping["Web scraping<br/>source adapters<br/>[fx_rates.py]"]
  node_database[("Database<br/>persistence layer")]
  node_db_sql[("SQL DB<br/>warehouse adapter<br/>[sql_database.py]")]
  node_db_postgres[("Postgres<br/>operational db")]
  node_data_control["Control center<br/>analysis controller"]
  node_forecast_core["Forecast core<br/>modeling stack"]
  node_exp_models["Exponential models<br/>forecast strategy"]
  node_regression_models["Regression models<br/>forecast strategy"]
  node_ml_models["ML models<br/>forecast strategy<br/>[lightgbm_model.py]"]
  node_graphing["Graphing<br/>reporting<br/>[graph.py]"]
end

subgraph group_external["External inputs"]
  node_external_sources(("External sources<br/>web feeds"))
end

subgraph group_storage["Data stores"]
  node_warehouse[("Warehouse<br/>analytical store")]
  node_operational_tables[("Operational tables<br/>forecast store")]
end

node_airflow -->|"triggers jobs"| node_api_runner
node_api_runner -->|"dispatches"| node_etl_interface
node_etl_interface -->|"implements"| node_forecast_etls
node_etl_interface -->|"implements"| node_market_etls
node_etl_interface -->|"implements"| node_elasticity_etl
node_market_etls -->|"fetches"| node_web_scraping
node_web_scraping -->|"reads from"| node_external_sources
node_forecast_etls -->|"loads"| node_database
node_market_etls -->|"loads"| node_database
node_elasticity_etl -->|"loads"| node_database
node_database -->|"includes"| node_db_sql
node_database -->|"includes"| node_db_postgres
node_db_sql -->|"persists"| node_warehouse
node_db_postgres -->|"persists"| node_operational_tables
node_api_runner -->|"delegates"| node_data_control
node_data_control -->|"coordinates"| node_forecast_core
node_forecast_core -->|"uses"| node_exp_models
node_forecast_core -->|"uses"| node_regression_models
node_forecast_core -->|"uses"| node_ml_models
node_forecast_core -->|"reports"| node_graphing
node_airflow -.->|"schedules"| node_market_etls
node_airflow -.->|"schedules"| node_forecast_etls
node_airflow -.->|"schedules"| node_elasticity_etl
node_data_control -->|"reads/writes"| node_database

click node_airflow "https://github.com/bmartinezsaiv/forecast-dm/blob/main/services/airflow/dags/upload_forecast_dataset.py"
click node_api_runner "https://github.com/bmartinezsaiv/forecast-dm/blob/main/services/api_runner/main.py"
click node_dev_workspace "https://github.com/bmartinezsaiv/forecast-dm/blob/main/services/dev_workspace/main.py"
click node_etl_interface "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/etl_interface.py"
click node_forecast_etls "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/forecast/forecast_dataset_etl.py"
click node_market_etls "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/fx/fx_official_etl.py"
click node_elasticity_etl "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/elasticity/elasticity_etl.py"
click node_web_scraping "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/web_scraping/src/web_scraping/fx_rates.py"
click node_database "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/database/src/database/database_interface.py"
click node_db_sql "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/database/src/database/sql_database.py"
click node_db_postgres "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/database/src/database/postgres_database.py"
click node_data_control "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/data_analysis/src/data_analysis/controller/data_control_center.py"
click node_forecast_core "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/preprocessing/training_preparation.py"
click node_exp_models "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/models/exponential_smoothing/exponential_model_base.py"
click node_regression_models "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/models/regression/elasticity_model.py"
click node_ml_models "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/models/lightgbm_model.py"
click node_graphing "https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/custom_grapher/custom_grapher/graph.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow,node_api_runner,node_dev_workspace toneBlue
class node_etl_interface,node_forecast_etls,node_market_etls,node_elasticity_etl,node_web_scraping,node_database,node_db_sql,node_db_postgres,node_data_control,node_forecast_core,node_exp_models,node_regression_models,node_ml_models,node_graphing toneAmber
class node_external_sources toneMint
class node_warehouse,node_operational_tables toneRose
```

## Components

### Services

Runtime entrypoints and schedulers

- [Airflow](https://github.com/bmartinezsaiv/forecast-dm/blob/main/services/airflow/dags/upload_forecast_dataset.py): scheduler · DAG orchestration and job sequencing · `services/airflow/dags/upload_forecast_dataset.py`
- [API runner](https://github.com/bmartinezsaiv/forecast-dm/blob/main/services/api_runner/main.py): execution API · HTTP entrypoint that executes pipeline actions · `services/api_runner/main.py`
- [Dev workspace](https://github.com/bmartinezsaiv/forecast-dm/blob/main/services/dev_workspace/main.py): local runtime · Separate development container entrypoint · `services/dev_workspace/main.py`

### Libraries

Reusable domain packages

- [ETL interface](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/etl_interface.py): pipeline contract · Shared Extract-Transform-Load lifecycle abstraction · `libraries/etls/src/etls/etl_interface.py`
- [Forecast ETLs](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/forecast/forecast_dataset_etl.py): domain ETLs · Forecast dataset and prediction loading workflows · `libraries/etls/src/etls/forecast/forecast_dataset_etl.py`
- [Market ETLs](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/fx/fx_official_etl.py): domain ETLs · FX, inflation, prices, and sales ingestion workflows · `libraries/etls/src/etls/fx/fx_official_etl.py`
- [Elasticity ETL](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/etls/src/etls/elasticity/elasticity_etl.py): domain ETL · Builds and loads elasticity inputs and outputs · `libraries/etls/src/etls/elasticity/elasticity_etl.py`
- [Web scraping](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/web_scraping/src/web_scraping/fx_rates.py): source adapters · Fetches market data from external websites and feeds · `libraries/web_scraping/src/web_scraping/fx_rates.py`
- [Database](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/database/src/database/database_interface.py): persistence layer · Shared database abstraction and backend implementations · `libraries/database/src/database/database_interface.py`
- [SQL DB](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/database/src/database/sql_database.py): warehouse adapter · Legacy warehouse and forecast view access · `libraries/database/src/database/sql_database.py`
- [Postgres](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/database/src/database/postgres_database.py): operational db · Operational forecast, elasticity, FX, and inflation tables · `libraries/database/src/database/postgres_database.py`
- [Control center](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/data_analysis/src/data_analysis/controller/data_control_center.py): analysis controller · Coordinates ETLs, database access, and modeling workflows · `libraries/data_analysis/src/data_analysis/controller/data_control_center.py`
- [Forecast core](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/preprocessing/training_preparation.py): modeling stack · Preprocessing, metrics, and model orchestration · `libraries/forecast/src/forecast/preprocessing/training_preparation.py`
- [Exponential models](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/models/exponential_smoothing/exponential_model_base.py): forecast strategy · Simple, double, and triple exponential smoothing · `libraries/forecast/src/forecast/models/exponential_smoothing/exponential_model_base.py`
- [Regression models](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/models/regression/elasticity_model.py): forecast strategy · Elasticity-oriented regression forecasting · `libraries/forecast/src/forecast/models/regression/elasticity_model.py`
- [ML models](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/forecast/src/forecast/models/lightgbm_model.py): forecast strategy · LightGBM, XGBoost, CatBoost, and DoubleML variants · `libraries/forecast/src/forecast/models/lightgbm_model.py`
- [Graphing](https://github.com/bmartinezsaiv/forecast-dm/blob/main/libraries/custom_grapher/custom_grapher/graph.py): reporting · Plotting and visualization utilities · `libraries/custom_grapher/custom_grapher/graph.py`

### External inputs

Web sources and operational data feeds

- **External sources**: web feeds · INE, BCB, and market websites used by scrapers

### Data stores

Operational databases and persistent outputs

- **Warehouse**: analytical store · Legacy analytical storage and forecast view
- **Operational tables**: forecast store · Forecast datasets, elasticity, FX, inflation, and predictions

## Connections

- Airflow → API runner: triggers jobs · DAGs call the HTTP API to execute scheduled pipeline steps
- API runner → ETL interface: dispatches · API runner invokes ETL actions through a common pipeline contract
- ETL interface → Forecast ETLs: implements · Forecast ETLs share the ETL lifecycle
- ETL interface → Market ETLs: implements · Market ETLs share the ETL lifecycle
- ETL interface → Elasticity ETL: implements · Elasticity ETL follows the shared ETL lifecycle
- Market ETLs → Web scraping: fetches · Market ETLs rely on scraping/download adapters for external inputs
- Web scraping → External sources: reads from · Scrapers collect data from external market and statistical sources
- Forecast ETLs → Database: loads · Forecast ETLs persist derived datasets and predictions
- Market ETLs → Database: loads · FX, inflation, prices, and sales ETLs write to storage
- Elasticity ETL → Database: loads · Elasticity ETL writes prepared elasticity data
- Database → SQL DB: includes · SQL backend for the warehouse and forecast view
- Database → Postgres: includes · Postgres backend for operational forecast data
- SQL DB → Warehouse: persists · Legacy warehouse storage and forecast view
- Postgres → Operational tables: persists · Operational tables for forecasts and market indicators
- API runner → Control center: delegates · Operational façade can hand off to higher-level analysis control
- Control center → Forecast core: coordinates · Central controller orchestrates preprocessing and model runs
- Forecast core → Exponential models: uses · Forecast core can train or infer with exponential smoothing
- Forecast core → Regression models: uses · Forecast core can use elasticity regression strategies
- Forecast core → ML models: uses · Forecast core can use machine-learning model families
- Forecast core → Graphing: reports · Forecast outputs can be visualized and graphed
- Airflow → Market ETLs: schedules · Daily DAGs run external indicator ingestion first
- Airflow → Forecast ETLs: schedules · Airflow coordinates forecast dataset and upload jobs
- Airflow → Elasticity ETL: schedules · Airflow runs elasticity-related workflows in sequence
- Control center → Database: reads/writes · Controller coordinates database access across workflows
