How bmartinezsaiv/forecast-dm works
1. Repository shape
- This is a monorepo for a forecasting/data-pipeline system, split into reusable Python libraries under libraries/ and runtime services under services/.
- The code is packaged as multiple installable Python projects (pyproject.toml in each library), which suggests the architecture is modular rather than a single app.
2. Top-level execution model
- The system is driven mainly by two services: Airflow for scheduling/orchestration and api_runner for executing pipeline actions.
- services/api_runner/main.py is the execution entrypoint exposed over HTTP; Airflow DAGs call this API to trigger specific ETLs and forecast jobs.
3. Orchestration boundary: Airflow
- services/airflow/dags/ contains one DAG per business process, including inflation, FX, elasticity, forecast dataset creation, and exponential forecast uploads.
- DAGs compute date ranges automatically and run in a daily sequence: external indicators first, then sales data, then elasticity, then forecasts.
- This is the main scheduling boundary in the system; it coordinates task order and failure handling, but does not contain the business logic itself.
4. Execution boundary: API runner
- services/api_runner acts as the operational façade for the data workflows.
- The README indicates it exposes callable actions with query parameters, and those actions invoke the ETL/model code in the libraries.
- This means Airflow is not directly implementing transformations; it delegates to the API, which in turn runs library code.
5. Database layer
- libraries/database defines a common Database abstraction plus implementations for SQL and Postgres.
- The SQL implementation is used for the old data warehouse and the forecast view; Postgres is used for operational forecast-related tables/views such as forecast_dataset, elasticidad, FX tables, inflation, and forecast prediction outputs.
- database_configurator.py indicates connection setup is environment-driven, matching the README’s .env/Portainer deployment model.
6. ETL subsystem
- libraries/etls is the core pipeline layer and is organized by domain: forecast, fx, inflation, prices, elasticity, sales_etl, plus utility ETLs.
- Each ETL follows a shared interface (etl_interface.py) with the classic Extract/Transform/Load lifecycle, and each file includes a local main entrypoint.
- These ETLs are the primary bridge between source systems/web data and the database.
7. External data acquisition
- libraries/web_scraping encapsulates scraping/downloading logic for market inputs like FX and inflation indexes.
- It targets external sources such as INE, BCB, and dolarboliviahoy.com, and downloads Excel or historical-data payloads for CPI, PPI, MPI, official FX, parallel FX, and referential FX.
- This is an important boundary because it separates unstable external web sources from the rest of the data pipeline.
8. Forecasting and modeling layer
- libraries/forecast contains the modeling stack: preprocessing, feature engineering, metrics, graphing, and multiple model families.
- Models include exponential smoothing variants, seasonal models, regression-based elasticity models, and ML models such as LightGBM, XGBoost, CatBoost, and DoubleML.
- The preprocessing modules (clean_sales_data, features_manager, training_preparation) sit between raw ETL outputs and model training/inference.
9. Business-specific forecast logic
- The repo supports at least two major forecast modes described in the README:
- plain exponential smoothing based on monthly sales in 9-liter boxes,
- exponential smoothing adjusted by elasticity and planned price changes.
- The forecast/models/regression and forecast/models/exponential_smoothing subpackages reflect that the forecast engine is built from composable model strategies rather than a single monolithic algorithm.
10. Data analysis / controller layer
- libraries/data_analysis/controller/data_control_center.py appears to centralize orchestration across ETLs, database access, and modeling components.
- This layer likely provides a higher-level entry point for running multi-step analyses without exposing each library directly.
- From an architecture standpoint, it sits above the specialized libraries and below the service layer.
11. Visualization and reporting
- libraries/custom_grapher and libraries/forecast/graphing provide plotting/graph-generation capabilities.
- These are supporting subsystems rather than core pipeline components, likely used for interpreting outputs and generating charts from forecasts or model results.
12. Operational deployment
- The repo is containerized with a root Dockerfile, docker-compose.yml, and a dedicated Airflow Dockerfile.
- Configuration is environment-based, and the README indicates deployment through Docker/Portainer with versioned images.
- There is also a services/dev_workspace container, which suggests a separate local development/runtime environment from the production API and Airflow services.
13. End-to-end data flow
- External data is scraped or fetched by ETLs, normalized, and loaded into Postgres or the warehouse.
- Sales and reference datasets are prepared via ETLs, then consumed by the forecasting library for training/inference.
- Forecast outputs and adjusted predictions are written back to database tables/views, where downstream processes or BI users can consume them.
14. Architectural takeaway
- The repo is structured as a layered data platform: orchestration at the top, API execution in the middle, reusable domain libraries underneath, and databases/external websites as the data sources/sinks.
- The clearest diagram should show separate boundaries for scheduling, API execution, ETL/data acquisition, model computation, and persistence.
Main entry points: database_interface.py Database · training_preparation.py Forecast core · upload_forecast_dataset.py Airflow · etl_interface.py ETL interface
Components 20
Services
Runtime entrypoints and schedulers
- Airflow · DAG orchestration and job sequencing
services/airflow/dags/upload_forecast_dataset.py - API runner · HTTP entrypoint that executes pipeline actions
services/api_runner/main.py - Dev workspace · Separate development container entrypoint
services/dev_workspace/main.py
Libraries
Reusable domain packages
- ETL interface · Shared Extract-Transform-Load lifecycle abstraction
libraries/etls/src/etls/etl_interface.py - Forecast ETLs · Forecast dataset and prediction loading workflows
libraries/etls/src/etls/forecast/forecast_dataset_etl.py - Market ETLs · FX, inflation, prices, and sales ingestion workflows
libraries/etls/src/etls/fx/fx_official_etl.py - Elasticity ETL · Builds and loads elasticity inputs and outputs
libraries/etls/src/etls/elasticity/elasticity_etl.py - Web scraping · Fetches market data from external websites and feeds
libraries/web_scraping/src/web_scraping/fx_rates.py - Database · Shared database abstraction and backend implementations
libraries/database/src/database/database_interface.py - SQL DB · Legacy warehouse and forecast view access
libraries/database/src/database/sql_database.py - Postgres · Operational forecast, elasticity, FX, and inflation tables
libraries/database/src/database/postgres_database.py - Control center · Coordinates ETLs, database access, and modeling workflows
libraries/data_analysis/src/data_analysis/controller/data_control_center.py - Forecast core · Preprocessing, metrics, and model orchestration
libraries/forecast/src/forecast/preprocessing/training_preparation.py - Exponential models · Simple, double, and triple exponential smoothing
libraries/forecast/src/forecast/models/exponential_smoothing/exponential_model_base.py - Regression models · Elasticity-oriented regression forecasting
libraries/forecast/src/forecast/models/regression/elasticity_model.py - ML models · LightGBM, XGBoost, CatBoost, and DoubleML variants
libraries/forecast/src/forecast/models/lightgbm_model.py - Graphing · Plotting and visualization utilities
libraries/custom_grapher/custom_grapher/graph.py
External inputs
Web sources and operational data feeds
- External sources · INE, BCB, and market websites used by scrapers
Data stores
Operational databases and persistent outputs
- Warehouse · Legacy analytical storage and forecast view
- Operational tables · Forecast datasets, elasticity, FX, inflation, and predictions
Connections 24
- Airflow → API runner: triggers jobs · DAGs call the HTTP API to execute scheduled pipeline steps
- API runner → ETL interface: dispatches · API runner invokes ETL actions through a common pipeline contract
- ETL interface → Forecast ETLs: implements · Forecast ETLs share the ETL lifecycle
- ETL interface → Market ETLs: implements · Market ETLs share the ETL lifecycle
- ETL interface → Elasticity ETL: implements · Elasticity ETL follows the shared ETL lifecycle
- Market ETLs → Web scraping: fetches · Market ETLs rely on scraping/download adapters for external inputs
- Web scraping → External sources: reads from · Scrapers collect data from external market and statistical sources
- Forecast ETLs → Database: loads · Forecast ETLs persist derived datasets and predictions
- Market ETLs → Database: loads · FX, inflation, prices, and sales ETLs write to storage
- Elasticity ETL → Database: loads · Elasticity ETL writes prepared elasticity data
- Database → SQL DB: includes · SQL backend for the warehouse and forecast view
- Database → Postgres: includes · Postgres backend for operational forecast data
- SQL DB → Warehouse: persists · Legacy warehouse storage and forecast view
- Postgres → Operational tables: persists · Operational tables for forecasts and market indicators
- API runner → Control center: delegates · Operational façade can hand off to higher-level analysis control
- Control center → Forecast core: coordinates · Central controller orchestrates preprocessing and model runs
- Forecast core → Exponential models: uses · Forecast core can train or infer with exponential smoothing
- Forecast core → Regression models: uses · Forecast core can use elasticity regression strategies
- Forecast core → ML models: uses · Forecast core can use machine-learning model families
- Forecast core → Graphing: reports · Forecast outputs can be visualized and graphed
- Airflow → Market ETLs: schedules · Daily DAGs run external indicator ingestion first
- Airflow → Forecast ETLs: schedules · Airflow coordinates forecast dataset and upload jobs
- Airflow → Elasticity ETL: schedules · Airflow runs elasticity-related workflows in sequence
- Control center → Database: reads/writes · Controller coordinates database access across workflows