How aaliasghar/nio-gap-reversion-engine works
This repository implements a scheduled NIO market-data pipeline: Airflow dispatches ingestion, Spark transformation, dbt modeling, and signal scanning in sequence. The code samples show ingestion of Yahoo Finance data into PostgreSQL, Spark reads and writes to that warehouse, and the scanner queries the gold signals table. README documents the medallion layers and signal criteria; backtesting is described, but no corresponding implementation path appears in the tree. The architecture below limits that capability to what the available paths support.
Main entry points: ingest_nio.py Ingest Pipeline · nio_dag.py NIO DAG · transform_nio_spark.py Spark Transform · gold_gap_signals.sql Gap Signals Model
Components 13
Orchestration
- Airflow Scheduler
- NIO DAG
dags/nio_dag.py
Data Pipeline
- Ingest Pipeline
scripts/ingest_nio.py - Bronze Prices
- Spark Transform
scripts/transform_nio_spark.py - Silver Prices
Signal Analytics
- dbt Models
sql/nio_strategies - Gap Signals Model
sql/nio_strategies/models/gold_gap_signals.sql - Gold Gap Signals
- Signal Scanner
scripts/scanner_nio.py - Gap Signal Output
External Systems
- Yahoo Finance
- PostgreSQL Warehouse
Connections 21
- Airflow Scheduler → NIO DAG: triggers
- NIO DAG → Ingest Pipeline: runs first
- Ingest Pipeline → PostgreSQL Warehouse: checks timestamp
- PostgreSQL Warehouse → Ingest Pipeline: returns timestamp
- Ingest Pipeline → Yahoo Finance: fetches prices
- Yahoo Finance → Ingest Pipeline: returns OHLCV
- Ingest Pipeline → Bronze Prices: appends raw prices
- Bronze Prices → PostgreSQL Warehouse: stored in
- NIO DAG → Spark Transform: runs next
- Spark Transform → Bronze Prices: reads
- Spark Transform → PostgreSQL Warehouse: reads and writes
- Spark Transform → Silver Prices: writes transformed prices
- Silver Prices → PostgreSQL Warehouse: stored in
- NIO DAG → dbt Models: runs after Spark
- dbt Models → Gap Signals Model: executes
- Gap Signals Model → Silver Prices: models analytics from
- Gap Signals Model → Gold Gap Signals: produces
- Gold Gap Signals → PostgreSQL Warehouse: stored in
- NIO DAG → Signal Scanner: runs after dbt
- Signal Scanner → Gold Gap Signals: queries latest signals
- Signal Scanner → Gap Signal Output: reports signal