# aaliasghar/nio-gap-reversion-engine architecture

> Architecture diagram of the GitHub repository aaliasghar/nio-gap-reversion-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aaliasghar/nio-gap-reversion-engine
- Repository: https://github.com/aaliasghar/nio-gap-reversion-engine

## Overview

This repository implements a scheduled NIO market-data pipeline: Airflow dispatches ingestion, Spark transformation, dbt modeling, and signal scanning in sequence. The code samples show ingestion of Yahoo Finance data into PostgreSQL, Spark reads and writes to that warehouse, and the scanner queries the gold signals table. README documents the medallion layers and signal criteria; backtesting is described, but no corresponding implementation path appears in the tree. The architecture below limits that capability to what the available paths support.

Main entry points: [`scripts/ingest_nio.py`](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/ingest_nio.py) (Ingest Pipeline), [`dags/nio_dag.py`](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/dags/nio_dag.py) (NIO DAG), [`scripts/transform_nio_spark.py`](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/transform_nio_spark.py) (Spark Transform), [`sql/nio_strategies/models/gold_gap_signals.sql`](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/sql/nio_strategies/models/gold_gap_signals.sql) (Gap Signals Model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_scheduler(("Airflow Scheduler"))
  node_dag["NIO DAG<br/>[nio_dag.py]"]
end

subgraph group_pipeline["Data Pipeline"]
  node_ingest["Ingest Pipeline<br/>[ingest_nio.py]"]
  node_bronze[("Bronze Prices")]
  node_spark["Spark Transform"]
  node_silver[("Silver Prices")]
end

subgraph group_analytics["Signal Analytics"]
  node_dbt["dbt Models"]
  node_gold_model["Gap Signals Model"]
  node_gold[("Gold Gap Signals")]
  node_scanner["Signal Scanner<br/>[scanner_nio.py]"]
  node_signals["Gap Signal Output"]
end

subgraph group_external["External Systems"]
  node_market_data["Yahoo Finance"]
  node_warehouse[("PostgreSQL Warehouse")]
end

node_scheduler -->|"triggers"| node_dag
node_dag -->|"runs first"| node_ingest
node_ingest -->|"checks timestamp"| node_warehouse
node_warehouse -->|"returns timestamp"| node_ingest
node_ingest -->|"fetches prices"| node_market_data
node_market_data -->|"returns OHLCV"| node_ingest
node_ingest -->|"appends raw prices"| node_bronze
node_bronze -->|"stored in"| node_warehouse
node_dag -->|"runs next"| node_spark
node_spark -->|"reads"| node_bronze
node_spark -->|"reads and writes"| node_warehouse
node_spark -->|"writes transformed prices"| node_silver
node_silver -->|"stored in"| node_warehouse
node_dag -->|"runs after Spark"| node_dbt
node_dbt -->|"executes"| node_gold_model
node_gold_model -->|"models analytics from"| node_silver
node_gold_model -->|"produces"| node_gold
node_gold -->|"stored in"| node_warehouse
node_dag -->|"runs after dbt"| node_scanner
node_scanner -->|"queries latest signals"| node_gold
node_scanner -->|"reports signal"| node_signals

click node_dag "https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/dags/nio_dag.py"
click node_ingest "https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/ingest_nio.py"
click node_spark "https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/transform_nio_spark.py"
click node_dbt "https://github.com/aaliasghar/nio-gap-reversion-engine/tree/main/sql/nio_strategies"
click node_gold_model "https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/sql/nio_strategies/models/gold_gap_signals.sql"
click node_scanner "https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/scanner_nio.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scheduler,node_dag toneBlue
class node_ingest,node_bronze,node_spark,node_silver toneAmber
class node_dbt,node_gold_model,node_gold,node_scanner,node_signals toneMint
class node_market_data,node_warehouse toneRose
```

## Components

### Orchestration

- **Airflow Scheduler**
- [NIO DAG](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/dags/nio_dag.py): `dags/nio_dag.py`

### Data Pipeline

- [Ingest Pipeline](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/ingest_nio.py): `scripts/ingest_nio.py`
- **Bronze Prices**
- [Spark Transform](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/transform_nio_spark.py): `scripts/transform_nio_spark.py`
- **Silver Prices**

### Signal Analytics

- [dbt Models](https://github.com/aaliasghar/nio-gap-reversion-engine/tree/main/sql/nio_strategies): `sql/nio_strategies`
- [Gap Signals Model](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/sql/nio_strategies/models/gold_gap_signals.sql): `sql/nio_strategies/models/gold_gap_signals.sql`
- **Gold Gap Signals**
- [Signal Scanner](https://github.com/aaliasghar/nio-gap-reversion-engine/blob/main/scripts/scanner_nio.py): `scripts/scanner_nio.py`
- **Gap Signal Output**

### External Systems

- **Yahoo Finance**
- **PostgreSQL Warehouse**

## Connections

- Airflow Scheduler → NIO DAG: triggers
- NIO DAG → Ingest Pipeline: runs first
- Ingest Pipeline → PostgreSQL Warehouse: checks timestamp
- PostgreSQL Warehouse → Ingest Pipeline: returns timestamp
- Ingest Pipeline → Yahoo Finance: fetches prices
- Yahoo Finance → Ingest Pipeline: returns OHLCV
- Ingest Pipeline → Bronze Prices: appends raw prices
- Bronze Prices → PostgreSQL Warehouse: stored in
- NIO DAG → Spark Transform: runs next
- Spark Transform → Bronze Prices: reads
- Spark Transform → PostgreSQL Warehouse: reads and writes
- Spark Transform → Silver Prices: writes transformed prices
- Silver Prices → PostgreSQL Warehouse: stored in
- NIO DAG → dbt Models: runs after Spark
- dbt Models → Gap Signals Model: executes
- Gap Signals Model → Silver Prices: models analytics from
- Gap Signals Model → Gold Gap Signals: produces
- Gold Gap Signals → PostgreSQL Warehouse: stored in
- NIO DAG → Signal Scanner: runs after dbt
- Signal Scanner → Gold Gap Signals: queries latest signals
- Signal Scanner → Gap Signal Output: reports signal