# varun-satyasanjeev2008/aeroindex architecture

> Architecture diagram of the GitHub repository varun-satyasanjeev2008/aeroindex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/varun-satyasanjeev2008/aeroindex
- Repository: https://github.com/varun-satyasanjeev2008/aeroindex

## Overview

AeroIndex collects domestic flight fares, cleans observations, computes a weighted daily price index, and optionally narrates significant changes. Its main workflow is orchestrated by `main.py`; the REST API and optional read-only MCP server expose stored results independently. The README documents calendar matching and Groq narration, while the sampled pipeline excerpt confirms the orchestrator’s stage calls. Some subsystem details, including API history/explanation queries and MCP tools, are documented but only partially sampled.

Main entry points: [`main.py`](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/main.py) (Pipeline runner), [`db/connection.py`](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/db/connection.py) (Database helpers), [`pipeline/spike_explainer.py`](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/spike_explainer.py) (Spike explanation), [`scraper/fetch.py`](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/scraper/fetch.py) (Fare scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Fare collection"]
  node_settings["Route and scrape config<br/>[config.py]"]
  node_robots["Robots policy check<br/>[robots_check.py]"]
  node_fare_fetch["Fare scraper<br/>[fetch.py]"]
end

subgraph group_processing["Index and explanation"]
  node_orchestrator["Pipeline runner<br/>[main.py]"]
  node_cleaner["Fare cleaning<br/>[clean.py]"]
  node_index_calc["Index computation<br/>[index.py]"]
  node_spike_logic["Spike explanation<br/>[spike_explainer.py]"]
  node_calendar["Event calendar<br/>[calendar.json]"]
end

subgraph group_persistence["Persistence"]
  node_fare_store["Fare persistence<br/>[store.py]"]
  node_db_helpers["Database helpers<br/>[connection.py]"]
  node_schema["Database schema<br/>[schema.sql]"]
  node_postgres[("PostgreSQL")]
end

subgraph group_interfaces["Read interfaces"]
  node_rest_api["REST API<br/>[main.py]"]
  node_mcp["MCP tools<br/>[server.py]"]
end

node_operator(("Pipeline operator"))
node_fare_sites["Fare websites"]
node_groq["Groq LLM"]
node_api_user(("API consumer"))
node_mcp_client(("MCP client"))

node_operator -->|"runs"| node_orchestrator
node_orchestrator -->|"loads settings"| node_settings
node_orchestrator -->|"bootstraps database"| node_db_helpers
node_orchestrator -->|"filters targets"| node_robots
node_orchestrator -->|"fetches fares"| node_fare_fetch
node_robots -->|"checks robots policy"| node_fare_sites
node_fare_fetch -->|"checks permission"| node_robots
node_fare_fetch -->|"scrapes fares"| node_fare_sites
node_orchestrator -->|"cleans observations"| node_cleaner
node_orchestrator -->|"stores cleaned fares"| node_fare_store
node_orchestrator -->|"computes and writes index"| node_index_calc
node_orchestrator -->|"checks for spike"| node_spike_logic
node_fare_store -->|"inserts fares"| node_postgres
node_index_calc -->|"reads and writes index data"| node_postgres
node_spike_logic -->|"matches events"| node_calendar
node_spike_logic -->|"requests narration"| node_groq
node_spike_logic -->|"stores explanation"| node_postgres
node_db_helpers -->|"connects and applies schema"| node_postgres
node_db_helpers -->|"executes DDL"| node_schema
node_api_user -->|"requests index data"| node_rest_api
node_rest_api -->|"opens connection"| node_db_helpers
node_rest_api -->|"queries index and explanations"| node_postgres
node_mcp_client -->|"invokes tools"| node_mcp
node_mcp -->|"opens connection"| node_db_helpers
node_mcp -->|"queries index and explanations"| node_postgres

click node_orchestrator "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/main.py"
click node_settings "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/config.py"
click node_robots "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/scraper/robots_check.py"
click node_fare_fetch "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/scraper/fetch.py"
click node_cleaner "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/clean.py"
click node_fare_store "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/store.py"
click node_index_calc "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/index.py"
click node_spike_logic "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/spike_explainer.py"
click node_calendar "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/data/calendar.json"
click node_db_helpers "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/db/connection.py"
click node_schema "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/db/schema.sql"
click node_rest_api "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/api/main.py"
click node_mcp "https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/mcp_server/server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_settings,node_robots,node_fare_fetch,node_mcp_client toneBlue
class node_orchestrator,node_cleaner,node_index_calc,node_spike_logic,node_calendar toneAmber
class node_fare_store,node_db_helpers,node_schema,node_postgres,node_api_user toneMint
class node_rest_api,node_mcp toneRose
class node_operator,node_fare_sites,node_groq toneIndigo
```

## Components

### Fare collection

- [Route and scrape config](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/config.py): `config.py`
- [Robots policy check](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/scraper/robots_check.py): `scraper/robots_check.py`
- [Fare scraper](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/scraper/fetch.py): `scraper/fetch.py`

### Index and explanation

- [Pipeline runner](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/main.py): `main.py`
- [Fare cleaning](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/clean.py): `pipeline/clean.py`
- [Index computation](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/index.py): `pipeline/index.py`
- [Spike explanation](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/spike_explainer.py): `pipeline/spike_explainer.py`
- [Event calendar](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/data/calendar.json): `data/calendar.json`

### Persistence

- [Fare persistence](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/pipeline/store.py): `pipeline/store.py`
- [Database helpers](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/db/connection.py): `db/connection.py`
- [Database schema](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/db/schema.sql): `db/schema.sql`
- **PostgreSQL**

### Read interfaces

- [REST API](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/api/main.py): `api/main.py`
- [MCP tools](https://github.com/varun-satyasanjeev2008/aeroindex/blob/main/mcp_server/server.py): `mcp_server/server.py`

### Other components

- **Pipeline operator**
- **Fare websites**
- **Groq LLM**
- **API consumer**
- **MCP client**

## Connections

- Pipeline operator → Pipeline runner: runs (evidence: `README.md`)
- Pipeline runner → Route and scrape config: loads settings (evidence: `main.py`)
- Pipeline runner → Database helpers: bootstraps database (evidence: `main.py`)
- Pipeline runner → Robots policy check: filters targets (evidence: `main.py`)
- Pipeline runner → Fare scraper: fetches fares (evidence: `main.py`)
- Robots policy check → Fare websites: checks robots policy (evidence: `scraper/robots_check.py`)
- Fare scraper → Robots policy check: checks permission (evidence: `scraper/fetch.py`)
- Fare scraper → Fare websites: scrapes fares (evidence: `scraper/fetch.py`)
- Pipeline runner → Fare cleaning: cleans observations (evidence: `main.py`)
- Pipeline runner → Fare persistence: stores cleaned fares (evidence: `main.py`)
- Pipeline runner → Index computation: computes and writes index (evidence: `main.py`)
- Pipeline runner → Spike explanation: checks for spike (evidence: `main.py`)
- Fare persistence → PostgreSQL: inserts fares (evidence: `pipeline/store.py`)
- Index computation → PostgreSQL: reads and writes index data (evidence: `pipeline/index.py`)
- Spike explanation → Event calendar: matches events (evidence: `README.md`)
- Spike explanation → Groq LLM: requests narration (evidence: `README.md`)
- Spike explanation → PostgreSQL: stores explanation (evidence: `pipeline/spike_explainer.py`)
- Database helpers → PostgreSQL: connects and applies schema (evidence: `db/connection.py`)
- Database helpers → Database schema: executes DDL (evidence: `db/connection.py`)
- API consumer → REST API: requests index data (evidence: `README.md`)
- REST API → Database helpers: opens connection (evidence: `api/main.py`)
- REST API → PostgreSQL: queries index and explanations (evidence: `api/main.py`)
- MCP client → MCP tools: invokes tools (evidence: `README.md`)
- MCP tools → Database helpers: opens connection (evidence: `mcp_server/server.py`)
- MCP tools → PostgreSQL: queries index and explanations (evidence: `mcp_server/server.py`)
