# deklonis/flight_detector architecture

> Architecture diagram of the GitHub repository deklonis/flight_detector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deklonis/flight_detector
- Repository: https://github.com/deklonis/flight_detector

## Overview

Flight Detector periodically collects aircraft state snapshots from OpenSky, normalizes them with PySpark, and loads them into PostgreSQL. Airflow schedules and orders the extract, transform, and load tasks; intermediate JSON and Parquet files carry data between those stages. PostgreSQL also exposes documented ground-plane and emergency-situation views. The graph reflects the shown DAG, pipeline, and database code; the SQL files are represented based on the README, which documents their purpose, rather than inspected SQL contents.

Main entry points: [`src/data_services/papeline.py`](https://github.com/deklonis/flight_detector/blob/main/src/data_services/papeline.py) (Pipeline stages), [`dags/airf.py`](https://github.com/deklonis/flight_detector/blob/main/dags/airf.py) (Flight DAG), [`src/db_settings/connection.py`](https://github.com/deklonis/flight_detector/blob/main/src/db_settings/connection.py) (DB connection), [`src/data_services/insert_data.py`](https://github.com/deklonis/flight_detector/blob/main/src/data_services/insert_data.py) (Normalize states)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_dag["Flight DAG<br/>[airf.py]"]
  node_pipeline["Pipeline stages<br/>[papeline.py]"]
end

subgraph group_acquisition["Acquisition"]
  node_extractor["OpenSky client<br/>[api_client.py]"]
end

subgraph group_processing["Processing"]
  node_normalizer["Normalize states<br/>[insert_data.py]"]
  node_spark["Spark session<br/>[sparkutil.py]"]
  node_rawjson["Raw snapshot JSON<br/>[raw_json.json]"]
  node_timejson["Snapshot time JSON<br/>[time_json.json]"]
  node_parquet["Normalized Parquet"]
end

subgraph group_storage["Storage and analytics"]
  node_loader["Database loader<br/>[insert_data.py]"]
  node_connection["DB connection<br/>[connection.py]"]
  node_postgres[("PostgreSQL")]
  node_schema["Database schema<br/>[schemas.sql]"]
  node_initializer["Schema initializer<br/>[main.py]"]
  node_views["Analytics views<br/>[views.sql]"]
end

node_scheduler(("Airflow scheduler"))
node_opensky["OpenSky API"]

node_scheduler -->|"triggers"| node_dag
node_dag -->|"runs extract"| node_pipeline
node_dag -->|"runs transform"| node_pipeline
node_dag -->|"runs load"| node_pipeline
node_dag -->|"orders tasks"| node_dag
node_pipeline -->|"extracts data"| node_normalizer
node_pipeline -->|"normalizes data"| node_normalizer
node_pipeline -->|"loads data"| node_loader
node_normalizer -->|"requests snapshot"| node_extractor
node_extractor -->|"fetches states"| node_opensky
node_normalizer -->|"creates dataframe"| node_spark
node_pipeline -->|"writes snapshot"| node_rawjson
node_pipeline -->|"reads snapshot"| node_rawjson
node_pipeline -->|"writes time"| node_timejson
node_pipeline -->|"reads time"| node_timejson
node_pipeline -->|"writes Parquet"| node_normalizer
node_pipeline -->|"stores normalized data"| node_parquet
node_pipeline -->|"reads for load"| node_parquet
node_loader -->|"reads Parquet"| node_spark
node_loader -->|"opens connection"| node_connection
node_connection -->|"connects"| node_postgres
node_initializer -->|"runs schema setup"| node_connection
node_connection -->|"reads SQL"| node_schema
node_connection -->|"executes schema"| node_postgres
node_loader -->|"writes snapshot records"| node_postgres
node_views -.->|"defines query views"| node_postgres

click node_dag "https://github.com/deklonis/flight_detector/blob/main/dags/airf.py"
click node_pipeline "https://github.com/deklonis/flight_detector/blob/main/src/data_services/papeline.py"
click node_extractor "https://github.com/deklonis/flight_detector/blob/main/src/api/api_client.py"
click node_normalizer "https://github.com/deklonis/flight_detector/blob/main/src/data_services/insert_data.py"
click node_spark "https://github.com/deklonis/flight_detector/blob/main/src/data_services/sparkutil.py"
click node_rawjson "https://github.com/deklonis/flight_detector/blob/main/tmp/raw_json.json"
click node_timejson "https://github.com/deklonis/flight_detector/blob/main/tmp/time_json.json"
click node_parquet "https://github.com/deklonis/flight_detector/tree/main/tmp/flights_parquet"
click node_loader "https://github.com/deklonis/flight_detector/blob/main/src/data_services/insert_data.py"
click node_connection "https://github.com/deklonis/flight_detector/blob/main/src/db_settings/connection.py"
click node_schema "https://github.com/deklonis/flight_detector/blob/main/src/db_settings/schemas/schemas.sql"
click node_initializer "https://github.com/deklonis/flight_detector/blob/main/main.py"
click node_views "https://github.com/deklonis/flight_detector/blob/main/views.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dag,node_pipeline toneBlue
class node_extractor toneAmber
class node_normalizer,node_spark,node_rawjson,node_timejson,node_parquet,node_opensky toneMint
class node_loader,node_connection,node_postgres,node_schema,node_initializer,node_views,node_scheduler toneRose
```

## Components

### Orchestration

- [Flight DAG](https://github.com/deklonis/flight_detector/blob/main/dags/airf.py): `dags/airf.py`
- [Pipeline stages](https://github.com/deklonis/flight_detector/blob/main/src/data_services/papeline.py): `src/data_services/papeline.py`

### Acquisition

- [OpenSky client](https://github.com/deklonis/flight_detector/blob/main/src/api/api_client.py): `src/api/api_client.py`

### Processing

- [Normalize states](https://github.com/deklonis/flight_detector/blob/main/src/data_services/insert_data.py): `src/data_services/insert_data.py`
- [Spark session](https://github.com/deklonis/flight_detector/blob/main/src/data_services/sparkutil.py): `src/data_services/sparkutil.py`
- [Raw snapshot JSON](https://github.com/deklonis/flight_detector/blob/main/tmp/raw_json.json): `tmp/raw_json.json`
- [Snapshot time JSON](https://github.com/deklonis/flight_detector/blob/main/tmp/time_json.json): `tmp/time_json.json`
- [Normalized Parquet](https://github.com/deklonis/flight_detector/tree/main/tmp/flights_parquet): `tmp/flights_parquet`

### Storage and analytics

- [Database loader](https://github.com/deklonis/flight_detector/blob/main/src/data_services/insert_data.py): `src/data_services/insert_data.py`
- [DB connection](https://github.com/deklonis/flight_detector/blob/main/src/db_settings/connection.py): `src/db_settings/connection.py`
- **PostgreSQL**
- [Database schema](https://github.com/deklonis/flight_detector/blob/main/src/db_settings/schemas/schemas.sql): `src/db_settings/schemas/schemas.sql`
- [Schema initializer](https://github.com/deklonis/flight_detector/blob/main/main.py): `main.py`
- [Analytics views](https://github.com/deklonis/flight_detector/blob/main/views.sql): `views.sql`

### Other components

- **Airflow scheduler**
- **OpenSky API**

## Connections

- Airflow scheduler → Flight DAG: triggers (evidence: `README.md`)
- Flight DAG → Pipeline stages: runs extract (evidence: `dags/airf.py`)
- Flight DAG → Pipeline stages: runs transform (evidence: `dags/airf.py`)
- Flight DAG → Pipeline stages: runs load (evidence: `dags/airf.py`)
- Flight DAG → Flight DAG: orders tasks (evidence: `dags/airf.py`)
- Pipeline stages → Normalize states: extracts data (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Normalize states: normalizes data (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Database loader: loads data (evidence: `src/data_services/papeline.py`)
- Normalize states → OpenSky client: requests snapshot (evidence: `src/data_services/insert_data.py`)
- OpenSky client → OpenSky API: fetches states (evidence: `src/api/api_client.py`)
- Normalize states → Spark session: creates dataframe (evidence: `src/data_services/insert_data.py`)
- Pipeline stages → Raw snapshot JSON: writes snapshot (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Raw snapshot JSON: reads snapshot (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Snapshot time JSON: writes time (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Snapshot time JSON: reads time (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Normalize states: writes Parquet (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Normalized Parquet: stores normalized data (evidence: `src/data_services/papeline.py`)
- Pipeline stages → Normalized Parquet: reads for load (evidence: `src/data_services/papeline.py`)
- Database loader → Spark session: reads Parquet (evidence: `src/data_services/insert_data.py`)
- Database loader → DB connection: opens connection (evidence: `src/data_services/insert_data.py`)
- DB connection → PostgreSQL: connects (evidence: `src/db_settings/connection.py`)
- Schema initializer → DB connection: runs schema setup (evidence: `main.py`)
- DB connection → Database schema: reads SQL (evidence: `src/db_settings/connection.py`)
- DB connection → PostgreSQL: executes schema (evidence: `src/db_settings/connection.py`)
- Database loader → PostgreSQL: writes snapshot records (evidence: `src/data_services/insert_data.py`)
- Analytics views → PostgreSQL: defines query views (evidence: `README.md`)
