How deklonis/flight_detector works
Flight Detector periodically collects aircraft state snapshots from OpenSky, normalizes them with PySpark, and loads them into PostgreSQL. Airflow schedules and orders the extract, transform, and load tasks; intermediate JSON and Parquet files carry data between those stages. PostgreSQL also exposes documented ground-plane and emergency-situation views. The graph reflects the shown DAG, pipeline, and database code; the SQL files are represented based on the README, which documents their purpose, rather than inspected SQL contents.
Main entry points: papeline.py Pipeline stages · airf.py Flight DAG · connection.py DB connection · insert_data.py Normalize states
Components 16
Orchestration
- Flight DAG
dags/airf.py - Pipeline stages
src/data_services/papeline.py
Acquisition
- OpenSky client
src/api/api_client.py
Processing
- Normalize states
src/data_services/insert_data.py - Spark session
src/data_services/sparkutil.py - Raw snapshot JSON
tmp/raw_json.json - Snapshot time JSON
tmp/time_json.json - Normalized Parquet
tmp/flights_parquet
Storage and analytics
- Database loader
src/data_services/insert_data.py - DB connection
src/db_settings/connection.py - PostgreSQL
- Database schema
src/db_settings/schemas/schemas.sql - Schema initializer
main.py - Analytics views
views.sql
Other components
- Airflow scheduler
- OpenSky API
Connections 26
- Airflow scheduler → Flight DAG: triggers
README.md - Flight DAG → Pipeline stages: runs extract
dags/airf.py - Flight DAG → Pipeline stages: runs transform
dags/airf.py - Flight DAG → Pipeline stages: runs load
dags/airf.py - Flight DAG → Flight DAG: orders tasks
dags/airf.py - Pipeline stages → Normalize states: extracts data
src/data_services/papeline.py - Pipeline stages → Normalize states: normalizes data
src/data_services/papeline.py - Pipeline stages → Database loader: loads data
src/data_services/papeline.py - Normalize states → OpenSky client: requests snapshot
src/data_services/insert_data.py - OpenSky client → OpenSky API: fetches states
src/api/api_client.py - Normalize states → Spark session: creates dataframe
src/data_services/insert_data.py - Pipeline stages → Raw snapshot JSON: writes snapshot
src/data_services/papeline.py - Pipeline stages → Raw snapshot JSON: reads snapshot
src/data_services/papeline.py - Pipeline stages → Snapshot time JSON: writes time
src/data_services/papeline.py - Pipeline stages → Snapshot time JSON: reads time
src/data_services/papeline.py - Pipeline stages → Normalize states: writes Parquet
src/data_services/papeline.py - Pipeline stages → Normalized Parquet: stores normalized data
src/data_services/papeline.py - Pipeline stages → Normalized Parquet: reads for load
src/data_services/papeline.py - Database loader → Spark session: reads Parquet
src/data_services/insert_data.py - Database loader → DB connection: opens connection
src/data_services/insert_data.py - DB connection → PostgreSQL: connects
src/db_settings/connection.py - Schema initializer → DB connection: runs schema setup
main.py - DB connection → Database schema: reads SQL
src/db_settings/connection.py - DB connection → PostgreSQL: executes schema
src/db_settings/connection.py - Database loader → PostgreSQL: writes snapshot records
src/data_services/insert_data.py - Analytics views → PostgreSQL: defines query views
README.md