ameliacmu/ameliaswim

1. What this repository is

AmeliaSWIM is a data preparation pipeline, not an application server. Its job is to turn raw FAA SWIM trajectory feeds into research-ready airport-surface movement datasets used by Amelia/Amelia-10/Amelia-42.

2. Main external dependency boundary

The pipeline starts from raw .njson.gz files downloaded from a public OpenStack Swift dataset. Everything else in the repo is about downloading, filtering, geometrically constraining, and converting those raw records into CSV-based scene windows.

3. Core processing path

The central flow is:

raw SWIM files → airport/month configuration selection → filtering and coordinate conversion → windowed CSV outputs.

The key executable for this is process.py, with process_parrallel.py providing the same logical pipeline for multi-core execution.

4. Configuration-driven architecture

The repo is heavily config-based. conf/config.yaml, conf/data/*, and conf/airports/* define the processing run.

- Month configs determine the time range and dataset slice.

- Airport configs define airport metadata such as reference latitude/longitude, altitude filtering, and geofence selection.

This makes the pipeline parameterized by airport and time period rather than hardcoded.

5. Airport ROI / geofence subsystem

Airport boundary logic lives in conf/fences/*.csv and conf/airports/*.yaml. These define the region of interest used to keep only relevant surface movement around each airport.

The repository treats the geofence as a first-class artifact: filtering is not just temporal, but spatial.

6. GIS asset subsystem

conf/qgist/ contains QGIS project and GIS source assets (.qgz, .shp, .osm, .dbf, .prj, etc.). These are the source-of-truth map artifacts used to create or edit airport polygons and ROI boundaries.

In the architecture, this is a separate GIS authoring boundary feeding the preprocessing pipeline.

7. Raw-data acquisition subsystem

download_raw.py handles fetching raw data over a time interval into a local directory such as swim_data/.

This is the ingestion boundary between the external Swift-hosted dataset and local processing storage.

8. Data transformation subsystem

The transform stage converts trajectory messages from raw JSON-like records into structured CSVs. Based on the configs described in the README, this stage:

- filters by airport and time window,

- applies altitude limits,

- converts to Cartesian coordinates using airport reference lat/lon,

- writes segmented CSV files for downstream model training and benchmarking.

9. Batch and scale-out execution

The scripts/ directory contains orchestration helpers for larger runs and HPC environments:

- run.sbatch, run_pair.sbatch, run_all.sh, run_all_slurm.sh

- load_azure.bash

- utility scripts like gen_split.py, gen_unix_stamps.py, compare_csvs.py

These are operational wrappers around the same preprocessing pipeline, used to scale dataset generation and validate outputs.

10. Datasets and output organization

The repo includes config/data collections for two major dataset families:

- Amelia10: benchmark subsets by month/airport

- Amelia42: the larger 42-airport corpus, split into many run-specific YAML fragments

The expected output of the pipeline is processed CSV data plus associated map assets, organized so downstream forecasting experiments can consume them consistently.

11. Validation and reproducibility boundary

test.py, compare_csvs.py, and the versioned config/assets indicate an emphasis on reproducible dataset generation. The repository’s outputs are meant to be deterministic for a given airport, time range, and configuration set.

12. Packaging/runtime

This is a Python project packaged with pyproject.toml, with environment definition in environment.yml. The repo is meant to be run locally or on batch compute, with configuration overrides supplied at launch time in a Hydra-like style.