# ameliacmu/ameliaswim architecture

> Architecture diagram of the GitHub repository ameliacmu/ameliaswim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ameliacmu/ameliaswim
- Repository: https://github.com/ameliacmu/ameliaswim

## Overview

1. **What this repository is**
   AmeliaSWIM is a **data preparation pipeline**, not an application server. Its job is to turn raw FAA SWIM trajectory feeds into research-ready airport-surface movement datasets used by Amelia/Amelia-10/Amelia-42.

2. **Main external dependency boundary**
   The pipeline starts from **raw `.njson.gz` files** downloaded from a **public OpenStack Swift** dataset. Everything else in the repo is about downloading, filtering, geometrically constraining, and converting those raw records into CSV-based scene windows.

3. **Core processing path**
   The central flow is:
   **raw SWIM files → airport/month configuration selection → filtering and coordinate conversion → windowed CSV outputs**.
   The key executable for this is `process.py`, with `process_parrallel.py` providing the same logical pipeline for multi-core execution.

4. **Configuration-driven architecture**
   The repo is heavily config-based. `conf/config.yaml`, `conf/data/*`, and `conf/airports/*` define the processing run.
   - **Month configs** determine the time range and dataset slice.
   - **Airport configs** define airport metadata such as reference latitude/longitude, altitude filtering, and geofence selection.
   This makes the pipeline parameterized by airport and time period rather than hardcoded.

5. **Airport ROI / geofence subsystem**
   Airport boundary logic lives in `conf/fences/*.csv` and `conf/airports/*.yaml`. These define the region of interest used to keep only relevant surface movement around each airport.
   The repository treats the geofence as a first-class artifact: filtering is not just temporal, but spatial.

6. **GIS asset subsystem**
   `conf/qgist/` contains QGIS project and GIS source assets (`.qgz`, `.shp`, `.osm`, `.dbf`, `.prj`, etc.). These are the source-of-truth map artifacts used to create or edit airport polygons and ROI boundaries.
   In the architecture, this is a separate GIS authoring boundary feeding the preprocessing pipeline.

7. **Raw-data acquisition subsystem**
   `download_raw.py` handles fetching raw data over a time interval into a local directory such as `swim_data/`.
   This is the ingestion boundary between the external Swift-hosted dataset and local processing storage.

8. **Data transformation subsystem**
   The transform stage converts trajectory messages from raw JSON-like records into structured CSVs. Based on the configs described in the README, this stage:
   - filters by airport and time window,
   - applies altitude limits,
   - converts to Cartesian coordinates using airport reference lat/lon,
   - writes segmented CSV files for downstream model training and benchmarking.

9. **Batch and scale-out execution**
   The `scripts/` directory contains orchestration helpers for larger runs and HPC environments:
   - `run.sbatch`, `run_pair.sbatch`, `run_all.sh`, `run_all_slurm.sh`
   - `load_azure.bash`
   - utility scripts like `gen_split.py`, `gen_unix_stamps.py`, `compare_csvs.py`
   These are operational wrappers around the same preprocessing pipeline, used to scale dataset generation and validate outputs.

10. **Datasets and output organization**
   The repo includes config/data collections for two major dataset families:
   - **Amelia10**: benchmark subsets by month/airport
   - **Amelia42**: the larger 42-airport corpus, split into many run-specific YAML fragments
   The expected output of the pipeline is processed CSV data plus associated map assets, organized so downstream forecasting experiments can consume them consistently.

11. **Validation and reproducibility boundary**
   `test.py`, `compare_csvs.py`, and the versioned config/assets indicate an emphasis on reproducible dataset generation. The repository’s outputs are meant to be deterministic for a given airport, time range, and configuration set.

12. **Packaging/runtime**
   This is a **Python** project packaged with `pyproject.toml`, with environment definition in `environment.yml`. The repo is meant to be run locally or on batch compute, with configuration overrides supplied at launch time in a Hydra-like style.

Main entry points: [`amelia_swim/process.py`](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/process.py) (Main process), [`scripts/run_all_slurm.sh`](https://github.com/ameliacmu/ameliaswim/blob/main/scripts/run_all_slurm.sh) (HPC wrappers), [`scripts/compare_csvs.py`](https://github.com/ameliacmu/ameliaswim/blob/main/scripts/compare_csvs.py) (CSV checks), [`amelia_swim/conf/config.yaml`](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/config.yaml) (Base config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["External data"]
  node_n1[("Swift SWIM<br/>remote dataset")]
  node_n2[("Local cache<br/>staging dir")]
end

subgraph group_g2["Config & GIS"]
  node_n3["Base config<br/>runtime config<br/>[config.yaml]"]
  node_n4["Dataset slices<br/>month configs<br/>[default.yaml]"]
  node_n5["Airport configs<br/>airport metadata<br/>[kdca.yaml]"]
  node_n6["Geofences<br/>roi boundaries<br/>[kdca.csv]"]
  node_n7["GIS assets<br/>qgis source<br/>[maps.qgz]"]
end

subgraph group_g3["Processing"]
  node_n8["Raw downloader<br/>ingest script<br/>[download_raw.py]"]
  node_n9["YAML generator<br/>config tool<br/>[generate_yaml.py]"]
  node_n10["Main process<br/>pipeline driver<br/>[process.py]"]
  node_n11["Parallel process<br/>parallel driver"]
  node_n12{{"Filter & gate<br/>transform stage"}}
  node_n13{{"Coordinate convert<br/>transform stage"}}
  node_n14{{"Window builder<br/>segmentation stage"}}
  node_n15[("Processed CSVs<br/>dataset output")]
end

subgraph group_g4["Ops & validation"]
  node_n16["HPC wrappers<br/>batch scripts<br/>[run_all_slurm.sh]"]
  node_n17["Azure load<br/>ops script<br/>[load_azure.bash]"]
  node_n18["CSV checks<br/>validation tools<br/>[compare_csvs.py]"]
end

node_n1 -->|"source"| node_n8
node_n8 -->|"writes"| node_n2
node_n3 -->|"drives"| node_n10
node_n4 -->|"selects"| node_n10
node_n5 -->|"parameterizes"| node_n10
node_n6 -->|"bounds"| node_n10
node_n7 -->|"authoring source"| node_n6
node_n9 -->|"generates"| node_n3
node_n2 -->|"input"| node_n10
node_n10 -->|"executes"| node_n12
node_n11 -->|"executes"| node_n12
node_n12 -->|"feeds"| node_n13
node_n13 -->|"feeds"| node_n14
node_n14 -->|"produces"| node_n15
node_n16 -->|"wraps"| node_n10
node_n16 -->|"wraps"| node_n11
node_n17 -->|"supports"| node_n2
node_n18 -->|"checks"| node_n15
node_n18 -->|"inspects"| node_n14

click node_n3 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/config.yaml"
click node_n4 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/data/default.yaml"
click node_n5 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/airports/kdca.yaml"
click node_n6 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/fences/kdca.csv"
click node_n7 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/qgist/maps.qgz"
click node_n8 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/download_raw.py"
click node_n9 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/generate_yaml.py"
click node_n10 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/process.py"
click node_n11 "https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/process_parrallel.py"
click node_n16 "https://github.com/ameliacmu/ameliaswim/blob/main/scripts/run_all_slurm.sh"
click node_n17 "https://github.com/ameliacmu/ameliaswim/blob/main/scripts/load_azure.bash"
click node_n18 "https://github.com/ameliacmu/ameliaswim/blob/main/scripts/compare_csvs.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2 toneBlue
class node_n3,node_n4,node_n5,node_n6,node_n7 toneAmber
class node_n8,node_n9,node_n10,node_n11,node_n12,node_n13,node_n14,node_n15 toneMint
class node_n16,node_n17,node_n18 toneRose
```

## Components

### External data

Raw FAA SWIM trajectory files and local staging for fetched inputs.

- **Swift SWIM**: remote dataset · Public OpenStack Swift source of raw .njson.gz trajectory feeds.
- **Local cache**: staging dir · Downloaded raw SWIM files stored locally for processing.

### Config & GIS

Airport, month, and geofence assets that parameterize dataset generation.

- [Base config](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/config.yaml): runtime config · Top-level pipeline defaults and shared configuration wiring. · `amelia_swim/conf/config.yaml`
- [Dataset slices](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/data/default.yaml): month configs · Amelia10 and Amelia42 run definitions that select time spans and corpus slices. · `amelia_swim/conf/data/default.yaml`
- [Airport configs](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/airports/kdca.yaml): airport metadata · Per-airport settings such as reference coordinates and altitude constraints. · `amelia_swim/conf/airports/kdca.yaml`
- [Geofences](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/fences/kdca.csv): roi boundaries · Airport region-of-interest polygons used to spatially filter movement records. · `amelia_swim/conf/fences/kdca.csv`
- [GIS assets](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/conf/qgist/maps.qgz): qgis source · QGIS projects and shapefile/osm assets used to author airport polygons. · `amelia_swim/conf/qgist/maps.qgz`

### Processing

Primary pipeline stages that download, filter, convert, and emit CSV windows.

- [Raw downloader](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/download_raw.py): ingest script · Fetches SWIM records over a requested time interval into local storage. · `amelia_swim/download_raw.py`
- [YAML generator](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/generate_yaml.py): config tool · Builds or expands processing configuration fragments for airports and dataset runs. · `amelia_swim/generate_yaml.py`
- [Main process](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/process.py): pipeline driver · Single-process end-to-end transformation from raw feeds to scene-window CSVs. · `amelia_swim/process.py`
- [Parallel process](https://github.com/ameliacmu/ameliaswim/blob/main/amelia_swim/process_parrallel.py): parallel driver · Multi-core variant of the same preprocessing pipeline. · `amelia_swim/process_parrallel.py`
- **Filter & gate**: transform stage · Applies airport, time-window, altitude, and ROI filtering to raw records.
- **Coordinate convert**: transform stage · Converts trajectories into airport-relative Cartesian coordinates.
- **Window builder**: segmentation stage · Segments processed trajectories into CSV scene windows for downstream use.
- **Processed CSVs**: dataset output · Final structured CSV outputs consumed by Amelia forecasting experiments.

### Ops & validation

Batch wrappers and output comparison helpers for scale-out runs.

- [HPC wrappers](https://github.com/ameliacmu/ameliaswim/blob/main/scripts/run_all_slurm.sh): batch scripts · SBATCH and shell entrypoints for repeated or large-scale dataset generation. · `scripts/run_all_slurm.sh`
- [Azure load](https://github.com/ameliacmu/ameliaswim/blob/main/scripts/load_azure.bash): ops script · Helper for moving data to or from cloud-backed batch environments. · `scripts/load_azure.bash`
- [CSV checks](https://github.com/ameliacmu/ameliaswim/blob/main/scripts/compare_csvs.py): validation tools · Utilities for splitting, timestamp generation, and output comparison. · `scripts/compare_csvs.py`

## Connections

- Swift SWIM → Raw downloader: source · Raw SWIM feeds are fetched by the downloader.
- Raw downloader → Local cache: writes · Downloaded files are stored in local staging.
- Base config → Main process: drives · Base configuration wires the main pipeline.
- Dataset slices → Main process: selects · Dataset slices determine the temporal and corpus scope.
- Airport configs → Main process: parameterizes · Airport metadata configures filtering and conversion.
- Geofences → Main process: bounds · Geofence polygons constrain the spatial filter.
- GIS assets → Geofences: authoring source · GIS assets feed the maintained fence boundaries.
- YAML generator → Base config: generates · YAML generation prepares run configuration fragments.
- Local cache → Main process: input · Local raw files are consumed by the processing driver.
- Main process → Filter & gate: executes · The main driver orchestrates filtering.
- Parallel process → Filter & gate: executes · The parallel driver runs the same logical stages.
- Filter & gate → Coordinate convert: feeds · Filtered trajectory records are converted to local coordinates.
- Coordinate convert → Window builder: feeds · Coordinate-transformed tracks are segmented into windows.
- Window builder → Processed CSVs: produces · Windowed CSV outputs are written for downstream experiments.
- HPC wrappers → Main process: wraps · Batch scripts launch the main preprocessing pipeline.
- HPC wrappers → Parallel process: wraps · Batch scripts also launch the parallel variant.
- Azure load → Local cache: supports · Cloud transfer helpers support staging input data.
- CSV checks → Processed CSVs: checks · Validation tools compare generated CSV outputs.
- CSV checks → Window builder: inspects · Utilities help verify segmentation and timestamping behavior.