# finkenils/processed-ais-data-baltic-sea-2020-v2 architecture

> Architecture diagram of the GitHub repository finkenils/processed-ais-data-baltic-sea-2020-v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/finkenils/processed-ais-data-baltic-sea-2020-v2
- Repository: https://github.com/finkenils/processed-ais-data-baltic-sea-2020-v2

## Overview

##### Project Type & Purpose
- This is a data processing and analytics project that ingests AIS signals (collected externally by the Danish Maritime Authority) and then processes and aggregates vessel data to produce two CSV outputs: supply.csv and idle.csv.
- It also includes a separate data file with geographical polygon definitions (stored in a pickle file) which is used to identify relevant coastal zones.

##### Main Components
a. External Data Sources:
- AIS Signals: Although the raw signals are not stored in the repository, they are mentioned as the original source. Represent this as an external data source.
- Vessel Database: Referenced (e.g., Vesselfinder) that provides the vessel deadweight (dwt) details.
b. Data Input/Storage:
- Processed Data Files: CSV files (supply.csv, idle.csv, zones.csv) and the polygon definitions file (baltic_sea_coastal_line_polygons.pkl) that serve as both input (for geospatial definitions) and output (aggregated results).
c. Processing Pipeline:
- Data Extraction Module: Gathers and pre-processes the required AIS parameters including timestamp, latitude, longitude, imo, mmsi, draught, and destination.
- Voyage Identification & Transformation Module: This module takes individual vessel journeys (ordered by timestamp) and determines where voyage changes occur (using changes in destination and increases in draft) to mark supply events.
- Geospatial Mapping Module: Uses the polygon definitions from the pickle file to map each event to the correct coastal zone.
- Aggregation Module: Processes the data into weekly aggregates — calculating the total cargo supplied (supply.csv) and computing average idle times within each zone (idle.csv).
d. Data Output:
- The CSV files (supply.csv and idle.csv) represent the aggregated results.
- The zones.csv file works in tandem with the polygon definitions to provide context to the geographic portions of the data.

##### Relationships and Data Flow
- Directional flow where raw AIS signals (with key vessel data) are processed to identify voyage segments.
- That once a voyage is identified, the processing flow splits:
- One stream maps vessel location data to zones using the Geospatial Mapping Module.
- Another stream processes vessel specific details (like draft changes) to flag the start of supply.
- From these modules, data flows into Aggregation, where calculations for supply in tons and idle times per week per zone are performed.

##### Architectural Patterns & Design Principles
- “pipeline” approach, where data moves sequentially through extraction, processing, mapping, and aggregation.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data Sources
    subgraph "External Data Sources"
        A["AIS Signals"]:::external
        B["Vessel Database"]:::external
    end

    %% Data Input/Storage
    subgraph "Data Input/Storage"
        F["Preprocessed Geospatial Definitions"]:::input
    end

    %% Data Processing Pipeline
    subgraph "Data Processing Pipeline"
        C["Data Extraction Module"]:::processing
        D["Voyage Identification & Transformation Module"]:::processing
        E["Geospatial Mapping Module"]:::processing
        G["Aggregation Module"]:::processing
        V["Visualization Module"]:::processing
    end

    %% Data Outputs
    subgraph "Data Output"
        H["Supply Output (supply.csv)"]:::output
        I["Idle Output (idle.csv)"]:::output
        J["Zones Context (zones.csv)"]:::output
    end

    %% Flow Connections
    A -->|"feeds"| C
    B -->|"feeds"| C
    C -->|"processes"| D
    D -->|"location_flow"| E
    D -->|"voyage_details"| G
    F -->|"polygon_defs"| E
    E -->|"mapped_data"| G
    G -->|"supply_aggregation"| H
    G -->|"idle_aggregation"| I
    G -->|"zone_context"| J
    G -->|"visualization_data"| V

    %% Click Events
    click F "https://github.com/finkenils/processed-ais-data-baltic-sea-2020-v2/blob/main/data/baltic_sea_coastal_line_polygons.pkl"
    click H "https://github.com/finkenils/processed-ais-data-baltic-sea-2020-v2/blob/main/data/supply.csv"
    click I "https://github.com/finkenils/processed-ais-data-baltic-sea-2020-v2/blob/main/data/idle.csv"
    click J "https://github.com/finkenils/processed-ais-data-baltic-sea-2020-v2/blob/main/data/zones.csv"

    %% Styles
    classDef external fill:#AEE1F9,stroke:#333,stroke-width:2px;
    classDef input fill:#FFF3B0,stroke:#333,stroke-width:2px;
    classDef processing fill:#C8E6C9,stroke:#333,stroke-width:2px;
    classDef output fill:#F8BBD0,stroke:#333,stroke-width:2px;
```
