How finkenils/processed-ais-data-baltic-sea-2020-v2 works
Project Type & Purpose
• This is a data processing and analytics project that ingests AIS signals (collected externally by the Danish Maritime Authority) and then processes and aggregates vessel data to produce two CSV outputs: supply.csv and idle.csv.
• It also includes a separate data file with geographical polygon definitions (stored in a pickle file) which is used to identify relevant coastal zones.
Main Components
a. External Data Sources:
• AIS Signals: Although the raw signals are not stored in the repository, they are mentioned as the original source. Represent this as an external data source.
• Vessel Database: Referenced (e.g., Vesselfinder) that provides the vessel deadweight (dwt) details.
b. Data Input/Storage:
• Processed Data Files: CSV files (supply.csv, idle.csv, zones.csv) and the polygon definitions file (baltic_sea_coastal_line_polygons.pkl) that serve as both input (for geospatial definitions) and output (aggregated results).
c. Processing Pipeline:
• Data Extraction Module: Gathers and pre-processes the required AIS parameters including timestamp, latitude, longitude, imo, mmsi, draught, and destination.
• Voyage Identification & Transformation Module: This module takes individual vessel journeys (ordered by timestamp) and determines where voyage changes occur (using changes in destination and increases in draft) to mark supply events.
• Geospatial Mapping Module: Uses the polygon definitions from the pickle file to map each event to the correct coastal zone.
• Aggregation Module: Processes the data into weekly aggregates — calculating the total cargo supplied (supply.csv) and computing average idle times within each zone (idle.csv).
d. Data Output:
• The CSV files (supply.csv and idle.csv) represent the aggregated results.
• The zones.csv file works in tandem with the polygon definitions to provide context to the geographic portions of the data.
Relationships and Data Flow
• Directional flow where raw AIS signals (with key vessel data) are processed to identify voyage segments.
• That once a voyage is identified, the processing flow splits:
• One stream maps vessel location data to zones using the Geospatial Mapping Module.
• Another stream processes vessel specific details (like draft changes) to flag the start of supply.
• From these modules, data flows into Aggregation, where calculations for supply in tons and idle times per week per zone are performed.
Architectural Patterns & Design Principles
• “pipeline” approach, where data moves sequentially through extraction, processing, mapping, and aggregation.