# ocean-data-lab/ocean-data-analysis architecture

> Architecture diagram of the GitHub repository ocean-data-lab/ocean-data-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ocean-data-lab/ocean-data-analysis
- Repository: https://github.com/ocean-data-lab/ocean-data-analysis

## Overview

##### Project Type and Purpose
- This is a data science / machine learning research project focused on marine acoustics. The goal is to extract isolated ship events from hydrophone audio data (supplemented with AIS and image data) and build an ML classification model for vessel types.
- The README explains the challenges with existing ship image datasets and how audio data can overcome those issues. It also describes the methods for isolating ship recordings from the background marine noise.

##### File Structure
- Notebooks: There are multiple Jupyter notebooks (e.g., “AIS data.ipynb”, “EDA_Central_Caldera.ipynb”, “Benchmark data-Offshore.ipynb”) indicating exploratory data analysis, experiments, and ML training.
- Python Files: The “functions.py” and __init__.py suggest shared code for common processing functions.
- Data Files: CSV files (e.g., vessel.csv, vessel_groups.csv, vessel_k5_1f.csv, etc.) contain AIS or related vessel metadata.
- Images: Directories like “images” and “ship_images” contain reference images for various ship types, supporting classification tasks.
- Checkpoints and temporary files: These indicate active development through Jupyter notebooks.

##### Core Components
A. Data Sources:
- AIS Data: CSV files that provide vessel identification, GPS coordinates, and timestamps.
- Hydrophone Recordings: Although not directly in the file tree, the README explains that audio data captured by hydrophones is a key input.
- Ship Images: Used to enrich the classification process or for validation.

B. Data Ingestion and Preprocessing:
- Data Import Modules: Code (in “functions.py” and notebooks) to load CSV data and potentially audio recordings.
- Preprocessing Steps: Filtering and cleaning of AIS data based on the three provided approaches (detailed in the README) to identify isolated ship events.
- Feature Extraction: Transform raw AIS data (and possibly audio features) into suitable inputs for ML training.

C. Exploratory Data Analysis (EDA) & Experimental Work:
- Jupyter Notebooks: A series of notebooks dedicated to exploring the data (e.g., “EDA _Axial.ipynb”, “EDA_Eastern_cald.ipynb”) and performing benchmarking.

D. Machine Learning Pipeline:
- Model Training: Notebooks and scripts where the ML model is built, training on the benchmark dataset.
- Model Evaluation: Analysis of results with performance metrics displayed or logged.
- Feedback Loop: Iteration and refinement guided by controlled experiments and threshold filtering (using dt and dt2 as described).

E. Dataset Generation and Publication:
- Output Datasets: The generation of public benchmark datasets of isolated ship recordings.
- Documentation: The README serves as part of the documentation and explanation for end users and contributors.

##### Relationships and Interactions
- ML Pipeline: From the EDA block, data is fed into model training and evaluation modules.

##### Architectural Design Patterns and Technologies
- Separation of Concerns: Although all components exist in a “flat” repository, conceptually separate the ingestion, preprocessing, analysis, and ML model training.
- Modular Design: Show common modules (e.g., functions.py) being used across different notebooks.
- Technologies: Note that the system is developed in Python using Jupyter Notebook as the primary interface. Additional libraries (like pandas, scikit-learn, matplotlib) may be inferred.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        subgraph "AIS Data (CSV)"
            file1["vessel.csv"]:::csv
            file2["vessel_groups.csv"]:::csv
            file3["vessel_k5_1f.csv"]:::csv
            file4["vessel_k8_1f.csv"]:::csv
            file5["vessel_k8_2f.csv"]:::csv
        end
        subgraph "Ship Images"
            images1["images"]:::dir
            images2["ship_images"]:::dir
        end
    end

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        proc1["functions.py"]:::file
        proc2["__init__.py"]:::file
    end

    %% Exploratory Data Analysis (EDA) & Experimental Work
    subgraph "Exploratory Data Analysis (EDA) & Experimental Work"
        n1["AIS data.ipynb"]:::ipynb
        n2["AIS data slope and hydrate.ipynb"]:::ipynb
        n3["EDA _Axial.ipynb"]:::ipynb
        n4["EDA_Central_Caldera.ipynb"]:::ipynb
        n5["EDA_Eastern_cald.ipynb"]:::ipynb
        n6["EDA _oregon_slope.ipynb"]:::ipynb
        n7["Benchmark data-Axial_Caldera.ipynb"]:::ipynb
        n8["Benchmark data-Offshore.ipynb"]:::ipynb
        n9["experiments.ipynb"]:::ipynb
        n10["spdf_isolated_ships.ipynb"]:::ipynb
    end

    %% Machine Learning Pipeline
    subgraph "Machine Learning Pipeline"
        ml1["Benchmark data-Offshore.ipynb"]:::ipynb
    end

    %% Dataset Generation & Publication
    subgraph "Dataset Generation & Publication"
        out1["Output CSV Datasets"]:::csv
        doc["README.md"]:::doc
    end

    %% Data Flow Connections
    file1 -->|"raw_data"| proc1
    images1 --> proc1
    images2 --> proc1
    proc1 -->|"processed_data"| n1
    n8 -->|"ml_input"| ml1
    ml1 -->|"results"| out1
    ml1 --> doc
    ml1 -->|"feedback"| proc1

    %% Class Definitions for Colour Coding
    classDef csv fill:#ffcccc,stroke:#333,stroke-width:2px;
    classDef dir fill:#ffe6cc,stroke:#333,stroke-width:2px;
    classDef file fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef ipynb fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef doc fill:#ffffcc,stroke:#333,stroke-width:2px;

    %% Click Events for Data Sources (AIS CSV)
    click file1 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/vessel.csv"
    click file2 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/vessel_groups.csv"
    click file3 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/vessel_k5_1f.csv"
    click file4 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/vessel_k8_1f.csv"
    click file5 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/vessel_k8_2f.csv"

    %% Click Events for Ship Images
    click images1 "https://github.com/ocean-data-lab/ocean-data-analysis/tree/main/images"
    click images2 "https://github.com/ocean-data-lab/ocean-data-analysis/tree/main/ship_images"

    %% Click Events for Data Ingestion & Preprocessing Module
    click proc1 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/functions.py"
    click proc2 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/__init__.py"

    %% Click Events for Exploratory Data Analysis (EDA) & Experimental Work Notebooks
    click n1 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/AIS data.ipynb"
    click n2 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/AIS data slope and hydrate.ipynb"
    click n3 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/EDA _Axial.ipynb"
    click n4 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/EDA_Central_Caldera.ipynb"
    click n5 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/EDA_Eastern_cald.ipynb"
    click n6 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/EDA _oregon_slope.ipynb"
    click n7 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/Benchmark data-Axial_Caldera.ipynb"
    click n8 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/Benchmark data-Offshore.ipynb"
    click n9 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/experiments.ipynb"
    click n10 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/spdf_isolated_ships.ipynb"

    %% Click Event for Machine Learning Pipeline Module
    click ml1 "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/Benchmark data-Offshore.ipynb"

    %% Click Event for Documentation
    click doc "https://github.com/ocean-data-lab/ocean-data-analysis/blob/main/README.md"
```
