How ocean-data-lab/ocean-data-analysis works
Project Type and Purpose
• This is a data science / machine learning research project focused on marine acoustics. The goal is to extract isolated ship events from hydrophone audio data (supplemented with AIS and image data) and build an ML classification model for vessel types.
• The README explains the challenges with existing ship image datasets and how audio data can overcome those issues. It also describes the methods for isolating ship recordings from the background marine noise.
File Structure
• Notebooks: There are multiple Jupyter notebooks (e.g., “AIS data.ipynb”, “EDA_Central_Caldera.ipynb”, “Benchmark data-Offshore.ipynb”) indicating exploratory data analysis, experiments, and ML training.
• Python Files: The “functions.py” and __init__.py suggest shared code for common processing functions.
• Data Files: CSV files (e.g., vessel.csv, vessel_groups.csv, vessel_k5_1f.csv, etc.) contain AIS or related vessel metadata.
• Images: Directories like “images” and “ship_images” contain reference images for various ship types, supporting classification tasks.
• Checkpoints and temporary files: These indicate active development through Jupyter notebooks.
Core Components
A. Data Sources:
• AIS Data: CSV files that provide vessel identification, GPS coordinates, and timestamps.
• Hydrophone Recordings: Although not directly in the file tree, the README explains that audio data captured by hydrophones is a key input.
• Ship Images: Used to enrich the classification process or for validation.
B. Data Ingestion and Preprocessing:
• Data Import Modules: Code (in “functions.py” and notebooks) to load CSV data and potentially audio recordings.
• Preprocessing Steps: Filtering and cleaning of AIS data based on the three provided approaches (detailed in the README) to identify isolated ship events.
• Feature Extraction: Transform raw AIS data (and possibly audio features) into suitable inputs for ML training.
C. Exploratory Data Analysis (EDA) & Experimental Work:
• Jupyter Notebooks: A series of notebooks dedicated to exploring the data (e.g., “EDA _Axial.ipynb”, “EDA_Eastern_cald.ipynb”) and performing benchmarking.
D. Machine Learning Pipeline:
• Model Training: Notebooks and scripts where the ML model is built, training on the benchmark dataset.
• Model Evaluation: Analysis of results with performance metrics displayed or logged.
• Feedback Loop: Iteration and refinement guided by controlled experiments and threshold filtering (using dt and dt2 as described).
E. Dataset Generation and Publication:
• Output Datasets: The generation of public benchmark datasets of isolated ship recordings.
• Documentation: The README serves as part of the documentation and explanation for end users and contributors.
Relationships and Interactions
• ML Pipeline: From the EDA block, data is fed into model training and evaluation modules.
Architectural Design Patterns and Technologies
• Separation of Concerns: Although all components exist in a “flat” repository, conceptually separate the ingestion, preprocessing, analysis, and ML model training.
• Modular Design: Show common modules (e.g., functions.py) being used across different notebooks.
• Technologies: Note that the system is developed in Python using Jupyter Notebook as the primary interface. Additional libraries (like pandas, scikit-learn, matplotlib) may be inferred.