# ocean-data-factory-sweden/kso architecture

> Architecture diagram of the GitHub repository ocean-data-factory-sweden/kso, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ocean-data-factory-sweden/kso
- Repository: https://github.com/ocean-data-factory-sweden/kso

## Overview

##### Project Type & Purpose
- This is an open-source, citizen science and machine learning system for analyzing subsea movies. It provides an end-to-end pipeline that moves and processes underwater footage (with associated metadata), integrates with the Zooniverse citizen science platform, and provides facilities for training and evaluating ML models (using YOLO variants) via modular Jupyter Notebooks.
- The system is used by marine scientists and citizen scientists to upload, annotate, process, analyze, and eventually publish analysis of underwater footage.

##### File Structure – Key Observations
- Top-level files and directories (e.g., README.md, Dockerfile, conda_env.yaml) indicate that the project supports containerized and conda-dependent environments.
- The “notebooks” directory is organized into four main functional groups: setup, classify, analyse, and publish. Each group represents a stage in the workflow.
- The “kso_utils” package contains key utility modules for operations such as database initialization (“db_starter”), frame and video processing, interfacing with external platforms (e.g., Zooniverse, Zenodo), and specialized tools for machine learning (e.g., YOLO, tracking, etc.).
- The “src” directory contains executable scripts that appear to be used for running real-time processing tasks such as detection, tracking, and data preparation.
- There are also examples and tests that illustrate usage and validate functionality.

a. Main Components
- User Interface & Workflow: Represented by the modular Jupyter Notebooks. This interface supports different tasks:
– Setup (e.g., metadata checking)
– Classification (upload footage, process classifications via Zooniverse)
– Analysis (training and evaluating models)
– Publishing (publishing models and exporting observations)
- Processing & Utilities Module (kso_utils): Contains libraries for data manipulation, video processing, interfacing with databases (via CSV templates in “db_starter”), and integrating with ML workflows.
- Core Processing Scripts (src): Scripts for detection, tracking, dataset preparation, and model execution. These perform the “heavy lifting” in video and image analysis using models like YOLO and other computer vision/ML techniques.
- External Integrations:
– Zooniverse: For citizen science annotations and classifications.
– Cloudina: A hosted JupyterHub-based environment supporting storage, notebooks (album), and MLFlow (vendor).
– Possibly other external data repositories (e.g., GBIF via “Publish_observations”).
- Supporting Infrastructure:
– Docker & Conda for environment reproducibility.
– GitHub workflows for CI/CD (linting, build/test pipelines).
– CSV “database” files and schema in the db_starter folder that initialize or seed projects.

b. Relationships & Data Flow
- Data Input: Underwater footage (videos, images) and associated metadata in CSV files flow into the system.
– The “db_starter” component sets up initial project details.
- Notebook Frontend: Users execute notebooks for data preparation, classification, analysis, and publishing.
– These tasks trigger utilities in kso_utils and processing scripts in src.
- ML & Processing Pipeline: Functions within kso_utils (e.g., for video reading, YOLO integrations, frame tracking) are used by scripts to process the media.
- External Services: Processed data is sent to Zooniverse (citizen science platform) for annotation and may also be published via Cloudina endpoints.
- CI/CD & Environment: Dockerfile, conda_env.yaml, and GitHub workflows support building, testing, and deploying the application.

##### Key Architectural Patterns & Design Principles
- Extensibility: The system is structured to allow researchers to plug in custom notebooks or extend utilities (e.g., support for new detectors).
- Integration with External Services: Clear concerns for data ingestion and output when interacting with external platforms (Zooniverse, Cloudina).
- Environment Agnosticism: Use of Docker and Conda indicates cross-platform deployments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion
    A["Underwater Footage & Metadata"]:::input

    %% User Interface Layer
    subgraph "User Interface (Jupyter Notebooks)"
        B["Setup"]:::notebook
        C["Classify"]:::notebook
        D["Analyse"]:::notebook
        E["Publish"]:::notebook
    end

    %% Processing & Utilities Module
    F["Utility Modules (kso_utils)"]:::utils
    G["DB Starter (CSV & Schema)"]:::utils

    %% Core Processing Scripts
    H["Processing Scripts (src)"]:::scripts

    %% External Integrations
    subgraph "External Integrations"
        I["Zooniverse API"]:::external
        J["Zenodo API"]:::external
    end

    %% Infrastructure & CI/CD
    subgraph "Infrastructure & CI/CD"
        K["Dockerfile"]:::infra
        L["conda_env.yaml"]:::infra
        M["GitHub Workflows"]:::infra
    end

    %% Flow connections
    A -->|"ingests"| B
    A -->|"ingests"| C
    A -->|"ingests"| D
    A -->|"ingests"| E

    %% Notebooks call utility functions
    B -->|"triggers"| F
    C -->|"triggers"| F
    D -->|"triggers"| F
    E -->|"triggers"| F

    %% Utility modules depend on DB Starter
    F -->|"initializes"| G

    %% Utilities call processing scripts
    F -->|"invokes"| H

    %% Processing scripts interact with External Integrations
    H -->|"sends data"| I
    H -->|"sends data"| J

    %% Also utilities interface with External Integrations
    F -->|"interfaces"| I
    F -->|"interfaces"| J

    %% External Integrations send results back to Notebooks
    I -->|"annotated data"| E
    J -->|"published info"| E

    %% Infrastructure supports all layers (side connectivity)
    K --- A
    K --- B
    L --- F
    M --- H

    %% Click Events for Component Mapping
    click B "https://github.com/ocean-data-factory-sweden/kso/tree/dev/notebooks/setup"
    click C "https://github.com/ocean-data-factory-sweden/kso/tree/dev/notebooks/classify"
    click D "https://github.com/ocean-data-factory-sweden/kso/tree/dev/notebooks/analyse"
    click E "https://github.com/ocean-data-factory-sweden/kso/tree/dev/notebooks/publish"
    click F "https://github.com/ocean-data-factory-sweden/kso/tree/dev/kso_utils/"
    click G "https://github.com/ocean-data-factory-sweden/kso/tree/dev/kso_utils/db_starter/"
    click H "https://github.com/ocean-data-factory-sweden/kso/tree/dev/src/"
    click I "https://github.com/ocean-data-factory-sweden/kso/blob/dev/kso_utils/zooniverse_utils.py"
    click J "https://github.com/ocean-data-factory-sweden/kso/blob/dev/kso_utils/zenodo_utils.py"
    click K "https://github.com/ocean-data-factory-sweden/kso/tree/dev/Dockerfile"
    click L "https://github.com/ocean-data-factory-sweden/kso/blob/dev/conda_env.yaml"
    click M "https://github.com/ocean-data-factory-sweden/kso/tree/dev/.github/"

    %% Styles
    classDef input fill:#e1f5fe,stroke:#0277bd,stroke-width:2px;
    classDef notebook fill:#fff9c4,stroke:#f9a825,stroke-width:2px,stroke-dasharray: 5 5;
    classDef utils fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef scripts fill:#ffcdd2,stroke:#c62828,stroke-width:2px;
    classDef external fill:#d1c4e9,stroke:#5e35b1,stroke-width:2px,stroke-dasharray: 2 2;
    classDef infra fill:#f0f4c3,stroke:#9e9d24,stroke-width:2px;
```
