# aoyamaxx/public-ais-stream-api architecture

> Architecture diagram of the GitHub repository aoyamaxx/public-ais-stream-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aoyamaxx/public-ais-stream-api
- Repository: https://github.com/aoyamaxx/public-ais-stream-api

## Overview

##### Project type and purpose
- This is a data‐pipeline / ETL‐style system (“AIS Data Collector with Cloud SQL”) that continuously ingests real‐time AIS WebSocket data from AISStream.io, writes it to a Google Cloud SQL (PostgreSQL) instance via a Cloud SQL Auth Proxy, and includes downstream batch processing and visualization components.

##### File structure
- Top‐level:
- Dockerfile, docker-compose.yml, Docker install/start/stop scripts → container orchestration
- ais_collector.py, lw_correction_service.py, combine_ship_data.py, download_tables.py → streaming ingestion & utilities
- manage_services.py → local orchestration helper
- data_process/ → separate ETL & visualization microservice, with its own Dockerfile, docker‐compose, scripts, and requirements
- navigational_status/ → one‐time or scheduled service loading static code tables into DB
- one_time_input/ & old_ais_data_one_time_clean/ → ad‐hoc or legacy data‐load scripts
- data/ → output artifacts (Parquet, CSV, HTML visualizations)
- analysis.ipynb & README.md → end‐user analysis and documentation

##### README for additional insights
- Real‐time WebSocket ingestion into `ships`, `ship_data`, `unknown_ships` tables
- Docker Compose brings up two containers: Python ingestion app + Cloud SQL Auth Proxy
- Data processing is decoupled: `data_process/process_ais_data.py` reads from Cloud SQL, writes Parquet, then `visualize_scrubber_data.py` generates HTML maps/plots
- Use cases: continuous ingestion, ad‐hoc batch analyses, geospatial visualizations

a. Main components
1. AISStream.io WebSocket API
2. Docker Compose “collector” service (Python app container)
3. Docker Compose “cloud_sql_proxy” service
4. Google Cloud SQL (PostgreSQL)
5. Data_Process Docker service (ETL & Visualization)
6. Data Lake / Artifact Store (Parquet files, CSVs, HTML outputs in `data/`)
7. Optional one‐time or scheduled services (navigational_status loader, legacy cleaning scripts)
8. User interfaces: Jupyter notebook, psql / Cloud Console query editor, HTML visualization front‐ends

b. Relationships and data flows
- AISStream.io WebSocket → Python Collector container (ais_collector.py)
- Collector → Cloud SQL via Auth Proxy container
- navigational_status loader → Cloud SQL (uploads code tables)
- Data_Process container:
- Extracts raw tables from Cloud SQL
- Transforms data (chunked processing, scrubber matching)
- Loads Parquet files into local `data/` directory
- Generates HTML visualizations in `data/`
- Users / Analysts:
- Query DB directly (psql or Cloud Console)
- Run analysis.ipynb against Cloud SQL
- View HTML reports in browser

c. Architectural patterns & design principles
- Microservices via Docker Compose: separation of concerns (ingest vs proxy vs ETL)
- Streaming ingestion + batch ETL pipeline
- Infrastructure as code: Dockerfiles and Compose for reproducibility
- Decoupled storage: Cloud SQL for raw, Parquet for analytical workloads
- Modularity: one‐time vs recurring processes isolated in subdirectories

d. Key technologies & frameworks
- Docker & Docker Compose
- Python (async WebSocket client, psycopg2/sqlalchemy)
- Google Cloud SQL (PostgreSQL) + Cloud SQL Auth Proxy
- pandas & pyarrow (Parquet)
- Jupyter Notebook for ad‐hoc analysis
- Folium / Bokeh / Plotly for map and Sankey visualizations

- Clear labels on each component (e.g., “Python Collector,” “Cloud SQL Proxy,” “Cloud SQL PG,” “ETL Service,” “Data Lake”)
- WebSocket → Collector → Proxy → DB
- DB → ETL → Parquet & HTML
- Analyst → psql/notebook → DB
- Ingestion (green), Storage (blue), Processing (orange), Visualization (purple), One‐time scripts (gray)

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Source
    AIS["AISStream.io WebSocket API"]:::external

    %% Ingestion Layer
    subgraph "Docker Compose: Collector Network"
        direction TB
        subgraph "Python Collector Service" 
            direction TB
            CollectorApp["ais_collector.py"]:::ingestion
            LW["lw_correction_service.py"]:::ingestion
            Combine["combine_ship_data.py"]:::ingestion
            Download["download_tables.py"]:::ingestion
            Manage["manage_services.py"]:::ingestion
            DockerfileC["Dockerfile"]:::ingestion
            ComposeC["docker-compose.yml"]:::ingestion
            StartS["start_services.sh"]:::ingestion
            StopS["stop_services.sh"]:::ingestion
            InstallS["docker_install.sh"]:::ingestion
        end
        AuthProxy["Cloud SQL Auth Proxy"]:::ingestion
    end

    %% Storage Layer
    CloudSQL["Google Cloud SQL (PostgreSQL)"]:::storage

    %% Batch Processing Layer
    subgraph "ETL & Visualization Service"
        direction TB
        DockerfileP["data_process/Dockerfile"]:::processing
        ComposeP["data_process/docker-compose-process.yml"]:::processing
        Process["data_process/process_ais_data.py"]:::processing
        Visualize["data_process/visualize_scrubber_data.py"]:::processing
        ImportPorts["data_process/import_ports_to_db.py"]:::processing
        RequirementsP["data_process/requirements.txt"]:::processing
    end

    %% Artifact Storage
    DataLake["Data Lake / Artifact Store (`data/`)"]:::visualization

    %% One-time Services
    subgraph "One-Time / Scheduled Services"
        direction TB
        NavCreate["navigational_status/create_navigational_status_csv.py"]:::onetime
        NavImport["navigational_status/import_navigational_status.py"]:::onetime
        Legacy["old_ais_data_one_time_clean/clean_historical_data.py"]:::onetime
        OneTimeInput["one_time_input/data_input.py"]:::onetime
    end

    %% Interfaces & Docs
    Notebook["analysis.ipynb"]:::user
    README["README.md"]:::user
    Requirements["requirements.txt"]:::user

    %% Data Flows
    AIS -->|stream data| CollectorApp
    CollectorApp --> Combine
    Combine --> LW
    LW --> Download
    Download -->|inserts data| AuthProxy
    Manage --> CollectorApp
    StartS --> CollectorApp
    StopS --> CollectorApp
    InstallS --> CollectorApp

    AuthProxy -->|secure connection| CloudSQL

    NavCreate -->|loads code tables| NavImport
    NavImport -->|uploads to| CloudSQL

    CloudSQL -->|extract raw tables| Process
    Process -->|write Parquet| DataLake
    Visualize -->|generate HTML| DataLake
    ImportPorts -->|load ports| CloudSQL

    OneTimeInput -->|ad-hoc load| CloudSQL
    Legacy -->|clean old data| CloudSQL

    Notebook -->|query| CloudSQL
    Notebook -->|reads/writes| DataLake
    README -->|documents| CollectorApp
    README -->|documents| Process
    Requirements -->|deps| CollectorApp
    Requirements -->|deps| Process

    %% Click Events
    click CollectorApp "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/ais_collector.py"
    click LW "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/lw_correction_service.py"
    click Combine "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/combine_ship_data.py"
    click Download "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/download_tables.py"
    click Manage "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/manage_services.py"
    click DockerfileC "https://github.com/aoyamaxx/public-ais-stream-api/tree/main/Dockerfile"
    click ComposeC "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/docker-compose.yml"
    click StartS "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/start_services.sh"
    click StopS "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/stop_services.sh"
    click InstallS "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/docker_install.sh"
    click AuthProxy "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/docker-compose.yml"
    click DockerfileP "https://github.com/aoyamaxx/public-ais-stream-api/tree/main/data_process/Dockerfile"
    click ComposeP "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/data_process/docker-compose-process.yml"
    click Process "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/data_process/process_ais_data.py"
    click Visualize "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/data_process/visualize_scrubber_data.py"
    click ImportPorts "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/data_process/import_ports_to_db.py"
    click RequirementsP "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/data_process/requirements.txt"
    click DataLake "https://github.com/aoyamaxx/public-ais-stream-api/tree/main/data/"
    click NavCreate "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/navigational_status/create_navigational_status_csv.py"
    click NavImport "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/navigational_status/import_navigational_status.py"
    click OneTimeInput "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/one_time_input/data_input.py"
    click Legacy "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/old_ais_data_one_time_clean/clean_historical_data.py"
    click Notebook "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/analysis.ipynb"
    click README "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/README.md"
    click Requirements "https://github.com/aoyamaxx/public-ais-stream-api/blob/main/requirements.txt"

    %% Styles
    classDef ingestion fill:#a8f0c6,stroke:#2d7a2d,color:#0b3d0b
    classDef storage fill:#c6e0f0,stroke:#28527a,color:#0b3d5a
    classDef processing fill:#f0d6a8,stroke:#7a622d,color:#5a3d0b
    classDef visualization fill:#e0c6f0,stroke:#622d7a,color:#3d0b5a
    classDef onetime fill:#d3d3d3,stroke:#888888,color:#444444
    classDef user fill:#fff1a8,stroke:#7a742d,color:#5a500b
    classDef external fill:#ffffff,stroke:#000000,color:#000000,stroke-dasharray: 3 3
```
