# aoyamaxx/public-north-sea-watch architecture

> Architecture diagram of the GitHub repository aoyamaxx/public-north-sea-watch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aoyamaxx/public-north-sea-watch
- Repository: https://github.com/aoyamaxx/public-north-sea-watch

## Overview

##### Project Type and Purpose
- It integrates real-time AIS data, agent-based modelling, policy data, and external services (web scrapers, LLM report generators) to support stakeholders and policymakers.

##### High-Level File-Structure Analysis
- Top-level “backend/” directory: Django project with apps for ABM (“apps/abm”), core logic (“apps/north_sea_watch”), common utilities, configuration, management commands, fixtures, migrations, Docker and compose files, scripts for cleanup and RPI deployment.
- Top-level “frontend/” directory: React/TypeScript app, feature-based organization (features/abm, map, home, privacy, etc.), global components, hooks, config, Docker and compose files, Nginx config, service YAML for production.
- Supporting directories: “cloudflared/” for Cloudflare tunnel setup, “data/” for placeholder CSV, “docs/” for setup guides, and root-level scripts for URL mapping and RPI deployment.
- CI/CD: GitHub workflows in .github/workflows for frontend/backend deploy and URL-map updates.

##### Key README Insights
- Backend: Django + Django REST Framework, ASGI/Uvicorn for SSE/WebSockets, Postgres (Cloud SQL), Scrapy/Selenium scrapers, LLM integration for reports, Prometheus/Grafana/cAdvisor for monitoring.
- Frontend: React, Mapbox GL JS for mapping, server-side rendering fallback via Rendertron, PWA manifest, hot-reload via Docker Compose.
- ABM Simulation: Mesa on backend powering agent steps; REST endpoints to control simulation; Canvas renderer on frontend; stateful grid updates via API.
- Deployment: Docker Compose for dev/prod/RPI, Google VM/Cloud Run, Cloudflare tunnel.

a. Main Components
- Frontend SPA (React + Mapbox GL)
- Backend API & WebSocket/SSE server (Django + DRF + Uvicorn/ASGI)
- ABM Simulation Engine (Mesa inside Django)
- PostgreSQL database (with PostGIS extension)
- External scrapers (Scrapy/Selenium)
- LLM report generator
- Caching/Message broker (if used—e.g., Redis for ASGI channels)
- Reverse proxy/load balancer (Nginx)
- CI/CD pipeline (GitHub Actions)
- Monitoring stack (Prometheus, Grafana, cAdvisor)
- Cloudflare Tunnel (cloudflared)

b. Component Interactions & Data Flow
1. User → React SPA
2. SPA → DRF REST endpoints for data (ship positions, emissions, ports)
3. SPA → WebSocket/SSE for real-time updates (simulation state, AIS feed)
4. DRF → PostgreSQL (read/write simulation parameters, AIS history, user settings)
5. Scheduled scrapers → Django management commands → update port data in DB
6. LLM service called via REST or internal service to generate markdown reports stored in DB or object storage
7. Monitoring agents (cAdvisor) push metrics to Prometheus → Grafana dashboards
8. GitHub Actions trigger Docker image builds → deploy to Cloud Run / VM behind Nginx proxy → Cloudflare tunnel exposes services

c. Architectural Patterns & Principles
- Separation of concerns: “frontend” vs. “backend” vs. “data ingestion” vs. “simulation”
- Feature-based modularity on both client (React features/) and server (Django apps/)
- Asynchronous processing: ASGI for live updates; background threads for simulation; management commands for scraping.
- Containerization & orchestration: Docker Compose for multi-container dev and prod with network bridging.
- Infrastructure as code: Compose files, config scripts, GitHub Actions.

d. Key Technologies & Frameworks
- React + TypeScript + Mapbox GL JS
- Django + Django REST Framework + Channels (ASGI/Uvicorn)
- Mesa (ABM)
- PostgreSQL + PostGIS
- Scrapy, Selenium, LLM library
- Docker, Docker Compose
- Nginx, Cloudflare tunnel
- GitHub Actions, Google Cloud Run / Google VM
- Prometheus, Grafana, cAdvisor

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client (Browser)"
        direction TB
        User["User"]:::external
        ReactSPA["React SPA"]:::frontend
        Mapbox["Mapbox GL JS"]:::external
    end

    %% Server Layer
    subgraph "Server (Containers)"
        direction TB

        subgraph "Reverse Proxy"
            direction TB
            Nginx["Nginx Proxy"]:::infra
        end

        subgraph "App Services"
            direction TB
            DjangoAPI["Django API & WebSocket/SSE"]:::backend
            ABMEngine["ABM Simulation Engine (Mesa)"]:::backend
            Scrapers["External Scrapers"]:::backend
            LLMService["LLM Report Generator"]:::backend
        end

        subgraph "Data Stores"
            direction TB
            Postgres["PostgreSQL (PostGIS)"]:::database
            Redis["Redis (Channels)"]:::database
        end

        subgraph "Monitoring"
            direction TB
            cAdvisor["cAdvisor"]:::infra
            Prometheus["Prometheus"]:::infra
            Grafana["Grafana"]:::infra
        end
    end

    %% CI/CD and Deployment
    subgraph "CI/CD & Deployment"
        direction TB
        GitHubActions["GitHub Actions"]:::infra
        DockerRegistry["Docker Registry"]:::infra
        Deployment["Cloud Run / VM"]:::infra
        Cloudflare["Cloudflare Tunnel"]:::infra
    end

    %% External Services
    subgraph "External Services"
        direction TB
        MapboxAPI["Mapbox API"]:::external
        LLMAPI["External LLM Endpoint"]:::external
    end

    %% Connections
    User -->|"opens"| ReactSPA
    ReactSPA -->|"HTTPS REST API"| DjangoAPI
    ReactSPA -->|"WebSocket/SSE"| DjangoAPI
    ReactSPA -->|"renders map"| MapboxAPI
    DjangoAPI -->|"reads/writes"| Postgres
    DjangoAPI -->|"pub/sub"| Redis
    Scrapers -->|"update data"| Postgres
    ABMEngine -->|"simulation state"| Postgres
    ABMEngine -->|"state updates"| DjangoAPI
    LLMService -->|"generate report"| LLMAPI
    LLMService -->|"store report"| Postgres
    cAdvisor -->|"metrics"| Prometheus
    Prometheus -->|"visualize"| Grafana
    GitHubActions -->|"build images"| DockerRegistry
    DockerRegistry -->|"deploy images"| Deployment
    Deployment -->|"serves through"| Nginx
    Cloudflare -->|"tunnel to"| Nginx

    %% Click Events - Frontend
    click ReactSPA "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/frontend/src/app/app.tsx"
    click ReactSPA "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/frontend/package.json"
    click ReactSPA "https://github.com/aoyamaxx/public-north-sea-watch/tree/main/frontend/src/features"
    click ReactSPA "https://github.com/aoyamaxx/public-north-sea-watch/tree/main/frontend/src/components"
    click ReactSPA "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/frontend/nginx.conf"
    click ReactSPA "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/frontend/service.yaml"

    %% Click Events - Backend API
    click DjangoAPI "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/config/asgi.py"
    click DjangoAPI "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/config/settings.py"
    click DjangoAPI "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/config/urls.py"
    click DjangoAPI "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/api/v1/views.py"
    click DjangoAPI "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/api/v1/serializers.py"
    click DjangoAPI "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/api/v1/urls.py"

    %% Click Events - ABM
    click ABMEngine "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/abm/mesa/mesa_model.py"
    click ABMEngine "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/abm/mesa/port.py"
    click ABMEngine "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/abm/mesa/ship.py"
    click ABMEngine "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/abm/views.py"
    click ABMEngine "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/abm/urls.py"

    %% Click Events - Database Config
    click Postgres "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/config/settings.py"
    click Postgres "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/db_routers.py"

    %% Click Events - Scrapers
    click Scrapers "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/management/commands/update_ship_emissions.py"
    click Scrapers "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/management/commands/update_ship_discharge_rates.py"
    click Scrapers "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/management/commands/cleanup_ship_data.py"
    click Scrapers "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/management/commands/clear_ship_emissions.py"
    click Scrapers "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/management/commands/debug_ships.py"

    %% Click Events - LLM Service
    click LLMService "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/north_sea_watch/services.py"

    %% Click Events - Redis
    click Redis "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/apps/config/settings.py"

    %% Click Events - Reverse Proxy
    click Nginx "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/frontend/nginx.conf"
    click Nginx "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/docker-compose.prod.yml"
    click Nginx "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/frontend/docker-compose.prod.yml"

    %% Click Events - CI/CD
    click GitHubActions "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/.github/workflows/deploy-backend.yml"
    click GitHubActions "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/.github/workflows/deploy-frontend.yml"
    click GitHubActions "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/.github/workflows/update-url-map.yml"

    %% Click Events - Monitoring
    click cAdvisor "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/docker-compose.yml"
    click Prometheus "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/docker-compose.prod.yml"
    click Grafana "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/scripts/update-url-map.sh"

    %% Click Events - Cloudflare
    click Cloudflare "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/cloudflared/config.yml"
    click Cloudflare "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/cloudflared/setup-tunnel.sh"

    %% Click Events - Deployment Scripts
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/tree/main/backend/Dockerfile"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/tree/main/frontend/Dockerfile"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/docker-compose.yml"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/frontend/docker-compose.yml"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/deploy-rpi.sh"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/scripts/start.sh"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/scripts/update-url-map.sh"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/entrypoint.sh"
    click Deployment "https://github.com/aoyamaxx/public-north-sea-watch/blob/main/backend/entrypoint.prod.sh"

    %% Styling
    classDef frontend fill:#cce5ff,stroke:#4da3ff,color:#004085;
    classDef backend fill:#d4edda,stroke:#28a745,color:#155724;
    classDef database fill:#fff3cd,stroke:#ffca2c,color:#856404;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#383d41;
    classDef infra fill:#e2dfff,stroke:#7a42f4,color:#3f004f;
```
