How aoyamaxx/public-north-sea-watch works
Project Type and Purpose
• It integrates real-time AIS data, agent-based modelling, policy data, and external services (web scrapers, LLM report generators) to support stakeholders and policymakers.
High-Level File-Structure Analysis
• Top-level “backend/” directory: Django project with apps for ABM (“apps/abm”), core logic (“apps/north_sea_watch”), common utilities, configuration, management commands, fixtures, migrations, Docker and compose files, scripts for cleanup and RPI deployment.
• Top-level “frontend/” directory: React/TypeScript app, feature-based organization (features/abm, map, home, privacy, etc.), global components, hooks, config, Docker and compose files, Nginx config, service YAML for production.
• Supporting directories: “cloudflared/” for Cloudflare tunnel setup, “data/” for placeholder CSV, “docs/” for setup guides, and root-level scripts for URL mapping and RPI deployment.
• CI/CD: GitHub workflows in .github/workflows for frontend/backend deploy and URL-map updates.
Key README Insights
• Backend: Django + Django REST Framework, ASGI/Uvicorn for SSE/WebSockets, Postgres (Cloud SQL), Scrapy/Selenium scrapers, LLM integration for reports, Prometheus/Grafana/cAdvisor for monitoring.
• Frontend: React, Mapbox GL JS for mapping, server-side rendering fallback via Rendertron, PWA manifest, hot-reload via Docker Compose.
• ABM Simulation: Mesa on backend powering agent steps; REST endpoints to control simulation; Canvas renderer on frontend; stateful grid updates via API.
• Deployment: Docker Compose for dev/prod/RPI, Google VM/Cloud Run, Cloudflare tunnel.
a. Main Components
• Frontend SPA (React + Mapbox GL)
• Backend API & WebSocket/SSE server (Django + DRF + Uvicorn/ASGI)
• ABM Simulation Engine (Mesa inside Django)
• PostgreSQL database (with PostGIS extension)
• External scrapers (Scrapy/Selenium)
• LLM report generator
• Caching/Message broker (if used—e.g., Redis for ASGI channels)
• Reverse proxy/load balancer (Nginx)
• CI/CD pipeline (GitHub Actions)
• Monitoring stack (Prometheus, Grafana, cAdvisor)
• Cloudflare Tunnel (cloudflared)
b. Component Interactions & Data Flow
1. User → React SPA
2. SPA → DRF REST endpoints for data (ship positions, emissions, ports)
3. SPA → WebSocket/SSE for real-time updates (simulation state, AIS feed)
4. DRF → PostgreSQL (read/write simulation parameters, AIS history, user settings)
5. Scheduled scrapers → Django management commands → update port data in DB
6. LLM service called via REST or internal service to generate markdown reports stored in DB or object storage
7. Monitoring agents (cAdvisor) push metrics to Prometheus → Grafana dashboards
8. GitHub Actions trigger Docker image builds → deploy to Cloud Run / VM behind Nginx proxy → Cloudflare tunnel exposes services
c. Architectural Patterns & Principles
• Separation of concerns: “frontend” vs. “backend” vs. “data ingestion” vs. “simulation”
• Feature-based modularity on both client (React features/) and server (Django apps/)
• Asynchronous processing: ASGI for live updates; background threads for simulation; management commands for scraping.
• Containerization & orchestration: Docker Compose for multi-container dev and prod with network bridging.
• Infrastructure as code: Compose files, config scripts, GitHub Actions.
d. Key Technologies & Frameworks
• React + TypeScript + Mapbox GL JS
• Django + Django REST Framework + Channels (ASGI/Uvicorn)
• Mesa (ABM)
• PostgreSQL + PostGIS
• Scrapy, Selenium, LLM library
• Docker, Docker Compose
• Nginx, Cloudflare tunnel
• GitHub Actions, Google Cloud Run / Google VM
• Prometheus, Grafana, cAdvisor