How aoyamaxx/public-ais-stream-api works
Project type and purpose
• This is a data‐pipeline / ETL‐style system (“AIS Data Collector with Cloud SQL”) that continuously ingests real‐time AIS WebSocket data from AISStream.io, writes it to a Google Cloud SQL (PostgreSQL) instance via a Cloud SQL Auth Proxy, and includes downstream batch processing and visualization components.
File structure
• Top‐level:
• Dockerfile, docker-compose.yml, Docker install/start/stop scripts → container orchestration
• ais_collector.py, lw_correction_service.py, combine_ship_data.py, download_tables.py → streaming ingestion & utilities
• manage_services.py → local orchestration helper
• data_process/ → separate ETL & visualization microservice, with its own Dockerfile, docker‐compose, scripts, and requirements
• navigational_status/ → one‐time or scheduled service loading static code tables into DB
• one_time_input/ & old_ais_data_one_time_clean/ → ad‐hoc or legacy data‐load scripts
• data/ → output artifacts (Parquet, CSV, HTML visualizations)
• analysis.ipynb & README.md → end‐user analysis and documentation
README for additional insights
• Real‐time WebSocket ingestion into ships, ship_data, unknown_ships tables
• Docker Compose brings up two containers: Python ingestion app + Cloud SQL Auth Proxy
• Data processing is decoupled: data_process/process_ais_data.py reads from Cloud SQL, writes Parquet, then visualize_scrubber_data.py generates HTML maps/plots
• Use cases: continuous ingestion, ad‐hoc batch analyses, geospatial visualizations
a. Main components
1. AISStream.io WebSocket API
2. Docker Compose “collector” service (Python app container)
3. Docker Compose “cloud_sql_proxy” service
4. Google Cloud SQL (PostgreSQL)
5. Data_Process Docker service (ETL & Visualization)
6. Data Lake / Artifact Store (Parquet files, CSVs, HTML outputs in data/)
7. Optional one‐time or scheduled services (navigational_status loader, legacy cleaning scripts)
8. User interfaces: Jupyter notebook, psql / Cloud Console query editor, HTML visualization front‐ends
b. Relationships and data flows
• AISStream.io WebSocket → Python Collector container (ais_collector.py)
• Collector → Cloud SQL via Auth Proxy container
• navigational_status loader → Cloud SQL (uploads code tables)
• Data_Process container:
• Extracts raw tables from Cloud SQL
• Transforms data (chunked processing, scrubber matching)
• Loads Parquet files into local data/ directory
• Generates HTML visualizations in data/
• Users / Analysts:
• Query DB directly (psql or Cloud Console)
• Run analysis.ipynb against Cloud SQL
• View HTML reports in browser
c. Architectural patterns & design principles
• Microservices via Docker Compose: separation of concerns (ingest vs proxy vs ETL)
• Streaming ingestion + batch ETL pipeline
• Infrastructure as code: Dockerfiles and Compose for reproducibility
• Decoupled storage: Cloud SQL for raw, Parquet for analytical workloads
• Modularity: one‐time vs recurring processes isolated in subdirectories
d. Key technologies & frameworks
• Docker & Docker Compose
• Python (async WebSocket client, psycopg2/sqlalchemy)
• Google Cloud SQL (PostgreSQL) + Cloud SQL Auth Proxy
• pandas & pyarrow (Parquet)
• Jupyter Notebook for ad‐hoc analysis
• Folium / Bokeh / Plotly for map and Sankey visualizations
• Clear labels on each component (e.g., “Python Collector,” “Cloud SQL Proxy,” “Cloud SQL PG,” “ETL Service,” “Data Lake”)
• WebSocket → Collector → Proxy → DB
• DB → ETL → Parquet & HTML
• Analyst → psql/notebook → DB
• Ingestion (green), Storage (blue), Processing (orange), Visualization (purple), One‐time scripts (gray)