How busig-the-sql/busig works
Project Type & Purpose
• This is a containerized, microservice-based full-stack application (“Busig”) for fetching, processing, and predicting city bus arrival times.
• It consists of:
• A React/Expo front-end (mobile & web)
• A Flask-based API Gateway
• Three backend services (training_data_collection, inference, bus_model)
• Persistent data stores (PostgreSQL, MongoDB)
• Docker Compose orchestration
File-Structure Analysis
• Top-level directories = microservices (training_data_collection, inference, bus_model, gateway), front-end, ORM (Django), benchmarking, inference/training code, docker-compose.yml.
• Each service has its own Dockerfile, requirements.txt, a Flask app.py, and supporting modules (e.g. GTFS fetchers, preprocessing pipelines, ML models).
• orm/ holds a Django project for data models and migrations (PostgreSQL schema).
• front-end/ holds an Expo React app with Redux slices, config, assets, screens, and hooks.
• docker-compose.yml brings up four Flask containers + a Postgres container + a Mongo container.
Key README Insights
• Four Flask services listening on ports:
– training_data_collection:5000
– inference:5001
– bus_model:5002
– gateway:5004
• .env supplies GTFS API keys
• training_data_collection builds Mongo collections from GTFS Realtime & Static
• bus_model builds/static Postgres GTFS database, and exposes route & schedule APIs
• inference loads pre-trained PyTorch models (.pth) to predict arrival times
• gateway routes incoming client calls to the above services
Main Components & Interactions
A. Front-end (React/Expo)
• Consumes REST endpoints exclusively on gateway: HTTP calls to http://localhost:5004/<endpoint>
• Dispatches user events (search stops/routes, view map) via Redux slices
B. API Gateway (Flask on 5004)
• Aggregator/orchestrator for all backend services
• Routes to:
– bus_model for static schedule & live status
– inference for predictive arrival times
– training_data_collection for raw data inspection (if exposed)
C. Bus Model Service (Flask on 5002)
• Builds & queries Postgres GTFS static DB
• Provides endpoints for: route list, stop arrivals (real-time + static)
• Runs update_GTFS_Static.sh to rebuild DB
D. Training Data Collection Service (Flask on 5000)
• Pulls GTFS Realtime & Static feeds
• Stores JSON in MongoDB
• Supplies raw data for model retraining & debugging
E. Inference Service (Flask on 5001)
• Preprocesses incoming JSON on-the-fly
• Loads PyTorch models (bus_time_prediction_model*.pth)
• Returns predicted arrival times
F. Databases & External Feeds
• PostgreSQL container → GTFS Static tables (built by bus_model & ORM migrations)
• MongoDB container → Collections of GTFS Realtime JSON (populated by training_data_collection)
• External GTFS APIs → polled by training_data_collection and bus_model scripts
Architectural Patterns & Principles
• Microservices: each concern in its own container + Flask app
• ETL pipeline: training_data_collection → preprocess → persist → model training
• ML inference service separate from data pipeline
• API Gateway: single ingress point for front-end, decouples UI from services
• Containerization for environment parity, orchestrated via docker-compose
Essential Elements Checklist
• Ports and protocols on connections (e.g., 5002, HTTPS/REST)
• Notes on triggers (e.g., “cron-like update_GTFS_Static.sh”)