How aaronteoh/streetcred works
Project Type and Purpose
• streetcred is an automated data-pipeline tool for ingesting, processing, and storing traffic and environmental data from Singapore’s LTA DataMall and NEA APIs.
• It runs on a Google Cloud Compute Engine instance, periodically downloads JSON/CSV datasets (and traffic images), processes them (including object detection via TensorFlow), and writes the results to a Google Cloud Storage (GCS) bucket.
File-Structure Analysis
• Top level
• clear_logs.py, logger.py, utils.py → shared utilities and logging
• crontab → scheduling file for periodic execution
• requirements.txt → Python dependencies
• applications/lta
• datamall-realtime_data.py → real-time LTA dataset downloader
• datamall-generate_metadata.py → generates structural metadata for raw datasets
• traffic_images-download_images.py → fetches images from LTA
• traffic_images-detect_objects.py → runs object-detection model on downloaded images
• traffic_images-generate_label_map_csv.py → builds class-to-label mapping CSV
• metadata/*.csv → static mapping and coordinate metadata
• traffic_images-visualise_detections.ipynb → exploratory notebook
• download_from_gcs.py → historic-data loader from GCS
• applications/nea
• nea-historic_data.py → NEA API historic data downloader
README Insights
• Pipeline runs every 15 minutes via crontab.
• Uses a service account for GCS reads/writes (“Requester pays” bucket).
• Requires TensorFlow Object Detection API (CenterNet Resnet101 V1 FPN 512×512).
• Outputs two classes of data:
a. Raw and filtered JSON/CSV feeds (carpark, taxi, incidents, speed bands, aggregated detections)
• Logging via Google Monitoring agent.
Main Components
• External APIs
• LTA DataMall REST endpoints (traffic-images, traffic-speed, carpark, taxi, incidents)
• NEA historic data REST
• Compute Layer (GCE instance)
• Cron Scheduler (crontab)
• Python ETL scripts (realtime_data, download_images, detect_objects, generate_metadata, nea_historic_data)
• Shared utilities: logger.py, utils.py
• TensorFlow Object Detection module
• Google Cloud Monitoring Agent (logging)
• Storage Layer
• Google Cloud Storage bucket (raw and processed datasets)
• Local filesystem (temporary storage for images, metadata CSVs)
• Metadata Repository
• metadata/*.csv (label maps, geocoordinates)
• Model and Dependencies
• TensorFlow models (object_detection directory)
• requirements.txt environment
Relationships and Data Flow
• Cron triggers the realtime_data.py and/or download_from_gcs.py scripts.
• realtime_data.py calls LTA DataMall API → writes JSON/CSV to GCS.
• traffic_images-download_images.py pulls image URLs from LTA API → downloads to local disk.
• detect_objects.py loads local images + metadata CSV → runs TensorFlow model → writes detection CSV to GCS.
• generate_metadata scripts enrich raw feeds into structured metadata tables → store to GCS.
• nea-historic_data.py fetches NEA API data → writes to GCS.
• Logger.py sends logs to Google Monitoring.
• clear_logs.py rotates or cleans local logs.
• Users/Analysts read aggregated datasets from GCS for downstream analysis.
Architectural Patterns & Principles
• ETL data pipeline: Extraction (APIs), Transformation (metadata generation, object detection), Load (GCS).
• Batch-oriented workflows orchestrated by cron.
• Separation of concerns: each script handles one data domain.
• Reusable utilities for logging and helper functions.
• Externalize static metadata in CSV for flexibility.
• Cloud-native storage (GCS) and monitoring.