# aaronteoh/streetcred architecture

> Architecture diagram of the GitHub repository aaronteoh/streetcred, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aaronteoh/streetcred
- Repository: https://github.com/aaronteoh/streetcred

## Overview

##### Project Type and Purpose
- streetcred is an automated data-pipeline tool for ingesting, processing, and storing traffic and environmental data from Singapore’s LTA DataMall and NEA APIs.
- It runs on a Google Cloud Compute Engine instance, periodically downloads JSON/CSV datasets (and traffic images), processes them (including object detection via TensorFlow), and writes the results to a Google Cloud Storage (GCS) bucket.

##### File-Structure Analysis
- Top level
- clear_logs.py, logger.py, utils.py → shared utilities and logging
- crontab → scheduling file for periodic execution
- requirements.txt → Python dependencies
- applications/lta
- datamall-realtime_data.py → real-time LTA dataset downloader
- datamall-generate_metadata.py → generates structural metadata for raw datasets
- traffic_images-download_images.py → fetches images from LTA
- traffic_images-detect_objects.py → runs object-detection model on downloaded images
- traffic_images-generate_label_map_csv.py → builds class-to-label mapping CSV
- metadata/*.csv → static mapping and coordinate metadata
- traffic_images-visualise_detections.ipynb → exploratory notebook
- download_from_gcs.py → historic-data loader from GCS
- applications/nea
- nea-historic_data.py → NEA API historic data downloader

##### README Insights
- Pipeline runs every 15 minutes via crontab.
- Uses a service account for GCS reads/writes (“Requester pays” bucket).
- Requires TensorFlow Object Detection API (CenterNet Resnet101 V1 FPN 512×512).
- Outputs two classes of data:
a. Raw and filtered JSON/CSV feeds (carpark, taxi, incidents, speed bands, aggregated detections)
- Logging via Google Monitoring agent.

##### Main Components
- External APIs
- LTA DataMall REST endpoints (traffic-images, traffic-speed, carpark, taxi, incidents)
- NEA historic data REST
- Compute Layer (GCE instance)
- Cron Scheduler (crontab)
- Python ETL scripts (realtime_data, download_images, detect_objects, generate_metadata, nea_historic_data)
- Shared utilities: logger.py, utils.py
- TensorFlow Object Detection module
- Google Cloud Monitoring Agent (logging)
- Storage Layer
- Google Cloud Storage bucket (raw and processed datasets)
- Local filesystem (temporary storage for images, metadata CSVs)
- Metadata Repository
- metadata/*.csv (label maps, geocoordinates)
- Model and Dependencies
- TensorFlow models (object_detection directory)
- requirements.txt environment

##### Relationships and Data Flow
- Cron triggers the realtime_data.py and/or download_from_gcs.py scripts.
- realtime_data.py calls LTA DataMall API → writes JSON/CSV to GCS.
- traffic_images-download_images.py pulls image URLs from LTA API → downloads to local disk.
- detect_objects.py loads local images + metadata CSV → runs TensorFlow model → writes detection CSV to GCS.
- generate_metadata scripts enrich raw feeds into structured metadata tables → store to GCS.
- nea-historic_data.py fetches NEA API data → writes to GCS.
- Logger.py sends logs to Google Monitoring.
- clear_logs.py rotates or cleans local logs.
- Users/Analysts read aggregated datasets from GCS for downstream analysis.

##### Architectural Patterns & Principles
- ETL data pipeline: Extraction (APIs), Transformation (metadata generation, object detection), Load (GCS).
- Batch-oriented workflows orchestrated by cron.
- Separation of concerns: each script handles one data domain.
- Reusable utilities for logging and helper functions.
- Externalize static metadata in CSV for flexibility.
- Cloud-native storage (GCS) and monitoring.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    LTAAPI["LTA DataMall API<br/>(traffic-images, speed, carpark, taxi, incidents)"]:::external
    NEAAPI["NEA Historic Data API"]:::external

    %% Storage Layer
    GCS["Google Cloud Storage Bucket<br/>(raw & processed datasets)"]:::storage
    LocalFS["Local Filesystem<br/>(tmp images & metadata)"]:::storage

    %% Metadata Repository
    subgraph "Metadata Repository"
        direction TB
        MetadataFiles["Static CSV Metadata<br/>(label maps, geocoordinates)"]:::files
    end

    %% GCE Instance
    subgraph "GCE Instance (Compute)" 
        direction TB
        Cron["Cron Scheduler<br/>(every 15 mins)"]:::compute
        ClearLogs["clear_logs.py<br/>(log rotation)"]:::compute
        LoggerUtil["logger.py"]:::compute
        Utils["utils.py"]:::compute
        Requirements["requirements.txt"]:::compute

        subgraph "LTA Domain Scripts"
            direction TB
            RealtimeData["datamall-realtime_data.py"]:::compute
            MetadataGen["datamall-generate_metadata.py"]:::compute
            DownloadGCS["download_from_gcs.py"]:::compute
            ImgDownloader["traffic_images-download_images.py"]:::compute
            DetectObjs["traffic_images-detect_objects.py"]:::compute
            LabelMapGen["traffic_images-generate_label_map_csv.py"]:::compute
            VisualNotebook["traffic_images-visualise_detections.ipynb"]:::compute
        end

        subgraph "NEA Domain Script"
            direction TB
            NEAHistoric["nea-historic_data.py"]:::compute
        end
    end

    %% Model Component
    TFModel["TensorFlow Object Detection Model"]:::model

    %% Monitoring
    Monitoring["Google Cloud Monitoring"]:::external

    %% Data Flows
    Cron -->|triggers| RealtimeData
    Cron -->|triggers| DownloadGCS
    Cron -->|triggers| ImgDownloader
    Cron -->|triggers| MetadataGen
    Cron -->|triggers| NEAHistoric
    Cron -->|triggers| ClearLogs

    RealtimeData -->|fetch JSON/CSV| LTAAPI
    RealtimeData -->|write JSON/CSV| GCS

    DownloadGCS -->|load historic| GCS

    ImgDownloader -->|fetch images| LTAAPI
    ImgDownloader -->|save images| LocalFS

    LabelMapGen -->|generate CSV| MetadataFiles
    MetadataGen -->|process feeds| GCS

    LocalFS -->|input images| DetectObjs
    MetadataFiles -->|input metadata| DetectObjs
    DetectObjs -->|run detection| TFModel
    TFModel -->|output detection CSV| GCS

    NEAHistoric -->|fetch JSON| NEAAPI
    NEAHistoric -->|write JSON| GCS

    %% Logging
    RealtimeData -->|logs| LoggerUtil
    DownloadGCS -->|logs| LoggerUtil
    ImgDownloader -->|logs| LoggerUtil
    DetectObjs -->|logs| LoggerUtil
    MetadataGen -->|logs| LoggerUtil
    NEAHistoric -->|logs| LoggerUtil
    ClearLogs -->|logs| LoggerUtil
    LoggerUtil -->|send logs| Monitoring

    %% Click Events
    click Cron "https://github.com/aaronteoh/streetcred/tree/master/crontab"
    click LoggerUtil "https://github.com/aaronteoh/streetcred/blob/master/logger.py"
    click Utils "https://github.com/aaronteoh/streetcred/blob/master/utils.py"
    click ClearLogs "https://github.com/aaronteoh/streetcred/blob/master/clear_logs.py"
    click Requirements "https://github.com/aaronteoh/streetcred/blob/master/requirements.txt"
    click RealtimeData "https://github.com/aaronteoh/streetcred/blob/master/applications/lta/datamall-realtime_data.py"
    click MetadataGen "https://github.com/aaronteoh/streetcred/blob/master/applications/lta/datamall-generate_metadata.py"
    click DownloadGCS "https://github.com/aaronteoh/streetcred/blob/master/applications/lta/download_from_gcs.py"
    click ImgDownloader "https://github.com/aaronteoh/streetcred/blob/master/applications/lta/traffic_images-download_images.py"
    click DetectObjs "https://github.com/aaronteoh/streetcred/blob/master/applications/lta/traffic_images-detect_objects.py"
    click LabelMapGen "https://github.com/aaronteoh/streetcred/blob/master/applications/lta/traffic_images-generate_label_map_csv.py"
    click VisualNotebook "https://github.com/aaronteoh/streetcred/blob/master/applications/lta/traffic_images-visualise_detections.ipynb"
    click NEAHistoric "https://github.com/aaronteoh/streetcred/blob/master/applications/nea/nea-historic_data.py"
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef storage fill:#007BFF,stroke:#333,stroke-width:1px
    classDef compute fill:#28A745,stroke:#333,stroke-width:1px
    classDef model fill:#6F42C1,stroke:#333,stroke-width:1px
    classDef files fill:#6C757D,stroke:#333,stroke-width:1px
```
