# robjective/transparentsf architecture

> Architecture diagram of the GitHub repository robjective/transparentsf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robjective/transparentsf
- Repository: https://github.com/robjective/transparentsf

## Overview

##### Project type and purpose
- TransparentSF is a full-stack, data-analysis & visualization web application combining AI agents with public San Francisco city data.
- It exposes two main user interfaces:
– An interactive chat UI for exploratory queries
– A backend/admin UI for configuration, scheduled analysis and report generation

##### High-level system components
From the file tree and README, the major components are:
- Web Client (Browser) – Serves static assets and templates under `ai/static` & `ai/templates`
- Python Backend Service – Entry point `ai/main.py` (or `start_services.sh`), modules under `ai/`
- Data Storage
– JSON files on disk (`ai/data/...`) for raw datasets, prompts, queries
– PostgreSQL for anomaly storage (initialized by `ai/tools/init_postgres_db.py`)
– Qdrant as a vector database for embeddings / time-series storage (launched via Docker in `start_services.sh`)
- External APIs
– OpenAI API (analysis, embedding, AI insights)
– Ghost CMS API for publishing (via a small Node.js “Ghost Bridge” service under `ai/tools/ghost_bridge/`)
- Background / Periodic Tasks
– Scripts under `ai/` (e.g. `monthly_report.py`, `periodic_analysis.py`)
– Vector loaders (`vector_loader_sfpublic.py`, `vector_loader_periodic.py`)
– Anomaly detection and storage (`anomalyAnalyzer.py`, tools in `ai/tools/`)

##### Architectural patterns & design principles
- Layered / Modular architecture:
– Presentation Layer: HTML/JS templates, CSS, static files
– Controller Layer: Flask (or similar) routes in `main.py`, `backend.py`, `webChat.py`
– Service Layer: Data prep, analysis, AI orchestration (`ai_dataprep.py`, `anomalyAnalyzer.py`, chart/message generation)
– Data Access Layer: DB utilities (`ai/tools/db_utils.py`), file-based data loaders, Qdrant client
- Microservice for CMS integration: The Ghost Bridge (Node.js) runs separately, decoupling CMS API calls from the Python monolith.
- Asynchronous / Scheduled processing: Periodic runners import vector loaders and analysis scripts to refresh data.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client
    BrowserChat["Browser Chat UI / Admin UI"]:::client

    %% Internal Services
    PythonBackend["Python Backend Service"]:::internal
    DataPrep["Data Prep Module"]:::internal
    AnomalyAnalyzer["Anomaly Analyzer"]:::internal
    ChartMessage["Chart & Message Generator"]:::internal
    ScheduledRunner["Scheduled Task Runner"]:::internal

    VectorLoaderPublic["Vector Loader SF Public"]:::internal
    VectorLoaderPeriodic["Vector Loader Periodic"]:::internal
    PeriodicAnalysis["Periodic Analysis"]:::internal
    MonthlyReport["Monthly Report"]:::internal

    DBUtils["DB Utilities"]:::internal
    StoreAnomalies["Store Anomalies"]:::internal
    StoreTimeSeries["Store Time Series"]:::internal
    VectorQuery["Vector Query Service"]:::internal

    GenGhostPost["Generate Ghost Post"]:::internal
    GenReportText["Generate Report Text"]:::internal

    %% Data Stores
    JSONData["JSON Data Store\n(ai/data/)"]:::db
    PostgresDB["PostgreSQL (Anomalies)"]:::db
    QdrantDB["Qdrant (Vector DB)"]:::db

    %% External Services
    OpenAI["OpenAI API"]:::external
    GhostBridge["Ghost Bridge (Node.js)"]:::micro
    GhostCMS["Ghost CMS"]:::external

    %% Main Flow (On-Demand)
    BrowserChat -->|"HTTP Requests"| PythonBackend
    PythonBackend -->|"JSON read"| JSONData
    PythonBackend -->|"vector insert/query"| QdrantDB
    PythonBackend -->|"read/write anomalies"| PostgresDB
    PythonBackend -->|"analysis & embeddings"| OpenAI
    PythonBackend -->|"publish request"| GhostBridge
    GhostBridge -->|"CMS API calls"| GhostCMS

    %% Scheduled Flows (Dashed)
    ScheduledRunner -.->|runs| VectorLoaderPublic
    ScheduledRunner -.->|runs| VectorLoaderPeriodic
    ScheduledRunner -.->|runs| PeriodicAnalysis
    ScheduledRunner -.->|runs| MonthlyReport

    VectorLoaderPublic -.->|"vector load"| QdrantDB
    VectorLoaderPeriodic -.->|"vector load"| QdrantDB
    PeriodicAnalysis -.->|"analysis"| QdrantDB
    PeriodicAnalysis -.->|"store anomalies"| PostgresDB

    %% Utility Connections
    PythonBackend --> DBUtils
    PythonBackend --> StoreAnomalies
    PythonBackend --> StoreTimeSeries
    PythonBackend --> VectorQuery
    PythonBackend --> DataPrep
    PythonBackend --> AnomalyAnalyzer
    PythonBackend --> ChartMessage
    PythonBackend --> GenGhostPost
    PythonBackend --> GenReportText

    %% Click Events
    click BrowserChat "https://github.com/robjective/transparentsf/tree/main/ai/templates/"
    click PythonBackend "https://github.com/robjective/transparentsf/blob/main/ai/main.py"
    click DataPrep "https://github.com/robjective/transparentsf/blob/main/ai/ai_dataprep.py"
    click AnomalyAnalyzer "https://github.com/robjective/transparentsf/blob/main/ai/anomalyAnalyzer.py"
    click ChartMessage "https://github.com/robjective/transparentsf/blob/main/ai/chart_message.py"
    click ScheduledRunner "https://github.com/robjective/transparentsf/blob/main/start_services.sh"
    click VectorLoaderPublic "https://github.com/robjective/transparentsf/blob/main/ai/vector_loader_sfpublic.py"
    click VectorLoaderPeriodic "https://github.com/robjective/transparentsf/blob/main/ai/vector_loader_periodic.py"
    click PeriodicAnalysis "https://github.com/robjective/transparentsf/blob/main/ai/periodic_analysis.py"
    click MonthlyReport "https://github.com/robjective/transparentsf/blob/main/ai/monthly_report.py"
    click JSONData "https://github.com/robjective/transparentsf/tree/main/ai/data/"
    click PostgresDB "https://github.com/robjective/transparentsf/blob/main/ai/tools/init_postgres_db.py"
    click DBUtils "https://github.com/robjective/transparentsf/blob/main/ai/tools/db_utils.py"
    click StoreAnomalies "https://github.com/robjective/transparentsf/blob/main/ai/tools/store_anomalies.py"
    click StoreTimeSeries "https://github.com/robjective/transparentsf/blob/main/ai/tools/store_time_series.py"
    click VectorQuery "https://github.com/robjective/transparentsf/blob/main/ai/tools/vector_query.py"
    click OpenAI "https://github.com/robjective/transparentsf/blob/main/ai/tools/embedding.py"
    click GhostBridge "https://github.com/robjective/transparentsf/blob/main/ai/tools/ghost_bridge/ghostBridge.js"
    click GenGhostPost "https://github.com/robjective/transparentsf/blob/main/ai/tools/genGhostPost.py"
    click GenReportText "https://github.com/robjective/transparentsf/blob/main/ai/tools/generate_report_text.py"

    %% Styles
    classDef internal fill:#cfe2f3,stroke:#2b5797,color:#073763;
    classDef db fill:#d9ead3,stroke:#38761d,color:#27560e;
    classDef external fill:#f2f2f2,stroke:#7f7f7f,color:#4f4f4f;
    classDef micro fill:#fce5cd,stroke:#e69138,color:#a64d00;
    classDef client fill:#eeeeee,stroke:#999999,color:#333333;
```
