How robjective/transparentsf works
Project type and purpose
• TransparentSF is a full-stack, data-analysis & visualization web application combining AI agents with public San Francisco city data.
• It exposes two main user interfaces:
– An interactive chat UI for exploratory queries
– A backend/admin UI for configuration, scheduled analysis and report generation
High-level system components
From the file tree and README, the major components are:
• Web Client (Browser) – Serves static assets and templates under ai/static & ai/templates
• Python Backend Service – Entry point ai/main.py (or start_services.sh), modules under ai/
• Data Storage
– JSON files on disk (ai/data/...) for raw datasets, prompts, queries
– PostgreSQL for anomaly storage (initialized by ai/tools/init_postgres_db.py)
– Qdrant as a vector database for embeddings / time-series storage (launched via Docker in start_services.sh)
• External APIs
– OpenAI API (analysis, embedding, AI insights)
– Ghost CMS API for publishing (via a small Node.js “Ghost Bridge” service under ai/tools/ghost_bridge/)
• Background / Periodic Tasks
– Scripts under ai/ (e.g. monthly_report.py, periodic_analysis.py)
– Vector loaders (vector_loader_sfpublic.py, vector_loader_periodic.py)
– Anomaly detection and storage (anomalyAnalyzer.py, tools in ai/tools/)
Architectural patterns & design principles
• Layered / Modular architecture:
– Presentation Layer: HTML/JS templates, CSS, static files
– Controller Layer: Flask (or similar) routes in main.py, backend.py, webChat.py
– Service Layer: Data prep, analysis, AI orchestration (ai_dataprep.py, anomalyAnalyzer.py, chart/message generation)
– Data Access Layer: DB utilities (ai/tools/db_utils.py), file-based data loaders, Qdrant client
• Microservice for CMS integration: The Ghost Bridge (Node.js) runs separately, decoupling CMS API calls from the Python monolith.
• Asynchronous / Scheduled processing: Periodic runners import vector loaders and analysis scripts to refresh data.