How dung0-o/llm-pulse-platform works
Local LLM Sentiment Tracker turns public Reddit discussions into model-family sentiment analytics. The main workflow is RSS scraping to raw GCS data, documented medallion transformations in BigQuery, sentiment backfill through a configurable inference backend, then cached API responses rendered in five dashboard views. The supplied tree has no dbt files, so its documented warehouse stages are represented without invented source paths. API and scraper calls are supported by excerpts; transformation and dashboard-to-API links rely on README documentation.
Main entry points: App.tsx View Router · main.py FastAPI Routes · backfill.py Sentiment Backfill · inference.py Inference Backend
Components 24
Ingestion
- RSS Scraper
scraper/reddit_scraper.py - Ingestion Runner
scraper/main.py - GCS Upload
scraper/gcs_client.py
Warehouse and ML
- Bronze JSONL
- Silver Posts
- Gold Features
- Sentiment Predictions
- Enriched View
- Sentiment Backfill
api/jobs/backfill.py - Inference Backend
api/inference.py
API Serving
- FastAPI Routes
api/main.py - BigQuery Queries
api/database.py - Redis Cache
api/cache.py
Dashboard
- API Client
dashboard/src/api/client.ts - View Router
dashboard/src/App.tsx - Leaderboard View
dashboard/src/pages/DashboardPage.tsx - Model Trend View
dashboard/src/pages/ModelPage.tsx - Share of Voice
dashboard/src/pages/ShareOfVoicePage.tsx - Posts Search
dashboard/src/pages/SearchPage.tsx - System Health
dashboard/src/pages/HealthPage.tsx
Other components
- Reddit RSS
- Hugging Face API
- Local Model
- Dashboard User
Connections 24
- Reddit RSS → RSS Scraper: provides feed
- Ingestion Runner → RSS Scraper: scrapes posts
scraper/main.py - Ingestion Runner → GCS Upload: uploads records
scraper/main.py - GCS Upload → Bronze JSONL: writes JSONL
scraper/gcs_client.py - Bronze JSONL → Silver Posts: transforms
- Silver Posts → Gold Features: transforms
- Gold Features → Enriched View: enriches view
- Sentiment Predictions → Enriched View: joins view
- Gold Features → Sentiment Backfill: queries mentions
- Sentiment Backfill → Inference Backend: scores aspects
api/jobs/backfill.py - Sentiment Backfill → BigQuery Queries: reads and merges
api/jobs/backfill.py - Inference Backend → Hugging Face API: requests scores
api/inference.py - Inference Backend → Local Model: runs locally
api/inference.py - FastAPI Routes → Sentiment Backfill: triggers backfill
api/main.py - FastAPI Routes → BigQuery Queries: queries analytics
api/main.py - FastAPI Routes → Redis Cache: reads and writes
api/main.py - Enriched View → BigQuery Queries: serves queries
api/database.py - Dashboard User → View Router: uses views
- View Router → Leaderboard View: routes to
dashboard/src/App.tsx - View Router → Model Trend View: routes to
dashboard/src/App.tsx - View Router → Share of Voice: routes to
dashboard/src/App.tsx - View Router → Posts Search: routes to
dashboard/src/App.tsx - View Router → System Health: routes to
dashboard/src/App.tsx - API Client → FastAPI Routes: requests endpoints