How dylantirandaz/forecast works
Forecast is an AI-assisted forecasting platform for NYC housing-policy outcomes, with a FastAPI backend, Celery workers, persistent data, and a React interface. The main workflow is question/scenario input, historical-prior and evidence analysis, forecast generation, then evaluation and calibration. Separate backtest, benchmark, experiment, replay, and cost features support model assessment. The graph also includes the standalone forecast/ LLM-evaluation toolkit. Some UI-to-API and unsampled subsystem connections are not visible in the excerpts, so those links are omitted or marked unverified.
Main entry points: evaluations.py Evaluation endpoints · replay_engine.py Replay engine · main.py FastAPI app · router.py API router
Components 34
User interface
- Forecast dashboard
frontend/src/app/page.tsx - Question views
frontend/src/app/questions/page.tsx - Evaluation views
frontend/src/app/eval/page.tsx
API and orchestration
- FastAPI app
backend/app/main.py - API router
backend/app/api/router.py - Question endpoints
backend/app/api/questions.py - Forecast endpoints
backend/app/api/forecasts.py - Scenario endpoints
backend/app/api/scenarios.py - Common schemas
backend/app/schemas/common.py - Forecast tasks
backend/app/tasks/forecast_tasks.py
Forecasting domain
- Forecast engine
backend/app/services/forecast_engine.py - Base-rate engine
backend/app/services/base_rate_engine.py - Evidence scoring
backend/app/services/evidence_scorer.py - Belief updating
backend/app/services/belief_updater.py - Scenario engine
backend/app/services/scenario_engine.py
Evaluation and learning
- Evaluation endpoints
backend/app/api/evaluations.py - Replay engine
backend/app/services/replay_engine.py - Evaluation metrics
backend/app/services/eval_metrics.py - Backtesting
backend/app/services/backtester.py - Calibration
backend/app/services/calibration.py - Benchmark harness
backend/app/services/benchmark_harness.py - Experiment tracking
backend/app/services/experiment_tracker.py - Ablation runner
backend/app/services/ablation_runner.py - Cost tracking
backend/app/services/cost_tracker.py
Data and persistence
- DB sessions
backend/app/database.py - PostgreSQL
- Data adapters
backend/app/adapters
LLM evaluation toolkit
- Toolkit config
forecast/config.py - LLM client
forecast/llm.py - Forecast prompts
forecast/prompts.py - Evidence search
forecast/search.py - Forecast metrics
forecast/metrics.py
Other components
- Forecaster
- LLM provider
Connections 37
- Forecaster → Forecast dashboard: uses
README.md - Forecast dashboard → FastAPI app: requests data
- Question views → FastAPI app: requests data
- Evaluation views → FastAPI app: requests data
- FastAPI app → API router: mounts routes
backend/app/main.py - API router → Question endpoints: registers
backend/app/api/router.py - API router → Forecast endpoints: registers
backend/app/api/router.py - API router → Scenario endpoints: registers
backend/app/api/router.py - API router → Evaluation endpoints: registers
backend/app/api/router.py - FastAPI app → DB sessions: checks connection
backend/app/main.py - DB sessions → PostgreSQL: opens sessions
backend/app/database.py - FastAPI app → Common schemas: validates health
backend/app/main.py - Forecast tasks → Forecast engine: runs forecast
backend/app/tasks/forecast_tasks.py - Forecast tasks → Backtesting: runs backtest
backend/app/tasks/forecast_tasks.py - Forecast tasks → Data adapters: dispatches ingestion
backend/app/tasks/forecast_tasks.py - Forecast engine → Base-rate engine: computes prior
backend/app/services/forecast_engine.py - Forecast engine → Evidence scoring: scores evidence
backend/app/services/forecast_engine.py - Forecast engine → Belief updating: updates belief
backend/app/services/forecast_engine.py - Forecast engine → Scenario engine: compares scenarios
backend/app/services/forecast_engine.py - Evaluation endpoints → Replay engine: runs replay
backend/app/api/evaluations.py - Evaluation endpoints → Evaluation metrics: computes metrics
backend/app/api/evaluations.py - Evaluation endpoints → DB sessions: uses session
backend/app/api/evaluations.py - Replay engine → Evidence scoring: scores evidence
- Replay engine → Belief updating: updates belief
- Replay engine → Base-rate engine: computes prior
- Replay engine → Calibration: uses calibration
- Replay engine → Cost tracking: tracks cost
- Toolkit config → LLM client: sets model costs
forecast/llm.py - Evidence search → Toolkit config: reads quality settings
forecast/search.py - Evidence search → LLM client: calls for review
forecast/search.py - Evidence search → Forecast prompts: uses prompts
forecast/search.py - LLM client → LLM provider: sends completion
forecast/llm.py - Experiment tracking → Ablation runner: runs ablations
backend/app/api/experiments.py - Evaluation endpoints → Benchmark harness: exposes evaluation
- Evaluation endpoints → Backtesting: exposes backtests
- Evaluation endpoints → Calibration: exposes calibration
- Forecast metrics → LLM provider: evaluates forecasts