How panshul42/forecasting_bot_q2 works
Project Type and Purpose
• This is a Python-based forecasting pipeline (“Forecasting bot”) that automates probabilistic predictions for Metaculus questions. It is not a traditional web app but a data-pipeline/multi-agent system. It ingests question text files, uses LLM agents plus web-retrieval and content-extraction services, runs parallel forecasts, and outputs calibrated probabilities.
File Structure Highlights
• Top-level directories/files:
• Bot/ → All core Python modules (question parsing, LLM calls, search integration, content extractors, forecasters, ensemble logic).
• Q2_tournament_forecasts/ → Dataset of question text files.
• new_benchmark_o1/ → Supplemental dataset.
• benchmark.py → End-to-end benchmarking harness.
• requirements.txt, README.md, LICENSE.
README Insights
• Multi-stage retrieval-augmented architecture: question analysis → query generation → async retrieval → content extraction → context synthesis → parallel forecast generation → ensemble aggregation → benchmarking.
• Integrates multiple external services: OpenAI GPT, Anthropic Claude, Google Search API, AskNews, Perplexity, Bright Data (proxy).
• Uses Python asyncio/aiohttp for concurrent API calls.
• Extraction modules: BeautifulSoup, Trafilatura, Readability, fallback strategies.
• Forecast modules for binary, multiple‐choice, numeric questions; statistical calibration; benchmarking.
A. Data Layer
– Question Store: Q2_tournament_forecasts/*.txt, new_benchmark_o1/*.txt
– Configuration: requirements.txt, .env/.env.example (API keys)
B. Orchestration / Entry Point
– Bot/main.py: CLI or scheduled runner
– benchmark.py: benchmarking harness
C. Question Analysis & Query Generation
– Bot/prompts.py, Bot/llm_calls.py: framing question; generating search queries
– Bot/forecaster_template.py: shared forecasting scaffolding
D. Information Retrieval
– Bot/search.py: orchestrates Google Search, Google News, AskNews, Perplexity APIs
– Async layer: Python asyncio + aiohttp
E. Content Extraction
– Bot/FastContentExtractor.py, Bot/HTMLContentExtractor.py: DOM analysis, readability, Trafilatura, fallback
F. Context Synthesis
– Bot/forecaster.py: filters, ranks, and compiles retrieved text into context payloads
G. Parallel Forecast Generation (LLM Agents)
– Bot/forecaster.py & specialized modules (binary.py, multiple_choice.py, numeric.py)
– Agent pool:
• Claude 3.7 Sonnet (2 instances)
• GPT-o4-mini (2 instances)
• GPT-o3 (1 instance, double weight)
H. Ensemble Aggregation
– Bot/forecaster.py: weighted averaging, calibration based on historical reliability
I. Output & Benchmarking
– Forecast files or console output (CLI)
– Bot/benchmark.py: simulates peer populations, calculates scores, visualizes performance
• Question Store → Orchestration
• Orchestration → Question Analysis → Query Generation
• Query Generation → Information Retrieval (parallel API calls)
• Retrieval → Content Extraction → Context Synthesis
• Context Synthesis → Parallel Forecast Generation (spawn 5 LLM agents)
• Forecast Generation → Ensemble Aggregation → Final Prediction
• Final Prediction → Output / Benchmarking module
Architectural Patterns and Principles
• Pipeline pattern: clear linear stages with asynchronous steps for I/O-bound tasks
• Parallelism: LLM calls and web fetches run concurrently using asyncio
• Modular separation: each concern (parsing, search, extraction, forecasting) lives in its own module
• Extensibility: easy to add new retrieval sources or forecasting agents by following existing interfaces
• Fault tolerance: retry/backoff strategies, graceful degradation for unavailable services
Technologies and Libraries
• Python 3.9+, asyncio, aiohttp
• LLM APIs: openai (GPT), anthropic (Claude)
• Search APIs: Google Serper, AskNews, Perplexity
• Extraction: BeautifulSoup, Trafilatura, Readability
• Statistical: numpy, scipy (for ensemble calibration), matplotlib/seaborn (benchmark visualizations)