# panshul42/forecasting_bot_q2 architecture

> Architecture diagram of the GitHub repository panshul42/forecasting_bot_q2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/panshul42/forecasting_bot_q2
- Repository: https://github.com/panshul42/forecasting_bot_q2

## Overview

##### Project Type and Purpose
- This is a Python-based forecasting pipeline (“Forecasting bot”) that automates probabilistic predictions for Metaculus questions. It is not a traditional web app but a data-pipeline/multi-agent system. It ingests question text files, uses LLM agents plus web-retrieval and content-extraction services, runs parallel forecasts, and outputs calibrated probabilities.

##### File Structure Highlights
- Top-level directories/files:
- Bot/ → All core Python modules (question parsing, LLM calls, search integration, content extractors, forecasters, ensemble logic).
- Q2_tournament_forecasts/ → Dataset of question text files.
- new_benchmark_o1/ → Supplemental dataset.
- benchmark.py → End-to-end benchmarking harness.
- requirements.txt, README.md, LICENSE.

##### README Insights
- Multi-stage retrieval-augmented architecture: question analysis → query generation → async retrieval → content extraction → context synthesis → parallel forecast generation → ensemble aggregation → benchmarking.
- Integrates multiple external services: OpenAI GPT, Anthropic Claude, Google Search API, AskNews, Perplexity, Bright Data (proxy).
- Uses Python asyncio/aiohttp for concurrent API calls.
- Extraction modules: BeautifulSoup, Trafilatura, Readability, fallback strategies.
- Forecast modules for binary, multiple‐choice, numeric questions; statistical calibration; benchmarking.

A. Data Layer
­– Question Store: Q2_tournament_forecasts/*.txt, new_benchmark_o1/*.txt
­– Configuration: requirements.txt, .env/.env.example (API keys)
B. Orchestration / Entry Point
­– Bot/main.py: CLI or scheduled runner
­– benchmark.py: benchmarking harness
C. Question Analysis & Query Generation
­– Bot/prompts.py, Bot/llm_calls.py: framing question; generating search queries
­– Bot/forecaster_template.py: shared forecasting scaffolding
D. Information Retrieval
­– Bot/search.py: orchestrates Google Search, Google News, AskNews, Perplexity APIs
­– Async layer: Python asyncio + aiohttp
E. Content Extraction
­– Bot/FastContentExtractor.py, Bot/HTMLContentExtractor.py: DOM analysis, readability, Trafilatura, fallback
F. Context Synthesis
­– Bot/forecaster.py: filters, ranks, and compiles retrieved text into context payloads
G. Parallel Forecast Generation (LLM Agents)
­– Bot/forecaster.py & specialized modules (binary.py, multiple_choice.py, numeric.py)
­– Agent pool:
- Claude 3.7 Sonnet (2 instances)
- GPT-o4-mini (2 instances)
- GPT-o3 (1 instance, double weight)
H. Ensemble Aggregation
­– Bot/forecaster.py: weighted averaging, calibration based on historical reliability
I. Output & Benchmarking
­– Forecast files or console output (CLI)
­– Bot/benchmark.py: simulates peer populations, calculates scores, visualizes performance

- Question Store → Orchestration
- Orchestration → Question Analysis → Query Generation
- Query Generation → Information Retrieval (parallel API calls)
- Retrieval → Content Extraction → Context Synthesis
- Context Synthesis → Parallel Forecast Generation (spawn 5 LLM agents)
- Forecast Generation → Ensemble Aggregation → Final Prediction
- Final Prediction → Output / Benchmarking module

##### Architectural Patterns and Principles
- Pipeline pattern: clear linear stages with asynchronous steps for I/O-bound tasks
- Parallelism: LLM calls and web fetches run concurrently using asyncio
- Modular separation: each concern (parsing, search, extraction, forecasting) lives in its own module
- Extensibility: easy to add new retrieval sources or forecasting agents by following existing interfaces
- Fault tolerance: retry/backoff strategies, graceful degradation for unavailable services

##### Technologies and Libraries
- Python 3.9+, asyncio, aiohttp
- LLM APIs: openai (GPT), anthropic (Claude)
- Search APIs: Google Serper, AskNews, Perplexity
- Extraction: BeautifulSoup, Trafilatura, Readability
- Statistical: numpy, scipy (for ensemble calibration), matplotlib/seaborn (benchmark visualizations)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        DS1["Question Store\n(Q2_tournament_forecasts/)"]:::data
        DS2["Supplemental Dataset\n(new_benchmark_o1/)"]:::data
        CFG["Configuration\n(requirements.txt, .env)"]:::data
    end

    %% Utilities
    subgraph "Utilities"
        DSLOADER["Dataset Loader\nBot/Dataset.py"]:::control
        META["Metaculus Bot Wrapper\nBot/metaculus_bot.py"]:::control
        TRY["Debug Scripts\nBot/try.py"]:::control
    end

    %% Orchestration & Control Flow
    subgraph "Orchestration"
        CLI["CLI Runner\nBot/main.py"]:::control
        BENCH_INIT["Benchmark Harness\nBot/benchmark.py"]:::control
    end

    %% Question Analysis & Query Generation
    subgraph "Question Analysis & Query Generation"
        PROMPTS["Prompt Definitions\nBot/prompts.py"]:::control
        LLMCALL["LLM Call Wrappers\nBot/llm_calls.py"]:::control
        SCAFFOLD["Forecasting Scaffolding\nBot/forecaster_template.py"]:::control
    end

    %% Information Retrieval
    subgraph "Information Retrieval"
        SEARCH["Search Orchestration\nBot/search.py"]:::retrieval
        BROWSER["Async HTTP Client\nBot/browser.py"]:::retrieval
        GIS["Google Search API"]:::external
        ANS["AskNews API"]:::external
        PLEX["Perplexity API"]:::external
    end

    %% Content Extraction
    subgraph "Content Extraction"
        FAST["FastContentExtractor\nBot/FastContentExtractor.py"]:::retrieval
        HTML["HTMLContentExtractor\nBot/HTMLContentExtractor.py"]:::retrieval
    end

    %% Context Synthesis
    CONTEXT["Context Synthesis\nBot/forecaster.py"]:::retrieval

    %% Parallel Forecast Generation
    subgraph "Agent Pool"
        BIN["Binary Agent\nBot/binary.py"]:::forecast
        MC["Multiple-Choice Agent\nBot/multiple_choice.py"]:::forecast
        NUM["Numeric Agent\nBot/numeric.py"]:::forecast
        CLAUDE["Anthropic Claude API"]:::external
        GPT_O4["GPT-o4-mini API"]:::external
        GPT_O3["GPT-o3 API"]:::external
    end

    %% Ensemble Aggregation
    AGG["Ensemble Aggregator\nBot/forecaster.py"]:::forecast

    %% Output & Benchmarking
    subgraph "Output & Benchmarking"
        OUT["Final Predictions\n(Console/Files)"]:::output
        BENCH2["Benchmarking & Visualization\nBot/benchmark.py"]:::output
    end

    %% Connections
    DS1 --> CLI
    DS2 --> CLI
    CFG --> CLI
    DSLOADER --> CLI
    META --> CLI
    TRY --> CLI

    CLI --> PROMPTS
    CLI --> LLMCALL
    CLI --> SCAFFOLD
    CLI --> BENCH_INIT

    PROMPTS --> LLMCALL
    LLMCALL --> SEARCH
    LLMCALL --> BROWSER

    SEARCH --> GIS
    SEARCH --> ANS
    SEARCH --> PLEX
    SEARCH --> BROWSER

    BROWSER --> FAST
    BROWSER --> HTML

    FAST --> CONTEXT
    HTML --> CONTEXT
    SCAFFOLD --> CONTEXT

    CONTEXT --> BIN
    CONTEXT --> MC
    CONTEXT --> NUM

    BIN --> CLAUDE
    BIN --> GPT_O4
    BIN --> GPT_O3

    MC --> CLAUDE
    MC --> GPT_O4
    MC --> GPT_O3

    NUM --> CLAUDE
    NUM --> GPT_O4
    NUM --> GPT_O3

    BIN --> AGG
    MC --> AGG
    NUM --> AGG

    AGG --> OUT
    OUT --> BENCH2

    BENCH_INIT --> BENCH2

    %% Click Events
    click DS1 "https://github.com/panshul42/forecasting_bot_q2/tree/main/Q2_tournament_forecasts/"
    click DS2 "https://github.com/panshul42/forecasting_bot_q2/tree/main/new_benchmark_o1/"
    click CFG "https://github.com/panshul42/forecasting_bot_q2/blob/main/requirements.txt"
    click DSLOADER "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/Dataset.py"
    click META "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/metaculus_bot.py"
    click TRY "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/try.py"
    click CLI "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/main.py"
    click BENCH_INIT "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/benchmark.py"
    click PROMPTS "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/prompts.py"
    click LLMCALL "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/llm_calls.py"
    click SCAFFOLD "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/forecaster_template.py"
    click SEARCH "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/search.py"
    click BROWSER "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/browser.py"
    click FAST "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/FastContentExtractor.py"
    click HTML "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/HTMLContentExtractor.py"
    click CONTEXT "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/forecaster.py"
    click BIN "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/binary.py"
    click MC "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/multiple_choice.py"
    click NUM "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/numeric.py"
    click AGG "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/forecaster.py"
    click BENCH2 "https://github.com/panshul42/forecasting_bot_q2/blob/main/Bot/benchmark.py"

    %% Styles
    classDef data fill:#ccffcc,stroke:#2d572c
    classDef control fill:#cce5ff,stroke:#2a5d9f
    classDef retrieval fill:#ffe6cc,stroke:#d97706
    classDef forecast fill:#e5ccff,stroke:#6b21a8
    classDef output fill:#f0f0f0,stroke:#5f5f5f
    classDef external fill:#fff5cc,stroke:#b59f00,stroke-dasharray: 2 2
```
