# jwang-580/asco_guideline_agents architecture

> Architecture diagram of the GitHub repository jwang-580/asco_guideline_agents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jwang-580/asco_guideline_agents
- Repository: https://github.com/jwang-580/asco_guideline_agents

## Overview

##### Project type and purpose
- This is an open-source evaluation tool for processing ASCO clinical guidelines. It implements a multi-agent framework (built with Autogen) that retrieves guideline PDFs, extracts relevant sections, formulates answers to clinical queries, and evaluates those answers against an expected QA dataset. It also provides non-agent, PDF-specific, and RAG-based evaluation pipelines.

##### File structure
- Top-level scripts:
- claude_autogen.py – sets up the multi-agent workflow (Coordinator, PDF Viewer, Reviewer) using Claude
- agent_eval.py – runs batch evaluations via the agent workflow, using GPT-4o as judge
- non_agent_eval.py – runs QA using various LLMs without agent orchestration
- pdf_viewer_eval.py – builds per-PDF LlamaIndex indices and evaluates retrieval QA
- RAG_eval.py – builds a unified RAG index across all PDFs and evaluates QA
- evaluate_answers.py – shared utilities to compare generated vs. expected answers and record results
- Data directories:
- data/ (guideline summaries, QA CSV)
- results/ (timestamped CSV evaluation outputs)
- Config:
- config.py loads environment variables (Anthropic/OpenAI API keys)
- Requirements & scripts reference libraries: Anthropic Claude, OpenAI, LlamaIndex, pandas, click/argparse

##### Extract insights from README
- Core workflow: user query → User Proxy agent → Tumor Board Coordinator agent (select guideline) → PDF Viewer agent (download & parse PDF) → Reviewer agent (formulate final answer)
- Evaluation modes: agent workflow, non-agent multi-model, per-PDF index, RAG index
- External services: Anthropic Claude 3.7, OpenAI GPT-4o, Gemini, DeepSeek
- Data stores: local PDF files, CSV QA dataset, LlamaIndex vector stores

a. Main components:
- CLI/entrypoint (agent_eval.py, non_agent_eval.py, pdf_viewer_eval.py, RAG_eval.py)
- Configuration loader (config.py)
- Multi-agent orchestration layer (claude_autogen.py):
- User Proxy agent
- Tumor Board Coordinator agent
- PDF Viewer agent
- Reviewer agent
- Retrieval services:
- PDF storage & parser
- LlamaIndex per-PDF indices
- Unified RAG index
- LLMs: Anthropic Claude, OpenAI GPT-4o, Gemini, DeepSeek
- Evaluation utility (evaluate_answers.py)
- Data storage:
- data/q_a.csv
- data/asco_guidelines.py (guideline metadata)
- results/*.csv
b. Relationships & interactions:
- CLI script reads config and CLI args, initializes agents or indexing modules
- For agent workflow: user query enters User Proxy → forwarded to Coordinator → Coordinator accesses guideline metadata → instructs PDF Viewer to load & parse PDF → parsed context passed to Reviewer → Reviewer calls Claude to generate answer → evaluate_answers compares answer to expected → write results
- For PDF eval: CLI triggers pdf_viewer_eval → builds per-PDF LlamaIndex → uses chosen LLM for answer → evaluate → results
- For RAG eval: CLI triggers RAG_eval → builds unified index → retrieval → answer → evaluation
- For non-agent: CLI triggers non_agent_eval → directly calls each LLM on question+context → evaluate
c. Architectural patterns and design principles:
- Modular agent pattern (decoupled responsibilities for query routing, retrieval, review)
- Pluggable evaluation pipelines (agent vs non-agent vs PDF vs RAG)
- Separation of concerns: config, orchestration, retrieval, LLM calls, evaluation
- Use of vector indexes for retrieval-augmented generation
d. Key technologies and frameworks:
- Python 3.x
- Autogen for multi-agent orchestration
- Anthropic Claude API, OpenAI API, other LLMs
- LlamaIndex for building and querying vector stores
- pandas for data loading and CSV output
- argparse or click for CLI

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Entrypoints
    subgraph "CLI Entrypoints"
        AgentEval["agent_eval.py"]:::cli
        NonAgentEval["non_agent_eval.py"]:::cli
        PDFViewerEval["pdf_viewer_eval.py"]:::cli
        RAGEval["RAG_eval.py"]:::cli
    end

    %% Configuration
    Config["config.py"]:::config

    %% Utilities
    Utils["utils.py"]:::util

    %% Multi-Agent Orchestration
    subgraph "Multi-Agent Orchestration"
        direction TB
        UserProxy["User Proxy Agent"]:::orchestration
        Coordinator["Tumor Board Coordinator Agent"]:::orchestration
        PDFViewerAgent["PDF Viewer Agent"]:::orchestration
        Reviewer["Reviewer Agent"]:::orchestration
    end

    %% Retrieval Services
    subgraph "Retrieval Services"
        direction TB
        PDFStore["Local PDF Store & Parser"]:::retrieval
        LlamaPerPDF["Per-PDF LlamaIndex"]:::retrieval
        RAGIndex["Unified RAG Index"]:::retrieval
    end

    %% LLM Services
    subgraph "LLM Services"
        direction TB
        Claude["Anthropic Claude"]:::llm
        GPT4o["OpenAI GPT-4o"]:::llm
        Gemini["Gemini"]:::llm
        DeepSeek["DeepSeek"]:::llm
    end

    %% Evaluation
    Evaluator["evaluate_answers.py"]:::evaluation

    %% Data Stores
    DataMeta["data/asco_guidelines.py"]:::data_store
    QAData["data/q_a.csv"]:::data_store
    Results["results/"]:::data_store

    %% CLI Dependencies
    AgentEval -->|reads config| Config
    NonAgentEval -->|reads config| Config
    PDFViewerEval -->|reads config| Config
    RAGEval -->|reads config| Config
    AgentEval --> Utils
    NonAgentEval --> Utils
    PDFViewerEval --> Utils
    RAGEval --> Utils

    %% Agent Workflow
    AgentEval --> UserProxy
    UserProxy --> Coordinator
    Coordinator --> DataMeta
    Coordinator --> PDFViewerAgent
    PDFViewerAgent --> PDFStore
    PDFViewerAgent --> Reviewer
    Reviewer -->|"generate answer"| Claude
    Reviewer -->|"generate answer"| GPT4o
    Reviewer -->|"generate answer"| Gemini
    Reviewer -->|"generate answer"| DeepSeek
    Reviewer --> Evaluator

    %% Non-Agent Workflow
    NonAgentEval -->|"direct QA"| Claude
    NonAgentEval -->|"direct QA"| GPT4o
    NonAgentEval -->|"direct QA"| Gemini
    NonAgentEval -->|"direct QA"| DeepSeek
    NonAgentEval --> Evaluator

    %% Per-PDF Retrieval Workflow
    PDFViewerEval --> LlamaPerPDF
    LlamaPerPDF -->|"retrieve context"| Claude
    LlamaPerPDF -->|"retrieve context"| GPT4o
    LlamaPerPDF --> Evaluator

    %% Unified RAG Workflow
    RAGEval --> RAGIndex
    RAGIndex -->|"retrieve context"| Claude
    RAGIndex -->|"retrieve context"| GPT4o
    RAGIndex --> Evaluator

    %% Evaluation to Results
    Evaluator --> Results

    %% Click Events
    click AgentEval "https://github.com/jwang-580/asco_guideline_agents/blob/main/agent_eval.py"
    click NonAgentEval "https://github.com/jwang-580/asco_guideline_agents/blob/main/non_agent_eval.py"
    click PDFViewerEval "https://github.com/jwang-580/asco_guideline_agents/blob/main/pdf_viewer_eval.py"
    click RAGEval "https://github.com/jwang-580/asco_guideline_agents/blob/main/RAG_eval.py"
    click Config "https://github.com/jwang-580/asco_guideline_agents/blob/main/config.py"
    click Utils "https://github.com/jwang-580/asco_guideline_agents/blob/main/utils.py"
    click DataMeta "https://github.com/jwang-580/asco_guideline_agents/blob/main/data/asco_guidelines.py"
    click QAData "https://github.com/jwang-580/asco_guideline_agents/blob/main/data/q_a.csv"
    click Evaluator "https://github.com/jwang-580/asco_guideline_agents/blob/main/evaluate_answers.py"
    click Results "https://github.com/jwang-580/asco_guideline_agents/tree/main/results/"
    
    %% Styles
    classDef cli fill:#f9f,stroke:#333,stroke-width:1px
    classDef config fill:#cff,stroke:#333,stroke-width:1px
    classDef orchestration fill:#fcf,stroke:#333,stroke-width:1px
    classDef retrieval fill:#ffc,stroke:#333,stroke-width:1px
    classDef llm fill:#cfc,stroke:#333,stroke-width:1px
    classDef evaluation fill:#ccf,stroke:#333,stroke-width:1px
    classDef data_store fill:#eef,stroke:#333,stroke-width:1px
    classDef util fill:#ddd,stroke:#333,stroke-width:1px
```
