How jwang-580/asco_guideline_agents works
Project type and purpose
• This is an open-source evaluation tool for processing ASCO clinical guidelines. It implements a multi-agent framework (built with Autogen) that retrieves guideline PDFs, extracts relevant sections, formulates answers to clinical queries, and evaluates those answers against an expected QA dataset. It also provides non-agent, PDF-specific, and RAG-based evaluation pipelines.
File structure
• Top-level scripts:
• claude_autogen.py – sets up the multi-agent workflow (Coordinator, PDF Viewer, Reviewer) using Claude
• agent_eval.py – runs batch evaluations via the agent workflow, using GPT-4o as judge
• non_agent_eval.py – runs QA using various LLMs without agent orchestration
• pdf_viewer_eval.py – builds per-PDF LlamaIndex indices and evaluates retrieval QA
• RAG_eval.py – builds a unified RAG index across all PDFs and evaluates QA
• evaluate_answers.py – shared utilities to compare generated vs. expected answers and record results
• Data directories:
• data/ (guideline summaries, QA CSV)
• results/ (timestamped CSV evaluation outputs)
• Config:
• config.py loads environment variables (Anthropic/OpenAI API keys)
• Requirements & scripts reference libraries: Anthropic Claude, OpenAI, LlamaIndex, pandas, click/argparse
Extract insights from README
• Core workflow: user query → User Proxy agent → Tumor Board Coordinator agent (select guideline) → PDF Viewer agent (download & parse PDF) → Reviewer agent (formulate final answer)
• Evaluation modes: agent workflow, non-agent multi-model, per-PDF index, RAG index
• External services: Anthropic Claude 3.7, OpenAI GPT-4o, Gemini, DeepSeek
• Data stores: local PDF files, CSV QA dataset, LlamaIndex vector stores
a. Main components:
• CLI/entrypoint (agent_eval.py, non_agent_eval.py, pdf_viewer_eval.py, RAG_eval.py)
• Configuration loader (config.py)
• Multi-agent orchestration layer (claude_autogen.py):
• User Proxy agent
• Tumor Board Coordinator agent
• PDF Viewer agent
• Reviewer agent
• Retrieval services:
• PDF storage & parser
• LlamaIndex per-PDF indices
• Unified RAG index
• LLMs: Anthropic Claude, OpenAI GPT-4o, Gemini, DeepSeek
• Evaluation utility (evaluate_answers.py)
• Data storage:
• data/q_a.csv
• data/asco_guidelines.py (guideline metadata)
• results/*.csv
b. Relationships & interactions:
• CLI script reads config and CLI args, initializes agents or indexing modules
• For agent workflow: user query enters User Proxy → forwarded to Coordinator → Coordinator accesses guideline metadata → instructs PDF Viewer to load & parse PDF → parsed context passed to Reviewer → Reviewer calls Claude to generate answer → evaluate_answers compares answer to expected → write results
• For PDF eval: CLI triggers pdf_viewer_eval → builds per-PDF LlamaIndex → uses chosen LLM for answer → evaluate → results
• For RAG eval: CLI triggers RAG_eval → builds unified index → retrieval → answer → evaluation
• For non-agent: CLI triggers non_agent_eval → directly calls each LLM on question+context → evaluate
c. Architectural patterns and design principles:
• Modular agent pattern (decoupled responsibilities for query routing, retrieval, review)
• Pluggable evaluation pipelines (agent vs non-agent vs PDF vs RAG)
• Separation of concerns: config, orchestration, retrieval, LLM calls, evaluation
• Use of vector indexes for retrieval-augmented generation
d. Key technologies and frameworks:
• Python 3.x
• Autogen for multi-agent orchestration
• Anthropic Claude API, OpenAI API, other LLMs
• LlamaIndex for building and querying vector stores
• pandas for data loading and CSV output
• argparse or click for CLI