How shan23chen/medbrowsecomp works
Project type and purpose
• MedBrowseComp is an open-source Python benchmarking and agentic‐system toolkit for multi-hop medical fact retrieval and synthesis.
• It glues together data files (ClinicalTrials.gov & PubMed), preprocessing utilities, LLM-based inference modules (Gemini, OpenAI, Sonar), and a Dockerized Streamlit demo agent.
File structure
• Top‐level scripts:
• process_NCT_predictions.py – orchestrates end-to-end predictions
• gemini_inference.py, openai_search_inference.py, sonar_inference.py – per-model adapters
• encode_decode.py, csv_generator.py – data encoding/decoding and sample generation
• Helpers: random_helper/ – utilities for data download, PubMed API, preprocessing, and notebooks
• Data: data/ – raw & encoded CSVs, prompts, EOBZIP files
• Agents demo: agents/computer-use-demo/ – Python package with loop.py, run_prompts.py, tools/ (bash, computer, edit, etc.), Streamlit UI, Dockerfile, entrypoint & HTTP server
• Results: results/ – output CSVs, analysis scripts, archived runs
README insights
• Core workflow: ingest benchmark CSV → optional encoding/decoding → run process_NCT_predictions with chosen model & flags → produce results CSV → analyze via notebooks/scripts
• External dependencies: GEMINI_API_KEY, OPENAI_API_KEY, SONAR_API_KEY, ClinicalTrials.gov API, Bio.Entrez (PubMed), optional Google Search tool
Main components
a. Data Sources & Storage
• Raw benchmark CSVs (data/final50.csv, data/final121.csv)
• Encoded CSV variants (data/..._b64.csv, _shift.csv)
• EOBZIP files, prompts CSVs
b. Data Preparation Layer
• encode_decode.py, csv_generator.py, random_helper/* utilities (download_nct_data, process_NCT_dataset, pubmed_utils)
c. Orchestrator / CLI
• process_NCT_predictions.py (accepts --csv_path, --model, --use_tools, --max_workers)
d. Inference Modules
• gemini_inference.py
• openai_search_inference.py
• sonar_inference.py
• (All call respective LLM APIs and external services)
e. Agent Demo (Dockerized)
• loop.py (agent control loop)
• run_prompts.py (invoke agent on prompts)
• tools/ (modular tool interfaces for bash, computer, edit, run)
• streamlit.py (web UI)
• HTTP server & static content (image/ folder)
f. External Services
• ClinicalTrials.gov API
• PubMed Bio.Entrez API
• LLM backends: Gemini, OpenAI, Sonar
• Optional Google Search tool
g. Results & Analysis
• results/ CSV outputs, merge_csv.py, hop_correctness_summary.py, notebooks
Relationships and interactions
• Data Files → Data Preparation → Orchestrator
• Orchestrator → Inference Module (selected per --model) → External LLM API + ClinicalTrials.gov/PubMed
• Inference returns predictions → Orchestrator writes Results CSV
• Post-processing → merge_csv.py, summary scripts → Notebooks & final analysis
• Separate “computer-use-demo” Docker container: accepts user input via Streamlit UI → loop.py invokes tools → displays agent actions in browser (via HTTP server + noVNC)
Architectural patterns and design principles
• Modular “one-script-per-responsibility” CLI pattern
• Agent pattern with pluggable tools (tools/*.py)
• Adapter pattern for multiple LLM backends
• Data pipeline flow (extract → transform → infer → load)
• Container boundary for demo agent