How poldrack/ai-peer-review works
Project type and purpose
• This is an open-source command-line tool (Python package) called “AI Peer Review.” Its job is to take a PDF of an academic paper, send it to multiple LLMs, collect individual reviews, synthesize a meta-review, and emit results in markdown, CSV, and JSON. It’s not a full-stack web app but a CLI-centric orchestration tool that integrates with external LLM APIs.
File structure
Top-level files:
• pyproject.toml – build/config for packaging
• README.md/.env.example – user documentation and sample environment variables
• src/ai_peer_review/cli.py – entrypoint for commands (config, review)
• src/ai_peer_review/utils/… – modules for configuration, PDF parsing
• src/ai_peer_review/llm_clients/… – one client per supported LLM, all subclassing a base_client
• src/ai_peer_review/review.py – core logic: orchestrate reviews, generate meta-review, write outputs
• tests/… – unit tests covering CLI, utils, clients, review logic
README for additional insights
• Workflow:
a) Load API keys (from CLI config or .env)
b) Parse PDF → text
c) For each configured LLM client: send prompt + paper → receive review
d) Aggregate reviews → meta-review via prompt
e) Produce files in output directory
Main components and their interactions
a. CLI Layer (cli.py)
• Parses user commands/flags
• Kicks off “review” command or “config” command
b. Configuration Manager (utils/config.py)
• Reads/writes ~/.ai-peer-review/config.json or environment variables
c. PDF Processor (utils/pdf.py)
• Extracts text from a PDF file path
d. Core Orchestrator (review.py)
• Instantiates all requested LLM clients
• Feeds them the review prompt + paper text
• Collects individual reviews
• Sends collected reviews to meta-review prompt
• Emits final artifacts via file I/O
e. LLM Clients (llm_clients/*.py)
• Each client implements a “generate_review” method calling a specific external API
• Share common interface defined in base_client.py
f. Output Writer (inside review.py)
• Writes review_{model}.md, meta_review.md, concerns_table.csv, results.json to disk
g. External Services
• OpenAI, Anthropic, Google AI, Together API – invoked by LLM clients
Architectural patterns / design principles
• Plugin-style client architecture: each LLM client is interchangeable, all inheriting from a base class
• Single-responsibility: CLI, config management, PDF parsing, orchestration, API clients, and I/O are decoupled
• Dependency injection: orchestrator loops over configured clients rather than hard-coding them