How anselmjeong/deepresearch works
Project type and purpose
• From the file tree, this is a standalone Python-based CLI/toolkit named “deepresearch.”
• It appears to automate literature ingestion and processing: PDF→markdown conversion, PubMed lookups (via DOI, BibTeX), citation “snowballing,” and stateful UI components for reviewing results.
File structure
• Top-level:
• deepresearch/ ← main package
• config.toml, rxconfig.py ← application configuration
• pyproject.toml, requirements.txt ← build/dependency specs
• Inside deepresearch/:
• deepresearch.py ← CLI entrypoint / orchestrator
• state.py ← application state management
• components/ ← UI rendering (Left/Middle/Right column)
• lib/ ← core services & utilities
– pdf_process.py ← PDF text extraction
– pdf_to_md/ ← text→markdown conversion
– process_directory.py ← bulk directory traversal
– pubmedder/ ← PubMed API integration (DOI finder, BibTeX parser, etc.)
– researcher.py ← high-level research workflows
– snowballer/ ← citation-expansion logic
Implicit architecture
• Orchestrator (deepresearch.py) initializes state, reads config, then invokes researcher workflows.
• researcher.py coordinates these sub-services: PDF ingestion → conversion → metadata lookup → snowballing.
• UI components subscribe to state changes and render columns of information.
• Config and state modules decouple environment settings from business logic.
• External systems: file system, PubMed API, local storage for markdown output.
a. Main components:
1) CLI Orchestrator
2) Configuration & State Manager
3) PDF Processor
4) Markdown Converter
5) Directory Scanner
6) PubMed API Client (pubmedder)
7) Citation Snowballer
8) UI Renderer (Left/Middle/Right columns)
9) File System / Assets / Output Store
b. Relationships & data flow:
• CLI → Config/State → Directory Scanner → PDF Processor → Markdown Converter → File System
• Markdown Converter → Researcher → PubMed API Client → Citation Snowballer → State
• State → UI Renderer → Terminal/Console
c. Architectural patterns & principles:
• Layered architecture: presentation (UI) / orchestration / services / infrastructure
• Single-responsibility: each lib/ submodule has one domain focus
• Dependency inversion: orchestrator depends on abstract service interfaces defined in lib/
• Extensibility: new components can be dropped into components/ or lib/ subpackages
d. Key technologies & libraries:
• Python 3.x (pyproject.toml)
• click or argparse (implied for CLI)
• pdfminer or PyPDF2 (in pdf_process.py)
• Markdown libraries (in pdf_to_md/fulltexter.py)
• Requests or Entrez API client (in pubmedder/*.py)
• Step 2: Beneath it, place Config/State Manager and Directory Scanner side by side.
• Step 3: Chain to PDF Processor → Markdown Converter in a vertical flow.
• Step 4: Branch from Markdown Converter to File System for output storage.
• The sequential data-processing pipeline (PDF → MD → storage)
• The research enrichment loop (metadata fetch & snowballing)
• The separation of concerns (orchestration vs. services vs. UI)
• Extension and integration points for future enhancements.