How seelebrn/deep-research-v2 works
Project Type and Purpose
• This is a Python-based research automation system (CLI tool) that orchestrates AI/ML models, academic database integrations, semantic storage, and an interactive chat interface.
• Its purpose is to take a user’s research query, break it down, search multiple academic and web sources, filter and compress content, store results in ChromaDB, and generate structured reports with citations.
File Structure
• Top-level scripts:
• start.py – primary entry point
• main.py – CLI interface for research pipelines and DB management
• knowledge_chat.py – interactive chat against persisted knowledge
• deep_research.py – orchestrator of the research pipeline
• academia.py – connectors to PubMed, HAL, arXiv, CrossRef, PEPITE
• deep_storage.py – ChromaDB persistence layer
• other scripts (debug.py, report_quality_enhancer.py, etc.) augment research flows
• Configuration / setup:
• .env.template / env.template – environment variables for model URLs, API keys, thresholds
• requirements.txt – Python dependencies (spaCy, chromadb, requests, OpenAI/LM SDK)
• Installer.bat – one-click installer on Windows
• setup.md – setup instructions
• External folders (implied by README):
• DBs/ – local vector databases per knowledge domain
• research_export_*/ – generated reports and export artifacts
Extract Architectural Insights from README
• Core pipeline stages:
1. Query Analysis
2. Multi-Cycle Source Discovery
3. Semantic Filtering & Compression
4. Knowledge Persistence (ChromaDB)
5. Synthesis & Export (reports, citations)
• Feedback loop for iterative refinement
• Integration points:
• LM Studio / OpenAI-like local server
• Search API (SearXNG)
• Academic database APIs
• spaCy for NLP
• ChromaDB for vector storage
Main Components and Their Interactions
a. User Interface
• CLI (start.py, main.py)
• knowledge_chat.py for interactive queries
b. Core Orchestrator
• deep_research.py invokes sub-modules for each pipeline stage
c. Academic Integrations
• academia.py calls external APIs for source discovery
d. Semantic Processing
• spaCy for keyword extraction
• deep_research.py and helper scripts for compression, verification
e. Persistence Layer
• deep_storage.py interacts with ChromaDB
f. External Services
• LM Studio / OpenAI server for generation and filtering models
• SearXNG (search API)
• Academic web APIs (PubMed, HAL, etc.)
g. Data Stores
• Local vector DBs (./DBs/{name}_knowledge_db/)
• Exported report files (research_export_*/)
Architectural Patterns and Design Principles
• Pipeline pattern: sequential stages with feedback loops
• Modular integration: separate modules for search, NLP, storage, synthesis
• Dependency injection via environment variables and CLI flags
• Layered architecture:
• Presentation (CLI)
• Application (orchestration)
• Domain (research logic, NLP)
• Infrastructure (storage, external APIs)