How epuerta9/deep-research-py works
Main Components
• User Interface (CLI): Represented by the main entry point (run.py) which gathers research topics, research breadth, depth, and follow-up questions.
• Core Research Logic: The “deep_research.py” module serves as the central brain, coordinating research activities.
• AI Services: The “ai” directory contains modules like providers.py (configuring the AI service clients) and text_splitter.py (for text preprocessing).
• Data Acquisition: The “data_acquisition” folder holds modules such as manager.py, scraper.py, search.py, and services.py that handle web searches and scraping.
• Prompt & Feedback Handling: Modules like prompt.py (for composing system-prompts) and feedback.py (for follow-up question generation).
• Configuration Module: The config.py and .env file indicate where API keys and runtime settings are managed.
• Deployment & Containerization: Dockerfile and docker-compose.yml illustrate that the application can be containerized and integrated with external tools (e.g., OpenWebUI).
Relationships & Interactions
• The CLI (run.py) gathers user inputs and triggers the core research process.
• The Core Research module (deep_research.py) orchestrates calling out to:
• The Data Acquisition layer (web search & scraping via data_acquisition/ modules) to fetch relevant source content.
• The AI Services layer (modules under ai/) which processes inputs and generates content based on prompts provided by prompt.py.
• The Feedback module that interacts with the user to refine the search or expand upon findings.
• There is a clear separation whereby configuration (config.py and .env) feeds parameters such as API keys into both the Data Acquisition and AI Services layers.
• Optionally, when deployed with Docker/OpenWebUI, an external UI layer is placed in front of the CLI, routing requests to the back-end API (configured via docker-compose and exposed endpoints).
Architectural Patterns and Technologies
• Separation of Concerns: The project cleanly separates its logic into distinct modules responsible for UI, research logic, data fetching, AI interfacing, and configuration.
• CLI-Driven Workflow: The primary user interaction is command-line oriented, with well-defined modules handling distinct research functionalities.
• Use of External APIs: The AI (OpenAI or DeepSeek) and web search (Firecrawl) integrations are configurable, implying the design emphasizes flexibility and extensibility.
• Containerization: Docker and docker-compose usage indicates a micro-deployment container model to facilitate integration with tools like OpenWebUI.