How kazdatahelp/automated-ai-web-researcher-ollama works
Project Type & Purpose
• This is a standalone Python-based automated research assistant that leverages locally running large language models (via Ollama) to conduct real-time, structured online research.
• It automatically breaks down user queries into multiple focused research areas; then, it performs targeted web searches, scrapes results, aggregates findings, and finally produces a comprehensive summary. In addition, it supports a follow-up Q&A conversation mode based on the accumulated research.
Analyzing the File Structure
• There is no separate directory hierarchy; rather, the project comprises several Python files at the top level. This suggests a monolithic design for the research automation logic.
• Key files include:
• Web-LLM.py: Likely the main entry point for initiating the research session and handling user interface (CLI-based).
• research_manager.py: Expected to manage the overall research workflow, including managing focus areas, progress tokens, and the orchestration of web search cycles.
• web_scraper.py: Responsible for interacting with external websites to fetch and scrape content based on search queries.
• llm_wrapper.py, llm_response_parser.py, and llm_config.py: These files encapsulate the integration with the local LLM (Ollama), handling configuration, making API calls, and parsing responses.
• Self_Improving_Search.py & strategic_analysis_parser.py: Likely provide additional logic such as refining research queries (self-improving capabilities) and extracting deeper insights from the data.
• Files like requirements.txt, README.md, LICENSE, and .gitignore set up the necessary environment and documentation for the project.
a. Main Components:
• User Interface / CLI Input: Represents where users provide their research queries and commands (triggered by “@”, “s”, “p”, “q”, etc.).
• Research Manager: The central orchestrator that coordinates the overall research flow. It receives user input, distributes tasks, and aggregates results.
• LLM Controller:
• llm_config.py: Contains LLM settings (e.g., model name, temperature, context length) for the Ollama integration.
• llm_wrapper.py and llm_response_parser.py: Abstract and wrap the interactions with the LLM, including sending queries and processing responses.
• Web Scraper / Search Module: Responsible for formulating search URLs, triggering web searches, and scraping and parsing information from selected web pages.
• Self-Improvement / Analysis Module: (via Self_Improving_Search.py and strategic_analysis_parser.py) Further refines research queries and analysis of gathered information.
• Output Generator: Handles the aggregation of research results and generates the final summary report as well as the research session’s text file (documenting content and source URLs).
b. Relationships and Data Flow:
• The User Interface passes the initial research query and commands to the Research Manager.
• The Research Manager decomposes the query into multiple focused areas.
• For each focus area, the Research Manager sends targeted searches to the Web Scraper module.
• Parallelly, the LLM Controller is called (via llm_wrapper) to:
• Query,
• Assist in generating search queries or focus follow-ups,
• Parse responses using llm_response_parser.
• The Web Scraper fetches relevant content from the internet and returns the data to the Research Manager.
• The Self-Improvement Module refines the process by re-evaluating and generating new focus areas if needed.
• Once content is collected, the Research Manager uses the LLM again to synthesize and compile the final summary.
• Finally, the summary and the complete research document are output and (optionally) the system moves to a conversation mode for Q&A.
c. Architectural Patterns & Design Principles:
• Modular design even though all modules live in a single file directory: separating responsibilities (UI, orchestration, LLM integration, web scraping, and output generation).
• Clear service boundaries between the web scraping logic and the LLM interfacing, supporting maintainability and potential future refactoring.
• A feedback loop for self-improvement (query refinement) and dynamic research expansion.
• Use of configuration (in llm_config.py) enabling flexibility with different LLM models and parameters.
d. Relevant Technologies & Frameworks:
• Python (core language) for the entire implementation.
• Ollama as the local LLM engine.
• Standard libraries for web-request handling and text parsing (likely via requests or similar modules internally).
• No explicit web framework appears to be used; the script is driven by CLI inputs.