How khanz9664/personal_research_assistant works
Step 1: Identify the Project Type and Purpose
• This is a Python-based “Personal Research Assistant.”
• It behaves like a full-stack (CLI/GUI front end + back-end logic) application that automates data retrieval (APIs, scraping), decision making (agent), orchestration (workflow), and report generation.
• Key use case: researchers interact via the UI, trigger a coordinated pipeline that fetches data, reasons about it, and outputs summaries/reports.
Step 2: Analyze the File Structure
• Top-level directories:
• ui/ – user interface (app.py)
• orchestrator/ – workflow coordinator
• agent/ – decision-making, planning, memory, tool registry
• data_access/ – modules for external data sources (search APIs, web scraping, LLM summarizer)
• output/ – report/summarization generator
• agent/tools/search contains sub-modules integrating with ArXiv, GitHub, HuggingFace Hub, PapersWithCode
• Requirements.txt and .env hint at external dependencies (HTTP libraries, LLM SDKs, scraping tools)
Step 3: Examine the README for Additional Insights
• Confirms modules: Agent, Data Access, Orchestrator, Output, UI
• No explicit database: “memory” is in-memory components (working vs. long-term)
Step 4: Main Components and Their Interactions
a. Components:
1. UI Layer (ui/app.py)
2. Orchestrator (orchestrator/orchestrator.py)
3. Agent Core
– Decomposer, DecisionMaker, Planner
– Memory: WorkingMemory & LongTermMemory
– Tool registry & search tools
– Evaluation (source_evaluator)
4. Data Access
– SearchAPI, WebScraper, LLMSummarizer
5. Output
– ReportGenerator
6. External Services
– ArXiv API, GitHub API, HuggingFace Hub, PapersWithCode, LLM provider, target websites
b. Relationships & Data Flows:
1. User → UI
2. UI → Orchestrator
3. Orchestrator invokes Agent
4. Agent decomposes user request → Planner → DecisionMaker
5. DecisionMaker calls Data Access or Agent.Tools
6. Data Access ↔ External Services
7. Fetched data → Agent.Evaluation → Agent.Memory
8. Orchestrator collects Agent outputs → Output.ReportGenerator
9. Report → UI → User
c. Architectural Patterns & Principles:
– Layered architecture: UI / Orchestrator / Agent / DataAccess / Output
– Single Responsibility: each module handles one concern
– Mediator (Orchestrator) coordinates loosely coupled modules
– Plugin pattern for search tools (registry + individual search modules)
– In-memory state management (working vs. long-term memory)
d. Technologies & Frameworks:
– Python 3.x
– HTTP clients (e.g., requests, aiohttp) for APIs and scraping
– LLM SDK (OpenAI, HuggingFace transformers) for summarization
– Possibly Flask or Typer for UI (if CLI/HTTP)
– Data serialization (JSON)
– Logging/Config via .env
• Treat ui/ as the “frontend” layer (even if CLI/web)
• Orchestrator + agent + data_access + output as backend services
• Workflow sequence: UI → Orchestrator → Agent → DataAccess → External → Agent → Output → UI
Step 6: Diagramming Guidelines
Use clear labels for each module (e.g., “Agent: Decomposer, Planner, DecisionMaker”).
Step 7: Emphasize Detail
• Every folder as a sub-component if space allows
• Tool registry pattern under Agent
• Two memory types (working vs. long-term)
• Don’t omit evaluation step (source_evaluator)
• Flow of data from External Services back into Agent memory and then onward to report generation