How amritkochar/gtmind works
Project Type and Purpose
• GTMind is a Python-based “autonomous research agent.”
• It implements a linear pipeline (Search → Parse → Extract → Aggregate → Persist/Serve) and exposes three user interfaces: CLI, HTTP API, and a Streamlit UI.
• It integrates with external services (Serper.dev for search, OpenAI for content extraction) and persists history to SQLite.
File Structure
• src/gtmind/core: pipeline modules (search.py, parse.py, extract.py, aggregate.py, settings.py)
• src/gtmind/api: Typer (CLI) + FastAPI (HTTP endpoint)
• src/gtmind/ui: Streamlit front-end
• src/gtmind/persistence.py: SQLite persistence logic
• sample_outputs.json: example report artifacts
• tests/: unit + integration tests validating each core component and end-to-end pipeline
README Insights
• The tech stack per layer is enumerated (httpx, trafilatura, OpenAI, RapidFuzz, sqlmodel, FastAPI, Typer, Streamlit).
Main Components & Their Relationships
A. External Dependencies
1. Serper.dev search API
2. The Web (HTML endpoints)
3. OpenAI GPT-4o API
B. Core Pipeline Container
1. Search Service (core/search.py)
2. Downloader + Boilerplate Stripper (core/parse.py)
3. Extractor (core/extract.py)
4. Aggregator (core/aggregate.py)
C. Persistence Layer
1. SQLite Database via SQLModel (persistence.py)
D. Serving Interfaces
1. CLI (Typer)
2. FastAPI HTTP Server
3. Streamlit UI
Data Flow (sequential):
[User → Interface] → Pipeline.Search → Pipeline.Parse → Pipeline.Extract → Pipeline.Aggregate → ResearchReport JSON → {CLI output, Database write, HTTP response, Streamlit render}
Architectural Patterns & Principles
• Pipeline pattern: each core module performs one transformation step.
• Separation of concerns: core logic vs. persistence vs. serving.
• Dependency injection via settings (core/settings.py) for API keys.
• Asynchronous I/O for HTTP calls (httpx in parse, search, and FastAPI).
Essential Elements
• Clear component labels and module paths (e.g., src/gtmind/core/extract.py).
• Versioned externals (OpenAI GPT-4o, Serper.dev).
• Data formats (JSON, SQLite).
– Rectangles for micro-services or Python modules
– Clouds for external APIs