How ra189zor/mindflow works
Project Type and Purpose
• MindFlow is a Python-based, AI-powered content generation pipeline with a Streamlit UI front-end.
• It orchestrates multiple specialized “agents” (Idea Generator, Filter, Researcher, Writer, Validator) via CrewAI/LangChain, persists research in a vector store, and exposes a live dashboard.
File Structure
• Top‐level files:
• app.py – Streamlit application (UI + orchestration)
• requirements.txt / .env.example – dependencies and API key configuration
• agents/ directory:
• Five agent modules (boss_agent.py, filter_agent.py, idea_agent.py, research_agent.py, writer_agent.py) – encapsulated AI tasks
• vectorstore/ directory:
• chroma_setup.py – ChromaDB initialization for caching research embeddings
Key README Insights
• Pipeline stages are linear with an iterative feedback loop for validation.
• Uses CrewAI + LangChain for agent definitions.
• External services: OpenAI LLM, DuckDuckGo scraper (Medium trends), ChromaDB, optional SQLite/InMemory cache.
• UI: Streamlit panels, status badges, live progress bar.
Main Components
a. Streamlit Frontend (app.py)
– Sidebar inputs (niche, keywords, parameters)
– Status pane & progress bar
– Orchestrator invoking each agent in sequence
b. Agent Modules
– Idea Agent (scrape Medium via DuckDuckGo → brainstorm)
– Filter Agent (rank & JSON‐output)
– Research Agent (gather facts, use vectorstore cache)
– Writer Agent (draft content)
– Boss/Validator Agent (JSON feedback, revision loop)
c. Vector Store
– ChromaDB instance (via chroma_setup.py) for embedding storage
– Optional in-memory or SQLite fallback
d. External Services
– OpenAI API (LLM calls with tenacity retries & concurrency throttling)
– DuckDuckGo search / Medium scraping
Relationships & Data Flows
• User → Streamlit UI
• UI → Agent Orchestrator → sequential AI calls
• Research Agent ↔ Vectorstore (read/write cache)
• All Agents → OpenAI API (model_name, max_concurrency)
• Idea Agent → DuckDuckGo/Medium scraper
• Validator Agent → Writer Agent (feedback loop, up to max_revisions)
• Final content → UI + export
Architectural Patterns & Design Principles
• Pipeline Pattern: discrete, single‐responsibility agents chained in order
• Feedback Loop: Boss Agent triggers Writer Agent revisions until approval
• Caching Layer: vectorstore for idempotency and performance
• Concurrency Control & Retry Logic: tenacity + max_concurrency parameters
• Separation of Concerns: UI, orchestration, agent logic, and persistence are modular
Technologies & Frameworks
• Streamlit (UI)
• CrewAI & LangChain (agent definitions & orchestration)
• ChromaDB (vectorstore)
• DuckDuckGo search API or web scraping
• tenacity (retry logic)
• InMemoryCache / SQLiteCache (LLM caching)