How fareedkhan-dev/autonomous-agentic-rag works
Project Type and Purpose
• This is a research‐prototype Jupyter Notebook implementing a self-improving, agentic RAG (Retrieval-Augmented Generation) pipeline for clinical trial design.
• It orchestrates multiple LLM roles (planner, researcher, SQL coder, director), builds vector stores, runs a multi-agent “Guild,” evaluates outputs on five dimensions, and evolves its own SOP (Standard Operating Procedure).
File Structure Analysis
• Top-level:
• LICENSE
• README.md (detailed tutorial and architecture description)
• code.ipynb (the implementation in notebook form)
• No separate “src” or “frontend/backend” directories – everything lives in the notebook, but logically the notebook contains modules for:
• Environment & config (.env, API keys)
• LLM model registry (“foundry” via Ollama)
• Knowledge ingestion (PubMed, FDA PDF, ethics text, MIMIC-III → DuckDB)
• Vector store creation (FAISS + Embeddings)
• Inner-loop RAG graph (LangGraph)
• Multi-dimensional evaluation suite
• Outer-loop evolution engine (diagnostician, architect, gene pool)
Key Architecture Components
A. Knowledge Infrastructure
• Data sources: PubMed abstracts, FDA guidelines (PDF→text), ethics summary (text), MIMIC-III CSVs → DuckDB.
• Vector stores: FAISS indexes for unstructured text (PubMed, FDA, ethics).
• Database: DuckDB file for structured clinical data.
B. Model Registry (“LLM Foundry”)
• Ollama clients for different model roles: planner (Llama 8B–json), drafter (Qwen 7B), SQL coder (Qwen 7B), director (Llama 70B–json), embedding model.
C. Inner-Loop Agentic Workflow (LangGraph)
1. Planner Agent: high-level JSON plan → tasks for specialists
2. Specialist Agents (parallelizable):
• Regulatory Specialist (FDA retriever)
• Medical Researcher (PubMed retriever)
• Ethics Specialist (ethics retriever)
• Patient Cohort Analyst (Text→SQL → DuckDB query)
3. Criteria Synthesizer: collates all agent outputs → final inclusion/exclusion document
D. Evaluation Gauntlet
• LLM-as-judge evaluators (director model) for Scientific Rigor, Regulatory Compliance, Ethical Soundness
• Programmatic evaluators for Recruitment Feasibility (SQL count normalized) and Operational Simplicity (keyword penalty)
• Aggregator: produces 5-dimensional score vector
E. Outer-Loop Evolution Engine
1. GenePool: stores SOP versions + their evaluation vectors + lineage
2. Performance Diagnostician: identifies primary weakness from 5D vector → structured Diagnosis
3. SOP Architect: mutates current SOP (prompts, flags, retriever k, model choices) → 2–3 candidate SOPs
4. Loop: each candidate runs inner-loop + evaluation → added to GenePool