How vik-u/synbiokb-agent works
Project type & purpose
• This is a pipeline‐oriented, full-stack (CLI + Web API + Nextflow) application that builds an evidence-grounded biosynthesis knowledge base.
• It consists of a multi-agent, LLM-first workflow: Strategist → Search/OA → Crawl → Extract → Summarize → Validate → Store → Compose → Report.
Top-level components
A. Clients & Orchestration
• CLI “biosyn-kb” entrypoint
• Web API / UI (FastAPI + Jinja2 templates + queue)
• Nextflow orchestration + Docker Compose for containerized runs
B. Pipeline Agents (Python modules under src/biosyn_kb/)
1) Strategist (LLM input: user query → subqueries + OA hints)
2) Search & OA Enricher (SerpAPI/Brave + Unpaywall)
3) Crawler (httpx AsyncClient + robots.txt; optional Playwright headless)
4) Extractor (trafilatura → BS4 fallback; PDF via pdfminer/pdfplumber)
5) Summarizer (LLM generates PageSummary JSONL under Pydantic schema)
6) Validator (deterministic code checks + LLM-RAG on stored paragraphs)
7) Store (SQLAlchemy → SQLite DB)
8) Composer (LLM → consolidated Markdown report)
C. External Services & Libraries
• LLM Provider (Ollama local or OpenAI)
• Search APIs (SerpAPI or Brave Search HTTP)
• Unpaywall API for OA enrichment
• Vector store: ChromaDB + Sentence-Transformers embeddings
D. Data Stores & Artifacts
• JSONL artifacts/ (clean text, summaries)
• SQLite files under artifacts/
• Checkpoints under work/<run>/checkpoints/
• Generated Markdown reports under work/<run>/report.md
Relationships & data flow
[User CLI / Web UI]
→ invokes
[Orchestrator / Queue]
→ dispatches
[Strategist]
→ outputs plan.json →
[Search & OA]
→ outputs urls.json →
[Crawler]
→ saves HTML pages + crawl_report.json →
[Extractor]
→ clean JSONL + extract_report.json →
[Summarizer]
→ PageSummary JSONL + summaries_quality.json →
[Validator]
→ validation_report.json →
[Store]
→ SQLite DB →
[Composer]
→ Markdown report
• Overlay calls to external APIs at each stage (e.g., SerpAPI, Unpaywall, Ollama).
Architectural patterns & principles
• Modular, single-responsibility agents
• Asynchronous pipelines (asyncio + checkpoints)
• Pluggable LLM clients (factory pattern under src/biosyn_kb/llm/)
• Checkpointing for auditability (each stage writes JSON)
• Multi-modal data handling (HTML, PDF, JSON, DB, Markdown)
• Infrastructure-agnostic orchestration (CLI, Web, Nextflow, Docker Compose)
Technology call-outs
• Python 3.11+, asyncio, httpx, Playwright
• trafilatura, beautifulsoup4, pdfminer.six, pdfplumber
• Pydantic v2 for schema validation
• SQLAlchemy 2 + SQLite
• chromadb + sentence-transformers for embeddings/RAG
• FastAPI, uvicorn, Jinja2 for Web API / UI
• Nextflow + Docker Compose for containerized, reproducible runs
Essential elements
• Clear component names matching directory names (e.g., Crawl → src/biosyn_kb/crawl)