# vik-u/synbiokb-agent architecture

> Architecture diagram of the GitHub repository vik-u/synbiokb-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vik-u/synbiokb-agent
- Repository: https://github.com/vik-u/synbiokb-agent

## Overview

##### Project type & purpose
- This is a pipeline‐oriented, full-stack (CLI + Web API + Nextflow) application that builds an evidence-grounded biosynthesis knowledge base.
- It consists of a multi-agent, LLM-first workflow: Strategist → Search/OA → Crawl → Extract → Summarize → Validate → Store → Compose → Report.

##### Top-level components
A. Clients & Orchestration
- CLI “biosyn-kb” entrypoint
- Web API / UI (FastAPI + Jinja2 templates + queue)
- Nextflow orchestration + Docker Compose for containerized runs
B. Pipeline Agents (Python modules under src/biosyn_kb/)
1) Strategist (LLM input: user query → subqueries + OA hints)
2) Search & OA Enricher (SerpAPI/Brave + Unpaywall)
3) Crawler (httpx AsyncClient + robots.txt; optional Playwright headless)
4) Extractor (trafilatura → BS4 fallback; PDF via pdfminer/pdfplumber)
5) Summarizer (LLM generates PageSummary JSONL under Pydantic schema)
6) Validator (deterministic code checks + LLM-RAG on stored paragraphs)
7) Store (SQLAlchemy → SQLite DB)
8) Composer (LLM → consolidated Markdown report)
C. External Services & Libraries
- LLM Provider (Ollama local or OpenAI)
- Search APIs (SerpAPI or Brave Search HTTP)
- Unpaywall API for OA enrichment
- Vector store: ChromaDB + Sentence-Transformers embeddings
D. Data Stores & Artifacts
- JSONL artifacts/ (clean text, summaries)
- SQLite files under artifacts/
- Checkpoints under work/<run>/checkpoints/
- Generated Markdown reports under work/<run>/report.md

##### Relationships & data flow
[User CLI / Web UI]
→ invokes
[Orchestrator / Queue]
→ dispatches
[Strategist]
→ outputs plan.json →
[Search & OA]
→ outputs urls.json →
[Crawler]
→ saves HTML pages + crawl_report.json →
[Extractor]
→ clean JSONL + extract_report.json →
[Summarizer]
→ PageSummary JSONL + summaries_quality.json →
[Validator]
→ validation_report.json →
[Store]
→ SQLite DB →
[Composer]
→ Markdown report
- Overlay calls to external APIs at each stage (e.g., SerpAPI, Unpaywall, Ollama).

##### Architectural patterns & principles
- Modular, single-responsibility agents
- Asynchronous pipelines (asyncio + checkpoints)
- Pluggable LLM clients (factory pattern under src/biosyn_kb/llm/)
- Checkpointing for auditability (each stage writes JSON)
- Multi-modal data handling (HTML, PDF, JSON, DB, Markdown)
- Infrastructure-agnostic orchestration (CLI, Web, Nextflow, Docker Compose)

##### Technology call-outs
- Python 3.11+, asyncio, httpx, Playwright
- trafilatura, beautifulsoup4, pdfminer.six, pdfplumber
- Pydantic v2 for schema validation
- SQLAlchemy 2 + SQLite
- chromadb + sentence-transformers for embeddings/RAG
- FastAPI, uvicorn, Jinja2 for Web API / UI
- Nextflow + Docker Compose for containerized, reproducible runs

##### Essential elements
- Clear component names matching directory names (e.g., Crawl → src/biosyn_kb/crawl)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration Layer
    subgraph "Orchestration & Entry Points"
        direction TB
        CLI["CLI: biosyn-kb"]:::orchestration
        WebUI["Web API / UI"]:::orchestration
        Queue["Queue"]:::orchestration
        Nextflow["Nextflow Workflow"]:::orchestration
        DockerCompose["Docker Compose"]:::orchestration
    end

    %% Pipeline Agents
    subgraph "Pipeline Agents"
        direction TB
        Strategist["Strategist"]:::agent
        SearchOA["Search & OA Enricher"]:::agent
        Crawler["Crawler"]:::agent
        Extractor["Extractor"]:::agent
        Summarizer["Summarizer"]:::agent
        Validator["Validator"]:::agent
        StoreAgent["Store"]:::agent
        Composer["Composer"]:::agent
    end

    %% Data Stores & Artifacts
    subgraph "Data Stores & Artifacts"
        direction TB
        JSONL["JSONL Artifacts"]:::datastore
        SQLiteDB["SQLite DB"]:::datastore
        Reports["Markdown Report"]:::datastore
        Checkpoints["Checkpoints"]:::datastore
    end

    %% External Services
    subgraph "External Services"
        direction TB
        SerpAPI["SerpAPI"]:::external
        Brave["Brave Search"]:::external
        Unpaywall["Unpaywall API"]:::external
        Ollama["Ollama"]:::external
        OpenAI["OpenAI"]:::external
        ChromaDB["ChromaDB"]:::external
    end

    %% Flows
    CLI -->|"invoke"| Queue
    WebUI -->|"invoke"| Queue
    Nextflow -->|"calls CLI"| CLI
    Queue -->|"dispatch"| Strategist

    Strategist -->|"plan.json"| SearchOA
    SearchOA -->|"urls.json"| Crawler
    Crawler -->|"html + crawl_report.json"| Extractor
    Extractor -->|"clean.jsonl + extract_report.json"| Summarizer
    Summarizer -->|"pagesummary.jsonl + quality.json"| Validator
    Validator -->|"validation_report.json"| StoreAgent
    StoreAgent -->|"writes DB"| SQLiteDB
    StoreAgent -->|"checkpoint"| Checkpoints
    StoreAgent -->|"artifact JSONL"| JSONL
    StoreAgent --> Composer
    Composer -->|"report.md"| Reports

    %% External calls at stages
    SearchOA -->|"SerpAPI call"| SerpAPI
    SearchOA -->|"Brave HTTP"| Brave
    SearchOA -->|"Unpaywall API"| Unpaywall
    Strategist -->|"LLM call"| Ollama
    Strategist -->|"LLM call"| OpenAI
    Summarizer -->|"LLM call"| Ollama
    Summarizer -->|"LLM call"| OpenAI
    Validator -->|"RAG on ChromaDB"| ChromaDB

    %% Click Events
    click CLI "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/cli.py"
    click WebUI "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/webapp/main.py"
    click Queue "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/queue/dbq.py"
    click Nextflow "https://github.com/vik-u/synbiokb-agent/blob/main/nextflow/main.nf"
    click DockerCompose "https://github.com/vik-u/synbiokb-agent/blob/main/docker-compose.yml"
    click Strategist "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/agents/orchestrator.py"
    click SearchOA "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/search/base.py"
    click SerpAPI "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/search/serpapi.py"
    click Brave "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/search/brave.py"
    click Unpaywall "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/search/open_access.py"
    click Crawler "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/crawl/fetcher.py"
    click Extractor "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/extract/clean.py"
    click Summarizer "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/summarize/pipeline.py"
    click Validator "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/validate/validator.py"
    click StoreAgent "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/store/db.py"
    click Composer "https://github.com/vik-u/synbiokb-agent/blob/main/src/biosyn_kb/report/compose.py"
    click SQLiteDB "https://github.com/vik-u/synbiokb-agent/blob/main/artifacts/db.sqlite"
    click JSONL "https://github.com/vik-u/synbiokb-agent/tree/main/artifacts/jsonl/"
    click Checkpoints "https://github.com/vik-u/synbiokb-agent/tree/main/work/<run>/checkpoints/"
    click Reports "https://github.com/vik-u/synbiokb-agent/blob/main/work/<run>/report.md"

    %% Styles
    classDef orchestration fill:#cce5ff,stroke:#3399ff,stroke-width:2px
    classDef agent fill:#d4edda,stroke:#28a745,stroke-width:2px
    classDef datastore fill:#fff3cd,stroke:#ffcc00,stroke-width:2px
    classDef external fill:#e2e3e5,stroke:#6c757d,stroke-width:2px
```
