# jolovicdev/shandu architecture

> Architecture diagram of the GitHub repository jolovicdev/shandu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jolovicdev/shandu
- Repository: https://github.com/jolovicdev/shandu

## Overview

##### Project type and purpose
- Shandu is a Python-based CLI library/tool, not a web application. It provides an AI-powered, state-based workflow (via LangGraph) for deep, multi-source research—driving search, scraping, content synthesis, and report generation.

##### File structure
- Top-level:
– setup.py, requirements.txt, MANIFEST.in, LICENSE → packaging & distribution
– cli.py → command-line entry point
– config.py & prompts.py → global configuration and prompt templates
– examples/ → sample usage and output
– tests/ → unit tests for core utilities (citation registry, report generator)

- shandu/agents/ → implements the LangGraph state graph
– graph/builder.py & wrapper.py → builds and executes the state graph
– nodes/ → each file is one node in the research pipeline (initialize, generate_queries, search, source_selection, citations, reflect, report_generation)
– processors/ → post-node processing: content_processor, report_generator
– utils/ → shared helpers (citation_manager, logger, agent_utils)

- shandu/research/ → high-level ResearchGraph API (researcher.py)
- shandu/search/ → AI-powered search adapters (ai_search.py) and generic search interface (search.py)
- shandu/scraper/ → web scraper module with JS rendering (scraper.py)
- shandu/utils/ → logging, caching, rate limiting

##### README for additional insights
Input (Q, Depth, Breadth) → Deep Research (DR) → SERP Queries (SQ) → Process Results (PR) → Learnings & Directions → decision node → either loop back or output Markdown report (MR).
- Architecture description highlights: state-based workflow, iterative exploration, multi-source synthesis, parallel processing, citation management.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI Layer"
        CLI["CLI Layer\n(cli.py)"]:::cli
    end

    subgraph "Configuration & Prompt Store"
        Config["Configuration\n(config.py)"]:::core
        Prompts["Prompt Store\n(prompts.py)"]:::core
    end

    subgraph "Core Research Engine"
        ResearchGraph["ResearchGraph API\n(researcher.py)"]:::core
    end

    subgraph "LangGraph Workflow Engine"
        Builder["Graph Builder\n(builder.py)"]:::core
        Wrapper["Graph Wrapper\n(wrapper.py)"]:::core
    end

    subgraph "Node Pipeline"
        Initialize["initialize()"]:::node
        Generate["generate_queries()"]:::node
        SearchNode["search()"]:::node
        SourceSel["source_selection()"]:::node
        Citations["citations()"]:::node
        Reflect["reflect()"]:::node
        ReportNode["report_generation()"]:::node
    end

    subgraph "Node Processors"
        ContentProc["Content Processor\n(content_processor.py)"]:::processor
        ReportGen["Report Generator\n(report_generator.py)"]:::output
    end

    subgraph "External Services"
        LLM["LLM Provider\n(OpenAI/GPT)"]:::external
        SearchEngines["Search Engines\n(Google, DuckDuckGo, Wikipedia, ArXiv)"]:::external
        Websites["Websites\n(scraped sources)"]:::external
    end

    subgraph "Citation Manager & Registry"
        CitMgr["Citation Manager\n(citation_manager.py)"]:::storage
        CitReg["Citation Registry\n(citation_registry.py)"]:::storage
        AgentUtils["Agent Utilities\n(agent_utils.py)"]:::storage
    end

    Scraper["Scraper Service\n(scraper.py)"]:::external
    Logger["Logging & Caching\n(logger.py)"]:::cross

    %% Connections
    CLI -->|"user input"| ResearchGraph
    Config --> ResearchGraph
    Prompts --> ResearchGraph
    ResearchGraph --> Builder
    Builder --> Wrapper
    Wrapper --> Initialize
    Initialize --> Generate
    Generate --> SearchNode
    SearchNode -->|"parallel"| SourceSel
    SourceSel --> Citations
    SourceSel --> ContentProc
    Citations --> CitMgr
    CitMgr --> CitReg
    ContentProc --> Scraper
    Scraper --> Websites
    ContentProc --> Reflect
    Reflect -->|"if depth>0"| Generate
    Reflect --> ReportNode
    ReportNode --> ReportGen
    ReportGen --> CLI

    %% External calls
    Generate --> LLM
    SearchNode --> SearchEngines
    ContentProc --> LLM

    %% Cross-cutting
    Config --- Logger
    Prompts --- Logger
    Builder --- Logger
    Wrapper --- Logger
    Initialize --- Logger
    Generate --- Logger
    SearchNode --- Logger
    SourceSel --- Logger
    Citations --- Logger
    ContentProc --- Logger
    Reflect --- Logger
    ReportNode --- Logger

    %% Click events
    click CLI "https://github.com/jolovicdev/shandu/blob/main/shandu/cli.py"
    click Config "https://github.com/jolovicdev/shandu/blob/main/shandu/config.py"
    click Prompts "https://github.com/jolovicdev/shandu/blob/main/shandu/prompts.py"
    click ResearchGraph "https://github.com/jolovicdev/shandu/blob/main/shandu/research/researcher.py"
    click Builder "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/graph/builder.py"
    click Wrapper "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/graph/wrapper.py"
    click Initialize "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/nodes/initialize.py"
    click Generate "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/nodes/generate_queries.py"
    click SearchNode "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/nodes/search.py"
    click SourceSel "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/nodes/source_selection.py"
    click Citations "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/nodes/citations.py"
    click Reflect "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/nodes/reflect.py"
    click ReportNode "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/nodes/report_generation.py"
    click ContentProc "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/processors/content_processor.py"
    click ReportGen "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/processors/report_generator.py"
    click SearchEngines "https://github.com/jolovicdev/shandu/blob/main/shandu/search/ai_search.py"
    click SearchEngines "https://github.com/jolovicdev/shandu/blob/main/shandu/search/search.py"
    click Scraper "https://github.com/jolovicdev/shandu/blob/main/shandu/scraper/scraper.py"
    click CitMgr "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/utils/citation_manager.py"
    click CitReg "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/utils/citation_registry.py"
    click AgentUtils "https://github.com/jolovicdev/shandu/blob/main/shandu/agents/utils/agent_utils.py"
    click Logger "https://github.com/jolovicdev/shandu/blob/main/shandu/utils/logger.py"

    %% Styles
    classDef cli fill:#D6E9C6,stroke:#333,stroke-width:1px
    classDef core fill:#AED6F1,stroke:#333,stroke-width:1px
    classDef node fill:#F9E79F,stroke:#333,stroke-width:1px
    classDef processor fill:#FAD7A0,stroke:#333,stroke-width:1px
    classDef output fill:#F5B7B1,stroke:#333,stroke-width:1px
    classDef external fill:#ABEBC6,stroke:#333,stroke-width:1px
    classDef storage fill:#FAD7A0,stroke:#333,stroke-width:1px
    classDef cross fill:#D2B4DE,stroke:#333,stroke-width:1px
```
