# thunlp/deepnote architecture

> Architecture diagram of the GitHub repository thunlp/deepnote, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thunlp/deepnote
- Repository: https://github.com/thunlp/deepnote

## Overview

##### Project type and purpose
- DeepNote is a research-oriented adaptive Retrieval-Augmented Generation (RAG) framework that orchestrates multi-stage pipelines: building retrieval indices, retrieving documents, generating and refining “notes,” and training or evaluating generative models. It is not a web app but a modular CLI/toolchain focused on knowledge exploration and robust QA performance across datasets (HotpotQA, MusiQue, 2WikiMQA, ASQA, StrategyQA).

##### File structure
- Top-level
- config/ (config.yaml for API keys, ds_config_zero3.json for DeepSpeed)
- prompts/ (templated queries for different QA tasks and retrieval strategies)
- src/
- build_index/es (BM25 index builders using ElasticSearch)
- build_index/emb (dense-embedding FAISS index builder)
- es_retrieve.py (ElasticSearch retrieval module)
- eval.py (evaluation metrics)
- gen_dpo_data.py (generate training examples via LLaMA or API)
- select_dpo_data.py (filter and process examples)
- train.py & train.sh (model fine-tuning pipeline)
- main.py (entrypoint for running DeepNote inference: adaptive retrieval + generation)
- utils.py (shared helpers)

##### Insights from README
- The process:
a. Prepare corpora and evaluation files under /data
b. Build indices (BM25 vs. dense)
c. Configure via config/config.yaml
d. Generate training data → filter/select it → train model
e. Run inference with main.py (deepnote method) and evaluate
- External services: ElasticSearch server; optional precomputed FAISS embeddings; LLM APIs (OpenAI or LLaMA).

a. Data Storage Layer
- Raw corpus (/data/corpus/wiki, datasets)
- Precomputed embeddings (optional .pkl)
- Generated notes & training data (/data/dpo_data, /data/dpo)
- Model checkpoints & outputs (output/{dataset}/)
b. Indexing Layer
- ElasticSearch cluster (BM25 indices via src/build_index/es)
- FAISS index (dense embeddings via src/build_index/emb)
c. Retrieval Engine
- es_retrieve.py → queries ES
- FAISS retrieval via embedded index.py
d. Management & Adaptive Loop (core DeepNote)
- main.py orchestrates:
– Initialize note (prompts/init_note)
– Retrieval trigger logic (based on note quality)
– refine_note, gen_new_query, gen_answer templates
– Stop criteria (max_step, max_fail_step)
- Prompts serve as templates for each stage
e. Language Model Interface
- LLaMA or OpenAI calls (via API key in config.yaml)
- wrapper in utils.py for sending prompts and receiving completions
f. Training Pipeline
1. gen_dpo_data.py (initial data generation with LLM)
2. select_dpo_data.py (filter/select examples)
3. train.py (fine-tune or RLHF with DeepSpeed)
4. train.sh (convenience script)
g. Evaluation Module
- eval.py (compute metrics on inference results)

– Raw corpus → Indexing → Retrieval Engine
– Retrieval Engine + Prompts + Model Interface → Note Loop → Final Answer
– Note Loop → Generated notes stored → optionally fed back into retrieval
– Training pipeline: Gen Data → Select Data → Train Model → Outputs stored
– Inference pipeline: Model Checkpoint → main.py → output results → eval.py

##### Architectural patterns & principles
- Modular pipeline: clear separation of indexing, retrieval, generation, training, evaluation
- Pluggable backends: ElasticSearch vs. FAISS; LLaMA vs. OpenAI
- Prompt-driven design: all query formulations and refinements managed as templates
- Iterative/loop pattern (note-centric adaptive retrieval)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    ESCluster["ElasticSearch Cluster"]:::external
    LLMAPI["OpenAI / LLaMA API"]:::external

    %% Configuration Management
    Config["Configuration Management"]:::config
    click Config "https://github.com/thunlp/deepnote/blob/main/config/config.yaml"

    %% Data Storage Layer
    subgraph "Data Storage Layer"
        direction TB
        RawCorpus["Raw Corpus (/data/corpus)"]:::storage
        Embeddings["Precomputed Embeddings (.pkl)"]:::storage
        DPOData["Generated DPO Data (/data/dpo_data)"]:::storage
        Checkpoints["Model Checkpoints & Outputs"]:::storage
    end

    %% Indexing Layer
    subgraph "Indexing Layer"
        direction TB
        subgraph "ElasticSearch BM25 Indexes"
            direction TB
            HotpotIndex["HotpotQA Index"]:::index
            MusiqueIndex["MusiQue Index"]:::index
            TwoWikiIndex["2WikiMQA Index"]:::index
        end
        FAISSIndex["FAISS Dense Embedding Index"]:::index
    end

    click HotpotIndex "https://github.com/thunlp/deepnote/blob/main/src/build_index/es/index_hotpotqa.py"
    click MusiqueIndex "https://github.com/thunlp/deepnote/blob/main/src/build_index/es/index_musique.py"
    click TwoWikiIndex "https://github.com/thunlp/deepnote/blob/main/src/build_index/es/index_2wiki.py"
    click FAISSIndex "https://github.com/thunlp/deepnote/blob/main/src/build_index/emb/index.py"

    %% Retrieval Engine
    subgraph "Retrieval Engine"
        direction TB
        ESRetrieve["ES Retrieval Module"]:::retrieval
        FAISSRetrieve["FAISS Retrieval Module"]:::retrieval
    end

    click ESRetrieve "https://github.com/thunlp/deepnote/blob/main/src/es_retrieve.py"
    click FAISSRetrieve "https://github.com/thunlp/deepnote/blob/main/src/build_index/emb/index.py"

    %% Prompt Templates
    Prompts["Prompt Templates"]:::prompts
    click Prompts "https://github.com/thunlp/deepnote/tree/main/prompts/"

    %% Note Management & Adaptive Loop
    NoteLoop["Adaptive Note Loop (main.py)"]:::loop
    click NoteLoop "https://github.com/thunlp/deepnote/blob/main/src/main.py"

    %% Language Model Interface
    Utils["LM Interface & Utilities (utils.py)"]:::interface
    click Utils "https://github.com/thunlp/deepnote/blob/main/src/utils.py"

    %% Training & Evaluation
    subgraph "Training & Evaluation"
        direction TB
        GenData["Generate DPO Data"]:::pipeline
        SelectData["Select DPO Data"]:::pipeline
        TrainModel["Train Model (train.py)"]:::pipeline
        TrainScript["Train Script (train.sh)"]:::pipeline
        EvalModule["Evaluation Module"]:::pipeline
    end

    click GenData "https://github.com/thunlp/deepnote/blob/main/src/gen_dpo_data.py"
    click SelectData "https://github.com/thunlp/deepnote/blob/main/src/select_dpo_data.py"
    click TrainModel "https://github.com/thunlp/deepnote/blob/main/src/train.py"
    click TrainScript "https://github.com/thunlp/deepnote/blob/main/src/train.sh"
    click EvalModule "https://github.com/thunlp/deepnote/blob/main/src/eval.py"

    %% README & Requirements
    Docs["README & Requirements"]:::docs
    click Docs "https://github.com/thunlp/deepnote/blob/main/README.md"

    %% Connections
    RawCorpus -->|feeds| HotpotIndex
    RawCorpus -->|feeds| MusiqueIndex
    RawCorpus -->|feeds| TwoWikiIndex
    Embeddings -->|feeds| FAISSIndex
    HotpotIndex -->|BM25 indices| ESRetrieve
    MusiqueIndex -->|BM25 indices| ESRetrieve
    TwoWikiIndex -->|BM25 indices| ESRetrieve
    FAISSIndex -->|dense index| FAISSRetrieve

    ESRetrieve -->|retrieved docs| NoteLoop
    FAISSRetrieve -->|retrieved docs| NoteLoop
    Prompts -->|templates| NoteLoop
    NoteLoop -->|prompts| Utils
    NoteLoop -->|notes JSON| DPOData
    NoteLoop -->|final answer| Checkpoints

    Checkpoints -->|checkpoint| NoteLoop
    Checkpoints -->|model for inference| NoteLoop

    Config --> NoteLoop
    Config --> TrainModel
    Config --> GenData

    ESCluster -->|query| ESRetrieve
    LLMAPI -->|completion| Utils

    %% Training Pipeline Flow
    GenData -->|raw examples| SelectData
    SelectData -->|filtered examples| TrainModel
    TrainModel -->|fine-tuned model| Checkpoints
    TrainScript --> TrainModel

    %% Evaluation
    Checkpoints -->|inference| EvalModule
    EvalModule --> Docs

    %% Styles
    classDef storage fill:#cce5ff,stroke:#004085,color:#004085
    classDef index fill:#ffe5b4,stroke:#cc7a00,color:#cc7a00
    classDef retrieval fill:#d4edda,stroke:#155724,color:#155724
    classDef loop fill:#d4edda,stroke:#155724,color:#155724
    classDef interface fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef pipeline fill:#f8f9fa,stroke:#343a40,color:#343a40
    classDef external fill:#f5c6cb,stroke:#721c24,color:#721c24
    classDef config fill:#fff3cd,stroke:#856404,color:#856404
    classDef prompts fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef docs fill:#e2e3e5,stroke:#6c757d,color:#6c757d
```
