How thunlp/deepnote works
Project type and purpose
• DeepNote is a research-oriented adaptive Retrieval-Augmented Generation (RAG) framework that orchestrates multi-stage pipelines: building retrieval indices, retrieving documents, generating and refining “notes,” and training or evaluating generative models. It is not a web app but a modular CLI/toolchain focused on knowledge exploration and robust QA performance across datasets (HotpotQA, MusiQue, 2WikiMQA, ASQA, StrategyQA).
File structure
• Top-level
• config/ (config.yaml for API keys, ds_config_zero3.json for DeepSpeed)
• prompts/ (templated queries for different QA tasks and retrieval strategies)
• src/
• build_index/es (BM25 index builders using ElasticSearch)
• build_index/emb (dense-embedding FAISS index builder)
• es_retrieve.py (ElasticSearch retrieval module)
• eval.py (evaluation metrics)
• gen_dpo_data.py (generate training examples via LLaMA or API)
• select_dpo_data.py (filter and process examples)
• train.py & train.sh (model fine-tuning pipeline)
• main.py (entrypoint for running DeepNote inference: adaptive retrieval + generation)
• utils.py (shared helpers)
Insights from README
• The process:
a. Prepare corpora and evaluation files under /data
b. Build indices (BM25 vs. dense)
c. Configure via config/config.yaml
d. Generate training data → filter/select it → train model
e. Run inference with main.py (deepnote method) and evaluate
• External services: ElasticSearch server; optional precomputed FAISS embeddings; LLM APIs (OpenAI or LLaMA).
a. Data Storage Layer
• Raw corpus (/data/corpus/wiki, datasets)
• Precomputed embeddings (optional .pkl)
• Generated notes & training data (/data/dpo_data, /data/dpo)
• Model checkpoints & outputs (output/{dataset}/)
b. Indexing Layer
• ElasticSearch cluster (BM25 indices via src/build_index/es)
• FAISS index (dense embeddings via src/build_index/emb)
c. Retrieval Engine
• es_retrieve.py → queries ES
• FAISS retrieval via embedded index.py
d. Management & Adaptive Loop (core DeepNote)
• main.py orchestrates:
– Initialize note (prompts/init_note)
– Retrieval trigger logic (based on note quality)
– refine_note, gen_new_query, gen_answer templates
– Stop criteria (max_step, max_fail_step)
• Prompts serve as templates for each stage
e. Language Model Interface
• LLaMA or OpenAI calls (via API key in config.yaml)
• wrapper in utils.py for sending prompts and receiving completions
f. Training Pipeline
1. gen_dpo_data.py (initial data generation with LLM)
2. select_dpo_data.py (filter/select examples)
3. train.py (fine-tune or RLHF with DeepSpeed)
4. train.sh (convenience script)
g. Evaluation Module
• eval.py (compute metrics on inference results)
– Raw corpus → Indexing → Retrieval Engine
– Retrieval Engine + Prompts + Model Interface → Note Loop → Final Answer
– Note Loop → Generated notes stored → optionally fed back into retrieval
– Training pipeline: Gen Data → Select Data → Train Model → Outputs stored
– Inference pipeline: Model Checkpoint → main.py → output results → eval.py
Architectural patterns & principles
• Modular pipeline: clear separation of indexing, retrieval, generation, training, evaluation
• Pluggable backends: ElasticSearch vs. FAISS; LLaMA vs. OpenAI
• Prompt-driven design: all query formulations and refinements managed as templates
• Iterative/loop pattern (note-centric adaptive retrieval)