# iluko/docindex architecture

> Architecture diagram of the GitHub repository iluko/docindex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iluko/docindex
- Repository: https://github.com/iluko/docindex

## Overview

1. **Overall shape**
   This repo is a multi-stage document RAG system with three major planes: **ingestion/indexing**, **query/retrieval/answering**, and **observability/evaluation**. The production runtime is mostly Python, with a separate **React/Vite frontend** and a **Streamlit UI** for interactive use.

2. **Primary user entrypoints**
   The main operational entrypoints are `cli.py` for command-line ingestion/querying, `app.py` for Streamlit, `frontend/` for the React app, and `api/` for a Python API layer. These all share the same underlying runtime components rather than duplicating logic.

3. **Ingestion pipeline**
   The ingestion subsystem lives in `ingestion/` and converts source files into indexable document structures. It handles document parsing/enrichment for `.pdf`, `.md`, `.markdown`, and `.docx`, with DOCX conversion preserving headings. PageIndex is the parsing foundation that produces a hierarchical per-document tree, which then feeds downstream indexing.

4. **Document index artifacts**
   Ingested content is materialized into multiple durable artifacts under `data/` or MongoDB depending on storage backend: uploaded originals, per-document trees, derived markdown, and a **master tree** that acts as the cross-document routing index. The repo is explicitly project-scoped, so each knowledge base gets isolated artifacts and its own `master_tree.json` and related index files.

5. **Master tree and routing metadata**
   The `master_tree/` subsystem defines the document-level routing representation. It stores document summaries plus routing hints such as topics, relationships, and newer routing facets like workflows, actors, systems, edge cases, and authority hints. This is the key boundary between ingestion and retrieval: ingestion enriches documents into a compact corpus-wide index that query-time routing can reason over.

6. **Retrieval pipeline**
   The `retrieval/` package implements the standard hybrid query flow: **router → navigator → verifier → fetcher → answer synthesis**. Router selects documents from the master tree, navigator selects sections within those docs, verifier optionally filters false positives, and fetcher pulls raw text chunks with context windows and token budgeting. The pipeline is deterministic in standard mode and traceable step by step.

7. **Advanced and alternate retrieval modes**
   There are two retrieval modes in the repo: **hybrid** and **pageindex**. Hybrid is the default bounded pipeline; pageindex is an agentic tool-use loop that iteratively inspects document structure and content. Advanced retrieval adds optional planning, adaptive width, and node expansion on top of the hybrid path, but these are explicitly opt-in and designed for broader, workflow-heavy questions.

8. **Storage boundary**
   The `storage/` package abstracts persistence behind a common interface. The default backend is local filesystem storage under `data/`, but the repo also supports MongoDB via `storage/mongo_store.py`. This backend boundary is important architecturally: the rest of the application should not care whether indexes and traces are on disk or in MongoDB.

9. **Traceability and audit trail**
   The `traces/` package records retrieval-time observability data, including routed docs, selected nodes, fetched chunks, token budgets, and mode-specific metrics for pageindex exploration. This is not just logging; it is part of the product’s architecture because every query returns a trace object used by the UI and CLI for auditability.

10. **Frontend surfaces**
   The `frontend/` directory is a standalone React app built with Vite and TypeScript. It provides separate pages for asking questions, ingesting documents, and inspecting results, with components for chat threads, inspector panels, and selection controls. It talks to the backend through a small API client layer under `frontend/src/api/`.

11. **Streamlit and Python API surfaces**
   The Streamlit app in `app.py` is a lightweight operational UI that exposes upload, query, and trace inspection without the full React stack. The `api/` package and `index_registry.py`/`model_registry.py` provide reusable runtime assembly so scripts, UI, and services all construct the same storage, tree, and retrieval objects.

12. **Experiments and evaluation subsystem**
   The `experiments/` tree is a separate evaluation harness, not the production query path. It contains retrieval variants, build presets, datasets, judges, scoring, comparison reports, and run registries. This makes it a parallel subsystem for offline benchmarking and regression analysis against curated corpora and golden datasets.

13. **Test and fixture boundaries**
   The `tests/` directory covers ingestion, retrieval, master tree logic, registries, verifier behavior, and experiments. Fixtures include sample document trees and sample master trees, which mirrors the repo’s core data model and helps validate the boundaries between parsing, indexing, retrieval, and storage.

14. **End-to-end data flow**
   The core runtime flow is: source document → ingestion/parsing → per-doc tree + markdown → master-node generation → project-scoped storage → query-time routing over master tree → section navigation → optional verification/expansion → raw chunk fetch with token limits → LLM answer synthesis → trace output. The key architectural idea is that retrieval is reasoning over structured document trees, not vector similarity search.

15. **Important architectural distinction**
   The repo cleanly separates **retrieval from answering** at the pipeline level, even though the default query path synthesizes a final response. That means the same retrieval machinery can be used for traceable answer generation, inspection-only workflows, or evaluation runs, which is why the storage, trace, and experiments subsystems are first-class citizens.

Main entry points: [`ingestion/ingest.py`](https://github.com/iluko/docindex/blob/staging/ingestion/ingest.py) (Ingest pipeline), [`retrieval/query_engine.py`](https://github.com/iluko/docindex/blob/staging/retrieval/query_engine.py) (Query engine), [`traces/service.py`](https://github.com/iluko/docindex/blob/staging/traces/service.py) (Traces), [`app.py`](https://github.com/iluko/docindex/blob/staging/app.py) (Streamlit app)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_runtime["Runtime core"]
  node_ingest["Ingest pipeline<br/>ingestion pipeline<br/>[ingest.py]"]
  node_docx_conv["DOCX convert<br/>converter<br/>[docx_converter.py]"]
  node_pdf_enrich["PDF enrich<br/>enrichment<br/>[pdf_enricher.py]"]
  node_image_tools["Image tools<br/>extraction<br/>[image_extractor.py]"]
  node_master_gen["Master nodes<br/>index builder<br/>[master_node_gen.py]"]
  node_storage[("Storage<br/>persistence boundary<br/>[base.py]")]
  node_router["Router<br/>retrieval stage<br/>[router.py]"]
  node_navigator["Navigator<br/>retrieval stage<br/>[navigator.py]"]
  node_verifier["Verifier<br/>retrieval stage<br/>[verifier.py]"]
  node_fetcher["Fetcher<br/>retrieval stage<br/>[fetcher.py]"]
  node_query_engine["Query engine<br/>answer pipeline<br/>[query_engine.py]"]
  node_pageindex_engine["PageIndex engine<br/>agentic retrieval"]
  node_planner["Planner<br/>advanced retrieval<br/>[planner.py]"]
  node_traces[("Traces<br/>observability<br/>[service.py]")]
  node_index_registry["Index registry<br/>runtime registry<br/>[index_registry.py]"]
  node_model_registry["Model registry<br/>runtime registry<br/>[model_registry.py]"]
end

subgraph group_g_ui["User interfaces"]
  node_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_streamlit_app["Streamlit app<br/>ui entrypoint<br/>[app.py]"]
  node_api_app["API app<br/>api entrypoint<br/>[app.py]"]
  node_frontend["React UI<br/>frontend app<br/>[App.tsx]"]
  node_api_runtime["API runtime<br/>runtime wiring<br/>[runtime.py]"]
end

subgraph group_g_data["Index artifacts"]
  node_master_tree[("Master tree<br/>routing index<br/>[master_tree.py]")]
  node_relationships["Relationships<br/>schema logic<br/>[relationships.py]"]
  node_local_store[("Local store<br/>filesystem backend<br/>[store.py]")]
  node_mongo_store[("Mongo store<br/>mongo backend<br/>[mongo_store.py]")]
end

subgraph group_g_eval["Experiments"]
  node_experiments_hub["Experiments<br/>evaluation harness<br/>[README.md]"]
  node_evals["Evaluators<br/>evaluation core<br/>[judges.py]"]
  node_retrieval_variants["Retrieval variants<br/>variant registry<br/>[hybrid.py]"]
  node_run_registry["Run registry<br/>experiment registry<br/>[registry.py]"]
end

node_cli -->|"ingest/query"| node_ingest
node_cli -->|"ask"| node_query_engine
node_streamlit_app -->|"upload"| node_ingest
node_streamlit_app -->|"query"| node_query_engine
node_streamlit_app -->|"inspect"| node_traces
node_api_app -->|"serve"| node_api_runtime
node_api_runtime -->|"assemble"| node_index_registry
node_api_runtime -->|"select"| node_model_registry
node_frontend -->|"calls"| node_api_app
node_frontend -->|"view"| node_traces
node_ingest -->|"convert"| node_docx_conv
node_ingest -->|"enrich"| node_pdf_enrich
node_ingest -->|"extract"| node_image_tools
node_ingest -->|"build"| node_master_gen
node_master_gen -->|"write"| node_master_tree
node_master_tree -->|"link"| node_relationships
node_storage -->|"default"| node_local_store
node_storage -->|"alt"| node_mongo_store
node_ingest -->|"persist"| node_storage
node_query_engine -->|"route"| node_router
node_router -->|"narrow"| node_navigator
node_navigator -->|"filter"| node_verifier
node_verifier -->|"fetch"| node_fetcher
node_fetcher -->|"context"| node_query_engine
node_query_engine -->|"emit"| node_traces
node_pageindex_engine -->|"inspect"| node_master_tree
node_planner -->|"extend"| node_pageindex_engine
node_experiments_hub -->|"score"| node_evals
node_experiments_hub -->|"compare"| node_retrieval_variants
node_experiments_hub -->|"track"| node_run_registry
node_retrieval_variants -->|"mirror"| node_query_engine
node_evals -->|"analyze"| node_traces

click node_cli "https://github.com/iluko/docindex/blob/staging/cli.py"
click node_streamlit_app "https://github.com/iluko/docindex/blob/staging/app.py"
click node_api_app "https://github.com/iluko/docindex/blob/staging/api/app.py"
click node_frontend "https://github.com/iluko/docindex/blob/staging/frontend/src/App.tsx"
click node_api_runtime "https://github.com/iluko/docindex/blob/staging/api/runtime.py"
click node_ingest "https://github.com/iluko/docindex/blob/staging/ingestion/ingest.py"
click node_docx_conv "https://github.com/iluko/docindex/blob/staging/ingestion/docx_converter.py"
click node_pdf_enrich "https://github.com/iluko/docindex/blob/staging/ingestion/pdf_enricher.py"
click node_image_tools "https://github.com/iluko/docindex/blob/staging/ingestion/image_extractor.py"
click node_master_gen "https://github.com/iluko/docindex/blob/staging/ingestion/master_node_gen.py"
click node_master_tree "https://github.com/iluko/docindex/blob/staging/master_tree/master_tree.py"
click node_relationships "https://github.com/iluko/docindex/blob/staging/master_tree/relationships.py"
click node_storage "https://github.com/iluko/docindex/blob/staging/storage/base.py"
click node_local_store "https://github.com/iluko/docindex/blob/staging/storage/store.py"
click node_mongo_store "https://github.com/iluko/docindex/blob/staging/storage/mongo_store.py"
click node_router "https://github.com/iluko/docindex/blob/staging/retrieval/router.py"
click node_navigator "https://github.com/iluko/docindex/blob/staging/retrieval/navigator.py"
click node_verifier "https://github.com/iluko/docindex/blob/staging/retrieval/verifier.py"
click node_fetcher "https://github.com/iluko/docindex/blob/staging/retrieval/fetcher.py"
click node_query_engine "https://github.com/iluko/docindex/blob/staging/retrieval/query_engine.py"
click node_pageindex_engine "https://github.com/iluko/docindex/blob/staging/retrieval/pageindex_engine.py"
click node_planner "https://github.com/iluko/docindex/blob/staging/retrieval/planner.py"
click node_traces "https://github.com/iluko/docindex/blob/staging/traces/service.py"
click node_index_registry "https://github.com/iluko/docindex/blob/staging/index_registry.py"
click node_model_registry "https://github.com/iluko/docindex/blob/staging/model_registry.py"
click node_experiments_hub "https://github.com/iluko/docindex/blob/staging/experiments/README.md"
click node_evals "https://github.com/iluko/docindex/blob/staging/experiments/evals/judges.py"
click node_retrieval_variants "https://github.com/iluko/docindex/blob/staging/experiments/retrieval/hybrid.py"
click node_run_registry "https://github.com/iluko/docindex/blob/staging/experiments/runs/registry.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest,node_docx_conv,node_pdf_enrich,node_image_tools,node_master_gen,node_storage,node_router,node_navigator,node_verifier,node_fetcher,node_query_engine,node_pageindex_engine,node_planner,node_traces,node_index_registry,node_model_registry toneBlue
class node_cli,node_streamlit_app,node_api_app,node_frontend,node_api_runtime toneAmber
class node_master_tree,node_relationships,node_local_store,node_mongo_store toneMint
class node_experiments_hub,node_evals,node_retrieval_variants,node_run_registry toneRose
```

## Components

### Runtime core

Shared ingestion, retrieval, storage, and traceable query execution

- [Ingest pipeline](https://github.com/iluko/docindex/blob/staging/ingestion/ingest.py): ingestion pipeline · Converts source documents into hierarchical document structures · `ingestion/ingest.py`
- [DOCX convert](https://github.com/iluko/docindex/blob/staging/ingestion/docx_converter.py): converter · DOCX conversion with heading preservation · `ingestion/docx_converter.py`
- [PDF enrich](https://github.com/iluko/docindex/blob/staging/ingestion/pdf_enricher.py): enrichment · PDF-specific enrichment and extraction · `ingestion/pdf_enricher.py`
- [Image tools](https://github.com/iluko/docindex/blob/staging/ingestion/image_extractor.py): extraction · Image extraction and analysis helpers · `ingestion/image_extractor.py`
- [Master nodes](https://github.com/iluko/docindex/blob/staging/ingestion/master_node_gen.py): index builder · Builds document-level routing nodes from parsed content · `ingestion/master_node_gen.py`
- [Storage](https://github.com/iluko/docindex/blob/staging/storage/base.py): persistence boundary · Abstracts local filesystem and MongoDB persistence · `storage/base.py`
- [Router](https://github.com/iluko/docindex/blob/staging/retrieval/router.py): retrieval stage · Selects candidate documents from the master tree · `retrieval/router.py`
- [Navigator](https://github.com/iluko/docindex/blob/staging/retrieval/navigator.py): retrieval stage · Selects relevant sections within routed documents · `retrieval/navigator.py`
- [Verifier](https://github.com/iluko/docindex/blob/staging/retrieval/verifier.py): retrieval stage · Filters false positives before chunk fetching · `retrieval/verifier.py`
- [Fetcher](https://github.com/iluko/docindex/blob/staging/retrieval/fetcher.py): retrieval stage · Retrieves chunks with context windows and token budgeting · `retrieval/fetcher.py`
- [Query engine](https://github.com/iluko/docindex/blob/staging/retrieval/query_engine.py): answer pipeline · Coordinates routing, verification, fetching, and synthesis · `retrieval/query_engine.py`
- [PageIndex engine](https://github.com/iluko/docindex/blob/staging/retrieval/pageindex_engine.py): agentic retrieval · Tool-use retrieval loop for structured document exploration · `retrieval/pageindex_engine.py`
- [Planner](https://github.com/iluko/docindex/blob/staging/retrieval/planner.py): advanced retrieval · Optional planning and adaptive expansion for harder questions · `retrieval/planner.py`
- [Traces](https://github.com/iluko/docindex/blob/staging/traces/service.py): observability · Retrieval-time audit trail and query metrics · `traces/service.py`
- [Index registry](https://github.com/iluko/docindex/blob/staging/index_registry.py): runtime registry · Builds project-scoped index objects for scripts and services · `index_registry.py`
- [Model registry](https://github.com/iluko/docindex/blob/staging/model_registry.py): runtime registry · Centralizes model and runtime selection · `model_registry.py`

### User interfaces

CLI, Streamlit, API, and React frontend entrypoints

- [CLI](https://github.com/iluko/docindex/blob/staging/cli.py): entrypoint · Command-line ingestion and query entrypoint · `cli.py`
- [Streamlit app](https://github.com/iluko/docindex/blob/staging/app.py): ui entrypoint · Lightweight operational UI for upload, query, and trace inspection · `app.py`
- [API app](https://github.com/iluko/docindex/blob/staging/api/app.py): api entrypoint · Python API surface for shared runtime operations · `api/app.py`
- [React UI](https://github.com/iluko/docindex/blob/staging/frontend/src/App.tsx): frontend app · Vite + TypeScript client for asking, ingesting, and inspecting · `frontend/src/App.tsx`
- [API runtime](https://github.com/iluko/docindex/blob/staging/api/runtime.py): runtime wiring · Shared API contracts and runtime assembly · `api/runtime.py`

### Index artifacts

Project-scoped durable document and routing artifacts

- [Master tree](https://github.com/iluko/docindex/blob/staging/master_tree/master_tree.py): routing index · Corpus-wide routing and summary structure · `master_tree/master_tree.py`
- [Relationships](https://github.com/iluko/docindex/blob/staging/master_tree/relationships.py): schema logic · Defines links and routing relationships between documents · `master_tree/relationships.py`
- [Local store](https://github.com/iluko/docindex/blob/staging/storage/store.py): filesystem backend · Default project-scoped storage under data/ · `storage/store.py`
- [Mongo store](https://github.com/iluko/docindex/blob/staging/storage/mongo_store.py): mongo backend · MongoDB-backed persistence for documents, trees, and traces · `storage/mongo_store.py`

### Experiments

Offline evaluation, comparison, and retrieval variants

- [Experiments](https://github.com/iluko/docindex/blob/staging/experiments/README.md): evaluation harness · Offline benchmarking, retrieval variants, scoring, and reports · `experiments/README.md`
- [Evaluators](https://github.com/iluko/docindex/blob/staging/experiments/evals/judges.py): evaluation core · Judges, suites, deterministic scoring, and exports · `experiments/evals/judges.py`
- [Retrieval variants](https://github.com/iluko/docindex/blob/staging/experiments/retrieval/hybrid.py): variant registry · Hybrid, pageindex, rag, and vector retrieval implementations · `experiments/retrieval/hybrid.py`
- [Run registry](https://github.com/iluko/docindex/blob/staging/experiments/runs/registry.py): experiment registry · Tracks and compares experiment runs and reports · `experiments/runs/registry.py`

## Connections

- CLI → Ingest pipeline: ingest/query · CLI drives the shared runtime for ingestion and querying
- CLI → Query engine: ask · CLI queries use the standard answer pipeline
- Streamlit app → Ingest pipeline: upload · Streamlit exposes ingestion of source documents
- Streamlit app → Query engine: query · Streamlit queries go through the shared retrieval pipeline
- Streamlit app → Traces: inspect · Streamlit surfaces retrieval traces for auditability
- API app → API runtime: serve · API entrypoint uses shared runtime assembly
- API runtime → Index registry: assemble · Runtime builds project-scoped index objects
- API runtime → Model registry: select · Runtime selects configured models and modes
- React UI → API app: calls · React client talks to backend endpoints
- React UI → Traces: view · Frontend displays query traces and inspection panels
- Ingest pipeline → DOCX convert: convert · DOCX conversion is part of ingestion
- Ingest pipeline → PDF enrich: enrich · PDF enrichment feeds the document tree pipeline
- Ingest pipeline → Image tools: extract · Image extraction supports richer document parsing
- Ingest pipeline → Master nodes: build · Parsed documents are turned into master-tree nodes
- Master nodes → Master tree: write · Generated routing nodes are persisted as the master tree
- Master tree → Relationships: link · Master tree stores document-level relationships and hints
- Storage → Local store: default · Filesystem storage is the default backend
- Storage → Mongo store: alt · MongoDB is an alternate persistence backend
- Ingest pipeline → Storage: persist · Ingestion materializes trees and artifacts through storage
- Query engine → Router: route · Query execution starts by selecting candidate documents
- Router → Navigator: narrow · Routed documents are narrowed to relevant sections
- Navigator → Verifier: filter · Candidate sections can be verified before fetching
- Verifier → Fetcher: fetch · Verified targets are expanded into context chunks
- Fetcher → Query engine: context · Fetched chunks feed answer synthesis
- Query engine → Traces: emit · Query execution emits a trace object
- PageIndex engine → Master tree: inspect · Agentic mode reasons over structured document trees
- Planner → PageIndex engine: extend · Planning can expand the agentic PageIndex loop
- Experiments → Evaluators: score · Experiment harness uses judges and suites for scoring
- Experiments → Retrieval variants: compare · Harness compares alternate retrieval implementations
- Experiments → Run registry: track · Runs and reports are recorded for analysis
- Retrieval variants → Query engine: mirror · Experimental variants mirror production retrieval stages
- Evaluators → Traces: analyze · Evaluation consumes traces and outputs for regression analysis
