How iluko/docindex works
1. Overall shape
This repo is a multi-stage document RAG system with three major planes: ingestion/indexing, query/retrieval/answering, and observability/evaluation. The production runtime is mostly Python, with a separate React/Vite frontend and a Streamlit UI for interactive use.
2. Primary user entrypoints
The main operational entrypoints are cli.py for command-line ingestion/querying, app.py for Streamlit, frontend/ for the React app, and api/ for a Python API layer. These all share the same underlying runtime components rather than duplicating logic.
3. Ingestion pipeline
The ingestion subsystem lives in ingestion/ and converts source files into indexable document structures. It handles document parsing/enrichment for .pdf, .md, .markdown, and .docx, with DOCX conversion preserving headings. PageIndex is the parsing foundation that produces a hierarchical per-document tree, which then feeds downstream indexing.
4. Document index artifacts
Ingested content is materialized into multiple durable artifacts under data/ or MongoDB depending on storage backend: uploaded originals, per-document trees, derived markdown, and a master tree that acts as the cross-document routing index. The repo is explicitly project-scoped, so each knowledge base gets isolated artifacts and its own master_tree.json and related index files.
5. Master tree and routing metadata
The master_tree/ subsystem defines the document-level routing representation. It stores document summaries plus routing hints such as topics, relationships, and newer routing facets like workflows, actors, systems, edge cases, and authority hints. This is the key boundary between ingestion and retrieval: ingestion enriches documents into a compact corpus-wide index that query-time routing can reason over.
6. Retrieval pipeline
The retrieval/ package implements the standard hybrid query flow: router → navigator → verifier → fetcher → answer synthesis. Router selects documents from the master tree, navigator selects sections within those docs, verifier optionally filters false positives, and fetcher pulls raw text chunks with context windows and token budgeting. The pipeline is deterministic in standard mode and traceable step by step.
7. Advanced and alternate retrieval modes
There are two retrieval modes in the repo: hybrid and pageindex. Hybrid is the default bounded pipeline; pageindex is an agentic tool-use loop that iteratively inspects document structure and content. Advanced retrieval adds optional planning, adaptive width, and node expansion on top of the hybrid path, but these are explicitly opt-in and designed for broader, workflow-heavy questions.
8. Storage boundary
The storage/ package abstracts persistence behind a common interface. The default backend is local filesystem storage under data/, but the repo also supports MongoDB via storage/mongo_store.py. This backend boundary is important architecturally: the rest of the application should not care whether indexes and traces are on disk or in MongoDB.
9. Traceability and audit trail
The traces/ package records retrieval-time observability data, including routed docs, selected nodes, fetched chunks, token budgets, and mode-specific metrics for pageindex exploration. This is not just logging; it is part of the product’s architecture because every query returns a trace object used by the UI and CLI for auditability.
10. Frontend surfaces
The frontend/ directory is a standalone React app built with Vite and TypeScript. It provides separate pages for asking questions, ingesting documents, and inspecting results, with components for chat threads, inspector panels, and selection controls. It talks to the backend through a small API client layer under frontend/src/api/.
11. Streamlit and Python API surfaces
The Streamlit app in app.py is a lightweight operational UI that exposes upload, query, and trace inspection without the full React stack. The api/ package and index_registry.py/model_registry.py provide reusable runtime assembly so scripts, UI, and services all construct the same storage, tree, and retrieval objects.
12. Experiments and evaluation subsystem
The experiments/ tree is a separate evaluation harness, not the production query path. It contains retrieval variants, build presets, datasets, judges, scoring, comparison reports, and run registries. This makes it a parallel subsystem for offline benchmarking and regression analysis against curated corpora and golden datasets.
13. Test and fixture boundaries
The tests/ directory covers ingestion, retrieval, master tree logic, registries, verifier behavior, and experiments. Fixtures include sample document trees and sample master trees, which mirrors the repo’s core data model and helps validate the boundaries between parsing, indexing, retrieval, and storage.
14. End-to-end data flow
The core runtime flow is: source document → ingestion/parsing → per-doc tree + markdown → master-node generation → project-scoped storage → query-time routing over master tree → section navigation → optional verification/expansion → raw chunk fetch with token limits → LLM answer synthesis → trace output. The key architectural idea is that retrieval is reasoning over structured document trees, not vector similarity search.
15. Important architectural distinction
The repo cleanly separates retrieval from answering at the pipeline level, even though the default query path synthesizes a final response. That means the same retrieval machinery can be used for traceable answer generation, inspection-only workflows, or evaluation runs, which is why the storage, trace, and experiments subsystems are first-class citizens.
Main entry points: ingest.py Ingest pipeline · query_engine.py Query engine · service.py Traces · app.py Streamlit app
Components 29
Runtime core
Shared ingestion, retrieval, storage, and traceable query execution
- Ingest pipeline · Converts source documents into hierarchical document structures
ingestion/ingest.py - DOCX convert · DOCX conversion with heading preservation
ingestion/docx_converter.py - PDF enrich · PDF-specific enrichment and extraction
ingestion/pdf_enricher.py - Image tools · Image extraction and analysis helpers
ingestion/image_extractor.py - Master nodes · Builds document-level routing nodes from parsed content
ingestion/master_node_gen.py - Storage · Abstracts local filesystem and MongoDB persistence
storage/base.py - Router · Selects candidate documents from the master tree
retrieval/router.py - Navigator · Selects relevant sections within routed documents
retrieval/navigator.py - Verifier · Filters false positives before chunk fetching
retrieval/verifier.py - Fetcher · Retrieves chunks with context windows and token budgeting
retrieval/fetcher.py - Query engine · Coordinates routing, verification, fetching, and synthesis
retrieval/query_engine.py - PageIndex engine · Tool-use retrieval loop for structured document exploration
retrieval/pageindex_engine.py - Planner · Optional planning and adaptive expansion for harder questions
retrieval/planner.py - Traces · Retrieval-time audit trail and query metrics
traces/service.py - Index registry · Builds project-scoped index objects for scripts and services
index_registry.py - Model registry · Centralizes model and runtime selection
model_registry.py
User interfaces
CLI, Streamlit, API, and React frontend entrypoints
- CLI · Command-line ingestion and query entrypoint
cli.py - Streamlit app · Lightweight operational UI for upload, query, and trace inspection
app.py - API app · Python API surface for shared runtime operations
api/app.py - React UI · Vite + TypeScript client for asking, ingesting, and inspecting
frontend/src/App.tsx - API runtime · Shared API contracts and runtime assembly
api/runtime.py
Index artifacts
Project-scoped durable document and routing artifacts
- Master tree · Corpus-wide routing and summary structure
master_tree/master_tree.py - Relationships · Defines links and routing relationships between documents
master_tree/relationships.py - Local store · Default project-scoped storage under data/
storage/store.py - Mongo store · MongoDB-backed persistence for documents, trees, and traces
storage/mongo_store.py
Experiments
Offline evaluation, comparison, and retrieval variants
- Experiments · Offline benchmarking, retrieval variants, scoring, and reports
experiments/README.md - Evaluators · Judges, suites, deterministic scoring, and exports
experiments/evals/judges.py - Retrieval variants · Hybrid, pageindex, rag, and vector retrieval implementations
experiments/retrieval/hybrid.py - Run registry · Tracks and compares experiment runs and reports
experiments/runs/registry.py
Connections 32
- CLI → Ingest pipeline: ingest/query · CLI drives the shared runtime for ingestion and querying
- CLI → Query engine: ask · CLI queries use the standard answer pipeline
- Streamlit app → Ingest pipeline: upload · Streamlit exposes ingestion of source documents
- Streamlit app → Query engine: query · Streamlit queries go through the shared retrieval pipeline
- Streamlit app → Traces: inspect · Streamlit surfaces retrieval traces for auditability
- API app → API runtime: serve · API entrypoint uses shared runtime assembly
- API runtime → Index registry: assemble · Runtime builds project-scoped index objects
- API runtime → Model registry: select · Runtime selects configured models and modes
- React UI → API app: calls · React client talks to backend endpoints
- React UI → Traces: view · Frontend displays query traces and inspection panels
- Ingest pipeline → DOCX convert: convert · DOCX conversion is part of ingestion
- Ingest pipeline → PDF enrich: enrich · PDF enrichment feeds the document tree pipeline
- Ingest pipeline → Image tools: extract · Image extraction supports richer document parsing
- Ingest pipeline → Master nodes: build · Parsed documents are turned into master-tree nodes
- Master nodes → Master tree: write · Generated routing nodes are persisted as the master tree
- Master tree → Relationships: link · Master tree stores document-level relationships and hints
- Storage → Local store: default · Filesystem storage is the default backend
- Storage → Mongo store: alt · MongoDB is an alternate persistence backend
- Ingest pipeline → Storage: persist · Ingestion materializes trees and artifacts through storage
- Query engine → Router: route · Query execution starts by selecting candidate documents
- Router → Navigator: narrow · Routed documents are narrowed to relevant sections
- Navigator → Verifier: filter · Candidate sections can be verified before fetching
- Verifier → Fetcher: fetch · Verified targets are expanded into context chunks
- Fetcher → Query engine: context · Fetched chunks feed answer synthesis
- Query engine → Traces: emit · Query execution emits a trace object
- PageIndex engine → Master tree: inspect · Agentic mode reasons over structured document trees
- Planner → PageIndex engine: extend · Planning can expand the agentic PageIndex loop
- Experiments → Evaluators: score · Experiment harness uses judges and suites for scoring
- Experiments → Retrieval variants: compare · Harness compares alternate retrieval implementations
- Experiments → Run registry: track · Runs and reports are recorded for analysis
- Retrieval variants → Query engine: mirror · Experimental variants mirror production retrieval stages
- Evaluators → Traces: analyze · Evaluation consumes traces and outputs for regression analysis