# jubayerrafin/deep-doc-extractor architecture

> Architecture diagram of the GitHub repository jubayerrafin/deep-doc-extractor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jubayerrafin/deep-doc-extractor
- Repository: https://github.com/jubayerrafin/deep-doc-extractor

## Overview

Deep Doc Extractor turns PDF manuals into structured Markdown and table/image data, generates and filters Q&A pairs, augments them with paraphrases, then builds an offline, citation-grounded RAG chatbot. The main user workflow starts with a manual and ends with dataset artifacts or chatbot answers; stages exchange files so they can run independently. Evaluation is a separate capability for measuring dataset faithfulness, question quality, and diversity. The README and sampled orchestrators document the stage boundaries; some internal connections and the Streamlit wiring are shown only where documented, not inferred from imports.

Main entry points: [`stage4/stage4_pipeline.py`](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/stage4_pipeline.py) (Index and Query Workflow), [`stage2/stage2_pipeline.py`](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/stage2_pipeline.py) (Q&A Workflow), [`pipeline.py`](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/pipeline.py) (Extraction Workflow), [`stage3/stage3_pipeline.py`](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/stage3_pipeline.py) (Augmentation Workflow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_extraction["PDF Extraction"]
  node_stage1["Extraction Workflow<br/>[pipeline.py]"]
  node_validator["PDF Validation<br/>[file_validator.py]"]
  node_text["Text Extraction<br/>[text_extractor.py]"]
  node_images["Image Extraction<br/>[image_extractor.py]"]
  node_tables["Table Extraction<br/>[table_extractor.py]"]
  node_compiler["Markdown Compiler"]
  node_markdown["Structured Markdown"]
  node_tabledata["Table JSON"]
end

subgraph group_dataset["Q&amp;A Construction"]
  node_stage2["Q&amp;A Workflow<br/>[stage2_pipeline.py]"]
  node_parser["Markdown Parser<br/>[markdown_parser.py]"]
  node_chunker["Content Chunker<br/>[chunker.py]"]
  node_classifier["Semantic Classifier"]
  node_generator["Q&amp;A Generator<br/>[qa_generator.py]"]
  node_filters["Quality Filters<br/>[quality_filters.py]"]
  node_qapairs["Q&amp;A JSONL"]
end

subgraph group_augmentation["Dataset Augmentation"]
  node_stage3["Augmentation Workflow<br/>[stage3_pipeline.py]"]
  node_rephraser["Question Rephraser<br/>[llm_rephraser.py]"]
  node_augmentfilter["Diversity Filter<br/>[augment_filter.py]"]
  node_exporter["Dataset Exporter<br/>[final_exporter.py]"]
  node_augmented["Augmented JSONL"]
end

subgraph group_rag["Offline RAG"]
  node_stage4["Index and Query Workflow<br/>[stage4_pipeline.py]"]
  node_mdchunker["Markdown Chunker"]
  node_embedder["Embedding Engine"]
  node_indexer["Chroma Indexer<br/>[chroma_indexer.py]"]
  node_chromadb[("Chroma Index")]
  node_ui["Chat Interface<br/>[streamlit_app.py]"]
  node_retriever["RAG Retriever<br/>[rag_retriever.py]"]
  node_answer["Answer Generator"]
  node_offline["Offline Guard<br/>[offline_guard.py]"]
end

subgraph group_evaluation["Dataset Evaluation"]
  node_evaluation["Dataset Evaluators<br/>[evaluate_all.py]"]
end

node_manual["PDF Manual"]
node_pipeline["Pipeline Orchestrator<br/>[pipeline.py]"]
node_ollama["Local Ollama"]
node_user(("Chat User"))

node_manual -->|"provides PDF"| node_pipeline
node_pipeline -->|"dispatches"| node_stage1
node_stage1 -->|"validates"| node_validator
node_stage1 -->|"extracts text"| node_text
node_stage1 -->|"extracts images"| node_images
node_stage1 -->|"extracts tables"| node_tables
node_stage1 -->|"compiles elements"| node_compiler
node_tables -->|"writes JSON"| node_tabledata
node_compiler -->|"writes Markdown"| node_markdown
node_pipeline -->|"dispatches"| node_stage2
node_markdown -->|"reads"| node_parser
node_tabledata -->|"reads tables"| node_parser
node_stage2 -->|"parses source"| node_parser
node_stage2 -->|"chunks content"| node_chunker
node_stage2 -->|"classifies content"| node_classifier
node_stage2 -->|"generates pairs"| node_generator
node_stage2 -->|"filters pairs"| node_filters
node_stage2 -.->|"requests generation"| node_ollama
node_stage2 -->|"writes accepted pairs"| node_qapairs
node_pipeline -->|"dispatches"| node_stage3
node_qapairs -->|"reads pairs"| node_stage3
node_stage3 -->|"requests variations"| node_rephraser
node_rephraser -.->|"requests paraphrases"| node_ollama
node_stage3 -->|"filters variations"| node_augmentfilter
node_stage3 -->|"exports records"| node_exporter
node_exporter -->|"writes dataset"| node_augmented
node_pipeline -->|"dispatches"| node_stage4
node_markdown -->|"indexes source"| node_stage4
node_augmented -->|"indexes pairs"| node_stage4
node_stage4 -->|"chunks Markdown"| node_mdchunker
node_stage4 -->|"embeds content"| node_embedder
node_stage4 -->|"upserts records"| node_indexer
node_indexer -->|"writes and queries"| node_chromadb
node_user -->|"asks question"| node_ui
node_ui -->|"submits query"| node_stage4
node_stage4 -->|"retrieves evidence"| node_retriever
node_retriever -->|"queries index"| node_indexer
node_retriever -->|"provides hits"| node_answer
node_stage4 -->|"generates answer"| node_answer
node_answer -.->|"requests response"| node_ollama
node_ui -->|"guards network"| node_offline
node_pipeline -.->|"runs separately"| node_evaluation
node_qapairs -.->|"evaluates pairs"| node_evaluation
node_augmented -.->|"evaluates pairs"| node_evaluation

click node_pipeline "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/pipeline.py"
click node_stage1 "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/pipeline.py"
click node_validator "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/file_validator.py"
click node_text "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/text_extractor.py"
click node_images "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/image_extractor.py"
click node_tables "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/table_extractor.py"
click node_compiler "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/markdown_compiler.py"
click node_stage2 "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/stage2_pipeline.py"
click node_parser "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/markdown_parser.py"
click node_chunker "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/chunker.py"
click node_classifier "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/semantic_classifier.py"
click node_generator "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/qa_generator.py"
click node_filters "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/quality_filters.py"
click node_stage3 "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/stage3_pipeline.py"
click node_rephraser "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/llm_rephraser.py"
click node_augmentfilter "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/augment_filter.py"
click node_exporter "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/final_exporter.py"
click node_stage4 "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/stage4_pipeline.py"
click node_mdchunker "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/markdown_chunker.py"
click node_embedder "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/embedding_engine.py"
click node_indexer "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/chroma_indexer.py"
click node_ui "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/streamlit_app.py"
click node_retriever "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/rag_retriever.py"
click node_answer "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/answer_generator.py"
click node_offline "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/offline_guard.py"
click node_evaluation "https://github.com/jubayerrafin/deep-doc-extractor/blob/main/evaluation/evaluate_all.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_stage1,node_validator,node_text,node_images,node_tables,node_compiler,node_markdown,node_tabledata,node_user toneBlue
class node_stage2,node_parser,node_chunker,node_classifier,node_generator,node_filters,node_qapairs toneAmber
class node_stage3,node_rephraser,node_augmentfilter,node_exporter,node_augmented toneMint
class node_stage4,node_mdchunker,node_embedder,node_indexer,node_chromadb,node_ui,node_retriever,node_answer,node_offline toneRose
class node_evaluation,node_manual,node_ollama toneIndigo
class node_pipeline toneTeal
```

## Components

### PDF Extraction

- [Extraction Workflow](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/pipeline.py): `pipeline.py`
- [PDF Validation](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/file_validator.py): `stage1/file_validator.py`
- [Text Extraction](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/text_extractor.py): `stage1/text_extractor.py`
- [Image Extraction](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/image_extractor.py): `stage1/image_extractor.py`
- [Table Extraction](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/table_extractor.py): `stage1/table_extractor.py`
- [Markdown Compiler](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage1/markdown_compiler.py): `stage1/markdown_compiler.py`
- **Structured Markdown**
- **Table JSON**

### Q&A Construction

- [Q&A Workflow](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/stage2_pipeline.py): `stage2/stage2_pipeline.py`
- [Markdown Parser](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/markdown_parser.py): `stage2/markdown_parser.py`
- [Content Chunker](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/chunker.py): `stage2/chunker.py`
- [Semantic Classifier](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/semantic_classifier.py): `stage2/semantic_classifier.py`
- [Q&A Generator](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/qa_generator.py): `stage2/qa_generator.py`
- [Quality Filters](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage2/quality_filters.py): `stage2/quality_filters.py`
- **Q&A JSONL**

### Dataset Augmentation

- [Augmentation Workflow](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/stage3_pipeline.py): `stage3/stage3_pipeline.py`
- [Question Rephraser](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/llm_rephraser.py): `stage3/llm_rephraser.py`
- [Diversity Filter](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/augment_filter.py): `stage3/augment_filter.py`
- [Dataset Exporter](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage3/final_exporter.py): `stage3/final_exporter.py`
- **Augmented JSONL**

### Offline RAG

- [Index and Query Workflow](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/stage4_pipeline.py): `stage4/stage4_pipeline.py`
- [Markdown Chunker](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/markdown_chunker.py): `stage4/markdown_chunker.py`
- [Embedding Engine](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/embedding_engine.py): `stage4/embedding_engine.py`
- [Chroma Indexer](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/chroma_indexer.py): `stage4/chroma_indexer.py`
- **Chroma Index**
- [Chat Interface](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/streamlit_app.py): `stage4/streamlit_app.py`
- [RAG Retriever](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/rag_retriever.py): `stage4/rag_retriever.py`
- [Answer Generator](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/answer_generator.py): `stage4/answer_generator.py`
- [Offline Guard](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/stage4/offline_guard.py): `stage4/offline_guard.py`

### Dataset Evaluation

- [Dataset Evaluators](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/evaluation/evaluate_all.py): `evaluation/evaluate_all.py`

### Other components

- **PDF Manual**
- [Pipeline Orchestrator](https://github.com/jubayerrafin/deep-doc-extractor/blob/main/pipeline.py): `pipeline.py`
- **Local Ollama**
- **Chat User**

## Connections

- PDF Manual → Pipeline Orchestrator: provides PDF
- Pipeline Orchestrator → Extraction Workflow: dispatches
- Extraction Workflow → PDF Validation: validates
- Extraction Workflow → Text Extraction: extracts text
- Extraction Workflow → Image Extraction: extracts images
- Extraction Workflow → Table Extraction: extracts tables
- Extraction Workflow → Markdown Compiler: compiles elements
- Table Extraction → Table JSON: writes JSON
- Markdown Compiler → Structured Markdown: writes Markdown
- Pipeline Orchestrator → Q&A Workflow: dispatches
- Structured Markdown → Markdown Parser: reads
- Table JSON → Markdown Parser: reads tables
- Q&A Workflow → Markdown Parser: parses source
- Q&A Workflow → Content Chunker: chunks content
- Q&A Workflow → Semantic Classifier: classifies content
- Q&A Workflow → Q&A Generator: generates pairs
- Q&A Workflow → Quality Filters: filters pairs
- Q&A Workflow → Local Ollama: requests generation
- Q&A Workflow → Q&A JSONL: writes accepted pairs
- Pipeline Orchestrator → Augmentation Workflow: dispatches
- Q&A JSONL → Augmentation Workflow: reads pairs
- Augmentation Workflow → Question Rephraser: requests variations
- Question Rephraser → Local Ollama: requests paraphrases
- Augmentation Workflow → Diversity Filter: filters variations
- Augmentation Workflow → Dataset Exporter: exports records
- Dataset Exporter → Augmented JSONL: writes dataset
- Pipeline Orchestrator → Index and Query Workflow: dispatches
- Structured Markdown → Index and Query Workflow: indexes source
- Augmented JSONL → Index and Query Workflow: indexes pairs
- Index and Query Workflow → Markdown Chunker: chunks Markdown
- Index and Query Workflow → Embedding Engine: embeds content
- Index and Query Workflow → Chroma Indexer: upserts records
- Chroma Indexer → Chroma Index: writes and queries
- Chat User → Chat Interface: asks question
- Chat Interface → Index and Query Workflow: submits query
- Index and Query Workflow → RAG Retriever: retrieves evidence
- RAG Retriever → Chroma Indexer: queries index
- RAG Retriever → Answer Generator: provides hits
- Index and Query Workflow → Answer Generator: generates answer
- Answer Generator → Local Ollama: requests response
- Chat Interface → Offline Guard: guards network
- Pipeline Orchestrator → Dataset Evaluators: runs separately
- Q&A JSONL → Dataset Evaluators: evaluates pairs
- Augmented JSONL → Dataset Evaluators: evaluates pairs