How jubayerrafin/deep-doc-extractor works
Deep Doc Extractor turns PDF manuals into structured Markdown and table/image data, generates and filters Q&A pairs, augments them with paraphrases, then builds an offline, citation-grounded RAG chatbot. The main user workflow starts with a manual and ends with dataset artifacts or chatbot answers; stages exchange files so they can run independently. Evaluation is a separate capability for measuring dataset faithfulness, question quality, and diversity. The README and sampled orchestrators document the stage boundaries; some internal connections and the Streamlit wiring are shown only where documented, not inferred from imports.
Main entry points: stage4_pipeline.py Index and Query Workflow · stage2_pipeline.py Q&A Workflow · pipeline.py Extraction Workflow · stage3_pipeline.py Augmentation Workflow
Components 34
PDF Extraction
- Extraction Workflow
pipeline.py - PDF Validation
stage1/file_validator.py - Text Extraction
stage1/text_extractor.py - Image Extraction
stage1/image_extractor.py - Table Extraction
stage1/table_extractor.py - Markdown Compiler
stage1/markdown_compiler.py - Structured Markdown
- Table JSON
Q&A Construction
- Q&A Workflow
stage2/stage2_pipeline.py - Markdown Parser
stage2/markdown_parser.py - Content Chunker
stage2/chunker.py - Semantic Classifier
stage2/semantic_classifier.py - Q&A Generator
stage2/qa_generator.py - Quality Filters
stage2/quality_filters.py - Q&A JSONL
Dataset Augmentation
- Augmentation Workflow
stage3/stage3_pipeline.py - Question Rephraser
stage3/llm_rephraser.py - Diversity Filter
stage3/augment_filter.py - Dataset Exporter
stage3/final_exporter.py - Augmented JSONL
Offline RAG
- Index and Query Workflow
stage4/stage4_pipeline.py - Markdown Chunker
stage4/markdown_chunker.py - Embedding Engine
stage4/embedding_engine.py - Chroma Indexer
stage4/chroma_indexer.py - Chroma Index
- Chat Interface
stage4/streamlit_app.py - RAG Retriever
stage4/rag_retriever.py - Answer Generator
stage4/answer_generator.py - Offline Guard
stage4/offline_guard.py
Dataset Evaluation
- Dataset Evaluators
evaluation/evaluate_all.py
Other components
- PDF Manual
- Pipeline Orchestrator
pipeline.py - Local Ollama
- Chat User
Connections 44
- PDF Manual → Pipeline Orchestrator: provides PDF
- Pipeline Orchestrator → Extraction Workflow: dispatches
- Extraction Workflow → PDF Validation: validates
- Extraction Workflow → Text Extraction: extracts text
- Extraction Workflow → Image Extraction: extracts images
- Extraction Workflow → Table Extraction: extracts tables
- Extraction Workflow → Markdown Compiler: compiles elements
- Table Extraction → Table JSON: writes JSON
- Markdown Compiler → Structured Markdown: writes Markdown
- Pipeline Orchestrator → Q&A Workflow: dispatches
- Structured Markdown → Markdown Parser: reads
- Table JSON → Markdown Parser: reads tables
- Q&A Workflow → Markdown Parser: parses source
- Q&A Workflow → Content Chunker: chunks content
- Q&A Workflow → Semantic Classifier: classifies content
- Q&A Workflow → Q&A Generator: generates pairs
- Q&A Workflow → Quality Filters: filters pairs
- Q&A Workflow → Local Ollama: requests generation
- Q&A Workflow → Q&A JSONL: writes accepted pairs
- Pipeline Orchestrator → Augmentation Workflow: dispatches
- Q&A JSONL → Augmentation Workflow: reads pairs
- Augmentation Workflow → Question Rephraser: requests variations
- Question Rephraser → Local Ollama: requests paraphrases
- Augmentation Workflow → Diversity Filter: filters variations
- Augmentation Workflow → Dataset Exporter: exports records
- Dataset Exporter → Augmented JSONL: writes dataset
- Pipeline Orchestrator → Index and Query Workflow: dispatches
- Structured Markdown → Index and Query Workflow: indexes source
- Augmented JSONL → Index and Query Workflow: indexes pairs
- Index and Query Workflow → Markdown Chunker: chunks Markdown
- Index and Query Workflow → Embedding Engine: embeds content
- Index and Query Workflow → Chroma Indexer: upserts records
- Chroma Indexer → Chroma Index: writes and queries
- Chat User → Chat Interface: asks question
- Chat Interface → Index and Query Workflow: submits query
- Index and Query Workflow → RAG Retriever: retrieves evidence
- RAG Retriever → Chroma Indexer: queries index
- RAG Retriever → Answer Generator: provides hits
- Index and Query Workflow → Answer Generator: generates answer
- Answer Generator → Local Ollama: requests response
- Chat Interface → Offline Guard: guards network
- Pipeline Orchestrator → Dataset Evaluators: runs separately
- Q&A JSONL → Dataset Evaluators: evaluates pairs
- Augmented JSONL → Dataset Evaluators: evaluates pairs