How awasthiutk564/idp-420 works
This is a hybrid document-extraction engine for PDFs and images. The principal workflow is file selection, extraction and classification, staged enrichment of a document graph, then formatted output. Its stages cover layout, text, equations, tables, metadata, semantics, chunking, fusion, validation, and output. PDF-library adapters and OCR/model components provide extraction capabilities; hierarchy and graph utilities support structured results. The README is empty, and main.py plus several stages are only partially excerpted, so some component links are intentionally omitted where invocation is not visible.
Main entry points: main.py CLI application · document_graph.py Document graph · pipeline.py Pipeline runner · hierarchy_builder.py Hierarchy builder
Components 27
Application
- CLI application
main.py - Pipeline runner
pipeline/pipeline.py
Input and extraction
- PDF adapters
adapters - Image extractor
adapters/image_extractor.py - PDF processors
processors - Image preprocessing
preprocess/image_preprocessor.py
Document enrichment
- Extraction stages
pipeline - Equation processing
pipeline/stage_math.py - Layout processing
pipeline/stage_layout.py - Text processing
pipeline/stage_text.py - Semantic processing
pipeline/stage_semantic.py - Table processing
pipeline/stage_tables.py - Extraction fusion
pipeline/stage_fusion.py - Metadata processing
pipeline/stage_metadata.py - Chunk generation
pipeline/stage_chunks.py - Validation stage
pipeline/stage_validation.py - Output stage
pipeline/stage_output.py
Models and classification
- Document classifiers
classifiers - OCR models
models/ocr_model.py - Layout model
models/layout_model.py - Equation model
models/equation_model.py - Table model
models/table_model.py - Chart model
models/chart_model.py
Document structure
- Document graph
utils/document_graph.py - Hierarchy builder
utils/hierarchy_builder.py - Result formatter
utils/formatter.py
Other components
- Document user
Connections 23
- Document user → CLI application: selects file
main.py - CLI application → Pipeline runner: runs pipeline
main.py - CLI application → PDF adapters: configures adapters
main.py - CLI application → Document classifiers: configures classifiers
main.py - CLI application → OCR models: uses OCR
main.py - CLI application → Layout model: uses model
main.py - CLI application → Equation model: uses model
main.py - CLI application → Table model: uses model
main.py - CLI application → Chart model: uses model
main.py - CLI application → Image preprocessing: uses preprocessing
main.py - Pipeline runner → Extraction stages: runs stages
pipeline/pipeline.py - Pipeline runner → Document graph: creates graph
pipeline/pipeline.py - Equation processing → Document graph: updates graph
pipeline/stage_math.py - Layout processing → Hierarchy builder: uses hierarchy
pipeline/stage_layout.py - Semantic processing → Hierarchy builder: uses hierarchy
pipeline/stage_semantic.py - Text processing → Document graph: processes graph
pipeline/stage_text.py - Table processing → Document graph: processes graph
pipeline/stage_tables.py - Extraction fusion → Document graph: processes graph
pipeline/stage_fusion.py - Metadata processing → Document graph: processes graph
pipeline/stage_metadata.py - Chunk generation → Document graph: processes graph
pipeline/stage_chunks.py - Output stage → Document graph: processes graph
pipeline/stage_output.py - Validation stage → Document graph: processes graph
- CLI application → Result formatter: formats results
main.py