How iamaayushijain/adobe_hackathon works
Project Type and Purpose
• This is a specialized, offline batch-processing tool (not a web app) for extracting structured data from PDFs and, in its Round-1B mode, semantically ranking sections per “persona & task.”
• It’s organized as a CLI-driven pipeline, with two entry points:
• main2.py for outline-only extraction (Round-1A)
• main.py for persona-based ranking (Round-1B)
• Key goals: full-text capture, hierarchical outline creation, table extraction, and optional semantic ranking via a local MiniLM model.
High-Level File Structure and Architectural Patterns
• Top-level orchestration scripts: main2.py, main.py, pipeline.py
• Core application modules under app/ (embedder, ranker, outline_to_refined_processor, subsection_selector)
• Extraction & parsing utilities: parser.py, raw_text_extractor.py, table_extractor.py, ocr_utils.py
• Model folder (models/all-MiniLM-L6-v2) for semantic embedding
• Input/output directories plus Challenge_1b collections for sample data
• Utilities and config: utils.py, output_writer.py, requirements.txt, Dockerfiles
This structure follows a linear pipeline pattern:
[Ingestion] → [Parsing] → [Outline Generation] → [Ranking] → [Output]
Key Components and Their Relationships
a. Ingestion Layer
• Input PDFs (input/ or Challenge_1b/Collection*/PDFs)
• main2.py or main.py / Docker entrypoint
b. Parsing Layer
• parser.py (PDFOutlineParser) orchestrates four PDF libs (pdfplumber, PyMuPDF, pdfminer, Camelot)
• raw_text_extractor.py collects all pages’ text
• table_extractor.py identifies and serializes tables
• ocr_utils.py (optional OCR fallback)
c. Generation
• DocumentPipeline (pipeline.py) integrates parsers, builds JSON outline with title, headings, raw_text, tables
d. Ranking Layer (Round-1B only)
• embedder.py loads MiniLM model from models/…
• ranker.py computes embeddings & relevance scores
• subsection_selector.py picks top sections per persona/task
• outline_to_refined_processor.py merges headings with refined text
e. Output Layer
• output_writer.py writes final JSON to output/ or Challenge_1b folders
• Sample outputs: challenge1b_outline_only.json, challenge1b_refined_output.json