# iamaayushijain/adobe_hackathon architecture

> Architecture diagram of the GitHub repository iamaayushijain/adobe_hackathon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iamaayushijain/adobe_hackathon
- Repository: https://github.com/iamaayushijain/adobe_hackathon

## Overview

##### Project Type and Purpose
- This is a specialized, offline batch-processing tool (not a web app) for extracting structured data from PDFs and, in its Round-1B mode, semantically ranking sections per “persona & task.”
- It’s organized as a CLI-driven pipeline, with two entry points:
- main2.py for outline-only extraction (Round-1A)
- main.py for persona-based ranking (Round-1B)
- Key goals: full-text capture, hierarchical outline creation, table extraction, and optional semantic ranking via a local MiniLM model.

##### High-Level File Structure and Architectural Patterns
- Top-level orchestration scripts: main2.py, main.py, pipeline.py
- Core application modules under `app/` (embedder, ranker, outline_to_refined_processor, subsection_selector)
- Extraction & parsing utilities: parser.py, raw_text_extractor.py, table_extractor.py, ocr_utils.py
- Model folder (`models/all-MiniLM-L6-v2`) for semantic embedding
- Input/output directories plus Challenge_1b collections for sample data
- Utilities and config: utils.py, output_writer.py, requirements.txt, Dockerfiles

This structure follows a linear pipeline pattern:
[Ingestion] → [Parsing] → [Outline Generation] → [Ranking] → [Output]

##### Key Components and Their Relationships
a. Ingestion Layer
- Input PDFs (input/ or Challenge_1b/Collection*/PDFs)
- main2.py or main.py / Docker entrypoint

b. Parsing Layer
- parser.py (PDFOutlineParser) orchestrates four PDF libs (pdfplumber, PyMuPDF, pdfminer, Camelot)
- raw_text_extractor.py collects all pages’ text
- table_extractor.py identifies and serializes tables
- ocr_utils.py (optional OCR fallback)

c. Generation
- DocumentPipeline (pipeline.py) integrates parsers, builds JSON outline with title, headings, raw_text, tables

d. Ranking Layer (Round-1B only)
- embedder.py loads MiniLM model from models/…
- ranker.py computes embeddings & relevance scores
- subsection_selector.py picks top sections per persona/task
- outline_to_refined_processor.py merges headings with refined text

e. Output Layer
- output_writer.py writes final JSON to output/ or Challenge_1b folders
- Sample outputs: challenge1b_outline_only.json, challenge1b_refined_output.json

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion & Entry Points"
        direction TB
        PDFs["PDF Files\n(input/ & Challenge_1b/Collection*/PDFs/)"]:::storage
        Main2["main2.py"]:::ingestion
        Main["main.py"]:::ingestion
    end

    %% Parsing Layer
    subgraph "Parsing Layer"
        direction TB
        Parser["PDFOutlineParser\n(parser.py)"]:::parsing
        RawText["RawTextExtractor\n(raw_text_extractor.py)"]:::parsing
        TableExt["TableExtractor\n(table_extractor.py)"]:::parsing
        OCR["OCR Utilities\n(ocr_utils.py)"]:::parsing
    end

    %% Core Pipeline
    subgraph "Outline Generation\n(Core Pipeline)"
        direction TB
        Pipeline["DocumentPipeline\n(pipeline.py)"]:::core
        OutlineJSON["Outline JSON\n(temporary)"]:::storage
    end

    %% Ranking Layer
    subgraph "Ranking Layer\n(Round-1B Only)"
        direction TB
        Embedder["Embedder\n(app/embedder.py)"]:::ranking
        Ranker["Ranker\n(app/ranker.py)"]:::ranking
        Selector["SubsectionSelector\n(app/subsection_selector.py)"]:::ranking
        Refiner["OutlineToRefinedProcessor\n(app/outline_to_refined_processor.py)"]:::ranking
        ModelStore["MiniLM Model\n(models/all-MiniLM-L6-v2/)"]:::storage
        RefinedJSON["Refined JSON Output"]:::storage
    end

    %% Output Layer
    subgraph "Output Layer"
        direction TB
        Writer["OutputWriter\n(output_writer.py)"]:::storage
        OutlineOut["Round-1A JSON\n(output2/)"]:::storage
        ChallengeOut["Challenge1B Outputs\n(Challenge_1b/Collection*/challenge1b_*.json)"]:::storage
    end

    %% Data Flow
    PDFs --> Main2
    PDFs --> Main
    Main2 --> Parser
    Main --> Parser

    Parser --> RawText
    Parser --> TableExt
    Parser --> OCR
    RawText --> Pipeline
    TableExt --> Pipeline
    OCR --> Pipeline

    Pipeline --> OutlineJSON
    OutlineJSON --> Writer
    Writer --> OutlineOut

    OutlineJSON --> Embedder
    Embedder --> Ranker
    Ranker --> Selector
    Selector --> Refiner
    Refiner --> Writer
    Writer --> ChallengeOut

    Embedder --> ModelStore

    %% Click Events
    click Main2 "https://github.com/iamaayushijain/adobe_hackathon/blob/main/main2.py"
    click Main "https://github.com/iamaayushijain/adobe_hackathon/blob/main/main.py"
    click Parser "https://github.com/iamaayushijain/adobe_hackathon/blob/main/parser.py"
    click RawText "https://github.com/iamaayushijain/adobe_hackathon/blob/main/raw_text_extractor.py"
    click TableExt "https://github.com/iamaayushijain/adobe_hackathon/blob/main/table_extractor.py"
    click OCR "https://github.com/iamaayushijain/adobe_hackathon/blob/main/ocr_utils.py"
    click Pipeline "https://github.com/iamaayushijain/adobe_hackathon/blob/main/pipeline.py"
    click Embedder "https://github.com/iamaayushijain/adobe_hackathon/blob/main/app/embedder.py"
    click Ranker "https://github.com/iamaayushijain/adobe_hackathon/blob/main/app/ranker.py"
    click Selector "https://github.com/iamaayushijain/adobe_hackathon/blob/main/app/subsection_selector.py"
    click Refiner "https://github.com/iamaayushijain/adobe_hackathon/blob/main/app/outline_to_refined_processor.py"
    click ModelStore "https://github.com/iamaayushijain/adobe_hackathon/tree/main/models/all-MiniLM-L6-v2/"
    click PDFs "https://github.com/iamaayushijain/adobe_hackathon/tree/main/input/"
    click ChallengeOut "https://github.com/iamaayushijain/adobe_hackathon/tree/main/Challenge_1b/Collection*/PDFs/"
    click Writer "https://github.com/iamaayushijain/adobe_hackathon/blob/main/output_writer.py"

    %% Styles
    classDef ingestion fill:#add8e6,stroke:#000,stroke-width:1px
    classDef parsing fill:#add8e6,stroke:#000,stroke-width:1px
    classDef core fill:#90ee90,stroke:#000,stroke-width:1px
    classDef ranking fill:#ffcc99,stroke:#000,stroke-width:1px
    classDef storage fill:#d3d3d3,stroke:#000,stroke-width:1px

    %% Legend
    subgraph "Legend"
        direction TB
        LI["Ingestion/Parsing"]:::ingestion
        LC["Core Pipeline"]:::core
        LR["Ranking"]:::ranking
        LS["Storage/Model"]:::storage
    end
```
