# sar-hal/icdcit-hackathon architecture

> Architecture diagram of the GitHub repository sar-hal/icdcit-hackathon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sar-hal/icdcit-hackathon
- Repository: https://github.com/sar-hal/icdcit-hackathon

## Overview

##### Project type and purpose
- This is a standalone Python-based content-processing tool (“PDF Course Content Minimizer”) designed to ingest educational PDF documents, extract text, intelligently chunk that text, call out to an LLM via the Groq API under rate limits, and produce concise summaries that optimize token usage.

##### High-level architecture and main components
a. **Orchestrator (CLI/Script Entry Point)**
- File: `Course_Reduction_V2.py` in `Minimization_Final_Version/`
- Responsibilities: read input paths, load configuration, kick off the processing pipeline, write outputs.
b. **PDF Extraction Module**
- Uses PyMuPDF for text extraction (and earlier in the “old” branch, EasyOCR/Tesseract notebooks existed).
- Input: raw PDF file(s)
- Output: raw text string
c. **Text Chunking Module**
- Splits long text into chunks sized to match LLM token limits, using smart boundary detection.
- Reads chunk‐size settings (from code or config).
d. **Rate‐Limiter / API Client**
- Enforces Groq API rate limits and retry logic.
- Encapsulated in a wrapper around the HTTP client.
e. **Summarization Module**
- Sends each chunk to the Groq API, collects partial summaries.
- May further combine chunk summaries into a global summary.
f. **Output Aggregator & Writer**
- Merges partial results, applies any final formatting or post-processing, writes to disk (e.g., `final_summary_v2.txt`).
g. **Configuration & Utilities**
- `.env` for Groq API key
- Utility functions for logging, error handling, token counting, JSON I/O

##### Relationships and data flow
- User invokes the orchestrator script →
1. Orchestrator loads config & input PDF list
2. For each PDF:
- PDF Extraction → raw text →
- Text Chunking → [chunk1, chunk2,…] →
- For each chunk: Rate-Limiter → API Client → Groq API → chunk summary →
- Aggregate chunk summaries → final summary
3. Write final summary to file
- External service: Groq API (over HTTPS)

##### Architectural patterns and design principles
- **Pipeline pattern**: clear stages (extract → chunk → summarize → aggregate)
- **Single-responsibility modules**: each component performs one function
- **Separation of concerns**: application logic is decoupled from API communication and configuration
- **Config-driven behavior**: environment variables for sensitive data, easily tunable chunk sizes

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Pipeline stages
    subgraph "Content Processing Pipeline"
        direction TB
        Orchestrator["Orchestrator\n(Course_Reduction_V2.py)"]:::internal
        PDFInput["PDF Input Folder"]:::files
        Extractor["PDF Extraction Module"]:::internal
        Chunker["Text Chunking Module"]:::internal
        RateLimiter["Rate-Limiter / API Client"]:::internal
        Summarizer["Summarization Module\n(Batch Requester & Parser)"]:::internal
        Aggregator["Output Aggregator & Writer"]:::internal
        OutputV2["Final Summary v2.txt"]:::files
        OutputV3["Final Summary v3.txt"]:::files
        Config["Configuration & Utilities\n(Workflow.md & Utils)"]:::files
    end

    %% External service
    GroqAPI["Groq API"]:::external

    %% Legacy notebooks
    subgraph "Legacy Notebooks\n(Minimization_Old)"
        direction TB
        NB1["PDF_Extraction_Tesseract.ipynb"]:::files
        NB2["PDF_Extraction_EasyOcr.ipynb"]:::files
        NB3["PDF_Extraction_Ver2.ipynb"]:::files
    end

    %% Data flows
    Config -->|loads settings| Orchestrator
    Orchestrator -->|reads PDFs| PDFInput
    PDFInput -->|raw PDF| Extractor
    Extractor -->|raw text| Chunker
    Chunker -->|text chunks| RateLimiter
    RateLimiter -->|requests| GroqAPI
    GroqAPI -->|chunk summaries| RateLimiter
    RateLimiter -->|partial summaries| Summarizer
    Summarizer -->|combined summary| Aggregator
    Aggregator -->|write summary| OutputV2
    Aggregator -->|write summary| OutputV3

    %% Development lifecycle
    NB1 -->|inspired| Orchestrator
    NB2 -->|inspired| Orchestrator
    NB3 -->|inspired| Orchestrator

    %% Click events
    click Orchestrator "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/Course_Reduction_V2.py"
    click Extractor "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/Course_Reduction_V2.py"
    click Chunker "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/Course_Reduction_V2.py"
    click RateLimiter "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/Course_Reduction_V2.py"
    click Summarizer "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/Course_Reduction_V2.py"
    click Aggregator "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/Course_Reduction_V2.py"
    click OutputV2 "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/final_summary_v2.txt"
    click OutputV3 "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/final_summary_v3.txt"
    click Config "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Final_Version/Workflow.md"
    click NB1 "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Old/Initial Notebooks/PDF_Extraction_Tesseract.ipynb"
    click NB2 "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Old/Testing_Stuff/PDF_Extraction_EasyOcr.ipynb"
    click NB3 "https://github.com/sar-hal/icdcit-hackathon/blob/main/Minimization_Old/Testing_Stuff/PDF_Extraction_Ver2.ipynb"

    %% Styles
    classDef internal fill:#BBDEFB,stroke:#0288D1,color:#000;
    classDef external fill:#C8E6C9,stroke:#2E7D32,color:#000;
    classDef files fill:#EEEEEE,stroke:#616161,color:#000;
```
