How sar-hal/icdcit-hackathon works
Project type and purpose
• This is a standalone Python-based content-processing tool (“PDF Course Content Minimizer”) designed to ingest educational PDF documents, extract text, intelligently chunk that text, call out to an LLM via the Groq API under rate limits, and produce concise summaries that optimize token usage.
High-level architecture and main components
a. Orchestrator (CLI/Script Entry Point)
• File: Course_Reduction_V2.py in Minimization_Final_Version/
• Responsibilities: read input paths, load configuration, kick off the processing pipeline, write outputs.
b. PDF Extraction Module
• Uses PyMuPDF for text extraction (and earlier in the “old” branch, EasyOCR/Tesseract notebooks existed).
• Input: raw PDF file(s)
• Output: raw text string
c. Text Chunking Module
• Splits long text into chunks sized to match LLM token limits, using smart boundary detection.
• Reads chunk‐size settings (from code or config).
d. Rate‐Limiter / API Client
• Enforces Groq API rate limits and retry logic.
• Encapsulated in a wrapper around the HTTP client.
e. Summarization Module
• Sends each chunk to the Groq API, collects partial summaries.
• May further combine chunk summaries into a global summary.
f. Output Aggregator & Writer
• Merges partial results, applies any final formatting or post-processing, writes to disk (e.g., final_summary_v2.txt).
g. Configuration & Utilities
• .env for Groq API key
• Utility functions for logging, error handling, token counting, JSON I/O
Relationships and data flow
• User invokes the orchestrator script →
1. Orchestrator loads config & input PDF list
2. For each PDF:
• PDF Extraction → raw text →
• Text Chunking → [chunk1, chunk2,…] →
• For each chunk: Rate-Limiter → API Client → Groq API → chunk summary →
• Aggregate chunk summaries → final summary
3. Write final summary to file
• External service: Groq API (over HTTPS)
Architectural patterns and design principles
• Pipeline pattern: clear stages (extract → chunk → summarize → aggregate)
• Single-responsibility modules: each component performs one function
• Separation of concerns: application logic is decoupled from API communication and configuration
• Config-driven behavior: environment variables for sensitive data, easily tunable chunk sizes