How choudhryfrompak/qa-dataset-generator works
Project Components Identification
• Main Application Controller – represented by main.py. This file serves as the entry point, orchestrating the overall workflow.
• Document Processing Module – this part is responsible for reading PDF or text files and splitting them into chunks. Mark an element here that deals with file ingestion and pre-processing.
• QA Generation Module – implemented within agents.py and using prompts from prompts.py. This module interacts with Groq LLM to generate question-answer pairs.
• Validation & Logging Module – even if not contained in a separate file, include a component that handles error checks, confidence scoring, and logging (both console and file-based).
• Output Handling – a module that formats and writes the generated QA pairs into two output formats (CSV and JSON) inside the output directory.
• User Interface Layer – the Gradio-based UI that lets users customize parameters (Groq API key, input file, chunk size, etc.) and interacts with the main application via port 7860.
• Docker Container – encapsulates the entire application for deployment. Highlight the role of Dockerfile, exposing port 7860 and mounting the output directory.
Relationships and Interactions
• From the Document Processing component, the processed text chunks are passed to the QA Generation component (agents.py and prompts.py) where the Groq LLM is invoked.
• After QA pair generation, data flows to a Validation/Logging unit, where confidence scoring and error handling take place.
• Finally, the validated results are sent to the Output Handler to generate both CSV and JSON files.
Architectural Patterns & Design Principles
• Separation of Concerns: Each distinct functionality (input handling, processing, QA generation, validation, output, UI) should be represented as separate components.
• Modularity: Highlight how agents.py and prompts.py isolate the logic for interacting with the LLM, making it easier to update if the LLM integration changes.
• Containerization: Mention the use of Docker for environment consistency and ease of deployment. Emphasize that the Dockerfile ensures that dependencies are installed automatically, and output data is mounted to the host for data persistence.
• Logging and Error Handling: Although not modularized into a separate file, create a node for tracking errors and logging flows to indicate robust error monitoring.
Technologies and Frameworks
• Python 3.9+: The core programming language.
• Groq LLM: For generating the QA content.
• Gradio: To build the user-friendly web interface.
• PyPDF2 (and possibly others): For processing PDFs.
• Docker: For containerized deployments.
• Standard libraries for CSV/JSON handling and logging.