# choudhryfrompak/qa-dataset-generator architecture

> Architecture diagram of the GitHub repository choudhryfrompak/qa-dataset-generator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/choudhryfrompak/qa-dataset-generator
- Repository: https://github.com/choudhryfrompak/qa-dataset-generator

## Overview

##### Project Components Identification
- Main Application Controller – represented by main.py. This file serves as the entry point, orchestrating the overall workflow.
- Document Processing Module – this part is responsible for reading PDF or text files and splitting them into chunks. Mark an element here that deals with file ingestion and pre-processing.
- QA Generation Module – implemented within agents.py and using prompts from prompts.py. This module interacts with Groq LLM to generate question-answer pairs.
- Validation & Logging Module – even if not contained in a separate file, include a component that handles error checks, confidence scoring, and logging (both console and file-based).
- Output Handling – a module that formats and writes the generated QA pairs into two output formats (CSV and JSON) inside the output directory.
- User Interface Layer – the Gradio-based UI that lets users customize parameters (Groq API key, input file, chunk size, etc.) and interacts with the main application via port 7860.
- Docker Container – encapsulates the entire application for deployment. Highlight the role of Dockerfile, exposing port 7860 and mounting the output directory.

##### Relationships and Interactions
- From the Document Processing component, the processed text chunks are passed to the QA Generation component (agents.py and prompts.py) where the Groq LLM is invoked.
- After QA pair generation, data flows to a Validation/Logging unit, where confidence scoring and error handling take place.
- Finally, the validated results are sent to the Output Handler to generate both CSV and JSON files.

##### Architectural Patterns & Design Principles
- Separation of Concerns: Each distinct functionality (input handling, processing, QA generation, validation, output, UI) should be represented as separate components.
- Modularity: Highlight how agents.py and prompts.py isolate the logic for interacting with the LLM, making it easier to update if the LLM integration changes.
- Containerization: Mention the use of Docker for environment consistency and ease of deployment. Emphasize that the Dockerfile ensures that dependencies are installed automatically, and output data is mounted to the host for data persistence.
- Logging and Error Handling: Although not modularized into a separate file, create a node for tracking errors and logging flows to indicate robust error monitoring.

##### Technologies and Frameworks
- Python 3.9+: The core programming language.
- Groq LLM: For generating the QA content.
- Gradio: To build the user-friendly web interface.
- PyPDF2 (and possibly others): For processing PDFs.
- Docker: For containerized deployments.
- Standard libraries for CSV/JSON handling and logging.

## Diagram (Mermaid)

```mermaid
flowchart TD
    User["User"]:::external

    subgraph "Docker Container"
        DC["Docker Container"]:::docker
        UI["Gradio UI (port 7860)"]:::ui
        Controller["Main Controller (main.py)"]:::controller
        subgraph "Backend Processing"
            Processor["Document Processing"]:::processor
            QAGen["QA Generation (agents & prompts)"]:::qa
            ValidLog["Validation & Logging"]:::log
            Output["Output Formatter (CSV/JSON)"]:::output
        end
    end

    GroqLLM["Groq LLM"]:::external

    User -->|"provides_input"| UI
    DC -->|"exposes:7860"| UI
    UI -->|"triggers_processing"| Controller
    Controller -->|"ingests_file"| Processor
    Processor -->|"sends_chunks"| QAGen
    QAGen -->|"invokes"| GroqLLM
    GroqLLM -->|"returns_QA"| QAGen
    QAGen -->|"sends_QApairs"| ValidLog
    ValidLog -->|"logs_validates"| Output

    click Controller "https://github.com/choudhryfrompak/qa-dataset-generator/blob/main/main.py"
    click Processor "https://github.com/choudhryfrompak/qa-dataset-generator/blob/main/main.py"
    click QAGen "https://github.com/choudhryfrompak/qa-dataset-generator/blob/main/agents.py"
    click QAGen "https://github.com/choudhryfrompak/qa-dataset-generator/blob/main/prompts.py"
    click ValidLog "https://github.com/choudhryfrompak/qa-dataset-generator/blob/main/main.py"
    click Output "https://github.com/choudhryfrompak/qa-dataset-generator/blob/main/main.py"
    click UI "https://github.com/choudhryfrompak/qa-dataset-generator/blob/main/main.py"
    click DC "https://github.com/choudhryfrompak/qa-dataset-generator/tree/main/Dockerfile"

    classDef external fill:#F0E68C,stroke:#333,stroke-width:2px;
    classDef docker fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef ui fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef controller fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef processor fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef qa fill:#AFEEEE,stroke:#333,stroke-width:2px;
    classDef log fill:#D3D3D3,stroke:#333,stroke-width:2px;
    classDef output fill:#FFA07A,stroke:#333,stroke-width:2px;
```
