# ekatraone/alpaca-dataset-generator architecture

> Architecture diagram of the GitHub repository ekatraone/alpaca-dataset-generator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ekatraone/alpaca-dataset-generator
- Repository: https://github.com/ekatraone/alpaca-dataset-generator

## Overview

##### Project type and purpose
- This is a Python-based data‐processing pipeline/tool (not a web app).
- Its goal: ingest text/PDF/Word files, generate Alpaca-style QA examples, validate them, and write raw + validated JSONL outputs.
- Key features: multi-threaded loading, batch processing, optional GPU acceleration, progress tracking, and customizable config.

##### File structure
- src/ contains all core modules:
- config.py – global settings
- data_loader.py – parallel file reading/parsing
- dataset_generator.py & edu_dataset-generator.py – instruction templates + example assembly
- model_setup.py – GPU detection & setup
- validation.py & utils.py – output filtering and helper functions
- main.py – entry point / orchestrator
- data/input/ – source files (.txt/.pdf/.docx)
- output/ – raw_dataset.jsonl and validated_dataset.jsonl
- requirements.txt – external dependencies
- NLTK resource step in setup implies an external data dependency.

##### README insights
- Emphasizes performance (multi‐thread, batch, GPU).
- Describes two output streams (raw vs validated).
- Mentions customization points: instruction list in dataset_generator.py, validation logic in utils.py.

##### Main components & interactions
a. Components
1) Configuration Manager (config.py)
2) Input File System (data/input directory + file parsers)
3) Data Loader (data_loader.py) with ThreadPoolExecutor
4) Pre-processing Utilities (utils.py + edu_utils.py)
5) Model Setup (model_setup.py → detects/initializes GPU)
6) Dataset Generator (dataset_generator.py / edu_dataset-generator.py → applies instruction templates, batching)
7) Validation Module (validation.py → filters examples via is_valid_output)
8) Output Writers (to raw_dataset.jsonl and validated_dataset.jsonl)
9) Progress Tracker / Logging (embedded in main.py)
10) External Dependencies (NLTK corpora, PDF/Word parsing libs, GPU drivers)

b. Relationships & data flow
- From Dataset Generator → Validation → Validated Output Writer
- Config Manager feeds all components with parameters (batch_size, max_workers, file paths).
- Model Setup component hooks into the Dataset Generator for GPU acceleration.
- Progress Tracker monitors each stage and logs to console/file.

c. Architectural patterns & design principles
- Modular pipeline: clear separation of concerns between loading, processing, generation, validation, and output.
- Config-driven behavior: single source of truth for parameters.
- Concurrency pattern: producer–consumer with a thread pool for I/O and batching for compute.
- Extensibility points: swap in new file parsers or instruction templates by updating data_loader or dataset_generator.

d. Key technologies & libraries
- Python 3.x
- NLTK (punkt, stopwords)
- PDF parser (e.g., PyPDF2)
- python-docx
- ThreadPoolExecutor (concurrent.futures)
- JSONL output
- (Optional) PyTorch/TensorFlow for GPU acceleration or torch.cuda for model_setup.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration and External
    Config["Configuration Manager"]:::config
    External["External Dependencies"]:::external

    %% Input
    InputFS["Input File System\n(data/input)"]:::input

    %% Processing Modules
    DataLoader["Data Loader\n(multi-thread)"]:::processing
    Utils["Pre-processing Utils\n(utils.py)"]:::processing
    EduUtils["Educational Utils\n(edu_utils.py)"]:::processing
    ModelSetup["Model Setup\n(GPU Detection)"]:::processing
    DatasetGenCore["Dataset Generator\n(core)"]:::processing
    DatasetGenEdu["Dataset Generator\n(edu)"]:::processing
    Validation["Validation Module"]:::processing

    %% Output
    RawWriter["Raw Output Writer\n(raw_dataset.jsonl)"]:::output
    ValidatedWriter["Validated Output Writer\n(validated_dataset.jsonl)"]:::output

    %% Orchestrator
    Main["Orchestrator / Progress Tracker / Output Writers"]:::processing

    %% Layout and Flow
    Config --> InputFS
    External -.-> Config
    External -.-> DataLoader
    InputFS --> DataLoader
    DataLoader --> Utils
    Utils --> EduUtils
    EduUtils --> DatasetGenCore
    EduUtils --> DatasetGenEdu
    ModelSetup --> DatasetGenCore
    ModelSetup --> DatasetGenEdu
    DatasetGenCore --> RawWriter
    DatasetGenCore --> Validation
    DatasetGenEdu --> RawWriter
    DatasetGenEdu --> Validation
    Validation --> ValidatedWriter
    RawWriter --> Main
    ValidatedWriter --> Main

    %% Click Events
    click Config "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/config.py"
    click DataLoader "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/data_loader.py"
    click Utils "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/utils.py"
    click EduUtils "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/edu_utils.py"
    click ModelSetup "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/model_setup.py"
    click DatasetGenCore "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/dataset_generator.py"
    click DatasetGenEdu "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/edu_dataset-generator.py"
    click Validation "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/validation.py"
    click Main "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/src/main.py"
    click External "https://github.com/ekatraone/alpaca-dataset-generator/blob/main/requirements.txt"

    %% Styles
    classDef input fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#000,stroke-width:1px
    classDef output fill:#FFA500,stroke:#000,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
```
