# e-p-armstrong/augmentoolkit architecture

> Architecture diagram of the GitHub repository e-p-armstrong/augmentoolkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/e-p-armstrong/augmentoolkit
- Repository: https://github.com/e-p-armstrong/augmentoolkit

## Overview

The system architecture should be represented as follows:

##### Core Components (Top Level)
- [Pipeline Manager] -> Controls execution of pipelines
- [Config Manager] -> Handles configuration and settings
- [Engine Wrapper] -> Manages API/Model interactions

##### Pipeline Structure (Second Level)
- QA Generation Pipeline
- RPToolkit Pipeline
- Classifier Creator Pipeline
Each pipeline should show its internal steps as a flow:

For QA Pipeline:
- Text Input -> Chunking -> Question Generation -> Validation -> Answer Generation -> Output Dataset

For RPToolkit:
- Story Input -> Emotion/Feature Extraction -> Scene Generation -> Story Generation -> Rating -> Output Dataset

For Classifier:
- Text Input -> Rule Generation -> Classification -> Model Training -> Evaluation -> Output Model

##### Core Abstractions (Third Level)
Relationship between key classes:
- PipelineStep
- GenerationStep
- EngineWrapper

##### Data Flow (Overlay)
- Input processing flow
- API/Model interactions
- Output generation paths
- Configuration application points

##### External Integrations
Connections to:
- Various AI APIs (Together.ai, Fireworks, etc.)
- Local Model Servers
- File System (Input/Output)
- Training Systems (Axolotl)

Key Design Elements to Include:

##### Component Labels
- Clear, concise names
- Version/configuration indicators where relevant
- Input/output format specifications

##### Subsystem Boundaries
- Related components
- Isolation between pipelines
- Shared resources

##### Modular nature of the system
- Each pipeline is independent
- Common abstractions are shared
- Easy to add new pipelines

##### Resumability feature
- Intermediate storage points
- State management
- Recovery paths

##### Configuration flexibility
- Multiple model support
- API provider switching
- Custom prompt integration

##### Validation points
- Quality checks
- Error handling
- Fallback paths

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core Components
    subgraph Core["Core Components"]
        PM[Pipeline Manager]:::core
        CM[Config Manager]:::core
        EW[Engine Wrapper]:::core
    end
    click EW "https://github.com/e-p-armstrong/augmentoolkit/blob/master/augmentoolkit/generation_functions/engine_wrapper_class.py"

    %% Pipeline Abstractions
    subgraph Abstractions["Core Abstractions"]
        PS[PipelineStep]:::abstract
        GS[GenerationStep]:::abstract
    end
    click PS "https://github.com/e-p-armstrong/augmentoolkit/blob/master/augmentoolkit/generation_functions/pipeline_step_class.py"
    click GS "https://github.com/e-p-armstrong/augmentoolkit/blob/master/augmentoolkit/generation_functions/generation_step_class.py"

    %% QA Pipeline
    subgraph QA["QA Generation Pipeline"]
        direction TB
        QA1[Text Input]:::process
        QA2[Chunking]:::process
        QA3[Question Generation]:::process
        QA4[Validation]:::validation
        QA5[Answer Generation]:::process
        QA6[Output Dataset]:::output
        QA1 --> QA2 --> QA3 --> QA4 --> QA5 --> QA6
    end
    click QA "https://github.com/e-p-armstrong/augmentoolkit/tree/master/original/"
    click QA3 "https://github.com/e-p-armstrong/augmentoolkit/blob/master/original/steps.py"
    click QA1 "https://github.com/e-p-armstrong/augmentoolkit/blob/master/original/processing.py"

    %% RPToolkit Pipeline
    subgraph RP["RPToolkit Pipeline"]
        direction TB
        RP1[Story Input]:::process
        RP2[Emotion/Feature Extraction]:::process
        RP3[Scene Generation]:::process
        RP4[Story Generation]:::process
        RP5[Rating]:::validation
        RP6[Output Dataset]:::output
        RP1 --> RP2 --> RP3 --> RP4 --> RP5 --> RP6
    end
    click RP "https://github.com/e-p-armstrong/augmentoolkit/tree/master/rptoolkit/"
    click RP3 "https://github.com/e-p-armstrong/augmentoolkit/blob/master/rptoolkit/steps.py"
    click RP1 "https://github.com/e-p-armstrong/augmentoolkit/blob/master/rptoolkit/processing.py"

    %% Classifier Pipeline
    subgraph CL["Classifier Creator Pipeline"]
        direction TB
        CL1[Text Input]:::process
        CL2[Rule Generation]:::process
        CL3[Classification]:::process
        CL4[Model Training]:::process
        CL5[Evaluation]:::validation
        CL6[Output Model]:::output
        CL1 --> CL2 --> CL3 --> CL4 --> CL5 --> CL6
    end
    click CL "https://github.com/e-p-armstrong/augmentoolkit/tree/master/classifier_creator/"
    click CL3 "https://github.com/e-p-armstrong/augmentoolkit/blob/master/classifier_creator/steps.py"
    click CL1 "https://github.com/e-p-armstrong/augmentoolkit/blob/master/classifier_creator/processing.py"

    %% External Integrations
    subgraph External["External Integrations"]
        API[AI APIs]:::external
        LMS[Local Model Servers]:::external
        FS[File System]:::external
        TS[Training Systems]:::external
    end
    click API "https://github.com/e-p-armstrong/augmentoolkit/tree/master/original/config_overrides/"
    click TS "https://github.com/e-p-armstrong/augmentoolkit/tree/master/_model_training_configs/"

    %% Configurations
    subgraph Config["Configuration"]
        BC[Base Config]:::config
        PC[Pipeline Configs]:::config
    end
    click BC "https://github.com/e-p-armstrong/augmentoolkit/blob/master/super_config.yaml"
    click PC "https://github.com/e-p-armstrong/augmentoolkit/blob/master/original/config.yaml"

    %% Utils
    UTIL[Utility Functions]:::util
    click UTIL "https://github.com/e-p-armstrong/augmentoolkit/tree/master/augmentoolkit/utils/"

    %% Relationships
    Core --> Abstractions
    PM --> QA & RP & CL
    CM --> Config
    Config --> Core
    EW --> External
    Abstractions --> QA & RP & CL
    UTIL --> Core

    %% Styles
    classDef core fill:#2374ab,stroke:#000,stroke-width:2px
    classDef abstract fill:#457b9d,stroke:#000
    classDef process fill:#a8dadc,stroke:#000
    classDef validation fill:#e63946,stroke:#000
    classDef output fill:#90be6d,stroke:#000
    classDef external fill:#f4a261,stroke:#000
    classDef config fill:#ffb703,stroke:#000
    classDef util fill:#8ecae6,stroke:#000

    %% Legend
    subgraph Legend
        L1[Core Component]:::core
        L2[Abstract Class]:::abstract
        L3[Process Step]:::process
        L4[Validation Point]:::validation
        L5[Output]:::output
        L6[External System]:::external
        L7[Configuration]:::config
        L8[Utility]:::util
    end
```
