# idvorkin/nlp architecture

> Architecture diagram of the GitHub repository idvorkin/nlp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/idvorkin/nlp
- Repository: https://github.com/idvorkin/nlp

## Overview

##### Main Components

a) Core Processing Components:
- NLP Processing Engine (centered around files like `think.py`, `gpt.py`, `gpt3.py`)
- Language Model Integration (evidenced by OpenAI integration via `openai_wrapper.py`)
- Text Analysis Tools (shown by various notebooks and Python scripts)

b) Input/Output Systems:
- CLI Interface Layer (multiple CLI tools mentioned in README)
- Journal Processing System (referenced in `igor_journal.py`, `journal.py`)
- Voice/Audio Processing (`transcribe.py`, `tts.py`)

c) Bot Systems:
- Multiple Bot Implementations (`bestie-bot.py`, `improv_bot.py`)
- Conversation Management (`convos` directory)

d) Evaluation Framework:
- Testing Infrastructure (`tests` directory)
- Evaluation Tools (`eval` directory)
- Prompt Management System (`prompts.py`)

**┌─ User Interface Layer ─────────────────────┐**
- CLI Tools │ Notebooks │ Bot Interfaces     │
- Application Layer ──────────────────────┐ │
- Journal Processing │ NLP Analysis        │ │
- Conversation Management │ Bot Logic      │ │
- Core Services Layer ────────────────────┤ │
- OpenAI Integration │ Language Processing │ │
- Text Analysis │ Voice Processing        │ │
- ┘

##### Key Relationships to Highlight

- Data flow between journal entries and NLP processing
- How the bot systems interact with the core NLP services
- Demonstrate the evaluation framework's relationship with other components
- Integration points with external services (OpenAI, etc.)

##### Implementation Details

- Rectangles: Core services
- Hexagons: External services
- Rounded rectangles: User interfaces

Annotations for:
- File formats supported
- API endpoints
- Configuration points
- Testing boundaries

##### Additional Elements

- Version/environment information
- Dependencies and requirements
- Scaling/deployment considerations

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and classes
    classDef ui fill:#90EE90,stroke:#333,stroke-width:2px
    classDef core fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef bot fill:#FFA500,stroke:#333,stroke-width:2px
    classDef eval fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef data fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef external fill:#FFD700,stroke:#333,stroke-width:2px

    %% User Interface Layer
    subgraph UI["User Interface Layer"]
        CLI["CLI Tools"]:::ui
        NB["Notebooks"]:::ui
        VCLI["Voice CLI"]:::ui
    end

    %% Core Processing Layer
    subgraph Core["Core Processing Layer"]
        NLP["NLP Processing Engine"]:::core
        LMI["Language Model Integration"]:::core
        JP["Journal Processing"]:::core
        VP["Voice Processing"]:::core
    end

    %% Bot Systems Layer
    subgraph Bots["Bot Systems"]
        BB["Bestie Bot"]:::bot
        IB["Improv Bot"]:::bot
        CM["Conversation Management"]:::bot
    end

    %% Evaluation Layer
    subgraph Eval["Evaluation Framework"]
        EV["Evaluation Tools"]:::eval
        TI["Testing Infrastructure"]:::eval
        PM["Prompt Management"]:::eval
    end

    %% External Services
    ES["OpenAI API"]:::external

    %% Text Analysis Tools
    TA["Text Analysis Tools"]:::core

    %% Relationships
    CLI --> NLP
    CLI --> JP
    VCLI --> VP
    NB --> TA
    NLP --> LMI
    LMI --> ES
    JP --> NLP
    VP --> LMI
    BB --> LMI
    IB --> LMI
    CM --> BB
    CM --> IB
    EV --> NLP
    TI --> NLP
    PM --> LMI
    
    %% Click events
    click NLP "https://github.com/idvorkin/nlp/blob/main/think.py"
    click LMI "https://github.com/idvorkin/nlp/blob/main/gpt.py"
    click JP "https://github.com/idvorkin/nlp/blob/main/journal.py"
    click VP "https://github.com/idvorkin/nlp/blob/main/transcribe.py"
    click VCLI "https://github.com/idvorkin/nlp/blob/main/voice_cli.py"
    click BB "https://github.com/idvorkin/nlp/blob/main/bestie-bot.py"
    click IB "https://github.com/idvorkin/nlp/blob/main/improv_bot.py"
    click CM "https://github.com/idvorkin/nlp/tree/main/convos/"
    click EV "https://github.com/idvorkin/nlp/tree/main/eval/"
    click TI "https://github.com/idvorkin/nlp/tree/main/tests/"
    click PM "https://github.com/idvorkin/nlp/blob/main/prompts.py"
    click NB "https://github.com/idvorkin/nlp/blob/main/PlayEmbed.ipynb"

    %% Legend
    subgraph Legend
        L1["UI Components"]:::ui
        L2["Core Components"]:::core
        L3["Bot Systems"]:::bot
        L4["Evaluation"]:::eval
        L5["External Services"]:::external
    end
```
