# danieljamesdavies/turing-explorer-back-end architecture

> Architecture diagram of the GitHub repository danieljamesdavies/turing-explorer-back-end, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danieljamesdavies/turing-explorer-back-end
- Repository: https://github.com/danieljamesdavies/turing-explorer-back-end

## Overview

##### Main Components Identification

a) Frontend Application
- Represented as a client-side component
- Interfaces with the backend through API routes

b) Backend Server (Flask-based, indicated by app.py)
- Main API server handling requests
- Route handlers in the /routes directory

c) Core Components:
- TuringLLM Module (model inference and processing)
- SAE (Sparse Autoencoder) Module
- Latent Data Storage System

**[Frontend Client] ←→ [Backend API Server] ←→ [Core Processing Modules]**
- ↕
- [Data Storage]

##### Backend API Server (Middle Layer)
- Route endpoints:
- /get_latent_connections
- /get_latent_data
- /get_latent_preview
- /inference
- /search_latents

##### Core Processing Modules (Right Side)
- TuringLLM Module
- model.py
- inference.py
- tokenizer.py
- config.py
- SAE Module
- SAE_TopK.py

##### Data Storage (Bottom Layer)
- Latent Data Storage
- top_other_latents
- unembedding_frequencies

C. Connections and Data Flow:

##### Additional Elements

- Component types
- Data flow directions

##### Technical Details

Annotations for:
- Model size (254M parameters)
- Key dependencies
- Data storage formats
- API endpoint purposes

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef frontend fill:#e1f5fe,stroke:#01579b
    classDef backend fill:#c8e6c9,stroke:#1b5e20
    classDef core fill:#ffe0b2,stroke:#e65100
    classDef storage fill:#e1bee7,stroke:#4a148c
    classDef api fill:#ffcdd2,stroke:#b71c1c

    %% Frontend Layer
    WebUI["Web Interface"]:::frontend

    %% Backend Layer
    APIServer["Flask API Server"]:::backend
    click APIServer "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/app.py"

    %% API Routes
    subgraph APIRoutes["API Routes"]
        LC["Latent Connections"]:::api
        LD["Latent Data"]:::api
        LP["Latent Preview"]:::api
        INF["Inference"]:::api
        SL["Search Latents"]:::api
    end
    click LC "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/routes/get_latent_connections.py"
    click LD "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/routes/get_latent_data.py"
    click LP "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/routes/get_latent_preview.py"
    click INF "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/routes/inference.py"
    click SL "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/routes/search_latents.py"

    %% Core Processing Modules
    subgraph CoreModules["Core Processing"]
        TuringModule["TuringLLM Module"]:::core
        click TuringModule "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/tree/main/TuringLLM"
        
        subgraph TuringComponents
            Model["Model Component"]:::core
            Inference["Inference Component"]:::core
            Tokenizer["Tokenizer Component"]:::core
            Config["Configuration Component"]:::core
        end
        click Model "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/TuringLLM/model.py"
        click Inference "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/TuringLLM/inference.py"
        click Tokenizer "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/TuringLLM/tokenizer.py"
        click Config "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/TuringLLM/config.py"

        SAE["SAE Module"]:::core
        click SAE "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/blob/main/SAE/SAE_TopK.py"
    end

    %% Data Storage
    subgraph Storage["Data Storage"]
        LatentStore["Latent Connections Storage"]:::storage
        FreqStore["Frequency Storage"]:::storage
    end
    click LatentStore "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/tree/main/latent_data/top_other_latents"
    click FreqStore "https://github.com/DanielJamesDavies/Turing-Explorer-Back-End/tree/main/latent_data/unembedding_frequencies"

    %% Connections
    WebUI <--> APIServer
    APIServer --> APIRoutes
    APIRoutes --> CoreModules
    CoreModules <--> Storage

    %% Component Relationships
    TuringModule --> Model
    TuringModule --> Inference
    TuringModule --> Tokenizer
    TuringModule --> Config
    TuringModule <--> SAE

    %% Legend
    subgraph Legend
        Frontend["Frontend Component"]:::frontend
        BackendComp["Backend Component"]:::backend
        CoreComp["Core Component"]:::core
        StorageComp["Storage Component"]:::storage
        APIComp["API Component"]:::api
    end
```
