# jjgo/thunderpack architecture

> Architecture diagram of the GitHub repository jjgo/thunderpack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jjgo/thunderpack
- Repository: https://github.com/jjgo/thunderpack

## Overview

##### Main Components

a) Core Components:
- LMDB Database Engine (as the underlying storage mechanism)
- ThunderDB (Write Interface)
- ThunderReader (Read Interface)
- ThunderDataset (PyTorch Integration)

b) Format Handlers:
- Tensor Formats (numpy, torch, jax)
- Media Formats (image, audio)
- Document Formats (json, yaml, etc.)
- Compression Handlers (lz4, zstd, etc.)

**Top Layer: User Interfaces**

**Middle Layer: Core Components**

**Bottom Layer: Storage & Formats**

##### Specific Implementation Details

a) User Interface Layer:
- ThunderDB for write operations (marked in green)
- ThunderReader for read operations (marked in blue)
- ThunderDataset as PyTorch integration (marked in purple)

b) Core Processing Layer:
- Format detection and conversion pipeline
- Compression/decompression workflow
- Demonstrate the type-format mapping system

c) Storage Layer:
- LMDB database with memory mapping
- Format handlers for different data types
- Compression modules

##### Data Flow Representation

- Write path: Python objects → Format conversion → Compression → LMDB storage
- Read path: LMDB retrieval → Decompression → Format conversion → Python objects

##### Important Elements to Highlight

- Memory mapping mechanism
- Thread-safe concurrent access
- Automatic format detection and conversion
- Compression pipeline
- PyTorch integration points

##### Additional Notes

Callouts for:
- Performance optimization points
- Thread safety mechanisms
- Memory mapping details
- Supported format list

##### The integration with PyTorch's ecosystem

This design should reflect ThunderPack's primary goal of providing high-performance data access for deep learning applications while maintaining flexibility and ease of use.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Style Definitions
    classDef write fill:#90EE90
    classDef read fill:#ADD8E6
    classDef storage fill:#D3D3D3
    classDef format fill:#FFA500
    classDef compression fill:#FFFF99
    classDef pytorch fill:#E6E6FA

    %% Legend
    subgraph Legend
        L1[Write Interface]:::write
        L2[Read Interface]:::read
        L3[Storage Component]:::storage
        L4[Format Handler]:::format
        L5[Compression Module]:::compression
        L6[PyTorch Integration]:::pytorch
    end

    %% User Interface Layer
    subgraph "User Interface Layer"
        ThunderDB[ThunderDB]:::write
        ThunderReader[ThunderReader]:::read
        ThunderDataset[ThunderDataset]:::pytorch
    end

    %% Core Processing Layer
    subgraph "Core Processing Layer"
        FormatDetection[Format Detection & Conversion]:::format
        CompressionPipeline[Compression Pipeline]:::compression
        TypeMapping[Type-Format Mapping]:::format
    end

    %% Storage Layer
    subgraph "Storage Layer"
        LMDB[LMDB Database]:::storage
        FormatHandlers[Format Handlers]:::format
        CompressionModules[Compression Modules]:::compression
    end

    %% Format Handlers Subgroup
    subgraph "Format Handlers"
        Tensor[Tensor Formats]:::format
        Media[Media Formats]:::format
        Document[Document Formats]:::format
    end

    %% Data Flow
    ThunderDB --> FormatDetection
    FormatDetection --> CompressionPipeline
    CompressionPipeline --> LMDB
    LMDB --> CompressionModules
    CompressionModules --> FormatHandlers
    FormatHandlers --> ThunderReader
    ThunderReader --> ThunderDataset

    %% Component Relationships
    TypeMapping --> FormatDetection
    FormatHandlers --> FormatDetection
    CompressionModules --> CompressionPipeline

    %% Click Events
    click ThunderDB "https://github.com/JJGO/thunderpack/blob/main/thunderpack/thunder.py"
    click ThunderReader "https://github.com/JJGO/thunderpack/blob/main/thunderpack/thunder.py"
    click ThunderDataset "https://github.com/JJGO/thunderpack/blob/main/thunderpack/torch.py"
    click FormatHandlers "https://github.com/JJGO/thunderpack/blob/main/thunderpack/formats.py"
    click CompressionModules "https://github.com/JJGO/thunderpack/blob/main/thunderpack/compression.py"
    click TypeMapping "https://github.com/JJGO/thunderpack/blob/main/thunderpack/supported.py"
    click FormatDetection "https://github.com/JJGO/thunderpack/blob/main/thunderpack/common.py"
```
