# activeloopai/deeplake architecture

> Architecture diagram of the GitHub repository activeloopai/deeplake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/activeloopai/deeplake
- Repository: https://github.com/activeloopai/deeplake

## Overview

##### Project Type and Purpose
- Deep Lake is a Python client library (“Database for AI”) that provides a serverless, columnar storage format optimized for deep‐learning applications, vector search, data versioning, and integrations with popular ML tooling. It is not a web front‐end or standalone microservice; it’s a modular Python package intended to be embedded in ML codebases.

##### High‐Level Components
a. Client API Layer
- Exposed via python/deeplake/core.py and __init__.py
- Provides Dataset creation, read/write, slicing, indexing, version control, and query interfaces
b. Storage Adapter Layer
- python/deeplake/storage.py and storage.pyi define abstract interfaces
- Implementations for local filesystem and cloud (S3, GCP, Azure, Activeloop Cloud)
c. Data Format & Compression
- python/deeplake/formats.py handles columnar encoding, native compression (e.g., JPEG/PNG for images, MP4 for video)
- Maintains metadata in schemas.py for lazy NumPy-like indexing
d. Ingestion Pipelines
- python/deeplake/ingestion/* contains modules for COCO, custom format ingestion, streaming data into datasets
e. Integrations & Extensions
- python/deeplake/integrations/* includes adapters for Labelbox, MMDetection, MMSegmentation, LangChain, LlamaIndex, Weights & Biases, etc.
- Each integration module converts external data formats or hooks into external SDKs
f. TensorFlow & PyTorch Dataloaders
- _tensorflow.py and _torch.py provide Dataset wrappers for seamless model training
g. Testing & CI
- .github/langchain-tests runs compatibility tests with LangChain and other vector-store use cases
- Workflows defined in .github/workflows/*.yaml ensure linting, unit tests, and release automation

##### External Services and Tools
- Cloud storage backends (S3, GCP, Azure, Activeloop Cloud)
- Deep Lake Visualizer App (external web UI for dataset inspection)
- Third-party ML frameworks (PyTorch, TensorFlow)
- LLM and vector-store frameworks (LangChain, LlamaIndex)
- Data lineage tools (Weights & Biases)

##### Relationships & Data Flow
1. User code imports the Client API → instantiates or opens a Dataset
2. Client API delegates to Storage Adapter → reads/writes chunked arrays
3. Data Format module compresses/decompresses raw data on demand (lazy loading)
4. Ingestion modules parse external datasets (e.g., COCO) and write into Deep Lake format
5. Integrations expose the Dataset as a VectorStore for LangChain or as DataLoader for PyTorch/TensorFlow
6. External services (cloud, W&B, Visualizer) are invoked via integration adapters

##### Architectural Patterns & Principles
- Modular plugin architecture: each integration or ingestion module lives in its own subpackage
- Dependency inversion: abstract storage interfaces with multiple backend implementations
- Lazy evaluation & streaming: data loaded on access, not upfront
- Separation of concerns: core logic vs. format encoding vs. external adapters

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Modules
    subgraph "Core Modules"
        style Core fill:#D6E9F8,stroke:#31708F,stroke-width:2px
        Init["__init__.py"]:::core
        DatasetManager["Dataset Manager\n(core.py)"]:::core
    end

    %% Storage Layer
    subgraph "Storage Layer"
        style Storage fill:#DFF0D8,stroke:#3C763D,stroke-width:2px
        StorageIface["Storage Interface\n(storage.py, storage.pyi)"]:::storage
        LocalFS["LocalFS Adapter"]:::storage
        S3Adapter["S3 Adapter"]:::storage
        GCPAdapter["GCP Adapter"]:::storage
        AzureAdapter["Azure Adapter"]:::storage
        CloudAdapter["Activeloop Cloud Adapter"]:::storage
        StorageIface --> LocalFS
        StorageIface --> S3Adapter
        StorageIface --> GCPAdapter
        StorageIface --> AzureAdapter
        StorageIface --> CloudAdapter
    end

    %% Format Engine
    subgraph "Format Engine"
        style Format fill:#FCF8E3,stroke:#8A6D3B,stroke-width:2px
        FormatsPY["formats.py, formats.pyi"]:::format
        SchemasPY["schemas.py, schemas.pyi"]:::format
    end

    %% Ingestion Pipelines
    subgraph "Ingestion Pipelines"
        style Ingestion fill:#F2DEDE,stroke:#A94442,stroke-width:2px
        COCOIngestor["COCO Ingestor Module\n(from_coco.py,\ningest_coco.py,\nexceptions.py)"]:::ingestion
    end

    %% Integrations
    subgraph "Integrations & Extensions"
        style Integrations fill:#F5E6FF,stroke:#6E5494,stroke-width:2px
        LabelboxAdapter["Labelbox Adapter"]:::integration
        MMCommon["MM Generic Hooks"]:::integration
        MMDAdapter["MMDetection Adapter"]:::integration
        MMSAdapter["MMSegmentation Adapter"]:::integration
    end

    %% ML Framework Interfaces
    subgraph "ML Framework Interfaces"
        style MLFramework fill:#E8F8FF,stroke:#1F618D,stroke-width:2px
        TorchLoader["PyTorch DataLoader\n(_torch.py)"]:::ml
        TFLoader["TensorFlow DataLoader\n(_tensorflow.py)"]:::ml
    end

    %% External Services
    subgraph "External Services"
        style External fill:#ECECEC,stroke:#7F8C8D,stroke-width:2px
        CloudStorage["Cloud Storage Services\n(S3, GCP, Azure,\nActiveloop Cloud)"]:::external
        DeepLakeApp["Deep Lake Visualizer App"]:::external
        PyTorch["PyTorch"]:::external
        TensorFlow["TensorFlow"]:::external
        LangChain["LangChain"]:::external
        LlamaIndex["LlamaIndex"]:::external
        WandB["Weights & Biases"]:::external
    end

    %% Connections
    Init --> DatasetManager
    DatasetManager --> StorageIface
    StorageIface --> CloudStorage
    DatasetManager --> FormatsPY
    DatasetManager --> SchemasPY
    DatasetManager --> COCOIngestor
    DatasetManager --> LabelboxAdapter
    DatasetManager --> MMCommon
    DatasetManager --> MMDAdapter
    DatasetManager --> MMSAdapter
    DatasetManager --> TorchLoader
    DatasetManager --> TFLoader

    LabelboxAdapter --> LangChain
    LabelboxAdapter --> WandB
    MMCommon --> PyTorch
    MMCommon --> TensorFlow
    MMDAdapter --> PyTorch
    MMSAdapter --> PyTorch

    TorchLoader --> PyTorch
    TFLoader --> TensorFlow
    LangChain --> DeepLakeApp
    LlamaIndex --> DeepLakeApp

    %% Click Events
    click Init "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/__init__.py"
    click DatasetManager "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/core.py"
    click StorageIface "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/storage.py"
    click StorageIface "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/storage.pyi"
    click FormatsPY "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/formats.py"
    click FormatsPY "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/formats.pyi"
    click SchemasPY "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/schemas.py"
    click SchemasPY "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/schemas.pyi"
    click COCOIngestor "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/ingestion/coco/from_coco.py"
    click COCOIngestor "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/ingestion/coco/ingest_coco.py"
    click COCOIngestor "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/ingestion/coco/exceptions.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/labelbox_.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/converters.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/labelbox_utils.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/deeplake_utils.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/labelbox_converter.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/labelbox_metadata_utils.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/labelbox_azure_utils.py"
    click LabelboxAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/labelbox_debug.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/mm_common.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/mm_runners.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/get_indexes.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/ipc.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/upcast_array.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/worker_init_fn.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/exceptions.py"
    click MMCommon "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/warnings.py"
    click MMDAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmdet/mmdet_.py"
    click MMDAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmdet/mmdet_dataset_.py"
    click MMDAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmdet/mmdet_utils_.py"
    click MMDAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmdet/test_.py"
    click MMSAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmseg/mmseg_.py"
    click MMSAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmseg/mmseg_dataset_.py"
    click MMSAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmseg/compose_transform_.py"
    click MMSAdapter "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmseg/test_.py"
    click TorchLoader "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/_torch.py"
    click TFLoader "https://github.com/activeloopai/deeplake/blob/main/python/deeplake/_tensorflow.py"

    %% Styles
    classDef core fill:#D6E9F8,stroke:#31708F,color:#31708F;
    classDef storage fill:#DFF0D8,stroke:#3C763D,color:#3C763D;
    classDef format fill:#FCF8E3,stroke:#8A6D3B,color:#8A6D3B;
    classDef ingestion fill:#F2DEDE,stroke:#A94442,color:#A94442;
    classDef integration fill:#F5E6FF,stroke:#6E5494,color:#6E5494;
    classDef ml fill:#E8F8FF,stroke:#1F618D,color:#1F618D;
    classDef external fill:#ECECEC,stroke:#7F8C8D,color:#7F8C8D;
```
