# h2oai/datatable architecture

> Architecture diagram of the GitHub repository h2oai/datatable, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/h2oai/datatable
- Repository: https://github.com/h2oai/datatable

## Overview

##### Main Components

a. Core Processing Layer
- Column-oriented storage engine
- Native C++ implementation core
- Memory management system
- Multi-threading engine

b. Data Types Layer
- Numeric types (int, float)
- String types
- Date/Time types
- Categorical types
- Object types
- Void/Invalid types

c. Operations Layer
- Reading/Writing subsystem (CSV, Jay format)
- Sorting/Grouping engine
- Join operations
- Mathematical operations
- String operations
- Frame manipulations

d. Python Interface Layer
- API endpoints
- Expression system
- Frame object interface
- Options management
- Type system interface

##### Component Relationships

**[Python Interface Layer]**
- ↕
**[Operations Layer]**
- ↕
**[Data Types Layer]**
- ↕
**[Core Processing Layer]**
- ↕
**[Storage/Memory Layer]**

##### Key Design Elements

- Blue: Python interface components
- Green: Data processing operations
- Yellow: Core C++ components
- Gray: Storage and memory components
- Purple: Multi-threading components

b. Connection Types:
- Double lines: Data flow paths

##### Specific Implementation Details

Key subsystems:
**[Frame Operations]**
- CSV Reader/Writer
- Jay Format Handler
- Pandas Converter

**[Expression Engine]**
- F-Expressions
- Reducers
- Aggregators

**[Core Engine]**
- Memory Manager
- Thread Pool
- Column Store
- Row Index

##### Integration Points

Interfaces with:
- NumPy
- Pandas
- Python native types

##### Performance Features

Highlight:
- Multi-threading paths
- Memory-mapping capabilities
- Native C++ implementation areas
- Zero-copy operations

- Nested components
- Different line styles
- Clear labeling
- Proper grouping of related components

Remember to:
- Use consistent styling
- Brief descriptions for complex components
- Version compatibility requirements

This design should emphasize datatable's key features: high performance, big data support, and efficient memory usage, while clearly showing how the Python interface interacts with the native C++ core.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Legend"
        L1[Python Interface]:::python
        L2[Operations]:::operations
        L3[Core C++]:::core
        L4[Storage/Memory]:::storage
        L5[Threading]:::threading
    end

    subgraph "Python Interface Layer"
        API["API Endpoints"]:::python
        FI["Frame Interface"]:::python
        EI["Expression Interface"]:::python
        OM["Options Management"]:::python
        TSI["Type System Interface"]:::python
    end

    subgraph "Operations Layer"
        RS["Reading Subsystem"]:::operations
        WS["Writing Subsystem"]:::operations
        SE["Sorting Engine"]:::operations
        ES["Expression System"]:::operations
        SO["String Operations"]:::operations
        FO["Frame Operations"]:::operations
    end

    subgraph "Data Types Layer"
        TS["Type System"]:::core
        NT["Numeric Types"]:::core
        ST["String Types"]:::core
        DT["Date/Time Types"]:::core
        CT["Categorical Types"]:::core
        OT["Object Types"]:::core
    end

    subgraph "Core Processing Layer"
        MM["Memory Management"]:::storage
        MTE["Multi-threading Engine"]:::threading
        CS["Column Storage"]:::storage
        NC["Native C++ Core"]:::core
    end

    subgraph "Integration Layer"
        AI["Arrow Integration"]:::operations
        PI["Pandas Integration"]:::operations
        NI["NumPy Integration"]:::operations
    end

    %% Connections
    API --> FI
    API --> EI
    API --> OM
    API --> TSI

    FI --> FO
    EI --> ES
    TSI --> TS

    FO --> RS
    FO --> WS
    FO --> SE
    FO --> AI
    FO --> PI
    FO --> NI

    RS --> CS
    WS --> CS
    SE --> MTE
    ES --> NC
    SO --> ST

    TS --> NT
    TS --> ST
    TS --> DT
    TS --> CT
    TS --> OT

    NC --> MM
    NC --> MTE
    NC --> CS

    %% Styles
    classDef python fill:#3776AB,color:#fff
    classDef operations fill:#4CAF50,color:#fff
    classDef core fill:#FFD700,color:#000
    classDef storage fill:#808080,color:#fff
    classDef threading fill:#9C27B0,color:#fff

    %% Click events
    click API "https://github.com/h2oai/datatable/tree/main/src/datatable/"
    click FI "https://github.com/h2oai/datatable/blob/main/src/datatable/frame.py"
    click EI "https://github.com/h2oai/datatable/tree/main/src/datatable/expr/"
    click OM "https://github.com/h2oai/datatable/blob/main/src/datatable/options.py"
    click TSI "https://github.com/h2oai/datatable/blob/main/src/datatable/types.py"
    click RS "https://github.com/h2oai/datatable/tree/main/src/core/read/"
    click WS "https://github.com/h2oai/datatable/tree/main/src/core/write/"
    click SE "https://github.com/h2oai/datatable/tree/main/src/core/sort/"
    click ES "https://github.com/h2oai/datatable/tree/main/src/core/expr/"
    click SO "https://github.com/h2oai/datatable/tree/main/src/core/str/"
    click FO "https://github.com/h2oai/datatable/tree/main/src/core/frame/"
    click TS "https://github.com/h2oai/datatable/tree/main/src/core/types/"
    click NT "https://github.com/h2oai/datatable/blob/main/src/core/types/type_int.cc"
    click ST "https://github.com/h2oai/datatable/blob/main/src/core/types/type_string.cc"
    click DT "https://github.com/h2oai/datatable/blob/main/src/core/types/type_date.cc"
    click CT "https://github.com/h2oai/datatable/blob/main/src/core/types/type_categorical.cc"
    click OT "https://github.com/h2oai/datatable/blob/main/src/core/types/type_object.cc"
    click MM "https://github.com/h2oai/datatable/blob/main/src/core/mmm.cc"
    click MTE "https://github.com/h2oai/datatable/tree/main/src/core/parallel/"
    click CS "https://github.com/h2oai/datatable/tree/main/src/core/column/"
    click NC "https://github.com/h2oai/datatable/tree/main/src/core/"
    click AI "https://github.com/h2oai/datatable/blob/main/src/core/frame/to_arrow.cc"
    click PI "https://github.com/h2oai/datatable/blob/main/src/core/frame/to_pandas.cc"
    click NI "https://github.com/h2oai/datatable/blob/main/src/core/frame/to_numpy.cc"
```
