# holoviz/datashader architecture

> Architecture diagram of the GitHub repository holoviz/datashader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/holoviz/datashader
- Repository: https://github.com/holoviz/datashader

## Overview

##### Main Components and Architecture

a. Core Pipeline Components (Central to the design):
- Projection System
- Aggregation Engine
- Transformation Pipeline

b. Data Input Layer:
- Multiple Data Library Support Components (shown in data_libraries/)
- Pandas
- Dask
- xarray
- cuDF
- Dask-cuDF

c. Glyph System (shown in glyphs/):
- Points
- Lines
- Areas
- Polygons
- Quadmesh
- Trimesh

d. Transfer Functions (shown in transfer_functions/):
- CPU Utils
- CUDA Utils

##### Component Relationships

**[Data Sources] → [Data Library Adapters] → [Core Pipeline]**
- ↓
**[Glyphs] ─────────────────────────────────→ [Projection]**
- ↓
**[Reductions] ────────────────────────────→ [Aggregation]**
- ↓
**[Transfer Functions] ───────────────────→ [Transformation]**
- ↓
- [Output Image]

##### Additional Elements

- A "Testing Framework" component showing integration with benchmarks
- Optional GPU acceleration paths
- Parallel processing capabilities
- Integration points with external visualization libraries

##### Architectural Patterns to Highlight

- Pipeline Pattern (core processing)
- Strategy Pattern (different data libraries)
- Factory Pattern (glyph creation)
- Observer Pattern (for interactive visualizations)

##### Implementation Details

- Python version compatibility (3.9-3.12)
- Main dependencies
- GPU vs CPU execution paths
- Integration with visualization tools

This design should clearly communicate both the simplicity of the basic pipeline and the sophistication of the implementation that allows it to handle large-scale data efficiently.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Data_Input_Layer
        DS[Data Sources]
        DLA[Data Library Adapters]:::adapters
    end

    subgraph Library_Support
        direction TB
        PS["Pandas Support"]:::lib
        DKS["Dask Support"]:::lib
        XAS["XArray Support"]:::lib
        CDS["cuDF Support"]:::lib
        DCDS["Dask-cuDF Support"]:::lib
    end

    subgraph Core_Pipeline
        direction TB
        CP["Core Pipeline"]:::core
        CE["Core Engine"]:::core
        PS1["Projection System"]:::core
        AE["Aggregation Engine"]:::core
        TP["Transformation Pipeline"]:::core
    end

    subgraph Glyph_System
        direction TB
        PTS["Points"]:::glyph
        LNS["Lines"]:::glyph
        ARS["Areas"]:::glyph
        POLY["Polygons"]:::glyph
        QM["Quadmesh"]:::glyph
        TM["Trimesh"]:::glyph
    end

    subgraph Transfer_Functions
        direction TB
        CPU["CPU Utils"]:::transfer
        CUDA["CUDA Utils"]:::transfer
    end

    subgraph Performance_Layer
        direction TB
        COMP["Compiler"]:::perf
        RED["Reductions"]:::perf
    end

    subgraph Additional_Components
        direction TB
        AA["Antialiasing"]:::add
        CM["Color Management"]:::add
        CO["Composite Operations"]:::add
        LM["Layout Management"]:::add
    end

    DS --> DLA
    DLA --> CP

    DLA --> |Adapts| PS
    DLA --> |Adapts| DKS
    DLA --> |Adapts| XAS
    DLA --> |Adapts| CDS
    DLA --> |Adapts| DCDS

    CP --> CE
    CE --> PS1
    PS1 --> AE
    AE --> TP

    Glyph_System --> PS1
    Performance_Layer --> AE
    Transfer_Functions --> TP
    Additional_Components --> TP

    classDef core fill:#3498db
    classDef adapters fill:#2ecc71
    classDef lib fill:#27ae60
    classDef glyph fill:#f1c40f
    classDef transfer fill:#9b59b6
    classDef perf fill:#e74c3c
    classDef add fill:#e67e22

    click CP "https://github.com/holoviz/datashader/blob/main/datashader/pipeline.py"
    click CE "https://github.com/holoviz/datashader/blob/main/datashader/core.py"
    click PS "https://github.com/holoviz/datashader/blob/main/datashader/data_libraries/pandas.py"
    click DKS "https://github.com/holoviz/datashader/blob/main/datashader/data_libraries/dask.py"
    click XAS "https://github.com/holoviz/datashader/blob/main/datashader/data_libraries/xarray.py"
    click CDS "https://github.com/holoviz/datashader/blob/main/datashader/data_libraries/cudf.py"
    click DCDS "https://github.com/holoviz/datashader/blob/main/datashader/data_libraries/dask_cudf.py"
    click PTS "https://github.com/holoviz/datashader/blob/main/datashader/glyphs/points.py"
    click LNS "https://github.com/holoviz/datashader/blob/main/datashader/glyphs/line.py"
    click ARS "https://github.com/holoviz/datashader/blob/main/datashader/glyphs/area.py"
    click POLY "https://github.com/holoviz/datashader/blob/main/datashader/glyphs/polygon.py"
    click QM "https://github.com/holoviz/datashader/blob/main/datashader/glyphs/quadmesh.py"
    click TM "https://github.com/holoviz/datashader/blob/main/datashader/glyphs/trimesh.py"
    click CPU "https://github.com/holoviz/datashader/blob/main/datashader/transfer_functions/_cpu_utils.py"
    click CUDA "https://github.com/holoviz/datashader/blob/main/datashader/transfer_functions/_cuda_utils.py"
    click COMP "https://github.com/holoviz/datashader/blob/main/datashader/compiler.py"
    click RED "https://github.com/holoviz/datashader/blob/main/datashader/reductions.py"
    click AA "https://github.com/holoviz/datashader/blob/main/datashader/antialias.py"
    click CM "https://github.com/holoviz/datashader/blob/main/datashader/colors.py"
    click CO "https://github.com/holoviz/datashader/blob/main/datashader/composite.py"
    click LM "https://github.com/holoviz/datashader/blob/main/datashader/layout.py"
```
