# datalens-tech/datalens architecture

> Architecture diagram of the GitHub repository datalens-tech/datalens, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datalens-tech/datalens
- Repository: https://github.com/datalens-tech/datalens

## Overview

##### Main components
- UI Component – A Single Page Application and its supporting Node.js server. It acts as the front door for users, proxying requests and handling light data postprocessing for charts.
- Backend Services – A collection of Python applications and libraries responsible for connecting to data sources, generating queries, and processing the resulting data.
- UnitedStorage (US) Service – A Node.js service that uses PostgreSQL to persist metadata and configuration. This component is essential for saving user settings and storing metadata.
- Infrastructure and Deployment – Docker Compose files (docker-compose.yml, docker-compose-dev.yml, docker-compose.zitadel.yml), Kubernetes-related deployment setups including Helm charts (in the deploy/helm folder) and Terraform scripts (in deploy/terraform). These files illustrate the project’s containerized, orchestrated deployment strategy.
- Auxiliary Components – Additional utilities such as GitHub workflows for CI/CD, changelog generating scripts, and initialization scripts (e.g., init.sh) which configure authentication (Zitadel) and other environment settings.

##### Relationships and Interactions
- The UI communicates with the Backend Services via REST APIs or proxy layers, handling chart data retrieval and user interaction.
- Backend services connect to various data sources and then forward processed data to the UI.
- The UnitedStorage service manages metadata in a PostgreSQL database. The UI and Backend may indirectly interact with metadata for dynamic configuration or data persistence.

##### Architectural Patterns and Design Principles
- Containerization: The entire system is built as a multi-container Docker application, ensuring that each service runs in its own isolated environment.
- Microservices Separation: Logical separation of concerns is maintained across UI, backend, and metadata services. You may also indicate that services can potentially scale independently.
- CI/CD and Infrastructure as Code: With GitHub workflows, Helm templates, and Terraform configurations, the project employs modern DevOps practices.
- Cloud-Native and Orchestration: Emphasize the Kubernetes and Helm deployment strategy, which is critical for production deployments on platforms like Yandex Cloud.
- Extensibility: The services are designed to easily integrate additional connectors and features (e.g., extending frontend visualizations by replacing Highcharts with D3.js).

##### Relevant Technologies, Frameworks, and Libraries
- Docker and Docker Compose for container orchestration.
- Kubernetes (with Helm charts and Terraform scripts) for production deployments.
- Node.js for both the UI and UnitedStorage services.
- Python for the backend components.
- PostgreSQL as the metadata database.
- Third-party libraries: Highcharts (with license considerations) and D3.js for charting.
- Zitadel for authentication (optional, shown in the Zitadel-specific docker-compose file).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Level
    User["User"]:::external

    %% Frontend Layer
    subgraph "Frontend Layer"
        UI["UI Service (Node.js SPA)"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        BE["Backend API (Python Services)"]:::backend
    end

    %% Storage Layer
    subgraph "Storage Layer"
        US["UnitedStorage Service (Node.js)"]:::storage
        DB["PostgreSQL Database"]:::storage
    end

    %% Deployment & CI/CD Layer
    subgraph "Deployment & CI/CD"
        DC["Docker Compose Files"]:::deploy
        K8sUI["K8s Deployment UI"]:::deploy
        K8sBE["K8s Deployment Backend"]:::deploy
        K8sUS["K8s Deployment US"]:::deploy
        GH["GitHub Workflows & CI/CD"]:::aux
        INIT["Init Script"]:::deploy
        UTIL["Changelog & Utility Scripts"]:::aux
    end

    %% External Dependencies
    subgraph "External Dependencies"
        CHART["Highcharts/D3.js"]:::external
        ZITA["Zitadel Authentication"]:::external
    end

    %% User Interaction Flow
    User -->|"accesses"| UI
    UI -->|"fetches/processes data"| BE
    BE -->|"returns data"| UI
    UI -->|"reads/saves metadata"| US
    BE -->|"updates metadata"| US
    US -->|"stores metadata"| DB

    %% External Dependencies Flow
    UI -->|"uses"| CHART
    UI -->|"authenticates via"| ZITA

    %% Deployment Flow Connections
    GH -->|"triggers"| DC
    GH -->|"triggers"| K8sUI
    GH -->|"triggers"| K8sBE
    GH -->|"triggers"| K8sUS
    INIT -->|"initializes"| DC
    UTIL -->|"assists in"| GH

    %% Deployment Connections to Services
    DC -->|"deploys container"| UI
    DC -->|"deploys container"| BE
    DC -->|"deploys container"| US
    K8sUI -->|"deploys"| UI
    K8sBE -->|"deploys"| BE
    K8sUS -->|"deploys"| US

    %% Class Definitions for Colors
    classDef frontend fill:#cfe2f3,stroke:#084298,stroke-width:2px;
    classDef backend fill:#d1e7dd,stroke:#0f5132,stroke-width:2px;
    classDef storage fill:#fff3cd,stroke:#664d03,stroke-width:2px;
    classDef deploy fill:#f8d7da,stroke:#842029,stroke-width:2px;
    classDef aux fill:#e2e3e5,stroke:#41464b,stroke-width:2px;
    classDef external fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px;

    %% Click Events for UI Component Deployments
    click DC "https://github.com/datalens-tech/datalens/blob/main/docker-compose.yml"
    click DC "https://github.com/datalens-tech/datalens/blob/main/docker-compose-dev.yml"
    click DC "https://github.com/datalens-tech/datalens/blob/main/docker-compose.zitadel.yml"
    click K8sUI "https://github.com/datalens-tech/datalens/blob/main/deploy/helm/templates/app-ui.yaml"
    click K8sUI "https://github.com/datalens-tech/datalens/blob/main/deploy/terraform/k8s-app-ui.tf"
    click K8sBE "https://github.com/datalens-tech/datalens/blob/main/deploy/helm/templates/app-data.yaml"
    click K8sBE "https://github.com/datalens-tech/datalens/blob/main/deploy/terraform/k8s-app-data-api.tf"
    click K8sUS "https://github.com/datalens-tech/datalens/blob/main/deploy/helm/templates/app-us.yaml"
    click K8sUS "https://github.com/datalens-tech/datalens/blob/main/deploy/terraform/k8s-app-us.tf"
    click INIT "https://github.com/datalens-tech/datalens/blob/main/init.sh"
    click GH "https://github.com/datalens-tech/datalens/tree/main/.github/workflows"
    click UTIL "https://github.com/datalens-tech/datalens/tree/main/.github/workflows/scripts/changelog"
    click UTIL "https://github.com/datalens-tech/datalens/tree/main/scripts"
```
