# epam/dial-xl architecture

> Architecture diagram of the GitHub repository epam/dial-xl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/epam/dial-xl
- Repository: https://github.com/epam/dial-xl

## Overview

##### Project Type and Purpose
- QuantGrid is a full-stack, monorepo platform for building and executing spreadsheet-style data transformations at scale.
- It comprises:
- A React-based web frontend (apps/quantgrid) for interactive spreadsheet editing and visualization
- A Spring Boot “web” backend exposing REST/WebSocket APIs and a pluggable compute layer
- A Java/Scala “engine” library supporting in-process and Spark-based distributed execution, plus ANTLR DSL parsing
- A Python assistant microservice offering AI-driven hints and completions
- A Python CLI client (xl-client) for programmatic integration
- Shared libraries (frontend/libs and backend/common) for UI components, parser APIs, and security utilities

##### File Structure
- Top-level “frontend”, “backend”, and “python” directories map to three primary services.
- backend/web is the Spring Boot REST/WebSocket service; backend/engine is the core compute library; backend/parser and backend/proto support DSL parsing and API generation.
- frontend/apps/quantgrid is a React+TypeScript application that consumes REST/WS endpoints.
- frontend/libs houses reusable UI and parser components shared across apps.
- python/assistant is a standalone Flask/FastAPI-style service for AI hints.
- python/xl-client is a CLI wrapper that calls the backend APIs.
- infrastructure holds Dockerfiles and k8s manifests for containerization and deployment.

##### Key Technologies and Patterns
- Frontend: React, TypeScript, Yarn v4, Monaco code editor, WebSocket for live updates
- Backend: Java 17, Spring Boot, gRPC/protobuf (backend/proto), REST Controllers, conditional bean wiring (Dial vs local storage), cluster compute API (local vs remote)
- Engine: Java & Scala with ANTLR-generated DSL parser, compute graph compilation, rule-based optimizer, pluggable execution backends (local memory, Spark, embedding search via OpenVINO), caching
- Python assistant: Poetry/MkDocs, LLM orchestration, internal graph-based solver, Pydantic models
- Deployment: Docker, Kubernetes Helm values, NGINX proxy

##### Main Components and Interactions
a. Frontend UI (React app)
- Calls: REST API (HTTP), subscription to WebSocket endpoints for progress & hints
b. API Gateway / NGINX
- Routes HTTP/WebSocket to Spring Boot backend
c. Spring Boot “web” service
- Controllers (Calculate, Compile, ParseSheet, SimilaritySearch)
- Services: ComputeService (LocalComputeService & RemoteComputeService via ClusterApi), ParserService, ProjectManager, DialLockService, HeartbeatService
- Invokes engine library (in-process) or delegates to remote Spark cluster via HTTP/gRPC
- Reads/writes project files via Dial storage abstraction
d. Compute Layer
- Engine (Java/Scala) compiles DSL -> logical/physical plan -> execution
- Execution backends:
- LocalCompute: direct in-JVM execution using engine.node.plan.local
- SparkCompute: spark cluster jobs via engine.node.plan.spark
- Embedding search module leveraging OpenVINO and vector search
e. Parser Layer
- ANTLR grammars in backend/parser and frontend/generateGrammar.js
- Shared DSL AST APIs in backend/common and frontend/libs/parser
f. Python Assistant Service
- Exposes WebSocket/REST for AI hints
- Orchestrates LLM calls, maintains conversation graph
g. Python xl-client
- CLI that invokes backend web endpoints for compile, calculate, export
h. Infrastructure
- Docker images per service, k8s manifests for web, engine, assistant
- Shared file storage for projects, optional S3 integration

## Diagram (Mermaid)

```mermaid
graph TD
    %% Frontend Layer
    subgraph "Frontend (Blue)"
        direction TB
        Browser["React SPA\n(apps/quantgrid)"]:::frontend
        subgraph "Shared UI Libraries"
            direction TB
            Canvas["Canvas Spreadsheet Lib"]:::frontend
            CodeEditor["Code Editor Lib"]:::frontend
            CommonUILib["Common UI Utils"]:::frontend
            ParserLib["Frontend Parser API"]:::frontend
        end
        GramGen["ANTLR Grammar Generation"]:::frontend
    end

    %% API Gateway / Proxy
    NGINX["NGINX Proxy\n(infrastructure)"]:::infra

    %% Backend Layer
    subgraph "Backend/API (Green)"
        direction TB
        AppStarter["ApplicationStarter"]:::backend
        Controllers["Controllers\n(Calculate, Compile, ParseSheet, SimilaritySearch)"]:::backend
        ComputeAdapter["ComputeService Adapter\n(LocalComputeService & RemoteComputeService)"]:::backend
        ProjectService["Project & Dial Storage Services"]:::backend
        DialLock["DialLockService"]:::backend
        Heartbeat["HeartbeatService & DialCleanupService"]:::backend
        Proto["gRPC/Protobuf Definitions"]:::backend
        subgraph "Common Utilities & Security"
            direction TB
            CommonSec["Security Utilities"]:::backend
            CommonUtil["Utility Functions"]:::backend
            CommonType["Shared Types"]:::backend
        end
        subgraph "Parser Layer (Backend)"
            direction TB
            Grammars["ANTLR Grammars"]:::backend
            ParserAPI["Parser API & AST Models"]:::backend
        end
    end

    %% Compute/Engine Layer
    subgraph "Compute/Engine (Orange)"
        direction TB
        EngineAPI["Core Engine API & Graph Compilation"]:::compute
        Compiler["Compiler & Rule-Based Optimizer"]:::compute
        LocalPlan["Local Execution Backend"]:::compute
        SparkPlan["Spark Execution Backend"]:::compute
        Embeddings["Embedding Search Module"]:::compute
        Cache["Caching"]:::compute
        StoreDial["Dial Storage Integration"]:::compute
    end

    %% AI Assistant Layer
    subgraph "AI Assistant (Purple)"
        direction TB
        AssistantApp["Assistant Service EntryPoint"]:::ai
        AssistantConfig["Configuration & LLM Models"]:::ai
        AssistantGraph["Conversation & Graph Solver"]:::ai
        AssistantModels["Pydantic Models & Formatting"]:::ai
        AssistantRuntime["Runtime & Export Engine"]:::ai
    end

    %% CLI Layer
    subgraph "CLI/External Tools (Gray)"
        direction TB
        CLITool["xl-client CLI"]:::cli
        CLICreds["Credentials & Client"]:::cli
        CLITests["CLI Tests"]:::cli
    end

    %% Infrastructure Layer
    subgraph "Infrastructure (Dashed)"
        direction TB
        InfraFront["Dockerfile: frontend"]:::infra
        InfraBack["Dockerfile: backend"]:::infra
        InfraCluster["Dockerfile: backend_cluster"]:::infra
        InfraSpark["Dockerfile: backend_spark"]:::infra
        InfraPy["Dockerfile: python"]:::infra
        NginxConf["NGINX Config"]:::infra
        Helm["Kubernetes Helm Values"]:::infra
    end

    %% Connections
    Browser -->|"HTTP/WS"| NGINX
    NGINX -->|"Routes"| Controllers
    Controllers -->|"calls"| ComputeAdapter
    ComputeAdapter -->|"in-process"| EngineAPI
    ComputeAdapter -->|"cluster API"| SparkPlan
    EngineAPI --> Compiler
    Compiler --> LocalPlan
    Compiler --> SparkPlan
    EngineAPI --> Embeddings
    EngineAPI --> Cache
    EngineAPI --> StoreDial
    Controllers -->|"uses"| ProjectService
    ProjectService -->|"locks"| DialLock
    Controllers -->|"heartbeat"| Heartbeat
    Controllers -->|"parses DSL"| ParserAPI
    GramGen --> ParserLib
    Browser -->|"uses"| ParserLib
    Browser -->|"syntax highlighting"| Canvas
    Browser --> CodeEditor
    Controllers -->|"gRPC"| Proto
    Controllers -->|"auth/util"| CommonSec
    Controllers --> CommonUtil
    Controllers --> CommonType
    Controllers -->|"API"| AssistantApp
    Browser -->|"hints via WS"| AssistantApp
    CLITool -->|"HTTP"| Controllers
    CLICreds --> CLITool
    CLITests --> CLITool
    InfraFront --> NGINX
    InfraBack --> Controllers
    InfraCluster --> SparkPlan
    InfraSpark --> SparkPlan
    InfraPy --> AssistantApp
    Helm --> InfraFront
    Helm --> InfraBack
    Helm --> AssistantApp
    Helm --> SparkPlan

    %% Click Events
    click Browser "https://github.com/epam/dial-xl/tree/main/frontend/apps/quantgrid"
    click Canvas "https://github.com/epam/dial-xl/tree/main/frontend/libs/canvasSpreadsheet"
    click CodeEditor "https://github.com/epam/dial-xl/tree/main/frontend/libs/code-editor"
    click CommonUILib "https://github.com/epam/dial-xl/tree/main/frontend/libs/common"
    click ParserLib "https://github.com/epam/dial-xl/tree/main/frontend/libs/parser"
    click GramGen "https://github.com/epam/dial-xl/blob/main/frontend/generateGrammar.js"
    click AppStarter "https://github.com/epam/dial-xl/blob/main/backend/web/src/main/java/com/epam/deltix/quantgrid/ApplicationStarter.java"
    click Controllers "https://github.com/epam/dial-xl/tree/main/backend/web/src/main/java/com/epam/deltix/quantgrid/web/controller"
    click ComputeAdapter "https://github.com/epam/dial-xl/tree/main/backend/web/src/main/java/com/epam/deltix/quantgrid/web/service/compute"
    click ProjectService "https://github.com/epam/dial-xl/tree/main/backend/web/src/main/java/com/epam/deltix/quantgrid/web/service/project"
    click DialLock "https://github.com/epam/dial-xl/blob/main/backend/web/src/main/java/com/epam/deltix/quantgrid/web/service/DialLockService.java"
    click Heartbeat "https://github.com/epam/dial-xl/blob/main/backend/web/src/main/java/com/epam/deltix/quantgrid/web/service/HeartbeatService.java"
    click DialCleanup "https://github.com/epam/dial-xl/blob/main/backend/web/src/main/java/com/epam/deltix/quantgrid/web/service/DialCleanupService.java"
    click Proto "https://github.com/epam/dial-xl/blob/main/backend/proto/src/main/proto/api.proto"
    click CommonSec "https://github.com/epam/dial-xl/tree/main/backend/common/src/main/java/com/epam/deltix/quantgrid/security"
    click CommonUtil "https://github.com/epam/dial-xl/tree/main/backend/common/src/main/java/com/epam/deltix/quantgrid/util"
    click CommonType "https://github.com/epam/dial-xl/tree/main/backend/common/src/main/java/com/epam/deltix/quantgrid/type"
    click EngineAPI "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine"
    click Compiler "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine/compiler"
    click Rule "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine/rule"
    click LocalPlan "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine/node/plan/local"
    click SparkPlan "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine/node/plan/spark"
    click Embeddings "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine/embeddings"
    click Cache "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine/cache"
    click StoreDial "https://github.com/epam/dial-xl/tree/main/backend/engine/src/main/java/com/epam/deltix/quantgrid/engine/store/dial"
    click Grammars "https://github.com/epam/dial-xl/tree/main/backend/parser/src/main/antlr"
    click ParserAPI "https://github.com/epam/dial-xl/tree/main/backend/parser/src/main/java/com/epam/deltix/quantgrid/parser"
    click AssistantApp "https://github.com/epam/dial-xl/blob/main/python/assistant/quantgrid/app.py"
    click AssistantConfig "https://github.com/epam/dial-xl/tree/main/python/assistant/quantgrid/configuration"
    click AssistantGraph "https://github.com/epam/dial-xl/tree/main/python/assistant/quantgrid/graph"
    click AssistantModels "https://github.com/epam/dial-xl/tree/main/python/assistant/quantgrid/models"
    click AssistantRuntime "https://github.com/epam/dial-xl/tree/main/python/assistant/quantgrid/python/runtime"
    click CLITool "https://github.com/epam/dial-xl/blob/main/python/xl-client/dial_xl/client.py"
    click CLICreds "https://github.com/epam/dial-xl/blob/main/python/xl-client/dial_xl/credentials.py"
    click CLITests "https://github.com/epam/dial-xl/tree/main/python/xl-client/tests"
    click InfraFront "https://github.com/epam/dial-xl/tree/main/infrastructure/frontend/Dockerfile"
    click InfraBack "https://github.com/epam/dial-xl/tree/main/infrastructure/backend/Dockerfile"
    click InfraCluster "https://github.com/epam/dial-xl/tree/main/infrastructure/backend_cluster/Dockerfile"
    click InfraSpark "https://github.com/epam/dial-xl/tree/main/infrastructure/backend_spark/Dockerfile"
    click InfraPy "https://github.com/epam/dial-xl/tree/main/infrastructure/python/Dockerfile"
    click NginxConf "https://github.com/epam/dial-xl/blob/main/infrastructure/frontend/nginx/nginx.conf"
    click Helm "https://github.com/epam/dial-xl/blob/main/infrastructure/k8s/quantgrid-*.yaml"

    %% Styles
    classDef frontend fill:#E0F0FF,stroke:#007acc,color:#003366
    classDef backend fill:#E8FFE0,stroke:#28a745,color:#155724
    classDef compute fill:#FFF4E0,stroke:#ff8c00,color:#663c00
    classDef ai fill:#F8E0FF,stroke:#9932CC,color:#4B0082
    classDef cli fill:#F0F0F0,stroke:#606060,color:#202020
    classDef infra stroke:#606060,stroke-dasharray:5 5,fill:none
```
