How epam/dial-xl works
Project Type and Purpose
• QuantGrid is a full-stack, monorepo platform for building and executing spreadsheet-style data transformations at scale.
• It comprises:
• A React-based web frontend (apps/quantgrid) for interactive spreadsheet editing and visualization
• A Spring Boot “web” backend exposing REST/WebSocket APIs and a pluggable compute layer
• A Java/Scala “engine” library supporting in-process and Spark-based distributed execution, plus ANTLR DSL parsing
• A Python assistant microservice offering AI-driven hints and completions
• A Python CLI client (xl-client) for programmatic integration
• Shared libraries (frontend/libs and backend/common) for UI components, parser APIs, and security utilities
File Structure
• Top-level “frontend”, “backend”, and “python” directories map to three primary services.
• backend/web is the Spring Boot REST/WebSocket service; backend/engine is the core compute library; backend/parser and backend/proto support DSL parsing and API generation.
• frontend/apps/quantgrid is a React+TypeScript application that consumes REST/WS endpoints.
• frontend/libs houses reusable UI and parser components shared across apps.
• python/assistant is a standalone Flask/FastAPI-style service for AI hints.
• python/xl-client is a CLI wrapper that calls the backend APIs.
• infrastructure holds Dockerfiles and k8s manifests for containerization and deployment.
Key Technologies and Patterns
• Frontend: React, TypeScript, Yarn v4, Monaco code editor, WebSocket for live updates
• Backend: Java 17, Spring Boot, gRPC/protobuf (backend/proto), REST Controllers, conditional bean wiring (Dial vs local storage), cluster compute API (local vs remote)
• Engine: Java & Scala with ANTLR-generated DSL parser, compute graph compilation, rule-based optimizer, pluggable execution backends (local memory, Spark, embedding search via OpenVINO), caching
• Python assistant: Poetry/MkDocs, LLM orchestration, internal graph-based solver, Pydantic models
• Deployment: Docker, Kubernetes Helm values, NGINX proxy
Main Components and Interactions
a. Frontend UI (React app)
• Calls: REST API (HTTP), subscription to WebSocket endpoints for progress & hints
b. API Gateway / NGINX
• Routes HTTP/WebSocket to Spring Boot backend
c. Spring Boot “web” service
• Controllers (Calculate, Compile, ParseSheet, SimilaritySearch)
• Services: ComputeService (LocalComputeService & RemoteComputeService via ClusterApi), ParserService, ProjectManager, DialLockService, HeartbeatService
• Invokes engine library (in-process) or delegates to remote Spark cluster via HTTP/gRPC
• Reads/writes project files via Dial storage abstraction
d. Compute Layer
• Engine (Java/Scala) compiles DSL -> logical/physical plan -> execution
• Execution backends:
• LocalCompute: direct in-JVM execution using engine.node.plan.local
• SparkCompute: spark cluster jobs via engine.node.plan.spark
• Embedding search module leveraging OpenVINO and vector search
e. Parser Layer
• ANTLR grammars in backend/parser and frontend/generateGrammar.js
• Shared DSL AST APIs in backend/common and frontend/libs/parser
f. Python Assistant Service
• Exposes WebSocket/REST for AI hints
• Orchestrates LLM calls, maintains conversation graph
g. Python xl-client
• CLI that invokes backend web endpoints for compile, calculate, export
h. Infrastructure
• Docker images per service, k8s manifests for web, engine, assistant
• Shared file storage for projects, optional S3 integration