# databricks/lilac architecture

> Architecture diagram of the GitHub repository databricks/lilac, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databricks/lilac
- Repository: https://github.com/databricks/lilac

## Overview

##### Project Type & Purpose
- Lilac is a full-stack tool designed for interactive exploration, curation, and quality control of datasets used with large language models (LLMs). It offers both a web-based UI (built with Svelte under the “web/blueprint” directory) and a Python API/CLI (located mainly in the “lilac” directory) to load, process, annotate, and search datasets.
- It supports multiple data operations – loading datasets from various data sources (HuggingFace, CSV, JSON, SQLite, etc.), computing signals (PII detection, language detection, near duplicate detection), clustering, embedding generation, and conceptual search. Additionally, there is integration to offload heavy computations to an external “Lilac Garden”.

##### Analyzing the File Structure for Architectural Components
- Backend Components:
- The “lilac” directory holds the core Python modules. It includes subdirectories and files for:
▪ Data processing (“lilac/data”): handling clustering, dataset loading, signal computations, and dataset utilities.
▪ Signals (“lilac/signals”): different modules for computing signals (e.g., PII, language detection, duplicate detection).
▪ Embeddings (“lilac/embeddings”): integration with vector stores, embedding computation through various providers (OpenAI, HuggingFace, Cohere, etc.).
▪ Concepts (“lilac/concepts”): modules to support conceptual search functionality.
▪ Routers and API endpoints (e.g., “lilac/router_*”) – these are likely integrated into the server side to expose REST/HTTP endpoints.
▪ The “server.py” file which starts and serves the API – central to the interactive web server.
- Deployment and Integration:
▪ “lilac_deployer”, Dockerfiles (Dockerfile; lilac/hf_docker), and CI/CD configurations (GitHub workflows, cloudbuild.yml, etc.) indicate containerization and automated deployment.
▪ Scripts in the “scripts” directory provide support for testing, linting, building docs, and deployments.
- Testing & Integration:
▪ “integration_tests”, “pytest.ini”, and various test files across directories show a strong testing strategy.

- Frontend Components:
- “web/blueprint” is a self-contained Svelte app that offers an interactive UI. It includes components for:
▪ Dataset exploration, labeling, clustering views, and signal computing visuals.
▪ Routes and layout files that define the client-side layout and navigation.
- “web/lib/fastapi_client” represents the client side’s tooling to interact with the API – generated TypeScript or JavaScript clients that wrap the backend’s endpoints.

- External/Offloaded Compute:

##### Key Relationships & Interactions
- The frontend (Svelte UI) communicates with the backend server through a REST/HTTP API (or FastAPI endpoints), using the TypeScript-based fastapi_client library as its client.
- The backend’s API server orchestrates data loading, computation of signals, clustering, and more. It interacts with internal modules (data, signals, embeddings, concepts) to process user requests.
- Optionally, for heavy computational tasks, the API layer communicates with the external “Lilac Garden” service.
- The deployment and CI/CD pipeline (configured via Dockerfiles, GitHub workflows, and cloudbuild.yml) manages both backend and frontend deployments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        FrontendUI["Frontend UI"]:::frontend
        ClientLib["Client Library"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        APIServer["Lilac API Server"]:::backend
        DataModule["Data Module"]:::backend
        SignalsModule["Signals Module"]:::backend
        EmbeddingsModule["Embeddings Module"]:::backend
        ConceptsModule["Concepts Module"]:::backend
    end

    %% Deployment & CI/CD Pipeline
    subgraph "Deployment & CI/CD Pipeline"
        DockerFile["Dockerfile"]:::deployment
        HFDocker["lilac/hf_docker"]:::deployment
        Cloudbuild["cloudbuild.yml"]:::deployment
        GithubWorkflows[".github/workflows"]:::deployment
    end

    %% External Service
    ExternalService["External Lilac Garden Service"]:::external

    %% Relationships
    FrontendUI -->|"REST_calls/API_requests"| ClientLib
    ClientLib -->|"Invokes"| APIServer

    APIServer -->|"Manages"| SignalsModule
    APIServer -->|"Manages"| EmbeddingsModule
    APIServer -->|"Manages"| ConceptsModule
    APIServer -->|"Manages"| DataModule

    APIServer -->|"Offloads_heavy_tasks"| ExternalService

    DockerFile ---|"Deploys"| APIServer
    DockerFile ---|"Deploys"| FrontendUI
    HFDocker ---|"Deploys"| APIServer
    HFDocker ---|"Deploys"| FrontendUI
    Cloudbuild ---|"Deploys"| APIServer
    Cloudbuild ---|"Deploys"| FrontendUI
    GithubWorkflows ---|"Deploys"| APIServer
    GithubWorkflows ---|"Deploys"| FrontendUI

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef backend fill:#d4edda,stroke:#333,stroke-width:2px;
    classDef external fill:#f8d7da,stroke:#333,stroke-width:2px;
    classDef deployment fill:#fff3cd,stroke:#333,stroke-width:2px;

    %% Click Events
    click FrontendUI "https://github.com/databricks/lilac/tree/main/web/blueprint"
    click APIServer "https://github.com/databricks/lilac/blob/main/lilac/server.py"
    click DataModule "https://github.com/databricks/lilac/tree/main/lilac/data"
    click SignalsModule "https://github.com/databricks/lilac/tree/main/lilac/signals"
    click EmbeddingsModule "https://github.com/databricks/lilac/tree/main/lilac/embeddings"
    click ConceptsModule "https://github.com/databricks/lilac/tree/main/lilac/concepts"
    click ClientLib "https://github.com/databricks/lilac/tree/main/web/lib/fastapi_client"
    click DockerFile "https://github.com/databricks/lilac/tree/main/Dockerfile"
    click HFDocker "https://github.com/databricks/lilac/tree/main/lilac/hf_docker"
    click Cloudbuild "https://github.com/databricks/lilac/blob/main/cloudbuild.yml"
    click GithubWorkflows "https://github.com/databricks/lilac/tree/main/.github/workflows"
```
