How databricks/lilac works
Project Type & Purpose
• Lilac is a full-stack tool designed for interactive exploration, curation, and quality control of datasets used with large language models (LLMs). It offers both a web-based UI (built with Svelte under the “web/blueprint” directory) and a Python API/CLI (located mainly in the “lilac” directory) to load, process, annotate, and search datasets.
• It supports multiple data operations – loading datasets from various data sources (HuggingFace, CSV, JSON, SQLite, etc.), computing signals (PII detection, language detection, near duplicate detection), clustering, embedding generation, and conceptual search. Additionally, there is integration to offload heavy computations to an external “Lilac Garden”.
Analyzing the File Structure for Architectural Components
• Backend Components:
• The “lilac” directory holds the core Python modules. It includes subdirectories and files for:
▪ Data processing (“lilac/data”): handling clustering, dataset loading, signal computations, and dataset utilities.
▪ Signals (“lilac/signals”): different modules for computing signals (e.g., PII, language detection, duplicate detection).
▪ Embeddings (“lilac/embeddings”): integration with vector stores, embedding computation through various providers (OpenAI, HuggingFace, Cohere, etc.).
▪ Concepts (“lilac/concepts”): modules to support conceptual search functionality.
▪ Routers and API endpoints (e.g., “lilac/router_*”) – these are likely integrated into the server side to expose REST/HTTP endpoints.
▪ The “server.py” file which starts and serves the API – central to the interactive web server.
• Deployment and Integration:
▪ “lilac_deployer”, Dockerfiles (Dockerfile; lilac/hf_docker), and CI/CD configurations (GitHub workflows, cloudbuild.yml, etc.) indicate containerization and automated deployment.
▪ Scripts in the “scripts” directory provide support for testing, linting, building docs, and deployments.
• Testing & Integration:
▪ “integration_tests”, “pytest.ini”, and various test files across directories show a strong testing strategy.
• Frontend Components:
• “web/blueprint” is a self-contained Svelte app that offers an interactive UI. It includes components for:
▪ Dataset exploration, labeling, clustering views, and signal computing visuals.
▪ Routes and layout files that define the client-side layout and navigation.
• “web/lib/fastapi_client” represents the client side’s tooling to interact with the API – generated TypeScript or JavaScript clients that wrap the backend’s endpoints.
• External/Offloaded Compute:
Key Relationships & Interactions
• The frontend (Svelte UI) communicates with the backend server through a REST/HTTP API (or FastAPI endpoints), using the TypeScript-based fastapi_client library as its client.
• The backend’s API server orchestrates data loading, computation of signals, clustering, and more. It interacts with internal modules (data, signals, embeddings, concepts) to process user requests.
• Optionally, for heavy computational tasks, the API layer communicates with the external “Lilac Garden” service.
• The deployment and CI/CD pipeline (configured via Dockerfiles, GitHub workflows, and cloudbuild.yml) manages both backend and frontend deployments.