# darklang/dark architecture

> Architecture diagram of the GitHub repository darklang/dark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darklang/dark
- Repository: https://github.com/darklang/dark

## Overview

##### Project type and purpose
- This repo (“dark-next”) is a full-stack, multi-component platform:
- A new programming language & interpreter (“Darklang”)
- A CLI toolchain for local and prod execution
- A VS Code extension (LSP client plus bundled server)
- A suite of backend microservices (web server, queue worker, cron checker, etc.)
- Infrastructure as code (Terraform) and container definitions for cloud deployment
- Its goal is to let users author, test, and deploy Darklang backends/CLIs locally or in the cloud.

##### Analysis of the file structure
- Top-level domains:
- /backend: F#/.NET solution housing language interpreter, built-in libs, service runtime, service plumbing, and microservices (BwdServer, QueueWorker, CronChecker, ProdExec, etc.)
- /vscode-extension: VS Code client (TypeScript) plus language-configuration and static assets
- /tree-sitter-darklang: Rust/JS grammar for syntax parsing (used by LSP server)
- /packages: Darklang standard libraries and language-server scripts in Darklang DSL
- /canvases & /user-code: sample or user-authored Darklang programs (the “app code”)
- /containers & /scripts: Dockerfiles and shell scripts to build/run/test all components
- /tf: Terraform modules defining cloud infra (GKE, CloudRun, PubSub, CloudStorage, IAM, VPC, etc.)
- Notice the use of layered service-libs under backend/src:
LibParser → LibExecution → LibCloud → LibService → microservices
- CI/CD configured under .circleci and scripts/deployment; dev containers under .devcontainer.

##### Key points from README
- Darklang = language + editor + infra to build backends/CLIs
- monorepo with source-available license, actively evolving
- Not yet production ready but designed to deploy via Terraform to GCP.

##### Components
a. Main components
1. Developer workstation:
- CLI tool (backend/src/Cli, ProdExec, LocalExec)
- Darklang REPL & build scripts
- VS Code extension (client ↔ LSP server)
- Tree-sitter parser
- Local Yugabyte/Postgres emulator
2. Cloud backend services (containerized):
- BwdServer (web API, routing, authentication)
- QueueWorker (async job processor)
- CronChecker (scheduled tasks)
- ProdExec (production execution engine for user code)
3. Shared runtime libs:
- LibParser (parsing AST)
- LibExecution (interpreter + type checker)
- LibCloud (ORM, migrations, queue, stats)
- LibService (HTTP middleware, telemetry, config)
4. Data stores & external services:
- YugabyteDB (multi-tenant DB)
- GCP Pub/Sub (event queue)
- GCP Cloud Storage (traces, artifacts)
- Rollbar/Honeycomb (telemetry & logging)
5. Infrastructure layer:
- Terraform modules (tf/*.tf)
- GKE / CloudRun / IAM / VPC / Secrets / Domains

b. Relationships & data flows
- VS Code extension → LSP server (spawned via CLI or backend service) → Tree-sitter + LibParser + LibExecution for diagnostics/completions
- Developer CLI → LocalExec for local testing → connects to local DB emulator
- In production: CLI or web API calls → BwdServer → LibCloud → DB / PubSub / Storage
- Asynchronous tasks: BwdServer enqueues messages → Pub/Sub → QueueWorker pulls → invokes LibExecution / ProdExec → writes back to DB/Storage
- CronChecker on schedule triggers jobs via Pub/Sub
- ProdExec container serves user HTTP handlers via Kestrel

c. Architectural patterns & principles
- Layered modular libs (Parser → Execution → Cloud → Service)
- Microservices for separation of concerns (web, jobs, scheduler)
- Shared code via package references (Paket)
- Infrastructure as code (Terraform) + containerization (Docker)
- Local emulation vs. cloud parity

d. Key technologies & frameworks
- F#/.NET 8, Paket for .NET dependency management
- Docker, Kubernetes / GCP Cloud Run
- Terraform (tf modules)
- TypeScript for VS Code extension
- Rust + Node.js for Tree-sitter grammar
- GCP Pub/Sub, Cloud Storage, IAM, VPC
- Honeycomb, Rollbar, OpenTelemetry

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Environment
    subgraph "Developer Environment"
        direction TB
        CLI["CLI Toolchain"]:::dev
        VSCode["VS Code Extension"]:::dev
        TreeSitterParser["Tree-sitter Parser"]:::dev
        LocalDB["Local DB Emulator"]:::dev
    end

    %% Language & Runtime Libraries
    subgraph "Language & Runtime Core"
        direction TB
        LSPServer["LSP Server Integration"]:::runtime
        LibParser["LibParser"]:::runtime
        LibExecution["LibExecution"]:::runtime
        LibCloud["LibCloud"]:::runtime
        LibCloudExec["LibCloudExecution"]:::runtime
        LibService["LibService"]:::runtime
    end

    %% Cloud Services
    subgraph "Cloud Services & Microservices"
        direction TB
        BwdServer["BwdServer (Web API)"]:::cloud
        QueueWorker["QueueWorker"]:::cloud
        CronChecker["CronChecker"]:::cloud
        ProdExec["ProdExec (Execution Engine)"]:::cloud
    end

    %% Infrastructure
    subgraph "Infrastructure"
        direction TB
        Terraform["Terraform Modules"]:::infra
        Registry["Container Registry"]:::infra
        K8s["GKE / Cloud Run"]:::infra
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        Yugabyte["YugabyteDB"]:::external
        PubSub["GCP Pub/Sub"]:::external
        Storage["Cloud Storage"]:::external
        Telemetry["Rollbar/Honeycomb"]:::external
    end

    %% Developer flows
    VSCode -->|"LSP (gRPC)"| LSPServer
    LSPServer -->|"uses"| TreeSitterParser
    TreeSitterParser -->|"parsing"| LibParser
    LibParser -->|"AST & types"| LibExecution

    CLI -->|"runs"| LocalExec["Local Execution"]:::dev
    LocalExec -->|"SQL"| LocalDB

    %% Runtime library flows
    LibExecution -->|"exec APIs"| LibCloud
    LibCloud -->|"queue,migrations"| PubSub
    LibCloud -->|"DB ORM"| Yugabyte
    LibCloudExec -->|"cloud helpers"| LibCloud
    LibService -->|"middleware"| LibExecution

    %% Cloud service flows
    BwdServer -->|"HTTP & telemetry"| LibService
    BwdServer -->|"DB queries"| Yugabyte
    BwdServer -->|"enqueue jobs"| PubSub
    QueueWorker -->|"pull jobs"| PubSub
    QueueWorker -->|"exec jobs"| LibExecution
    QueueWorker -->|"store results"| Storage
    CronChecker -->|"schedule"| PubSub
    ProdExec -->|"host user code"| LibService
    ProdExec -->|"data ops"| LibCloud

    %% Infra deployments
    Terraform -->|provisions| K8s
    Registry -->|images| K8s
    K8s -->|deploys| BwdServer
    K8s -->|deploys| QueueWorker
    K8s -->|deploys| CronChecker
    K8s -->|deploys| ProdExec

    %% External telemetry
    LibService -->|"emit"| Telemetry
    BwdServer -->|"emit"| Telemetry
    QueueWorker -->|"emit"| Telemetry
    CronChecker -->|"emit"| Telemetry
    ProdExec -->|"emit"| Telemetry

    %% Click Events
    click CLI "https://github.com/darklang/dark/blob/main/backend/src/Cli/Cli.fs"
    click LocalExec "https://github.com/darklang/dark/blob/main/backend/src/LocalExec/LocalExec.fs"
    click ProdExec "https://github.com/darklang/dark/blob/main/backend/src/ProdExec/ProdExec.fs"
    click VSCode "https://github.com/darklang/dark/blob/main/vscode-extension/client/src/extension.ts"
    click TreeSitterParser "https://github.com/darklang/dark/blob/main/tree-sitter-darklang/grammar.js"
    click LSPServer "https://github.com/darklang/dark/blob/main/backend/src/LibTreeSitter/TreeSitter.Darklang.fs"
    click LibParser "https://github.com/darklang/dark/blob/main/backend/src/LibParser/Parser.fs"
    click LibExecution "https://github.com/darklang/dark/blob/main/backend/src/LibExecution/Interpreter.fs"
    click LibCloud "https://github.com/darklang/dark/blob/main/backend/src/LibCloud/Queue.fs"
    click LibCloudExec "https://github.com/darklang/dark/blob/main/backend/src/LibCloudExecution/CloudExecution.fs"
    click LibService "https://github.com/darklang/dark/blob/main/backend/src/LibService/Kestrel.fs"
    click BwdServer "https://github.com/darklang/dark/blob/main/backend/src/BwdServer/Server.fs"
    click QueueWorker "https://github.com/darklang/dark/blob/main/backend/src/QueueWorker/QueueWorker.fs"
    click CronChecker "https://github.com/darklang/dark/blob/main/backend/src/CronChecker/CronChecker.fs"
    click Terraform "https://github.com/darklang/dark/blob/main/tf/main.tf"

    %% Styles
    classDef dev fill:#cfe2f3,stroke:#0366d6,color:#0366d6
    classDef runtime fill:#d0e9c6,stroke:#3c763d,color:#3c763d
    classDef cloud fill:#fce8b2,stroke:#f0ad4e,color:#8a6d3b
    classDef infra fill:#f4f4f4,stroke:#6c757d,color:#6c757d
    classDef external fill:#e2e2e2,stroke:#444444,color:#444444
```
