How darklang/dark works
Project type and purpose
• This repo (“dark-next”) is a full-stack, multi-component platform:
• A new programming language & interpreter (“Darklang”)
• A CLI toolchain for local and prod execution
• A VS Code extension (LSP client plus bundled server)
• A suite of backend microservices (web server, queue worker, cron checker, etc.)
• Infrastructure as code (Terraform) and container definitions for cloud deployment
• Its goal is to let users author, test, and deploy Darklang backends/CLIs locally or in the cloud.
Analysis of the file structure
• Top-level domains:
• /backend: F#/.NET solution housing language interpreter, built-in libs, service runtime, service plumbing, and microservices (BwdServer, QueueWorker, CronChecker, ProdExec, etc.)
• /vscode-extension: VS Code client (TypeScript) plus language-configuration and static assets
• /tree-sitter-darklang: Rust/JS grammar for syntax parsing (used by LSP server)
• /packages: Darklang standard libraries and language-server scripts in Darklang DSL
• /canvases & /user-code: sample or user-authored Darklang programs (the “app code”)
• /containers & /scripts: Dockerfiles and shell scripts to build/run/test all components
• /tf: Terraform modules defining cloud infra (GKE, CloudRun, PubSub, CloudStorage, IAM, VPC, etc.)
• Notice the use of layered service-libs under backend/src:
LibParser → LibExecution → LibCloud → LibService → microservices
• CI/CD configured under .circleci and scripts/deployment; dev containers under .devcontainer.
Key points from README
• Darklang = language + editor + infra to build backends/CLIs
• monorepo with source-available license, actively evolving
• Not yet production ready but designed to deploy via Terraform to GCP.
Components
a. Main components
1. Developer workstation:
• CLI tool (backend/src/Cli, ProdExec, LocalExec)
• Darklang REPL & build scripts
• VS Code extension (client ↔ LSP server)
• Tree-sitter parser
• Local Yugabyte/Postgres emulator
2. Cloud backend services (containerized):
• BwdServer (web API, routing, authentication)
• QueueWorker (async job processor)
• CronChecker (scheduled tasks)
• ProdExec (production execution engine for user code)
3. Shared runtime libs:
• LibParser (parsing AST)
• LibExecution (interpreter + type checker)
• LibCloud (ORM, migrations, queue, stats)
• LibService (HTTP middleware, telemetry, config)
4. Data stores & external services:
• YugabyteDB (multi-tenant DB)
• GCP Pub/Sub (event queue)
• GCP Cloud Storage (traces, artifacts)
• Rollbar/Honeycomb (telemetry & logging)
5. Infrastructure layer:
• Terraform modules (tf/*.tf)
• GKE / CloudRun / IAM / VPC / Secrets / Domains
b. Relationships & data flows
• VS Code extension → LSP server (spawned via CLI or backend service) → Tree-sitter + LibParser + LibExecution for diagnostics/completions
• Developer CLI → LocalExec for local testing → connects to local DB emulator
• In production: CLI or web API calls → BwdServer → LibCloud → DB / PubSub / Storage
• Asynchronous tasks: BwdServer enqueues messages → Pub/Sub → QueueWorker pulls → invokes LibExecution / ProdExec → writes back to DB/Storage
• CronChecker on schedule triggers jobs via Pub/Sub
• ProdExec container serves user HTTP handlers via Kestrel
c. Architectural patterns & principles
• Layered modular libs (Parser → Execution → Cloud → Service)
• Microservices for separation of concerns (web, jobs, scheduler)
• Shared code via package references (Paket)
• Infrastructure as code (Terraform) + containerization (Docker)
• Local emulation vs. cloud parity
d. Key technologies & frameworks
• F#/.NET 8, Paket for .NET dependency management
• Docker, Kubernetes / GCP Cloud Run
• Terraform (tf modules)
• TypeScript for VS Code extension
• Rust + Node.js for Tree-sitter grammar
• GCP Pub/Sub, Cloud Storage, IAM, VPC
• Honeycomb, Rollbar, OpenTelemetry