How dataform-co/dataform works
Project type and purpose
• This is Dataform Core: an open‑source TypeScript/Node.js “meta‑language” and CLI for defining, compiling, and running SQL transformations and dependency graphs on BigQuery. It also includes a VSCode extension and gRPC/protobuf APIs for remote orchestration.
File structure
• Top‑level Bazel config (BUILD, WORKSPACE), tsconfig.json, package.json, scripts/ and tools/ for repo-level automation.
• cli/: the CLI application, commands (build, compile, run, test, query, etc.), yargs wrapper, VM compile, dbadapters (BigQuery, generic SQL execution).
• core/: the heart of the system—Actions, Compilers, Session, Contextables, Path, WorkflowSettings. Transforms SQLX ASTs into SQL.
• common/: shared utilities—errors, promises, flags, protos definitions.
• protos/ & common/protos: .proto files describing RPC contracts (configs, compilation, execution, evaluation, profiles).
• sqlx/: custom SQLX parser/lexer.
• packages/: Bazel TS packages (@dataform/cli, @dataform/core), rollup configs, worker.ts.
• vscode/: the VSCode extension client (extension.ts) and language‑server (server.ts), grammar, schemas.
• tests/: unit, API, integration tests covering CLI->core APIs and BigQuery end‑to‑end.
• examples/: sample Dataform projects.
• scripts/ and tools/: CI/CD (cloudbuild, GitHub workflows), publishing, linting, versioning.
README
• Confirms purpose: dependency management, data‑quality testing, docs generation for BigQuery pipelines.
• Highlights two interfaces: the managed GCP environment and local CLI.
• Points to core features (tables, views, assertions, JavaScript API).