How nvidia-nemo/datadesigner works
Purpose and public entry points
NeMo Data Designer is a Python framework and CLI for dependency-aware synthetic dataset generation, from samplers, seed data, LLMs, and validators. The installed API centers on DataDesigner and configuration builders in packages/data-designer/src/data_designer/interface/data_designer.py and packages/data-designer-config/src/data_designer/config/config_builder.py. The CLI entry point is packages/data-designer/src/data_designer/cli/main.py, supporting generation/preview plus persistent model, provider, plugin, and MCP configuration.
Package layering and contracts
The repository is a three-package Python monorepo:
• packages/data-designer-config/src/data_designer/config/data_designer_config.py defines serializable user-facing configuration, column types, seeds, model/MCP settings, scheduling, and validator parameters.
• packages/data-designer-engine/src/data_designer/engine/compiler.py compiles that declarative configuration into executable work.
• packages/data-designer/src/data_designer/interface/data_designer.py is the façade that wires config, engine, results, CLI, and integrations.
This separates stable authoring/config contracts from execution internals and presentation.
Dataset compilation and async execution
The compiler resolves columns and dependencies into an execution graph; dataset construction then schedules row-group/cell work while permitting independent columns to overlap. Core control paths are packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py, packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py, and packages/data-designer-engine/src/data_designer/engine/dataset_builders/utils/execution_graph.py. Scheduling has explicit task/resource admission, queues, completion, and concurrency policies under packages/data-designer-engine/src/data_designer/engine/dataset_builders/scheduling/, forming the primary throughput and backpressure boundary.
Column generation, inputs, and quality gates
A registry selects implementations for sampler, seed-dataset, custom, expression, embedding, image, LLM-completion, and validation columns. Key paths: packages/data-designer-engine/src/data_designer/engine/column_generators/registry.py, packages/data-designer-engine/src/data_designer/engine/column_generators/generators/llm_completion.py, and packages/data-designer-engine/src/data_designer/engine/column_generators/generators/samplers.py. Seed and managed resources enter through packages/data-designer-engine/src/data_designer/engine/resources/seed_reader.py; person/entity sampling is a separate local-data generation subsystem in packages/data-designer-engine/src/data_designer/engine/sampling_gen/generator.py. Quality controls are pluggable Python, SQL, local-callable, and remote validators in packages/data-designer-engine/src/data_designer/engine/validators/.
Model, tool, and plugin boundaries
LLM access is abstracted behind a model façade/factory and provider-specific client adapters, including Anthropic and OpenAI-compatible HTTP endpoints: packages/data-designer-engine/src/data_designer/engine/models/facade.py, packages/data-designer-engine/src/data_designer/engine/models/clients/factory.py, and packages/data-designer-engine/src/data_designer/engine/models/clients/adapters/anthropic.py. Per-model request admission controls concurrency and pressure in packages/data-designer-engine/src/data_designer/engine/models/request_admission/controller.py. MCP tools are isolated behind their own façade and registry (packages/data-designer-engine/src/data_designer/engine/mcp/facade.py), while extensibility is governed by plugin contracts/registration in packages/data-designer-config/src/data_designer/plugins/plugin.py and packages/data-designer-config/src/data_designer/plugin_manager.py.
Outputs, observability, and integrations
Execution produces preview/results and persists dataset artifacts/media through packages/data-designer-engine/src/data_designer/engine/storage/artifact_storage.py and packages/data-designer-engine/src/data_designer/engine/storage/media_storage.py. Progress and telemetry are engine concerns (packages/data-designer-engine/src/data_designer/engine/progress/tracker.py, packages/data-designer-engine/src/data_designer/engine/models/telemetry.py); OpenTelemetry integration lives in packages/data-designer/src/data_designer/integrations/opentelemetry.py. External services include configured LLM endpoints (NVIDIA Build, OpenAI, OpenRouter, Anthropic-compatible services), MCP providers, and Hugging Face dataset publishing via packages/data-designer/src/data_designer/integrations/huggingface/client.py.
Documentation and delivery
Fern is the documentation source and publishing surface (fern/versions/latest/pages/); notebooks originate in docs/notebook_source/. GitHub Actions provides CI, documentation/notebook builds, release/publishing, dependency automation, and agentic maintenance workflows under .github/workflows/.
Main entry points: dataset_builder.py Dataset builder · data_designer.py DataDesigner façade · async_scheduler.py Async scheduler · compiler.py Configuration compiler
Components 29
Authoring & API
Public configuration and user entry points.
- Configuration contracts · Serializable dataset, column, model, scheduling, and validator settings.
packages/data-designer-config/src/data_designer/config/data_designer_config.py - Config builders
packages/data-designer-config/src/data_designer/config/config_builder.py - DataDesigner façade · Wires configuration, execution, results, and integrations.
packages/data-designer/src/data_designer/interface/data_designer.py - CLI · Generation, preview, and persistent service configuration.
packages/data-designer/src/data_designer/cli/main.py - Plugin contracts
packages/data-designer-config/src/data_designer/plugins/plugin.py - Plugin manager
packages/data-designer-config/src/data_designer/plugin_manager.py
Execution Engine
Compilation, scheduling, generation, and persistence.
- Configuration compiler · Resolves declarative columns and dependencies into executable work.
packages/data-designer-engine/src/data_designer/engine/compiler.py - Execution graph · Captures column and cell work dependencies.
packages/data-designer-engine/src/data_designer/engine/dataset_builders/utils/execution_graph.py - Dataset builder · Constructs datasets from compiled row-group and cell work.
packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py - Async scheduler · Runs independent column work concurrently.
packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py - Scheduling controls · Task/resource admission, completion, concurrency, and backpressure policies.
- Column generator registry · Selects generation implementations by column type.
packages/data-designer-engine/src/data_designer/engine/column_generators/registry.py - Sampler generators
packages/data-designer-engine/src/data_designer/engine/column_generators/generators/samplers.py - LLM completion generator
packages/data-designer-engine/src/data_designer/engine/column_generators/generators/llm_completion.py - Seed resources · Loads seed data and managed resources.
packages/data-designer-engine/src/data_designer/engine/resources/seed_reader.py - Entity sampling · Generates local person and entity data.
packages/data-designer-engine/src/data_designer/engine/sampling_gen/generator.py - Validators · Pluggable Python, SQL, local, and remote validation.
- Artifact storage · Persists dataset artifacts.
packages/data-designer-engine/src/data_designer/engine/storage/artifact_storage.py - Media storage · Persists generated media.
packages/data-designer-engine/src/data_designer/engine/storage/media_storage.py - Progress tracker
packages/data-designer-engine/src/data_designer/engine/progress/tracker.py
Models & Integrations
External model, tool, plugin, and publishing boundaries.
- Model façade · Abstracts configured LLM providers.
packages/data-designer-engine/src/data_designer/engine/models/facade.py - Provider client factory · Builds provider-specific clients such as Anthropic adapters.
packages/data-designer-engine/src/data_designer/engine/models/clients/factory.py - Model request admission · Controls per-model request pressure.
packages/data-designer-engine/src/data_designer/engine/models/request_admission/controller.py - MCP façade · Isolates MCP tools behind a façade and registry.
packages/data-designer-engine/src/data_designer/engine/mcp/facade.py - OpenTelemetry integration
packages/data-designer/src/data_designer/integrations/opentelemetry.py - Hugging Face publishing
packages/data-designer/src/data_designer/integrations/huggingface/client.py
Documentation & Delivery
Documentation sources and automation.
- Fern documentation · Documentation source and publishing surface.
- Notebook sources
- GitHub Actions · Builds, releases, publishes, and automates maintenance.
Connections 29
- Config builders → Configuration contracts: builds
- CLI → DataDesigner façade: invokes
- DataDesigner façade → Configuration contracts: accepts
- DataDesigner façade → Configuration compiler: compiles
- Plugin contracts → Plugin manager: defines extensions for
- Plugin manager → DataDesigner façade: extends
- Configuration contracts → Configuration compiler: declarative input
- Configuration compiler → Execution graph: builds
- Configuration compiler → Dataset builder: creates executable plan
- Execution graph → Async scheduler: dependency order
- Dataset builder → Async scheduler: dispatches work
- Async scheduler → Scheduling controls: uses admission policies
- Async scheduler → Column generator registry: executes columns
- Column generator registry → Sampler generators: selects
- Column generator registry → LLM completion generator: selects
- Sampler generators → Seed resources: reads seeds
- Sampler generators → Entity sampling: uses local data
- LLM completion generator → Model façade: requests completions
- LLM completion generator → MCP façade: uses tools
- Model façade → Model request admission: admitted by
- Model request admission → Provider client factory: permits requests
- Dataset builder → Validators: quality checks
- Dataset builder → Artifact storage: persists results
- Dataset builder → Media storage: persists media
- Async scheduler → Progress tracker: reports progress
- DataDesigner façade → OpenTelemetry integration: exports telemetry
- Artifact storage → Hugging Face publishing: publishes datasets
- Notebook sources → Fern documentation: source material
- GitHub Actions → Fern documentation: builds and publishes