# nvidia-nemo/datadesigner architecture

> Architecture diagram of the GitHub repository nvidia-nemo/datadesigner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia-nemo/datadesigner
- Repository: https://github.com/nvidia-nemo/datadesigner

## Overview

#### Purpose and public entry points
NeMo Data Designer is a Python framework and CLI for dependency-aware synthetic dataset generation, from samplers, seed data, LLMs, and validators. The installed API centers on `DataDesigner` and configuration builders in `packages/data-designer/src/data_designer/interface/data_designer.py` and `packages/data-designer-config/src/data_designer/config/config_builder.py`. The CLI entry point is `packages/data-designer/src/data_designer/cli/main.py`, supporting generation/preview plus persistent model, provider, plugin, and MCP configuration.

#### Package layering and contracts
The repository is a three-package Python monorepo:
- `packages/data-designer-config/src/data_designer/config/data_designer_config.py` defines serializable user-facing configuration, column types, seeds, model/MCP settings, scheduling, and validator parameters.
- `packages/data-designer-engine/src/data_designer/engine/compiler.py` compiles that declarative configuration into executable work.
- `packages/data-designer/src/data_designer/interface/data_designer.py` is the façade that wires config, engine, results, CLI, and integrations.
This separates stable authoring/config contracts from execution internals and presentation.

#### Dataset compilation and async execution
The compiler resolves columns and dependencies into an execution graph; dataset construction then schedules row-group/cell work while permitting independent columns to overlap. Core control paths are `packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py`, `packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py`, and `packages/data-designer-engine/src/data_designer/engine/dataset_builders/utils/execution_graph.py`. Scheduling has explicit task/resource admission, queues, completion, and concurrency policies under `packages/data-designer-engine/src/data_designer/engine/dataset_builders/scheduling/`, forming the primary throughput and backpressure boundary.

#### Column generation, inputs, and quality gates
A registry selects implementations for sampler, seed-dataset, custom, expression, embedding, image, LLM-completion, and validation columns. Key paths: `packages/data-designer-engine/src/data_designer/engine/column_generators/registry.py`, `packages/data-designer-engine/src/data_designer/engine/column_generators/generators/llm_completion.py`, and `packages/data-designer-engine/src/data_designer/engine/column_generators/generators/samplers.py`. Seed and managed resources enter through `packages/data-designer-engine/src/data_designer/engine/resources/seed_reader.py`; person/entity sampling is a separate local-data generation subsystem in `packages/data-designer-engine/src/data_designer/engine/sampling_gen/generator.py`. Quality controls are pluggable Python, SQL, local-callable, and remote validators in `packages/data-designer-engine/src/data_designer/engine/validators/`.

#### Model, tool, and plugin boundaries
LLM access is abstracted behind a model façade/factory and provider-specific client adapters, including Anthropic and OpenAI-compatible HTTP endpoints: `packages/data-designer-engine/src/data_designer/engine/models/facade.py`, `packages/data-designer-engine/src/data_designer/engine/models/clients/factory.py`, and `packages/data-designer-engine/src/data_designer/engine/models/clients/adapters/anthropic.py`. Per-model request admission controls concurrency and pressure in `packages/data-designer-engine/src/data_designer/engine/models/request_admission/controller.py`. MCP tools are isolated behind their own façade and registry (`packages/data-designer-engine/src/data_designer/engine/mcp/facade.py`), while extensibility is governed by plugin contracts/registration in `packages/data-designer-config/src/data_designer/plugins/plugin.py` and `packages/data-designer-config/src/data_designer/plugin_manager.py`.

#### Outputs, observability, and integrations
Execution produces preview/results and persists dataset artifacts/media through `packages/data-designer-engine/src/data_designer/engine/storage/artifact_storage.py` and `packages/data-designer-engine/src/data_designer/engine/storage/media_storage.py`. Progress and telemetry are engine concerns (`packages/data-designer-engine/src/data_designer/engine/progress/tracker.py`, `packages/data-designer-engine/src/data_designer/engine/models/telemetry.py`); OpenTelemetry integration lives in `packages/data-designer/src/data_designer/integrations/opentelemetry.py`. External services include configured LLM endpoints (NVIDIA Build, OpenAI, OpenRouter, Anthropic-compatible services), MCP providers, and Hugging Face dataset publishing via `packages/data-designer/src/data_designer/integrations/huggingface/client.py`.

#### Documentation and delivery
Fern is the documentation source and publishing surface (`fern/versions/latest/pages/`); notebooks originate in `docs/notebook_source/`. GitHub Actions provides CI, documentation/notebook builds, release/publishing, dependency automation, and agentic maintenance workflows under `.github/workflows/`.

Main entry points: [`packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py`](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py) (Dataset builder), [`packages/data-designer/src/data_designer/interface/data_designer.py`](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/interface/data_designer.py) (DataDesigner façade), [`packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py`](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py) (Async scheduler), [`packages/data-designer-engine/src/data_designer/engine/compiler.py`](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/compiler.py) (Configuration compiler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_authoring["Authoring & API"]
  node_config["Configuration contracts<br/>Python package"]
  node_config_builder["Config builders<br/>Python API<br/>[config_builder.py]"]
  node_data_designer["DataDesigner façade<br/>Python API<br/>[data_designer.py]"]
  node_cli["CLI<br/>[main.py]"]
  node_plugin_contracts["Plugin contracts<br/>Extension API<br/>[plugin.py]"]
  node_plugin_manager["Plugin manager<br/>Extension registry<br/>[plugin_manager.py]"]
end

subgraph group_engine["Execution Engine"]
  node_compiler{{"Configuration compiler<br/>[compiler.py]"}}
  node_execution_graph["Execution graph<br/>Dependency graph<br/>[execution_graph.py]"]
  node_dataset_builder["Dataset builder<br/>Orchestrator<br/>[dataset_builder.py]"]
  node_async_scheduler["Async scheduler<br/>[async_scheduler.py]"]
  node_scheduling["Scheduling controls<br/>Admission &amp; queues"]
  node_generator_registry["Column generator registry<br/>[registry.py]"]
  node_sampler_generator["Sampler generators<br/>Column generator<br/>[samplers.py]"]
  node_llm_generator["LLM completion generator<br/>Column generator<br/>[llm_completion.py]"]
  node_seed_resources["Seed resources<br/>Resource reader<br/>[seed_reader.py]"]
  node_entity_sampling["Entity sampling<br/>Local generator<br/>[generator.py]"]
  node_validators["Validators<br/>Quality gate"]
  node_artifact_storage[("Artifact storage")]
  node_media_storage[("Media storage<br/>[media_storage.py]")]
  node_progress["Progress tracker<br/>Observability<br/>[tracker.py]"]
end

subgraph group_integrations["Models &amp; Integrations"]
  node_model_facade["Model façade<br/>Model boundary<br/>[facade.py]"]
  node_model_clients["Provider client factory<br/>[factory.py]"]
  node_request_admission["Model request admission<br/>Concurrency control<br/>[controller.py]"]
  node_mcp_facade["MCP façade<br/>Tool boundary<br/>[facade.py]"]
  node_otel["OpenTelemetry integration<br/>[opentelemetry.py]"]
  node_huggingface["Hugging Face publishing<br/>Publishing integration<br/>[client.py]"]
end

subgraph group_delivery["Documentation &amp; Delivery"]
  node_docs["Fern documentation"]
  node_notebooks["Notebook sources<br/>Documentation source"]
  node_workflows["GitHub Actions<br/>CI/CD"]
end

node_config_builder -->|"builds"| node_config
node_cli -->|"invokes"| node_data_designer
node_data_designer -->|"accepts"| node_config
node_data_designer -->|"compiles"| node_compiler
node_plugin_contracts -->|"defines extensions for"| node_plugin_manager
node_plugin_manager -.->|"extends"| node_data_designer
node_config -->|"declarative input"| node_compiler
node_compiler -->|"builds"| node_execution_graph
node_compiler -->|"creates executable plan"| node_dataset_builder
node_execution_graph -->|"dependency order"| node_async_scheduler
node_dataset_builder -->|"dispatches work"| node_async_scheduler
node_async_scheduler -->|"uses admission policies"| node_scheduling
node_async_scheduler -->|"executes columns"| node_generator_registry
node_generator_registry -->|"selects"| node_sampler_generator
node_generator_registry -->|"selects"| node_llm_generator
node_sampler_generator -->|"reads seeds"| node_seed_resources
node_sampler_generator -->|"uses local data"| node_entity_sampling
node_llm_generator -->|"requests completions"| node_model_facade
node_llm_generator -.->|"uses tools"| node_mcp_facade
node_model_facade -->|"admitted by"| node_request_admission
node_request_admission -->|"permits requests"| node_model_clients
node_dataset_builder -->|"quality checks"| node_validators
node_dataset_builder -->|"persists results"| node_artifact_storage
node_dataset_builder -->|"persists media"| node_media_storage
node_async_scheduler -->|"reports progress"| node_progress
node_data_designer -.->|"exports telemetry"| node_otel
node_artifact_storage -.->|"publishes datasets"| node_huggingface
node_notebooks -->|"source material"| node_docs
node_workflows -->|"builds and publishes"| node_docs

click node_config "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/config/data_designer_config.py"
click node_config_builder "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/config/config_builder.py"
click node_data_designer "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/interface/data_designer.py"
click node_cli "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/cli/main.py"
click node_plugin_contracts "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/plugins/plugin.py"
click node_plugin_manager "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/plugin_manager.py"
click node_compiler "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/compiler.py"
click node_execution_graph "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/utils/execution_graph.py"
click node_dataset_builder "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py"
click node_async_scheduler "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py"
click node_generator_registry "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/column_generators/registry.py"
click node_sampler_generator "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/column_generators/generators/samplers.py"
click node_llm_generator "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/column_generators/generators/llm_completion.py"
click node_seed_resources "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/resources/seed_reader.py"
click node_entity_sampling "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/sampling_gen/generator.py"
click node_artifact_storage "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/storage/artifact_storage.py"
click node_media_storage "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/storage/media_storage.py"
click node_progress "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/progress/tracker.py"
click node_model_facade "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/models/facade.py"
click node_model_clients "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/models/clients/factory.py"
click node_request_admission "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/models/request_admission/controller.py"
click node_mcp_facade "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/mcp/facade.py"
click node_otel "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/integrations/opentelemetry.py"
click node_huggingface "https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/integrations/huggingface/client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_config_builder,node_data_designer,node_cli,node_plugin_contracts,node_plugin_manager toneBlue
class node_compiler,node_execution_graph,node_dataset_builder,node_async_scheduler,node_scheduling,node_generator_registry,node_sampler_generator,node_llm_generator,node_seed_resources,node_entity_sampling,node_validators,node_artifact_storage,node_media_storage,node_progress toneAmber
class node_model_facade,node_model_clients,node_request_admission,node_mcp_facade,node_otel,node_huggingface toneMint
class node_docs,node_notebooks,node_workflows toneRose
```

## Components

### Authoring & API

Public configuration and user entry points.

- [Configuration contracts](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/config/data_designer_config.py): Python package · Serializable dataset, column, model, scheduling, and validator settings. · `packages/data-designer-config/src/data_designer/config/data_designer_config.py`
- [Config builders](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/config/config_builder.py): Python API · `packages/data-designer-config/src/data_designer/config/config_builder.py`
- [DataDesigner façade](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/interface/data_designer.py): Python API · Wires configuration, execution, results, and integrations. · `packages/data-designer/src/data_designer/interface/data_designer.py`
- [CLI](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/cli/main.py): CLI entry point · Generation, preview, and persistent service configuration. · `packages/data-designer/src/data_designer/cli/main.py`
- [Plugin contracts](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/plugins/plugin.py): Extension API · `packages/data-designer-config/src/data_designer/plugins/plugin.py`
- [Plugin manager](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/src/data_designer/plugin_manager.py): Extension registry · `packages/data-designer-config/src/data_designer/plugin_manager.py`

### Execution Engine

Compilation, scheduling, generation, and persistence.

- [Configuration compiler](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/compiler.py): Compiler · Resolves declarative columns and dependencies into executable work. · `packages/data-designer-engine/src/data_designer/engine/compiler.py`
- [Execution graph](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/utils/execution_graph.py): Dependency graph · Captures column and cell work dependencies. · `packages/data-designer-engine/src/data_designer/engine/dataset_builders/utils/execution_graph.py`
- [Dataset builder](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py): Orchestrator · Constructs datasets from compiled row-group and cell work. · `packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py`
- [Async scheduler](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py): Scheduler · Runs independent column work concurrently. · `packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py`
- **Scheduling controls**: Admission & queues · Task/resource admission, completion, concurrency, and backpressure policies.
- [Column generator registry](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/column_generators/registry.py): Registry · Selects generation implementations by column type. · `packages/data-designer-engine/src/data_designer/engine/column_generators/registry.py`
- [Sampler generators](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/column_generators/generators/samplers.py): Column generator · `packages/data-designer-engine/src/data_designer/engine/column_generators/generators/samplers.py`
- [LLM completion generator](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/column_generators/generators/llm_completion.py): Column generator · `packages/data-designer-engine/src/data_designer/engine/column_generators/generators/llm_completion.py`
- [Seed resources](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/resources/seed_reader.py): Resource reader · Loads seed data and managed resources. · `packages/data-designer-engine/src/data_designer/engine/resources/seed_reader.py`
- [Entity sampling](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/sampling_gen/generator.py): Local generator · Generates local person and entity data. · `packages/data-designer-engine/src/data_designer/engine/sampling_gen/generator.py`
- **Validators**: Quality gate · Pluggable Python, SQL, local, and remote validation.
- [Artifact storage](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/storage/artifact_storage.py): Storage · Persists dataset artifacts. · `packages/data-designer-engine/src/data_designer/engine/storage/artifact_storage.py`
- [Media storage](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/storage/media_storage.py): Storage · Persists generated media. · `packages/data-designer-engine/src/data_designer/engine/storage/media_storage.py`
- [Progress tracker](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/progress/tracker.py): Observability · `packages/data-designer-engine/src/data_designer/engine/progress/tracker.py`

### Models & Integrations

External model, tool, plugin, and publishing boundaries.

- [Model façade](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/models/facade.py): Model boundary · Abstracts configured LLM providers. · `packages/data-designer-engine/src/data_designer/engine/models/facade.py`
- [Provider client factory](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/models/clients/factory.py): Client factory · Builds provider-specific clients such as Anthropic adapters. · `packages/data-designer-engine/src/data_designer/engine/models/clients/factory.py`
- [Model request admission](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/models/request_admission/controller.py): Concurrency control · Controls per-model request pressure. · `packages/data-designer-engine/src/data_designer/engine/models/request_admission/controller.py`
- [MCP façade](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/src/data_designer/engine/mcp/facade.py): Tool boundary · Isolates MCP tools behind a façade and registry. · `packages/data-designer-engine/src/data_designer/engine/mcp/facade.py`
- [OpenTelemetry integration](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/integrations/opentelemetry.py): Telemetry integration · `packages/data-designer/src/data_designer/integrations/opentelemetry.py`
- [Hugging Face publishing](https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/integrations/huggingface/client.py): Publishing integration · `packages/data-designer/src/data_designer/integrations/huggingface/client.py`

### Documentation & Delivery

Documentation sources and automation.

- **Fern documentation**: Documentation · Documentation source and publishing surface.
- **Notebook sources**: Documentation source
- **GitHub Actions**: CI/CD · Builds, releases, publishes, and automates maintenance.

## Connections

- Config builders → Configuration contracts: builds
- CLI → DataDesigner façade: invokes
- DataDesigner façade → Configuration contracts: accepts
- DataDesigner façade → Configuration compiler: compiles
- Plugin contracts → Plugin manager: defines extensions for
- Plugin manager → DataDesigner façade: extends
- Configuration contracts → Configuration compiler: declarative input
- Configuration compiler → Execution graph: builds
- Configuration compiler → Dataset builder: creates executable plan
- Execution graph → Async scheduler: dependency order
- Dataset builder → Async scheduler: dispatches work
- Async scheduler → Scheduling controls: uses admission policies
- Async scheduler → Column generator registry: executes columns
- Column generator registry → Sampler generators: selects
- Column generator registry → LLM completion generator: selects
- Sampler generators → Seed resources: reads seeds
- Sampler generators → Entity sampling: uses local data
- LLM completion generator → Model façade: requests completions
- LLM completion generator → MCP façade: uses tools
- Model façade → Model request admission: admitted by
- Model request admission → Provider client factory: permits requests
- Dataset builder → Validators: quality checks
- Dataset builder → Artifact storage: persists results
- Dataset builder → Media storage: persists media
- Async scheduler → Progress tracker: reports progress
- DataDesigner façade → OpenTelemetry integration: exports telemetry
- Artifact storage → Hugging Face publishing: publishes datasets
- Notebook sources → Fern documentation: source material
- GitHub Actions → Fern documentation: builds and publishes