# opendp/smartnoise-sdk architecture

> Architecture diagram of the GitHub repository opendp/smartnoise-sdk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opendp/smartnoise-sdk
- Repository: https://github.com/opendp/smartnoise-sdk

## Overview

##### Project type and purpose
- This is an open-source, multi-package Python SDK for differential privacy on tabular data.
- It exposes two main products:
a. smartnoise-sql: A DP-enabled SQL query engine and API
b. smartnoise-synth: A suite of DP synthetic data generators
- Secondary packages:
- dpsdgym: Benchmarking & attack simulations
- eval: Evaluation metrics and tooling

##### File structure
- Top-level directories:
- sql/ – core DP SQL engine
- synth/ – core DP synthetic data library
- dpsdgym/ – experimental gym for DP attacks and benchmarking
- eval/ – evaluation framework & metrics
- datasets/ – sample tabular data & metadata
- docs/ – Sphinx documentation for the overall SDK
- .github/workflows/ – CI/CD pipeline definitions
- Each library (sql, synth, dpsdgym, eval) follows a typical Python project layout: src code, docs, tests, config (pyproject.toml / setup.py), samples/notebooks.

##### Key architectural patterns
- Modular design: each product is a standalone Python package
- Plugin-style extensibility:
– sql/reader/* readers for different backends (pandas, MySQL, Spark, etc.)
– sql/sql/_mechanisms/* for different DP noise mechanisms
– synth/snsynth/* synthesizer implementations (MWEM, PATE-CTGAN, etc.)
- Data flow pipelines: user → API → transform/parse → DP mechanism → back-end execution → result

##### Main components and interactions
A. smartnoise-sql
1. Client API (sqsq or snsql entrypoints)
2. Privacy object (epsilon, delta)
3. Connection factory → returns a Reader instance
4. Reader.execute(sql)
– Parser & AST module (sql/snsql/_ast)
– Query validator & rewriter (private_reader, private_rewriter)
– DP mechanism layer (laplace, gaussian, etc.)
– Odometer (usage tracking)
– Dispatch to a physical engine via sql/reader/* (pandas, mysql, spark...)
5. Data sources (CSV, SQL databases, Spark tables)
6. Results returned as Pandas DataFrame

B. smartnoise-synth
1. Client API (snsynth entrypoints)
2. Synthesizer factory (snsynth.base)
3. Transform pipeline (snsynth/transform/*)
4. Synthesizer implementations (snsynth/mwem, snsynth/pytorch/*, snsynth/aim, etc.)
– Each implements fit(data) and sample(n)
5. Optional sub-module: PytorchDPSynthesizer uses PyTorch neural nets
6. Outputs synthetic dataset

C. dpsdgym
1. control.py / main.py – orchestrate gym experiments
2. benchmark.py, evaluate.py – attack & defense flows
3. metrics (pmse, wasserstein, sra)

D. eval
1. CLI / API for running evaluations over datasets
2. Dataset abstraction → evaluations of metrics
3. Reporting

##### Important design principles to capture
- Separation of concerns: parsing, rewriting, DP noise, execution backends
- Extensibility: pluggable readers and mechanisms, synthesizer architectures
- Data abstraction: metadata drives DP enforcement (PUMS.yaml, DataFrame metadata)
- Monitoring: odometer tracks privacy budget usage

##### Final touches
- Docs generation pipeline (Sphinx → HTML) pointing to docs/
- Test suites under each package feeding into CI

## Diagram (Mermaid)

```mermaid
flowchart TB
    U["User Code"]:::external

    subgraph "smartnoise-sql"
        direction TB
        SQL_API["Client API"]:::sql
        ConnFactory["Connection Factory"]:::sql
        Privacy["Privacy Object (ε,δ)"]:::sql
        ParserAST["Parser & AST Module"]:::sql
        ReaderComp["Private Reader"]:::sql
        RewriterComp["Private Rewriter"]:::sql
        Mechanisms["DP Mechanism Layer"]:::sql
        Odometer["Odometer (Usage Tracking)"]:::sql
        Backends["Execution Backends (Readers)"]:::sql
        Grammar["SQL Parser Grammar & Support"]:::sql
    end

    DS["Data Sources (CSV, DB, Spark)"]:::external
    DF["Pandas DataFrame"]:::external

    U --> SQL_API
    SQL_API --> ConnFactory
    ConnFactory --> ParserAST
    ParserAST --> ReaderComp
    ReaderComp --> RewriterComp
    RewriterComp --> Mechanisms
    Mechanisms --> Odometer
    Odometer --> Backends
    Backends --> DS
    Backends --> DF
    ParserAST --> Grammar

    subgraph "smartnoise-synth"
        direction TB
        SynthAPI["Client API Entrypoint"]:::synth
        SynthFactory["Synthesizer Factory"]:::synth
        Transform["Transform Pipeline"]:::synth
        MWEM["MWEM Synthesizer"]:::synth
        AIM["AIM Synthesizer"]:::synth
        MST["MST Synthesizer"]:::synth
        Quail["Quail Synthesizer"]:::synth
        AggSeed["Aggregate-Seeded Synthesizer"]:::synth
        PyTorchSynth["PyTorch Synthesizer"]:::synth
        NNMods["NN Modules"]:::synth
    end

    OutputSyn["Synthetic Dataset"]:::external

    U --> SynthAPI
    SynthAPI --> SynthFactory
    SynthFactory --> Transform
    Transform --> MWEM
    Transform --> AIM
    Transform --> MST
    Transform --> Quail
    Transform --> AggSeed
    Transform --> PyTorchSynth
    PyTorchSynth --> NNMods
    NNMods --> OutputSyn

    subgraph "dpsdgym"
        direction TB
        Control["Experiment Orchestration"]:::other
        BenchFlow["Benchmark Flows"]:::other
        AttackFlow["Attack Flows"]:::other
        DPSMetrics["Metrics"]:::other
    end

    GymOut["Gym Results/Outputs"]:::external

    U --> Control
    Control --> BenchFlow
    Control --> AttackFlow
    BenchFlow --> DPSMetrics
    AttackFlow --> DPSMetrics
    DPSMetrics --> GymOut

    subgraph "eval"
        direction TB
        EvalAPI["CLI/API Entrypoint"]:::other
        Dataset["Dataset Abstraction"]:::other
        EvalLogic["Evaluation Logic"]:::other
        MetricsFW["Metrics Framework"]:::other
        Reporting["Reporting"]:::other
    end

    EvalOut["Evaluation Reports"]:::external

    U --> EvalAPI
    EvalAPI --> Dataset
    Dataset --> EvalLogic
    EvalLogic --> MetricsFW
    MetricsFW --> Reporting
    Reporting --> EvalOut

    TestsSQL["sql/tests/"]:::external
    TestsSynth["synth/tests/"]:::external
    TestsEval["eval/tests/"]:::external

    SQL_API --> TestsSQL
    SynthAPI --> TestsSynth
    EvalAPI --> TestsEval

    CI["CI/CD Workflows"]:::external
    Docs["Documentation (Sphinx)"]:::external
    Samples["Sample Data & Metadata"]:::external

    CI --- SQL_API
    CI --- SynthAPI
    CI --- Control
    CI --- EvalAPI
    Docs --- SQL_API
    Docs --- SynthAPI
    Docs --- EvalAPI
    Samples --- DS
    Samples --- Dataset

    click SQL_API "https://github.com/opendp/smartnoise-sdk/blob/main/sql/snsql/connect.py"
    click Privacy "https://github.com/opendp/smartnoise-sdk/blob/main/sql/snsql/sql/privacy.py"
    click ParserAST "https://github.com/opendp/smartnoise-sdk/tree/main/sql/snsql/_ast/"
    click ReaderComp "https://github.com/opendp/smartnoise-sdk/blob/main/sql/snsql/sql/private_reader.py"
    click RewriterComp "https://github.com/opendp/smartnoise-sdk/blob/main/sql/snsql/sql/private_rewriter.py"
    click Mechanisms "https://github.com/opendp/smartnoise-sdk/tree/main/sql/snsql/sql/_mechanisms/"
    click Odometer "https://github.com/opendp/smartnoise-sdk/blob/main/sql/snsql/sql/odometer.py"
    click Backends "https://github.com/opendp/smartnoise-sdk/tree/main/sql/snsql/sql/reader/"
    click Grammar "https://github.com/opendp/smartnoise-sdk/tree/main/sql/snsql/sql/parser/"
    click SynthAPI "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/__init__.py"
    click SynthFactory "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/base.py"
    click Transform "https://github.com/opendp/smartnoise-sdk/tree/main/synth/snsynth/transform/"
    click MWEM "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/mwem.py"
    click AIM "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/aim/aim.py"
    click MST "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/mst/mst.py"
    click Quail "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/quail.py"
    click AggSeed "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/aggregate_seeded/aggregate_seeded.py"
    click PyTorchSynth "https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/pytorch/pytorch_synthesizer.py"
    click NNMods "https://github.com/opendp/smartnoise-sdk/tree/main/synth/snsynth/pytorch/nn/"
    click Control "https://github.com/opendp/smartnoise-sdk/blob/main/dpsdgym/control.py"
    click BenchFlow "https://github.com/opendp/smartnoise-sdk/blob/main/dpsdgym/benchmark.py"
    click AttackFlow "https://github.com/opendp/smartnoise-sdk/blob/main/dpsdgym/evaluate.py"
    click DPSMetrics "https://github.com/opendp/smartnoise-sdk/tree/main/dpsdgym/metrics/"
    click EvalAPI "https://github.com/opendp/smartnoise-sdk/blob/main/eval/sneval/__init__.py"
    click Dataset "https://github.com/opendp/smartnoise-sdk/blob/main/eval/sneval/dataset.py"
    click EvalLogic "https://github.com/opendp/smartnoise-sdk/blob/main/eval/sneval/evaluate.py"
    click MetricsFW "https://github.com/opendp/smartnoise-sdk/tree/main/eval/sneval/metrics/"
    click CI "https://github.com/opendp/smartnoise-sdk/tree/main/.github/workflows/"
    click Docs "https://github.com/opendp/smartnoise-sdk/tree/main/docs/"
    click Samples "https://github.com/opendp/smartnoise-sdk/tree/main/datasets/"
    click TestsSQL "https://github.com/opendp/smartnoise-sdk/tree/main/sql/tests/"
    click TestsSynth "https://github.com/opendp/smartnoise-sdk/tree/main/synth/tests/"
    click TestsEval "https://github.com/opendp/smartnoise-sdk/tree/main/eval/tests/"

    classDef sql fill:#cce5ff,stroke:#6699cc,color:#000
    classDef synth fill:#d4f7d4,stroke:#5fbb5f,color:#000
    classDef other fill:#ffe5cc,stroke:#ff9933,color:#000
    classDef external fill:#f0f0f0,stroke:#999,color:#000,font-style:italic
```
