How opendp/smartnoise-sdk works
Project type and purpose
• This is an open-source, multi-package Python SDK for differential privacy on tabular data.
• It exposes two main products:
a. smartnoise-sql: A DP-enabled SQL query engine and API
b. smartnoise-synth: A suite of DP synthetic data generators
• Secondary packages:
• dpsdgym: Benchmarking & attack simulations
• eval: Evaluation metrics and tooling
File structure
• Top-level directories:
• sql/ – core DP SQL engine
• synth/ – core DP synthetic data library
• dpsdgym/ – experimental gym for DP attacks and benchmarking
• eval/ – evaluation framework & metrics
• datasets/ – sample tabular data & metadata
• docs/ – Sphinx documentation for the overall SDK
• .github/workflows/ – CI/CD pipeline definitions
• Each library (sql, synth, dpsdgym, eval) follows a typical Python project layout: src code, docs, tests, config (pyproject.toml / setup.py), samples/notebooks.
Key architectural patterns
• Modular design: each product is a standalone Python package
• Plugin-style extensibility:
– sql/reader/* readers for different backends (pandas, MySQL, Spark, etc.)
– sql/sql/_mechanisms/* for different DP noise mechanisms
– synth/snsynth/* synthesizer implementations (MWEM, PATE-CTGAN, etc.)
• Data flow pipelines: user → API → transform/parse → DP mechanism → back-end execution → result
Main components and interactions
A. smartnoise-sql
1. Client API (sqsq or snsql entrypoints)
2. Privacy object (epsilon, delta)
3. Connection factory → returns a Reader instance
4. Reader.execute(sql)
– Parser & AST module (sql/snsql/_ast)
– Query validator & rewriter (private_reader, private_rewriter)
– DP mechanism layer (laplace, gaussian, etc.)
– Odometer (usage tracking)
– Dispatch to a physical engine via sql/reader/* (pandas, mysql, spark...)
5. Data sources (CSV, SQL databases, Spark tables)
6. Results returned as Pandas DataFrame
B. smartnoise-synth
1. Client API (snsynth entrypoints)
2. Synthesizer factory (snsynth.base)
3. Transform pipeline (snsynth/transform/*)
4. Synthesizer implementations (snsynth/mwem, snsynth/pytorch/*, snsynth/aim, etc.)
– Each implements fit(data) and sample(n)
5. Optional sub-module: PytorchDPSynthesizer uses PyTorch neural nets
6. Outputs synthetic dataset
C. dpsdgym
1. control.py / main.py – orchestrate gym experiments
2. benchmark.py, evaluate.py – attack & defense flows
3. metrics (pmse, wasserstein, sra)
D. eval
1. CLI / API for running evaluations over datasets
2. Dataset abstraction → evaluations of metrics
3. Reporting
Important design principles to capture
• Separation of concerns: parsing, rewriting, DP noise, execution backends
• Extensibility: pluggable readers and mechanisms, synthesizer architectures
• Data abstraction: metadata drives DP enforcement (PUMS.yaml, DataFrame metadata)
• Monitoring: odometer tracks privacy budget usage
Final touches
• Docs generation pipeline (Sphinx → HTML) pointing to docs/
• Test suites under each package feeding into CI