How msg555/subsetter works
Project Type and Purpose
• This is an open-source Python CLI utility (“subsetter”) for extracting consistent, relationally-correct subsets of a production database.
• It reads a user’s YAML config, generates a sampling “plan” of foreign-key-aware queries, then executes those queries against a source database and writes the resulting rows into a target (DB or JSON) while applying optional filters, compaction, merging, or multiplicity transformations.
High-Level File Structure Analysis
• Top-level files:
• Dockerfile, Makefile, CI workflows, pyproject.toml / setup.cfg — standard build, CI, packaging.
• subsetter/ — the application package.
• tests/ — integration and unit tests under various data fixtures.
• Inside subsetter/:
• __main__.py — CLI entry point.
• config_model.py, plan_model.py — Pydantic (or dataclass) models for config and plan.
• planner.py, solver.py — logic to generate sampling plans and resolve dependencies.
• sampler.py — logic to execute the plan, stream data between source and target, handle temporary tables.
• filters.py — built-in and plugin filters (anonymization via faker).
• common.py, metadata.py — shared utilities (e.g. database introspection, foreign-key metadata).
• _version.py — version info.
Key Architectural Patterns
• Modular, layered CLI design:
a. CLI Dispatcher →
b. Config Parser →
c. Plan Generator (“planner” + “solver”) →
d. Plan Executor (“sampler” + “filters”) →
e. Data I/O (DB connectors)
• Plugin and extension points for custom filters.
• Separation of concerns: config, plan modelling, planning, execution, and transformations.
Main Components
1. CLI / Entry Point
• subsetter/__main__.py
• Parses command-line arguments (plan, sample, subset) and loads config.
2. Configuration Module
• config_model.py
• Loads and validates subsetter.yaml via PyYAML + Pydantic.
3. Metadata & Introspection
• metadata.py, common.py
• Connects to source DB, reads schema/foreign-key graph.
4. Planner & Solver
• planner.py (builds a raw plan from “targets” in config).
• solver.py (orders tables, detects dependencies, applies single-query rule).
• plan_model.py (serializes plan as YAML AST).
5. Sampler / Executor
• sampler.py
• Phases: create tables (optional), truncate/merge, materialize temp tables, copy data.
• Interfaces with source and target DBs.
• Applies filters.py transformations, compaction, multiplicity.
6. Filters / Transformations
• filters.py + plugin API
• Faker-based PII anonymization, identifier compaction, merge logic.
7. External Systems
• Source Database (Postgres, MySQL, etc.)
• Target Database or JSON files
Component Interactions and Data Flow
• Filters plug into Sampler between read and write.