# msg555/subsetter architecture

> Architecture diagram of the GitHub repository msg555/subsetter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/msg555/subsetter
- Repository: https://github.com/msg555/subsetter

## Overview

##### Project Type and Purpose
- This is an open-source Python CLI utility (“subsetter”) for extracting consistent, relationally-correct subsets of a production database.
- It reads a user’s YAML config, generates a sampling “plan” of foreign-key-aware queries, then executes those queries against a source database and writes the resulting rows into a target (DB or JSON) while applying optional filters, compaction, merging, or multiplicity transformations.

##### High-Level File Structure Analysis
- Top-level files:
- Dockerfile, Makefile, CI workflows, pyproject.toml / setup.cfg — standard build, CI, packaging.
- subsetter/ — the application package.
- tests/ — integration and unit tests under various data fixtures.
- Inside subsetter/:
- __main__.py — CLI entry point.
- config_model.py, plan_model.py — Pydantic (or dataclass) models for config and plan.
- planner.py, solver.py — logic to generate sampling plans and resolve dependencies.
- sampler.py — logic to execute the plan, stream data between source and target, handle temporary tables.
- filters.py — built-in and plugin filters (anonymization via faker).
- common.py, metadata.py — shared utilities (e.g. database introspection, foreign-key metadata).
- _version.py — version info.

##### Key Architectural Patterns
- Modular, layered CLI design:
a. CLI Dispatcher →
b. Config Parser →
c. Plan Generator (“planner” + “solver”) →
d. Plan Executor (“sampler” + “filters”) →
e. Data I/O (DB connectors)
- Plugin and extension points for custom filters.
- Separation of concerns: config, plan modelling, planning, execution, and transformations.

##### Main Components
1. CLI / Entry Point
- subsetter/__main__.py
- Parses command-line arguments (plan, sample, subset) and loads config.
2. Configuration Module
- config_model.py
- Loads and validates subsetter.yaml via PyYAML + Pydantic.
3. Metadata & Introspection
- metadata.py, common.py
- Connects to source DB, reads schema/foreign-key graph.
4. Planner & Solver
- planner.py (builds a raw plan from “targets” in config).
- solver.py (orders tables, detects dependencies, applies single-query rule).
- plan_model.py (serializes plan as YAML AST).
5. Sampler / Executor
- sampler.py
- Phases: create tables (optional), truncate/merge, materialize temp tables, copy data.
- Interfaces with source and target DBs.
- Applies filters.py transformations, compaction, multiplicity.
6. Filters / Transformations
- filters.py + plugin API
- Faker-based PII anonymization, identifier compaction, merge logic.
7. External Systems
- Source Database (Postgres, MySQL, etc.)
- Target Database or JSON files

##### Component Interactions and Data Flow
- Filters plug into Sampler between read and write.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    CLI["subsetter CLI"]:::cli
    click CLI "https://github.com/msg555/subsetter/blob/main/subsetter/__main__.py"

    %% Configuration Layer
    Config["Config Parser"]:::config
    click Config "https://github.com/msg555/subsetter/blob/main/subsetter/config_model.py"

    CLI -->|"Load subsetter.yaml"| Config

    %% Metadata Layer
    Metadata["Metadata Collector"]:::config
    click Metadata "https://github.com/msg555/subsetter/blob/main/subsetter/metadata.py"
    Common["Common Utilities"]:::config
    click Common "https://github.com/msg555/subsetter/blob/main/subsetter/common.py"

    Config -->|"DB credentials"| Metadata
    Config -->|"DB credentials"| Common

    %% Planning Layer
    Planner["Planner"]:::planning
    click Planner "https://github.com/msg555/subsetter/blob/main/subsetter/planner.py"
    Solver["Solver"]:::planning
    click Solver "https://github.com/msg555/subsetter/blob/main/subsetter/solver.py"
    PlanModel["PlanModel (YAML AST)"]:::planning
    click PlanModel "https://github.com/msg555/subsetter/blob/main/subsetter/plan_model.py"

    Metadata -->|"Schema + targets"| Planner
    Planner -->|"Raw plan"| Solver
    Solver -->|"Resolved plan"| PlanModel

    %% Execution Layer
    subgraph "Sampler / Executor" 
        direction TB
        Sampler["Sampler"]:::execution
        click Sampler "https://github.com/msg555/subsetter/blob/main/subsetter/sampler.py"
        Create["Create/Truncate"]:::execution
        Materialize["Materialize Temp Tables"]:::execution
        Filters["Apply Filters"]:::execution
        click Filters "https://github.com/msg555/subsetter/blob/main/subsetter/filters.py"
        Copy["Copy Data"]:::execution

        Sampler --> Create
        Sampler --> Materialize
        Sampler --> Filters
        Sampler --> Copy
    end

    PlanModel -->|"Plan YAML"| Sampler

    %% Filter Plugin Boundary
    class Filters dashed-border;
    classDef dashed-border stroke-dasharray: 5 5;

    %% External Systems
    SourceDB(("Source DB")):::external
    TargetDB(("Target DB / JSON")):::external

    Create -->|"Execute SQL"| SourceDB
    Materialize -->|"Temp tables"| SourceDB
    Filters -->|"Filtered rows"| Copy
    Copy -->|"Write rows"| TargetDB

    %% Tests
    Tests["Tests Coverage"]:::tests
    click Tests "https://github.com/msg555/subsetter/tree/main/tests/"
    Tests --> Planner
    Tests --> Solver
    Tests --> Filters
    Tests --> Sampler

    %% Styles
    classDef cli fill:#e0f7fa,stroke:#006064,color:#000
    classDef config fill:#e8f5e9,stroke:#2e7d32,color:#000
    classDef planning fill:#e3f2fd,stroke:#1565c0,color:#000
    classDef execution fill:#fff3e0,stroke:#ef6c00,color:#000
    classDef external fill:#eeeeee,stroke:#757575,color:#000
    classDef tests fill:#f3e5f5,stroke:#6a1b9a,color:#000
```
