# bluegroundltd/rustic-witcher architecture

> Architecture diagram of the GitHub repository bluegroundltd/rustic-witcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bluegroundltd/rustic-witcher
- Repository: https://github.com/bluegroundltd/rustic-witcher

## Overview

##### Project Type and Purpose
- RusticWitcher is a Rust-based toolkit dedicated to data export and anonymization. Its main purpose is to handle large datasets (specifically in Parquet format stored on Amazon S3), perform robust data export, and apply sophisticated anonymization techniques. The README even emphasizes performance, seamless S3 integration, and advanced anonymization.
- You also see that the project includes orchestration (with Argo Workflows), containerization (Dockerfiles, docker-compose), and several microservice-like Rust crates that are individually focused (e.g., anonymization configuration, operators for anonymization, CDC processing, mongo buddy for MongoDB, and more).

##### Analysis of the File Structure
- The top-level files include common build files (Cargo.toml, Makefile, Dockerfile, docker-compose.yaml) and a set of documentation files (in the docs folder) that specify various configurations, environment setup, tests, and workflows.
- There are many subdirectories named “rustic-anonymization-config”, “rustic-anonymization-operator”, “rustic-base-transformations”, “rustic-cdc-operator”, “rustic-config-generator-cli”, “rustic-faker-transformations”, “rustic-mongo-buddy”, “rustic-result-validator”, “rustic-s3-config”, “rustic-shell”, “rustic-target-db”, “rustic-transformator”, “rustic-whole-table-transformator” and others. This indicates that the overall system is modular—each represents a microcrates or microservices each with its own specific functionality.
- Additionally, there is a central “src” directory (with main.rs) and scripts which suggest there’s also a coordinating application or entry point.

##### Key Components and Their Relationships
a. Core Orchestrator & Entry Points:
- The main entry point in the “src/main.rs” and the associated Dockerfile/dockers-compose imply there is one or several service(s) that control the overall execution.

b. Data Processing Modules:
- Anonymization Related:
- “rustic-anonymization-config” encapsulates configuration handling for anonymization.
- “rustic-anonymization-operator” applies anonymization logic on dataframes.
- Transformation Modules:
- “rustic-base-transformations”, “rustic-transformator”, and “rustic-whole-table-transformator” provide transformation capabilities and different granularities (per column vs. whole table).
- Data Validation:
- “rustic-result-validator” appears responsible for checking and validating the transformation and anonymization outcomes.
- Data Import/Export:
- “rustic-mongo-buddy” handles MongoDB-specific export/import operations.
- “rustic-target-db” manages writing data to target databases.
- “rustic-s3-config” likely configures interaction with Amazon S3.
- Additional Utilities:
- “rustic-config-generator-cli” helps generate configuration files.
- “rustic-faker-transformations” and “rustic-faker-types” provide fake or dummy data generation for testing or anonymization.
- “rustic-cdc-operator” appears to handle Change Data Capture processes.
- “rustic-shell” may be designed for command execution or scripting in an integrated shell environment.
- “rustic-duration” provides utilities around timing and duration calculations.

c. Integration and External Dependencies:
- The system integrates with Amazon S3 for Parquet file storage.
- The provided Dockerfiles and docker-compose.yaml suggest containerized deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Orchestration & Entry Point
    subgraph "Orchestration & Entry Point"
        Main["src/main.rs"]
        O1["Orchestrator/Main Controller"]
        Docker["Dockerfile"]
        Compose["docker-compose.yaml"]
    end

    %% External Services
    subgraph "External Services"
        S3["Amazon S3"]
        Mongo["MongoDB"]
    end

    %% Anonymization & Transformation Modules
    subgraph "Anonymization & Transformation Modules"
        AnonConfig["rustic-anonymization-config"]
        AnonOperator["rustic-anonymization-operator"]
        BaseTrans["rustic-base-transformations"]
        Transform["rustic-transformator"]
        WholeTable["rustic-whole-table-transformator"]
        FakerTrans["rustic-faker-transformations"]
        FakerTypes["rustic-faker-types"]
    end

    %% Other Processing Modules
    subgraph "Other Processing Modules"
        CDC["rustic-cdc-operator"]
        Validation["rustic-result-validator"]
        MongoBuddy["rustic-mongo-buddy"]
        TargetDB["rustic-target-db"]
    end

    %% Configuration & Utility
    subgraph "Configuration & Utility"
        ConfigGen["rustic-config-generator-cli"]
        S3Config["rustic-s3-config"]
        Shell["rustic-shell"]
        Duration["rustic-duration"]
    end

    %% Data Flows and Dependencies
    S3 -->|"reads_data"| O1
    O1 -->|"initiates"| AnonConfig
    O1 -->|"initiates"| AnonOperator
    O1 -->|"initiates"| BaseTrans
    O1 -->|"initiates"| Transform
    O1 -->|"initiates"| WholeTable
    O1 -->|"initiates"| FakerTrans
    O1 -->|"initiates"| FakerTypes
    O1 -->|"triggers"| CDC
    ConfigGen -->|"supplies_config"| AnonConfig
    S3Config -->|"configures"| O1
    AnonConfig -->|"configures_operator"| AnonOperator
    AnonOperator -->|"data_transformed"| Validation
    BaseTrans -->|"data_transformed"| Validation
    Transform -->|"data_transformed"| Validation
    WholeTable -->|"data_transformed"| Validation
    FakerTrans -->|"data_transformed"| Validation
    FakerTypes -->|"data_transformed"| Validation
    Validation -->|"outputs_to"| TargetDB
    O1 -->|"triggers_validation"| Validation
    Validation ---|"sends_feedback"| O1
    O1 -->|"triggers_export"| MongoBuddy
    Mongo -->|"data_source"| MongoBuddy
    O1 -->|"executes_scripts"| Shell
    Duration -->|"metrics_for"| BaseTrans

    %% Click Events
    click Main "https://github.com/bluegroundltd/rustic-witcher/blob/main/src/main.rs"
    click Docker "https://github.com/bluegroundltd/rustic-witcher/tree/main/Dockerfile"
    click Compose "https://github.com/bluegroundltd/rustic-witcher/blob/main/docker-compose.yaml"
    click AnonConfig "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-anonymization-config"
    click AnonOperator "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-anonymization-operator"
    click BaseTrans "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-base-transformations"
    click Transform "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-transformator"
    click WholeTable "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-whole-table-transformator"
    click FakerTrans "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-faker-transformations"
    click FakerTypes "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-faker-types"
    click CDC "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-cdc-operator"
    click Validation "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-result-validator"
    click MongoBuddy "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-mongo-buddy"
    click TargetDB "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-target-db"
    click S3Config "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-s3-config"
    click ConfigGen "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-config-generator-cli"
    click Shell "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-shell"
    click Duration "https://github.com/bluegroundltd/rustic-witcher/tree/main/rustic-duration"

    %% Styles
    class Main,O1,Docker,Compose orchestration
    class AnonConfig,AnonOperator,BaseTrans,Transform,WholeTable,FakerTrans,FakerTypes,CDC transformation
    class Validation,TargetDB,MongoBuddy output
    class ConfigGen,S3Config,Shell,Duration utility
    class S3,Mongo external

    classDef orchestration fill:#f9c74f,stroke:#000,stroke-width:2px;
    classDef transformation fill:#90be6d,stroke:#000,stroke-width:2px;
    classDef output fill:#90a4ae,stroke:#000,stroke-width:2px;
    classDef utility fill:#577590,stroke:#000,stroke-width:2px,color:#fff;
    classDef external fill:#f94144,stroke:#000,stroke-width:2px,color:#fff;
```
