# sodadata/soda-core architecture

> Architecture diagram of the GitHub repository sodadata/soda-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sodadata/soda-core
- Repository: https://github.com/sodadata/soda-core

## Overview

##### Project type and purpose
- Soda Core is an open-source, Python-based CLI tool and library for data quality testing.
- It parses user-defined checks (SodaCL)—written as YAML or DSL—turns them into SQL or engine-specific queries, runs them against SQL, Spark, or Pandas data sources, and reports pass/fail/warn results.

##### File structure
- Top-level: configuration (`.env.example`, `configuration.yml` in docs), packaging/manifests (`pyproject.toml`, `setup.py` in submodules), CI/CD (`.github/workflows`, Dockerfiles), docs (`docs/`), examples, scripts.
- `soda/` directory is a polyglot monorepo of packages:
- `soda/core` – the engine, CLI entrypoint, configuration parser, SodaCL parser (ANTLR grammar), execution framework, telemetry, cloud plugin.
- `soda/contracts` – data contract DSL, conversion, verification, custom plugin framework.
- `soda/<connector>` – one folder per data source (Postgres, Spark, Athena, BigQuery, etc.), each exposing a standardized DataSource interface.
- `soda/scientific` – optional anomaly detection models and simulations.
- Each package has a `setup.py` for independent installation (`pip install soda-core-postgres`, etc.).
- Tests in each module verify connectivity, query correctness, DSL parsing, and core behaviors.

##### README for additional insights
- CLI commands (`soda scan`) orchestrate scans.
- Library API for programmatic scans.
- Integration points: external data sources (various SQL engines, Spark, Pandas), optional Soda Cloud service, telemetry exporters.

##### Final advice
- Plugin boundaries where extensibility occurs.
- Sequence of operations for a scan to guide readers through the flow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    UserArtifacts["Config YAML & SodaCL Checks"]:::grey
    CLI["CLI / Python API Layer"]:::blue
    subgraph "Core Engine"
        scanOrch["Scan Orchestrator"]:::blue
        configParser["Configuration Parser"]:::blue
        configModel["Configuration Model"]:::blue
        sodaclGrammar["SodaCL ANTLR Grammar"]:::blue
        sodaclParser["SodaCL Parser & Visitor"]:::blue
        queryGen["Check & Metric Builders"]:::blue
        execMgr["Execution Manager"]:::blue
        telemetryAPI["Telemetry & Logging"]:::blue
    end
    subgraph "Plugins"
        cloudPlugin["Cloud Plugin"]:::blue
        contractsPlugin["Contracts Plugin"]:::green
        dataContracts["Data Contracts & Verification"]:::green
        sciAnom["Anomaly Detection (V1)"]:::green
        sciAnomV2["Anomaly Detection V2"]:::green
        sciSim["Simulation & Visualization"]:::green
    end
    subgraph "Connectors"
        postgres["Postgres Connector"]:::green
        spark["Spark Connector"]:::green
        bigquery["BigQuery Connector"]:::green
        athena["Athena Connector"]:::green
        otherConn["Other Connectors"]:::green
    end
    subgraph "External Systems"
        DBs["Databases (SQL)"]:::orange
        SparkCluster["Spark Cluster"]:::orange
        Pandas["Pandas DataFrames"]:::orange
        CloudService["Soda Cloud Service"]:::orange
        TelemetryBackend["Telemetry Backend"]:::orange
    end
    subgraph "Outputs"
        CLIout["CLI Output & Reports"]:::grey
        JSONrep["JSON/CSV Reports"]:::grey
        Logs["Logs"]:::grey
    end

    UserArtifacts --> CLI
    CLI --> scanOrch
    scanOrch --> configParser
    scanOrch --> sodaclParser
    configParser --> configModel
    sodaclGrammar --> sodaclParser
    sodaclParser --> queryGen
    queryGen --> execMgr
    execMgr --> postgres
    execMgr --> spark
    execMgr --> bigquery
    execMgr --> athena
    execMgr --> otherConn
    postgres --> DBs
    spark --> SparkCluster
    otherConn --> Pandas
    execMgr --> scanOrch
    scanOrch --> telemetryAPI
    telemetryAPI --> TelemetryBackend
    scanOrch --> cloudPlugin
    cloudPlugin --> CloudService
    scanOrch --> CLIout
    scanOrch --> JSONrep
    scanOrch --> Logs

    click CLI "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/cli/cli.py"
    click configParser "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/configuration/configuration_parser.py"
    click configModel "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/configuration/configuration.py"
    click sodaclGrammar "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/sodacl/antlr/SodaCLAntlr.g4"
    click sodaclParser "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/sodacl/sodacl_parser.py"
    click queryGen "https://github.com/sodadata/soda-core/tree/main/soda/core/soda/execution/check/"
    click queryGen "https://github.com/sodadata/soda-core/tree/main/soda/core/soda/execution/query/"
    click queryGen "https://github.com/sodadata/soda-core/tree/main/soda/core/soda/execution/metric/"
    click execMgr "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/execution/data_source_manager.py"
    click scanOrch "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/scan.py"
    click telemetryAPI "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/telemetry/soda_telemetry.py"
    click cloudPlugin "https://github.com/sodadata/soda-core/blob/main/soda/core/soda/cloud/soda_cloud.py"
    click contractsPlugin "https://github.com/sodadata/soda-core/blob/main/soda/contracts/soda/contracts/impl/plugin.py"
    click sciAnom "https://github.com/sodadata/soda-core/blob/main/soda/scientific/soda/scientific/anomaly_detection/anomaly_detector.py"
    click sciAnomV2 "https://github.com/sodadata/soda-core/blob/main/soda/scientific/soda/scientific/anomaly_detection_v2/anomaly_detector.py"
    click sciSim "https://github.com/sodadata/soda-core/blob/main/soda/scientific/soda/scientific/anomaly_detection_v2/simulate/app.py"

    classDef blue fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef green fill:#d4f4dd,stroke:#333,stroke-width:1px
    classDef orange fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef grey fill:#e0e0e0,stroke:#333,stroke-width:1px
```
