How sodadata/soda-core works
Project type and purpose
• Soda Core is an open-source, Python-based CLI tool and library for data quality testing.
• It parses user-defined checks (SodaCL)—written as YAML or DSL—turns them into SQL or engine-specific queries, runs them against SQL, Spark, or Pandas data sources, and reports pass/fail/warn results.
File structure
• Top-level: configuration (.env.example, configuration.yml in docs), packaging/manifests (pyproject.toml, setup.py in submodules), CI/CD (.github/workflows, Dockerfiles), docs (docs/), examples, scripts.
• soda/ directory is a polyglot monorepo of packages:
• soda/core – the engine, CLI entrypoint, configuration parser, SodaCL parser (ANTLR grammar), execution framework, telemetry, cloud plugin.
• soda/contracts – data contract DSL, conversion, verification, custom plugin framework.
• soda/<connector> – one folder per data source (Postgres, Spark, Athena, BigQuery, etc.), each exposing a standardized DataSource interface.
• soda/scientific – optional anomaly detection models and simulations.
• Each package has a setup.py for independent installation (pip install soda-core-postgres, etc.).
• Tests in each module verify connectivity, query correctness, DSL parsing, and core behaviors.
README for additional insights
• CLI commands (soda scan) orchestrate scans.
• Library API for programmatic scans.
• Integration points: external data sources (various SQL engines, Spark, Pandas), optional Soda Cloud service, telemetry exporters.
Final advice
• Plugin boundaries where extensibility occurs.
• Sequence of operations for a scan to guide readers through the flow.