How evidentlyai/evidently works
• Core Library and API Layer – This includes the bulk of the business logic found in the “src/evidently” directory. It is subdivided into multiple modules:
• Calculation Engine (src/evidently/calculation_engine): Implements the metric implementations, supporting different numerical methods and statistical tests.
• Metrics, Test Suites, and Presets (src/evidently/metrics, src/evidently/test_preset, src/evidently/test_suite): Responsible for computing data drift, data quality, classification/regression metrics, etc.
• Pipeline & Runner (src/evidently/pipeline, src/evidently/runner): Manage the orchestration of evaluations, generating Reports or executing tests.
• Utilities and Configuration (src/evidently/_config.py, src/evidently/options, src/evidently/utils): Common helper functions, configuration management, and shared utilities.
• Command Line Interface (CLI) – Found in “src/evidently/cli”, it enables users to run evaluations from the command line.
• Collector and Integration Modules – Includes “src/evidently/collector” and supporting directories; these modules handle data ingestion, storage, and external integrations. Also, the numerous examples/integrations directories illustrate how Evidently connects with external services (Airflow, MLflow, Grafana, Postgres, etc.).
• External Deployment and Development Infrastructure – This includes Docker setup (various scripts and docker folders), GitHub workflows, container configurations, and CI/CD integrations that help package, test, and deploy the system.
Relationships and Data Flow
• The Core Library acts as the backend engine that gathers input data (both “reference” and “current”) and computes metrics and tests via the Calculation Engine. This output is then packaged into Reports or Test Suite results.
• The CLI or API interfaces (from the Core Library) are used to run evaluations through one-off scripts or automated pipelines.
• The Collector component ingests data (via REST API calls or file payloads) and forwards it to the Core Library modules.
• The Dashboard/UI layer retrieves or listens to evaluation results (using the API defined in “src/evidently/ui/api”) and visualizes them via interactive widgets and reports.
• The Integration Modules provide bridges between Evidently and external systems (such as Spark jobs, Airflow pipelines, Grafana dashboards, and MLflow experiments) shown in the “examples/integrations” segments. These indicate how Evidently can be plugged into existing ML infrastructures.
• CI/CD pipelines and configuration files (for Docker, GitHub actions, and pre-commit hooks) ensure that the system is continuously integrated and deployed.
Architectural Patterns and Design Principles
• Modularity and Plug-ability – Evidently is highly modular; components for metrics, test suites, and UI are separated cleanly. This encourages extensions (e.g., custom tests or new integrations).
• Layered Architecture – There is a clear separation of concerns between the core evaluation logic (calculations, metrics, tests), the delivery layer (CLI, API, UI), and external integrations.
• Use of Pipelines – The evaluation process is structured as a pipeline (from data input to processing to output generation), which is intrinsic to how reports and test suites are executed.
• Open Architecture for Integration – The design outlines simple APIs and registries (as seen in directories for registries and presets) to allow integration with third-party tools.
Relevant Technologies, Frameworks, and Libraries
• Python – The primary language, with standard libraries and popular frameworks for data handling (e.g., Pandas, scikit-learn).
• Pydantic – For configuration and data validation (note files like src/evidently/_pydantic_compat.py).
• Docker and CI/CD Tools – Evident from the docker configurations and GitHub workflows to support containerization and automated testing.
• Frontend Frameworks – In the UI directory, using TypeScript/React (visible from “tsx” files, themes, hooks, and router utilities within “ui/packages/evidently-ui-lib”) for the dashboard and interactive interfaces.
Additional Details
• Directory structure abstraction by maybe grouping components that are conceptually related (for instance, grouping all “src/evidently” modules together under a “Core Evaluation Engine” section, while drawing the “ui” folder as a separate “Frontend” block).