# whylabs/whylogs architecture

> Architecture diagram of the GitHub repository whylabs/whylogs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/whylabs/whylogs
- Repository: https://github.com/whylabs/whylogs

## Overview

##### High-Level Components (Main Layer)

a. Core Components:
- Profile Engine (center)
- Schema Management
- Metrics System
- Constraints Engine

b. Language Implementations:
- Python Implementation
- Java Implementation
- Protocol Buffers Interface (connecting both implementations)

##### Integration Layer (Top)
- API Interfaces
- Writers (WhyLabs, S3, Local, GCS)
- Integration Points (MLflow, PySpark, Kafka, etc.)

##### Feature Layer (Bottom)
- Data Validation
- Profile Visualization
- Performance Metrics
- Custom Metrics

Use the following organization:

##### Component Relationships
- [Integration Layer]
- ↕
- [Core Components]
- ↕
- [Feature Layer]

##### Specific Elements

a. Core Components Section:
- Dataset Profile Manager
- Column Profile Handler
- Metric Registry
- Schema Validator
- Constraint Engine

b. Integration Layer:
- Writer Registry
- API Endpoints
- External System Connectors
- Serialization/Deserialization

c. Feature Layer:
- Visualization Components
- Statistical Analyzers
- Performance Metrics
- Custom Metric Extensions

##### Important Connections
- Bidirectional flow between Core and Integration layers
- Data transformation points
- Async vs sync operations
- Extension points

##### Additional Details
- Version compatibility notes
- Experimental features differently
- Configuration points

Key Design Considerations:

##### Modularity
- Plugin architecture
- Clear boundaries between components
- Extension points

##### Data Flow
- Primary data paths
- Transformation points
- Where profiles are generated/stored

##### Integration Points
- External system connections
- Protocol buffer interfaces
- Supported platforms

##### Scalability Aspects
- Parallel processing capabilities
- Batch vs streaming paths
- Distributed operation support

Remember to:
- Use consistent notation
- Provide clear labels
- Both static and dynamic relationships
- Version compatibility
- Security boundaries

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style Definitions
    classDef core fill:#b3d1ff,stroke:#333,stroke-width:2px
    classDef integration fill:#90EE90,stroke:#333,stroke-width:2px
    classDef feature fill:#FFB366,stroke:#333,stroke-width:2px
    classDef language fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef external fill:#C8A2C8,stroke:#333,stroke-width:2px

    subgraph Integration["Integration Layer"]
        API["API Interfaces"]:::integration
        Writers["Writers Registry"]:::integration
        IntegrationPoints["Integration Points"]:::integration
        Serialization["Serialization/Deserialization"]:::integration
    end

    subgraph Core["Core Components"]
        ProfileEngine["Profile Engine"]:::core
        SchemaManagement["Schema Management"]:::core
        MetricsSystem["Metrics System"]:::core
        ConstraintsEngine["Constraints Engine"]:::core
    end

    subgraph Languages["Language Implementations"]
        Python["Python Implementation"]:::language
        Java["Java Implementation"]:::language
        ProtoBuf["Protocol Buffers Interface"]:::language
    end

    subgraph Features["Feature Layer"]
        DataValidation["Data Validation"]:::feature
        Visualization["Profile Visualization"]:::feature
        PerfMetrics["Performance Metrics"]:::feature
        CustomMetrics["Custom Metrics"]:::feature
    end

    %% Connections
    API --> ProfileEngine
    Writers --> ProfileEngine
    IntegrationPoints --> API
    
    ProfileEngine --> SchemaManagement
    ProfileEngine --> MetricsSystem
    ProfileEngine --> ConstraintsEngine
    
    Python --> ProtoBuf
    Java --> ProtoBuf
    ProtoBuf --> ProfileEngine
    
    ProfileEngine --> DataValidation
    ProfileEngine --> Visualization
    MetricsSystem --> PerfMetrics
    MetricsSystem --> CustomMetrics

    %% Click Events
    click ProfileEngine "https://github.com/whylabs/whylogs/blob/mainline/python/whylogs/core/dataset_profile.py"
    click SchemaManagement "https://github.com/whylabs/whylogs/blob/mainline/python/whylogs/core/schema.py"
    click MetricsSystem "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/core/metrics/"
    click ConstraintsEngine "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/core/constraints/"
    click Python "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/"
    click Java "https://github.com/whylabs/whylogs/tree/mainline/java/core/"
    click ProtoBuf "https://github.com/whylabs/whylogs/blob/mainline/proto/src/whylogs_messages.proto"
    click API "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/api/"
    click Writers "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/api/writer/"
    click IntegrationPoints "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/api/pyspark/"
    click DataValidation "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/core/validators/"
    click Visualization "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/viz/"
    click PerfMetrics "https://github.com/whylabs/whylogs/tree/mainline/python/whylogs/core/model_performance_metrics/"
    click CustomMetrics "https://github.com/whylabs/whylogs/blob/mainline/python/whylogs/core/metrics/metrics.py"

    %% Legend
    subgraph Legend
        CoreL["Core Component"]:::core
        IntegrationL["Integration Component"]:::integration
        FeatureL["Feature Component"]:::feature
        LanguageL["Language Implementation"]:::language
    end
```
