How whylabs/whylogs works
High-Level Components (Main Layer)
a. Core Components:
• Profile Engine (center)
• Schema Management
• Metrics System
• Constraints Engine
b. Language Implementations:
• Python Implementation
• Java Implementation
• Protocol Buffers Interface (connecting both implementations)
Integration Layer (Top)
• API Interfaces
• Writers (WhyLabs, S3, Local, GCS)
• Integration Points (MLflow, PySpark, Kafka, etc.)
Feature Layer (Bottom)
• Data Validation
• Profile Visualization
• Performance Metrics
• Custom Metrics
Use the following organization:
Component Relationships
• [Integration Layer]
• ↕
• [Core Components]
• ↕
• [Feature Layer]
Specific Elements
a. Core Components Section:
• Dataset Profile Manager
• Column Profile Handler
• Metric Registry
• Schema Validator
• Constraint Engine
b. Integration Layer:
• Writer Registry
• API Endpoints
• External System Connectors
• Serialization/Deserialization
c. Feature Layer:
• Visualization Components
• Statistical Analyzers
• Performance Metrics
• Custom Metric Extensions
Important Connections
• Bidirectional flow between Core and Integration layers
• Data transformation points
• Async vs sync operations
• Extension points
Additional Details
• Version compatibility notes
• Experimental features differently
• Configuration points
Key Design Considerations:
Modularity
• Plugin architecture
• Clear boundaries between components
• Extension points
Data Flow
• Primary data paths
• Transformation points
• Where profiles are generated/stored
Integration Points
• External system connections
• Protocol buffer interfaces
• Supported platforms
Scalability Aspects
• Parallel processing capabilities
• Batch vs streaming paths
• Distributed operation support
Remember to:
• Use consistent notation
• Provide clear labels
• Both static and dynamic relationships
• Version compatibility
• Security boundaries