How airbytehq/airbyte-python-cdk works
Main Components
a. Core Framework Components:
• Source Connector Framework
• Destination Connector Framework
• Protocol Models Layer
• Configuration Management
• Logging System
b. Specialized CDKs:
• Low-Code CDK
• Concurrent CDK
• File-Based CDK
• Connector Builder
Component Relationships and Architecture
Top Layer - Entry Points:
• CLI Interface
• Connector Builder Interface
• Source/Destination Interfaces
Middle Layer - Specialized CDKs:
• Low-Code CDK (declarative)
• Concurrent CDK
• File-Based CDK
Core Layer:
• Base Connector Classes
• Protocol Models
• Utilities
• Authentication
• Error Handling
Key Design Elements
• Rectangles: Main components
• Rounded rectangles: Specialized CDKs
• Hexagons: Entry points
• Diamonds: Decision points
• Blue: Core components
• Green: Source-related components
• Orange: Destination-related components
• Purple: Utility components
• Yellow: Entry points
Detailed Component Breakdown
Sources Package:
Sources
• Abstract Source
• HTTP Source
• Authentication
• Error Handling
• Rate Limiting
• Declarative Source
• Manifest Parser
• Component Factory
• Stream Definitions
• Concurrent Source
• Thread Pool Manager
• Partition Reader
• State Management
• File-Based Source
• Parsers (CSV, JSON, etc.)
• Schema Management
• Validation Policies
Integration Points
Clear integration points for:
• External API connections
• Protocol message handling
• State management
• Error handling and logging
• Configuration management
Data Flow Representation
• Configuration flow
• Data extraction flow
• State management flow
• Error handling flow
• Logging flow
Additional Elements
Include:
• Version/compatibility information
• External dependencies
• Extension points
• Configuration points
Implementation Notes
• Modular architecture
• Extensibility points
• Clear separation of concerns
• Protocol compliance
• Error handling paths