# airbytehq/airbyte-python-cdk architecture

> Architecture diagram of the GitHub repository airbytehq/airbyte-python-cdk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/airbytehq/airbyte-python-cdk
- Repository: https://github.com/airbytehq/airbyte-python-cdk

## Overview

##### Main Components

a. Core Framework Components:
- Source Connector Framework
- Destination Connector Framework
- Protocol Models Layer
- Configuration Management
- Logging System

b. Specialized CDKs:
- Low-Code CDK
- Concurrent CDK
- File-Based CDK
- Connector Builder

##### Component Relationships and Architecture

Top Layer - Entry Points:
- CLI Interface
- Connector Builder Interface
- Source/Destination Interfaces

Middle Layer - Specialized CDKs:
- Low-Code CDK (declarative)
- Concurrent CDK
- File-Based CDK

Core Layer:
- Base Connector Classes
- Protocol Models
- Utilities
- Authentication
- Error Handling

##### Key Design Elements

- Rectangles: Main components
- Rounded rectangles: Specialized CDKs
- Hexagons: Entry points
- Diamonds: Decision points

- Blue: Core components
- Green: Source-related components
- Orange: Destination-related components
- Purple: Utility components
- Yellow: Entry points

##### Detailed Component Breakdown

Sources Package:
**Sources**
- Abstract Source
- HTTP Source
- Authentication
- Error Handling
- Rate Limiting
- Declarative Source
- Manifest Parser
- Component Factory
- Stream Definitions
- Concurrent Source
- Thread Pool Manager
- Partition Reader
- State Management
- File-Based Source
- Parsers (CSV, JSON, etc.)
- Schema Management
- Validation Policies

##### Integration Points

Clear integration points for:
- External API connections
- Protocol message handling
- State management
- Error handling and logging
- Configuration management

##### Data Flow Representation

- Configuration flow
- Data extraction flow
- State management flow
- Error handling flow
- Logging flow

##### Additional Elements

Include:
- Version/compatibility information
- External dependencies
- Extension points
- Configuration points

##### Implementation Notes

- Modular architecture
- Extensibility points
- Clear separation of concerns
- Protocol compliance
- Error handling paths

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Entry_Points
        CLI[CLI Interface]:::entrypoint
        CB[Connector Builder]:::entrypoint
        SI[Source Interface]:::entrypoint
        DI[Destination Interface]:::entrypoint
    end

    subgraph Specialized_CDKs
        LCD[Low-Code CDK]:::specialized
        CCD[Concurrent CDK]:::specialized
        FBD[File-Based CDK]:::specialized
    end

    subgraph Core_Framework
        SCF[Source Connector Framework]:::core
        DCF[Destination Connector Framework]:::core
        PML[Protocol Models Layer]:::core
        CM[Configuration Management]:::core
        LS[Logging System]:::core
    end

    subgraph Components
        AF[Authentication Framework]:::component
        EH[Error Handling]:::component
        HC[HTTP Components]:::component
        SM[Schema Management]:::component
        STM[State Management]:::component
        EP[Entry Point System]:::component
    end

    %% Relationships
    CLI --> EP
    CB --> EP
    SI --> EP
    DI --> EP

    EP --> SCF
    EP --> DCF

    LCD --> SCF
    CCD --> SCF
    FBD --> SCF

    SCF --> PML
    DCF --> PML
    
    SCF --> CM
    DCF --> CM
    
    SCF --> LS
    DCF --> LS

    LCD --> AF
    LCD --> EH
    LCD --> HC
    LCD --> SM
    LCD --> STM

    CCD --> STM
    FBD --> SM

    %% Click Events
    click SCF "https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/source.py"
    click DCF "https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/destinations/destination.py"
    click PML "https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/models/airbyte_protocol.py"
    click CM "https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/config.py"
    click LS "https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/logger.py"
    click LCD "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/sources/declarative/"
    click CCD "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/sources/concurrent_source/"
    click FBD "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/sources/file_based/"
    click CB "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/connector_builder/"
    click AF "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/sources/declarative/auth/"
    click EH "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/sources/declarative/requesters/error_handlers/"
    click HC "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/sources/streams/http/"
    click SM "https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/sources/declarative/schema/"
    click STM "https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/connector_state_manager.py"
    click EP "https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/entrypoint.py"

    %% Styling
    classDef core fill:#2374f7,stroke:#000,stroke-width:2px
    classDef specialized fill:#ff7f0e,stroke:#000,stroke-width:2px
    classDef component fill:#9467bd,stroke:#000,stroke-width:2px
    classDef entrypoint fill:#ffbb00,stroke:#000,stroke-width:2px

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[Specialized CDKs]:::specialized
        L3[Utility Components]:::component
        L4[Entry Points]:::entrypoint
    end
```
