# airbytehq/pyairbyte architecture

> Architecture diagram of the GitHub repository airbytehq/pyairbyte, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/airbytehq/pyairbyte
- Repository: https://github.com/airbytehq/pyairbyte

## Overview

##### Project type and purpose
- PyAirbyte is an open‐source Python library and CLI tool that enables developers to run Airbyte connectors directly in Python.
- It provides abstractions for connectors (sources/destinations), executors (local, Docker, Python), caching backends, SQL processors, and integration with Airbyte Cloud.

##### File structure
- Top‐level:
- airbyte/ (core package)
- examples/ (usage demos)
- docs/ (documentation)
- tests/ (unit, integration, lint)
- CI/CD workflows (.github/)
- Key subpackages under airbyte/:
- _executors – strategy implementations for running connectors
- _processors/sql – SQL translation layers for various warehouses
- caches – pluggable state/catalog caches (Postgres, DuckDB, BigQuery, etc.)
- cloud – helpers to talk to Airbyte Cloud REST API
- sources / destinations – base classes and registries
- secrets – secret‐management abstractions (env, GCP secret manager, etc.)
- writers – file and JSONL writers for output streams
- CLI entrypoint: airbyte/cli.py
- Configuration & build: pyproject.toml, poetry.toml

##### Extract insights from README
- PyAirbyte does not orchestrate; it’s a library to run connectors, handle state, and stream data.
- Supports any connector with a CLI interface on PATH, or local connector projects.
- Includes built‐in “cache” destinations to land data in popular warehouses.
- Works with orchestration frameworks but does not itself orchestrate.

a. Main components
- CLI Layer (airbyte/cli.py)
- Core Library Layer (airbyte package) with subcomponents:
- Connector Registry & Loader
- Executor Strategies (Local, Docker, Python, Declarative)
- Processors (SQL translation)
- Cache Abstractions (state & catalog persistence)
- Cloud Client (Airbyte Cloud API)
- Secrets Manager
- Writers (file, JSONL)
- External Systems & Services
- Airbyte Cloud API
- Source Systems (APIs, Databases)
- Destination Systems (Databases, Data Warehouses)
- Secret Backends (Env Vars, GCP Secret Manager)
b. Relationships & Interactions
- User invokes CLI → CLI parses commands → passes control to Library Core
- Core selects Connector via Registry → picks Executor strategy → Executor spins up connector process (Docker container, local script, or Python module)
- Data emitted by connector flows into Processors → optional caching backend writes state/catalog → Writers serialize output
- Cloud Client module can push/pull sync metadata to/from Airbyte Cloud
- Secrets Module supplies credentials to connectors, executors, and cloud client
c. Architectural patterns & design principles
- Plugin/Registry pattern for connectors
- Strategy pattern for Executors and Caches
- Adapter pattern for SQL translation across warehouses
- Separation of concerns: CLI vs Core Engine vs IO vs Caching vs Cloud Integration
d. Key technologies & libraries
- Python 3.x, Poetry for packaging
- Click or argparse (for CLI parsing)
- Docker SDK (for Docker executor)
- HTTP client (requests) for Airbyte Cloud API
- DB client libraries (psycopg2, duckdb, google-cloud-bigquery, snowflake-connector) for caches/processors

##### Final checklist
- All major modules from airbyte/ are represented and grouped logically.
- External dependencies (DBs, Cloud API, Docker daemon) are shown.
- Keep the layout readable: CLI at top or left, core engine central, external systems on the perimeter.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    CLI["CLI Layer\n(airbyte/cli.py)"]:::cli
    click CLI "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/cli.py"

    %% Core Engine
    subgraph "Core Library Layer"
        direction TB

        %% Connector Registry & Loader
        Registry["Connector Registry & Loader"]:::core
        click Registry "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/sources/registry.py"
        BaseConnector["Base Connector"]:::core
        click BaseConnector "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_connector_base.py"

        %% Executor Strategies
        subgraph "Executor Strategies"
            direction TB
            ExecBase["Base Strategy"]:::core
            click ExecBase "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_executors/base.py"
            DeclarativeExec["Declarative Executor"]:::core
            click DeclarativeExec "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_executors/declarative.py"
            DockerExec["Docker Executor"]:::core
            click DockerExec "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_executors/docker.py"
            LocalExec["Local Executor"]:::core
            click LocalExec "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_executors/local.py"
            PythonExec["Python Executor"]:::core
            click PythonExec "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_executors/python.py"
        end

        %% Processors
        subgraph "Processors (SQL Translation)"
            direction TB
            BigQueryProc["bigquery.py"]:::core
            click BigQueryProc "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_processors/sql/bigquery.py"
            DuckDBProc["duckdb.py"]:::core
            click DuckDBProc "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_processors/sql/duckdb.py"
            MotherduckProc["motherduck.py"]:::core
            click MotherduckProc "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_processors/sql/motherduck.py"
            PostgresProc["postgres.py"]:::core
            click PostgresProc "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_processors/sql/postgres.py"
            SnowflakeProc["snowflake.py"]:::core
            click SnowflakeProc "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_processors/sql/snowflake.py"
        end

        %% Cache Abstractions
        subgraph "Cache Abstractions"
            direction TB
            CacheBase["base.py"]:::core
            click CacheBase "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/base.py"
            StateBackend["_state_backend.py"]:::core
            click StateBackend "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/_state_backend.py"
            CatalogBackend["_catalog_backend.py"]:::core
            click CatalogBackend "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/_catalog_backend.py"
            PostgresCache["postgres.py"]:::core
            click PostgresCache "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/postgres.py"
            DuckDBCache["duckdb.py"]:::core
            click DuckDBCache "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/duckdb.py"
            BigQueryCache["bigquery.py"]:::core
            click BigQueryCache "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/bigquery.py"
            MotherduckCache["motherduck.py"]:::core
            click MotherduckCache "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/motherduck.py"
            SnowflakeCache["snowflake.py"]:::core
            click SnowflakeCache "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/caches/snowflake.py"
        end

        %% Cloud Client
        subgraph "Cloud Client"
            direction TB
            Connections["connections.py"]:::io
            click Connections "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/cloud/connections.py"
            ConnectorsCC["connectors.py"]:::io
            click ConnectorsCC "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/cloud/connectors.py"
            SyncResults["sync_results.py"]:::io
            click SyncResults "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/cloud/sync_results.py"
            Workspaces["workspaces.py"]:::io
            click Workspaces "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/cloud/workspaces.py"
            Constants["constants.py"]:::io
            click Constants "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/cloud/constants.py"
        end

        %% Secrets Manager
        subgraph "Secrets Manager"
            direction TB
            SecretBase["base.py"]:::core
            click SecretBase "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/secrets/base.py"
            ConfigSecret["config.py"]:::core
            click ConfigSecret "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/secrets/config.py"
            EnvVars["env_vars.py"]:::core
            click EnvVars "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/secrets/env_vars.py"
            GSM["google_gsm.py"]:::core
            click GSM "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/secrets/google_gsm.py"
            GoogleColab["google_colab.py"]:::core
            click GoogleColab "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/secrets/google_colab.py"
            CustomSecret["custom.py"]:::core
            click CustomSecret "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/secrets/custom.py"
            PromptSecret["prompt.py"]:::core
            click PromptSecret "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/secrets/prompt.py"
        end

        %% Writers
        subgraph "Writers"
            direction TB
            WriterBase["base.py"]:::core
            click WriterBase "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_writers/base.py"
            FileWriter["file_writers.py"]:::core
            click FileWriter "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_writers/file_writers.py"
            JSONLWriter["jsonl.py"]:::core
            click JSONLWriter "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_writers/jsonl.py"
        end

        %% Utilities & Shared
        SubUtil["Utility & Shared Modules"]:::core
        click SubUtil "https://github.com/airbytehq/pyairbyte/tree/main/airbyte/_util"
        Shared["Shared Modules"]:::core
        click Shared "https://github.com/airbytehq/pyairbyte/tree/main/airbyte/shared"
        
        %% Core Data Models & Helpers
        DataModels["Core Data Models & Helpers"]:::core
        click DataModels "https://github.com/airbytehq/pyairbyte/blob/main/airbyte/_batch_handles.py"
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        AirbyteCloudAPI["Airbyte Cloud API"]:::external
        SourceSystems["Source Systems\n(APIs, Databases)"]:::external
        DestinationSystems["Destination Systems\n(Databases, Warehouses)"]:::external
        SecretStores["Secret Stores\n(Environment, GCP)"]:::external
        DockerDaemon["Docker Daemon"]:::external
    end

    %% Data/Control Flows
    CLI -->|"parse commands"| Registry
    Registry -->|"load connector"| ExecBase
    ExecBase -->|"select strategy"| DockerExec
    ExecBase --> LocalExec
    ExecBase --> PythonExec
    ExecBase --> DeclarativeExec

    DockerExec -->|"invoke connector"| DockerDaemon
    LocalExec -->|"invoke connector"| SourceSystems
    PythonExec -->|"invoke module"| SourceSystems

    DockerExec -->|"emit data"| Processors
    LocalExec -->|"emit data"| Processors
    PythonExec -->|"emit data"| Processors

    Processors -->|"translate SQL"| CacheBase
    Processors -->|"stream data"| WriterBase

    CacheBase -->|"persist state"| DestinationSystems
    WriterBase -->|"write output"| DestinationSystems

    Registry --> SecretsManager
    SecretsManager --> SecretStores

    Registry --> Connections
    Connections --> AirbyteCloudAPI
    Connections --> Workspaces
    SyncResults --> AirbyteCloudAPI

    %% Styles
    classDef cli fill:#D0E8FF,stroke:#0366D6,color:#0366D6
    classDef core fill:#E6FFEB,stroke:#28A745,color:#28A745
    classDef io fill:#FFF5E6,stroke:#E98B39,color:#E98B39
    classDef external fill:#F0F0F0,stroke:#6A737D,color:#6A737D
```
