How airbytehq/pyairbyte works
Project type and purpose
• PyAirbyte is an open‐source Python library and CLI tool that enables developers to run Airbyte connectors directly in Python.
• It provides abstractions for connectors (sources/destinations), executors (local, Docker, Python), caching backends, SQL processors, and integration with Airbyte Cloud.
File structure
• Top‐level:
• airbyte/ (core package)
• examples/ (usage demos)
• docs/ (documentation)
• tests/ (unit, integration, lint)
• CI/CD workflows (.github/)
• Key subpackages under airbyte/:
• _executors – strategy implementations for running connectors
• _processors/sql – SQL translation layers for various warehouses
• caches – pluggable state/catalog caches (Postgres, DuckDB, BigQuery, etc.)
• cloud – helpers to talk to Airbyte Cloud REST API
• sources / destinations – base classes and registries
• secrets – secret‐management abstractions (env, GCP secret manager, etc.)
• writers – file and JSONL writers for output streams
• CLI entrypoint: airbyte/cli.py
• Configuration & build: pyproject.toml, poetry.toml
Extract insights from README
• PyAirbyte does not orchestrate; it’s a library to run connectors, handle state, and stream data.
• Supports any connector with a CLI interface on PATH, or local connector projects.
• Includes built‐in “cache” destinations to land data in popular warehouses.
• Works with orchestration frameworks but does not itself orchestrate.
a. Main components
• CLI Layer (airbyte/cli.py)
• Core Library Layer (airbyte package) with subcomponents:
• Connector Registry & Loader
• Executor Strategies (Local, Docker, Python, Declarative)
• Processors (SQL translation)
• Cache Abstractions (state & catalog persistence)
• Cloud Client (Airbyte Cloud API)
• Secrets Manager
• Writers (file, JSONL)
• External Systems & Services
• Airbyte Cloud API
• Source Systems (APIs, Databases)
• Destination Systems (Databases, Data Warehouses)
• Secret Backends (Env Vars, GCP Secret Manager)
b. Relationships & Interactions
• User invokes CLI → CLI parses commands → passes control to Library Core
• Core selects Connector via Registry → picks Executor strategy → Executor spins up connector process (Docker container, local script, or Python module)
• Data emitted by connector flows into Processors → optional caching backend writes state/catalog → Writers serialize output
• Cloud Client module can push/pull sync metadata to/from Airbyte Cloud
• Secrets Module supplies credentials to connectors, executors, and cloud client
c. Architectural patterns & design principles
• Plugin/Registry pattern for connectors
• Strategy pattern for Executors and Caches
• Adapter pattern for SQL translation across warehouses
• Separation of concerns: CLI vs Core Engine vs IO vs Caching vs Cloud Integration
d. Key technologies & libraries
• Python 3.x, Poetry for packaging
• Click or argparse (for CLI parsing)
• Docker SDK (for Docker executor)
• HTTP client (requests) for Airbyte Cloud API
• DB client libraries (psycopg2, duckdb, google-cloud-bigquery, snowflake-connector) for caches/processors
Final checklist
• All major modules from airbyte/ are represented and grouped logically.
• External dependencies (DBs, Cloud API, Docker daemon) are shown.
• Keep the layout readable: CLI at top or left, core engine central, external systems on the perimeter.