How dyvenia/viadot works
Project type and purpose
• This is an open-source Python library named Viadot for orchestrating and simplifying ETL/data-ingestion pipelines.
• It exposes:
• “sources” modules to extract data from APIs or databases,
• “orchestration” modules built on Prefect (flows & tasks) to sequence work,
• utility functions, and destination connectors (ADLS, S3, SQL, Parquet, etc.).
File structure
• Top-level folders:
• src/viadot – the library’s source code
– sources/ – connector classes for each data source (BigQuery, HubSpot, SAP, etc.)
– orchestration/prefect/flows – Prefect flow definitions combining tasks end-to-end
– orchestration/prefect/tasks – individual Prefect tasks (read/write operations)
– utils.py, config.py, exceptions, signals
• docs/ – MkDocs-based documentation
• docker/ – container definitions for local/dev environment
• .github/workflows – CI/CD pipelines
• tests/ – unit & integration tests
• config.yaml.example, pyproject.toml – config & packaging
Read README for additional insights
• The library uses Rye for environment management
• Data extraction: instantiate a Source, run query(), call to_df()
• Data load: instantiate a Destination, call from_df()
• Prefect flows tie extraction, transformation, and loading
a. Main components
• User code / consumer application
• Viadot library
– Sources layer
– Transformation/Utils
– Orchestration (Prefect flows & tasks)
– Destination connectors
• External systems
– Data sources (APIs, RDBMS, files)
– Data stores (Azure Data Lake, S3, SQL Server, Parquet store)
– Prefect backend (optional)
– Docker environment (for containerized runs)
b. Relationships & interactions
1. User code invokes Viadot APIs to configure and launch flows.
2. Flow runner (Prefect) loads flow definitions from the orchestration module.
3. Within a flow:
– Source tasks read from external data source → produce DataFrame
– Optional transform tasks (duckdb, pandas)
– Destination tasks write to target sinks
4. Config is read from ~/.config/viadot/config.yaml or ENV vars.
5. Docker compose can bundle all dependencies (ODBC, Prefect, examples).
c. Architectural patterns & design principles
• Modular plugin architecture: each source and destination implemented as a class
• Declarative workflows: Prefect flows define data pipelines as DAGs
• Separation of concerns:
• Sources vs. tasks vs. flows vs. utilities
• Configuration abstracted away from code
• Test-driven: comprehensive unit & integration tests
d. Key technologies
• Python packaging (pyproject.toml, Rye)
• Prefect (orchestration engine)
• Pandas/DataFrame for in-memory transforms
• Azure SDK, SQL connectors, MinIO, DuckDB, BigQuery, Redshift Spectrum, etc.
• Docker for containerized examples