# dyvenia/viadot architecture

> Architecture diagram of the GitHub repository dyvenia/viadot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dyvenia/viadot
- Repository: https://github.com/dyvenia/viadot

## Overview

##### Project type and purpose
- This is an open-source Python library named Viadot for orchestrating and simplifying ETL/data-ingestion pipelines.
- It exposes:
- “sources” modules to extract data from APIs or databases,
- “orchestration” modules built on Prefect (flows & tasks) to sequence work,
- utility functions, and destination connectors (ADLS, S3, SQL, Parquet, etc.).

##### File structure
- Top-level folders:
- src/viadot – the library’s source code
– sources/ – connector classes for each data source (BigQuery, HubSpot, SAP, etc.)
– orchestration/prefect/flows – Prefect flow definitions combining tasks end-to-end
– orchestration/prefect/tasks – individual Prefect tasks (read/write operations)
– utils.py, config.py, exceptions, signals
- docs/ – MkDocs-based documentation
- docker/ – container definitions for local/dev environment
- .github/workflows – CI/CD pipelines
- tests/ – unit & integration tests
- config.yaml.example, pyproject.toml – config & packaging

##### Read README for additional insights
- The library uses Rye for environment management
- Data extraction: instantiate a Source, run query(), call to_df()
- Data load: instantiate a Destination, call from_df()
- Prefect flows tie extraction, transformation, and loading

a. Main components
- User code / consumer application
- Viadot library
– Sources layer
– Transformation/Utils
– Orchestration (Prefect flows & tasks)
– Destination connectors
- External systems
– Data sources (APIs, RDBMS, files)
– Data stores (Azure Data Lake, S3, SQL Server, Parquet store)
– Prefect backend (optional)
– Docker environment (for containerized runs)

b. Relationships & interactions
1. User code invokes Viadot APIs to configure and launch flows.
2. Flow runner (Prefect) loads flow definitions from the orchestration module.
3. Within a flow:
– Source tasks read from external data source → produce DataFrame
– Optional transform tasks (duckdb, pandas)
– Destination tasks write to target sinks
4. Config is read from ~/.config/viadot/config.yaml or ENV vars.
5. Docker compose can bundle all dependencies (ODBC, Prefect, examples).

c. Architectural patterns & design principles
- Modular plugin architecture: each source and destination implemented as a class
- Declarative workflows: Prefect flows define data pipelines as DAGs
- Separation of concerns:
- Sources vs. tasks vs. flows vs. utilities
- Configuration abstracted away from code
- Test-driven: comprehensive unit & integration tests

d. Key technologies
- Python packaging (pyproject.toml, Rye)
- Prefect (orchestration engine)
- Pandas/DataFrame for in-memory transforms
- Azure SDK, SQL connectors, MinIO, DuckDB, BigQuery, Redshift Spectrum, etc.
- Docker for containerized examples

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User Application / CLI"]:::user
    ConfigService["Configuration Service\nreads config.yaml / ENV"]:::user

    %% Viadot Library
    subgraph "Viadot Library"
        direction TB
        Config["Config Module"]:::library
        SourcesLayer["Sources Layer"]:::library
        TransformUtils["General Utilities\n(src/viadot/utils.py)"]:::library
        PrefectUtils["Prefect Utilities\n(src/viadot/orchestration/prefect/utils.py)"]:::library
        PrefectTasks["Prefect Tasks"]:::library
        PrefectFlows["Prefect Flows"]:::library
        DestTasks["Destination Connectors"]:::library
        ExceptionsCore["Core Exceptions"]:::library
        ExceptionsPrefect["Prefect Exceptions"]:::library
        Signals["Signals"]:::library
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        DataSources["Data Sources\n(APIs, RDBMS, Files)"]:::external
        DataStores["Data Stores\n(ADLS, S3, SQL, Parquet)"]:::external
        PrefectBackend["Prefect Backend"]:::external
    end

    %% Infrastructure & Docs
    Docker["Docker Environment\ndefinitions"]:::infra
    Docs["Documentation Site"]:::infra
    CI["CI/CD Workflows"]:::infra
    Examples["Examples (SAP RFC)"]:::infra
    ConfigExample["config.yaml.example"]:::infra
    PackageDef["pyproject.toml"]:::infra
    Tests["Unit & Integration Tests"]:::infra

    %% Relationships
    User -->|"invokes"| PrefectFlows
    User -->|"uses"| ConfigService
    ConfigService -->|"reads config"| Config
    PrefectFlows -->|"uses tasks"| PrefectTasks
    PrefectFlows -->|"uses utils"| PrefectUtils
    PrefectTasks -->|"call sources"| SourcesLayer
    PrefectTasks -->|"use transforms"| TransformUtils
    PrefectTasks -->|"call destinations"| DestTasks
    SourcesLayer -->|"reads from"| DataSources
    DestTasks -->|"writes to"| DataStores
    PrefectFlows -->|"registers/schedules"| PrefectBackend

    %% Infrastructure connections
    Docker --> User
    Docker --> PrefectFlows
    Docs -.-> User
    CI -.-> PrefectFlows
    Examples --> User
    ConfigExample -.-> Config
    PackageDef -.-> Config
    Tests -.-> PrefectTasks
    Tests -.-> PrefectFlows

    %% Click Events
    click SourcesLayer "https://github.com/dyvenia/viadot/tree/main/src/viadot/sources/"
    click PrefectTasks "https://github.com/dyvenia/viadot/tree/main/src/viadot/orchestration/prefect/tasks/"
    click PrefectFlows "https://github.com/dyvenia/viadot/tree/main/src/viadot/orchestration/prefect/flows/"
    click TransformUtils "https://github.com/dyvenia/viadot/blob/main/src/viadot/utils.py"
    click PrefectUtils "https://github.com/dyvenia/viadot/blob/main/src/viadot/orchestration/prefect/utils.py"
    click Config "https://github.com/dyvenia/viadot/blob/main/src/viadot/config.py"
    click ExceptionsCore "https://github.com/dyvenia/viadot/blob/main/src/viadot/exceptions.py"
    click ExceptionsPrefect "https://github.com/dyvenia/viadot/blob/main/src/viadot/orchestration/prefect/exceptions.py"
    click Signals "https://github.com/dyvenia/viadot/blob/main/src/viadot/signals.py"
    click Examples "https://github.com/dyvenia/viadot/tree/main/src/viadot/examples/sap_rfc/"
    click Docs "https://github.com/dyvenia/viadot/tree/main/docs/"
    click Docker "https://github.com/dyvenia/viadot/tree/main/docker/"
    click CI "https://github.com/dyvenia/viadot/tree/main/.github/workflows/"
    click ConfigExample "https://github.com/dyvenia/viadot/blob/main/config.yaml.example"
    click PackageDef "https://github.com/dyvenia/viadot/blob/main/pyproject.toml"
    click Tests "https://github.com/dyvenia/viadot/tree/main/tests/"

    %% Styles
    classDef user fill:#f5b895,stroke:#333,stroke-width:1px
    classDef library fill:#add8e6,stroke:#333,stroke-width:1px
    classDef external fill:#a2f5a2,stroke:#333,stroke-width:1px
    classDef infra fill:#ffe599,stroke:#333,stroke-width:1px
```
