# datayoga-io/datayoga architecture

> Architecture diagram of the GitHub repository datayoga-io/datayoga, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datayoga-io/datayoga
- Repository: https://github.com/datayoga-io/datayoga

## Overview

##### Main Components Identification

The system should be represented with these key components:

a) Core Runtime Engine:
- Job Executor
- Block Manager
- Connection Manager
- Expression Engine
- Processing Strategies Handler

b) Block Types (Group by category):
- Input Blocks (read operations)
- Transformation Blocks
- Output Blocks (write operations)
- Utility Blocks

c) External Systems Integration:
- Databases (Relational & NoSQL)
- Message Queues
- File Systems
- Cloud Services
- HTTP Services

**┌─ Application Layer ─────────────────────┐**
- CLI Interface                           │
- Configuration Management                │
- ┬─┘
**┌─ Core Processing Layer ───────────────┴─┐**
- ┌─ Job Execution Engine ─┐              │
- Job Validator        │              │
- Step Orchestrator    │              │
- Block Manager        │              │
- ┘              │
- ┌─ Processing Strategies ┐              │
- Async Processing     │              │
- Multi-threading      │              │
- Multi-processing     │              │
- ┘              │
- ┬─┘
**┌─ Block Layer ────────────────────────┴─┐**
- ┌─ Input ─┐ ┌─ Transform ┐ ┌─ Output ┐ │
- CSV     │ │ Filter     │ │ Redis   │ │
- Redis   │ │ Map        │ │ DB      │ │
- DB      │ │ Template   │ │ HTTP    │ │
- ┘ └───────────┘ └─────────┘ │
- ┬─┘
**┌─ Integration Layer ──────────────────┴─┐**
- Database Connectors                    │
- Message Queue Adapters                 │
- File System Handlers                   │
- External Service Clients               │
- ┘

##### Key Relationships

- Data flow between steps in a job
- Block interaction with external systems
- Configuration management flow
- Error handling and monitoring paths
- Processing strategy implementation

##### Additional Details

Supporting elements:
- Metrics/Monitoring integration
- Configuration validation flow
- Error handling paths
- Resource management
- Security boundaries

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Application
        CLI["CLI Interface"]
        Config["Configuration Management"]
    end

    subgraph Core
        JE["Job Executor"]
        BM["Block Manager"]
        CM["Connection Manager"]
        EE["Expression Engine"]
        SO["Step Orchestrator"]
        
        subgraph ProcessingStrategies
            AP["Async Processing"]
            MT["Multi-threading"]
            MP["Multi-processing"]
        end
    end

    subgraph Blocks
        subgraph InputBlocks
            CSV["CSV Reader"]
            RedisRead["Redis Stream Reader"]
            DBRead["Database Reader"]
        end

        subgraph TransformBlocks
            Filter["Filter"]
            Map["Map"]
            Template["Jinja Template"]
        end

        subgraph OutputBlocks
            RedisWrite["Redis Writer"]
            DBWrite["Database Writer"]
            HTTPWrite["HTTP Writer"]
        end
    end

    subgraph Integration
        DB[(Database)]
        MQ[(Message Queue)]
        Cloud[(Cloud Services)]
        HTTP[(HTTP Services)]
    end

    subgraph Monitoring
        Metrics["Prometheus Metrics"]
    end

    %% Relationships
    CLI --> Config
    Config --> JE
    JE --> BM
    JE --> CM
    JE --> EE
    JE --> SO

    BM --> InputBlocks
    BM --> TransformBlocks
    BM --> OutputBlocks

    DBRead --> DB
    DBWrite --> DB
    RedisRead --> MQ
    RedisWrite --> MQ
    HTTPWrite --> HTTP

    SO --> ProcessingStrategies
    JE --> Metrics

    %% Styling
    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px
    classDef core fill:#b3e0ff,stroke:#333
    classDef input fill:#90EE90,stroke:#333
    classDef transform fill:#FFE5B4,stroke:#333
    classDef output fill:#FFB6C1,stroke:#333
    classDef external fill:#E6E6FA,stroke:#333
    classDef monitoring fill:#F0E68C,stroke:#333

    class JE,BM,CM,EE,SO,AP,MT,MP core
    class CSV,RedisRead,DBRead input
    class Filter,Map,Template transform
    class RedisWrite,DBWrite,HTTPWrite output
    class DB,MQ,Cloud,HTTP external
    class Metrics monitoring

    %% Click events
    click JE "https://github.com/datayoga-io/datayoga/blob/main/core/src/datayoga_core/job.py"
    click BM "https://github.com/datayoga-io/datayoga/blob/main/core/src/datayoga_core/block.py"
    click CM "https://github.com/datayoga-io/datayoga/blob/main/core/src/datayoga_core/connection.py"
    click EE "https://github.com/datayoga-io/datayoga/blob/main/core/src/datayoga_core/expression.py"
    click SO "https://github.com/datayoga-io/datayoga/blob/main/core/src/datayoga_core/step.py"
    click CSV "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/files/read_csv/"
    click RedisRead "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/redis/read_stream/"
    click DBRead "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/relational/read/"
    click Filter "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/filter/"
    click Map "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/map/"
    click Template "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/jinja_template/"
    click RedisWrite "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/redis/write/"
    click DBWrite "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/relational/write/"
    click HTTPWrite "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/http/write/"
    click DB "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/relational/"
    click MQ "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/redis/"
    click Cloud "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/azure/"
    click HTTP "https://github.com/datayoga-io/datayoga/tree/main/core/src/datayoga_core/blocks/http/"
    click Config "https://github.com/datayoga-io/datayoga/blob/main/core/src/datayoga_core/resources/schemas/job.schema.json"
    click CLI "https://github.com/datayoga-io/datayoga/blob/main/cli/src/datayoga/__main__.py"
    click Metrics "https://github.com/datayoga-io/datayoga/blob/main/core/src/datayoga_core/prometheus.py"
```
