# zeus-deus/data-engineering_workflow architecture

> Architecture diagram of the GitHub repository zeus-deus/data-engineering_workflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zeus-deus/data-engineering_workflow
- Repository: https://github.com/zeus-deus/data-engineering_workflow

## Overview

##### Main Components
- Data Ingestion Module: Represent components that fetch raw data from external sources. This is implemented under src/data_ingestion (scripts like reasoning_errors_to_azure.py and reasoning_errors_to_mongodb.py). Show that raw data is pushed to either MongoDB or Azure Blob Storage.
- ETL Pipelines: Clearly indicate the ETL process present under src/pipelines/reasoning_errors (modules for extract, transform, and load). This pipeline moves data between storage solutions (e.g., MongoDB/Azure Blob for raw data and PostgreSQL/Azure PostgreSQL for structured data).
- Workflow Orchestration: The project uses Prefect for task automation and orchestration. Include the Prefect flows from the prefect/ directory (such as prefect_flow.py, prefect_vectorization_flow.py) that manage the execution and scheduling of tasks.
- Vectorization and RAG Integration: Highlight that there is a dedicated module (src/vectorization) for converting text data to vectors and a separate module under src/rag for integrating RAG-based question answering that leverages Qdrant and the OpenAI API.
- Containerization: Docker is used to create consistent, isolated environments. Use the docker directory and docker-compose.yml file to illustrate how multiple containers (application, MongoDB, PostgreSQL, etc.) interact.
- External Services: Identify integration points with external systems and APIs such as:
- Azure Services (Blob Storage and Azure PostgreSQL) – indicated by the environmental toggles.
- Qdrant – as a vector store, and the use of the OpenAI API for embedding generation.
- Other libraries/tools (BeautifulSoup, pandas, SQLAlchemy).

##### Relationships and Interactions
- ETL pipelines should be depicted as transforming data from one format/location (raw in MongoDB/Azure Blob) to processed data stored in PostgreSQL or Azure PostgreSQL.
- Dependencies between the Prefect orchestrator and individual pipelines, hinting at monitoring and automation.
- Integration between the vectorization component and the RAG module where processed or vectorized data is sent to Qdrant for enabling advanced query matching.
- That Docker components encapsulate many of these services to allow for uniform deployment across environments.

##### Architectural Patterns and Design Principles
- Modularity: The separation of data ingestion, ETL, vectorization, and workflow orchestration.
- Decoupling: The common utilities and configuration management (e.g., reading environment variables to switch between local and cloud databases).
- Containerization: Showing services as isolated containers that communicate via defined interfaces.
- Orchestration: The use of Prefect to manage and monitor pipeline execution, which enforces a clear separation of tasks and error handling.
- Extensibility: The structure supports plugging in additional data sources or additional pipeline stages with minimal changes.

##### Relevant Technologies
- Python and its libraries (pandas, BeautifulSoup, SQLAlchemy).
- NoSQL (MongoDB) and RDBMS (PostgreSQL / Azure PostgreSQL).
- Cloud services (Azure Blob Storage).
- Prefect for orchestration.
- Qdrant and OpenAI API for vectorization and RAG.
- Docker and Docker Compose for containerizing and managing multi-container setups.

##### Detail and Clarity
- Every logical unit and interaction is represented as accurately as possible.
- Don’t shy away from decomposing the project into many small blocks if that accurately reflects the modular nature of the architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Cluster
    subgraph "Data Ingestion"
        DI["Data Ingestion Module"]:::ingestion
        subgraph "Raw Data Storage"
            M["MongoDB"]:::storage
            ABS["Azure Blob Storage"]:::storage
        end
    end

    %% ETL Pipelines Cluster
    subgraph "ETL Pipelines"
        EX["Extract"]:::etl
        TR["Transform"]:::etl
        LD["Load"]:::etl
        EX --> TR
        TR --> LD
        ST["Structured Storage (PostgreSQL/Azure PostgreSQL)"]:::storage
    end

    %% Common Utilities Cluster
    subgraph "Common Utilities"
        CU["Common Utilities"]:::utility
    end

    %% Workflow Orchestration Cluster
    subgraph "Workflow Orchestration (Prefect)"
        PDI["Prefect Data Ingestion Flow"]:::orchestration
        PP["Prefect Pipeline Flow"]:::orchestration
        PV["Prefect Vectorization Flow"]:::orchestration
    end

    %% Vectorization and RAG Integration Cluster
    subgraph "Vectorization & RAG"
        VM["Vectorization Module"]:::vector
        RI["RAG Integration"]:::rag
        QD["Qdrant"]:::external
        OAI["OpenAI API"]:::external
    end

    %% Containerization Services Cluster
    subgraph "Containerization Services"
        CS["Containerization Services"]:::container
    end

    %% External Source for Raw Data
    RDS["Raw Data Source"]:::external

    %% Data Flow Connections
    RDS -->|"ingests"| DI
    DI --> M
    DI --> ABS
    M -->|"raw data"| EX
    ABS -->|"raw data"| EX
    EX -->|"processing"| TR
    TR -->|"processing"| LD
    LD -->|"structured data"| ST
    CU --- M
    CU --- ST
    PDI -->|"triggers"| DI
    PP -->|"schedules"| EX
    ST -->|"feeds"| VM
    PV -->|"triggers"| VM
    VM -->|"vector output"| QD
    QD -->|"query-match"| RI
    RI -->|"API calls"| OAI

    %% Containerization connects to all internal components
    CS --- DI
    CS --- EX
    CS --- CU
    CS --- PDI
    CS --- PP
    CS --- PV
    CS --- VM
    CS --- RI

    %% Styles
    classDef ingestion fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef etl fill:#A9DFBF,stroke:#145A32,stroke-width:2px;
    classDef utility fill:#F9E79F,stroke:#7D6608,stroke-width:2px;
    classDef orchestration fill:#F5B7B1,stroke:#943126,stroke-width:2px;
    classDef vector fill:#E8DAEF,stroke:#6C3483,stroke-width:2px;
    classDef rag fill:#D2B4DE,stroke:#512E5F,stroke-width:2px;
    classDef container fill:#D5DBDB,stroke:#566573,stroke-width:2px;
    classDef storage fill:#F7DC6F,stroke:#B7950B,stroke-width:2px;
    classDef external fill:#FAD7A0,stroke:#CA6F1E,stroke-width:2px;

    %% Click Events
    click DI "https://github.com/zeus-deus/data-engineering_workflow/tree/main/src/data_ingestion"
    click EX "https://github.com/zeus-deus/data-engineering_workflow/tree/main/src/pipelines/reasoning_errors"
    click CU "https://github.com/zeus-deus/data-engineering_workflow/tree/main/src/common"
    click PDI "https://github.com/zeus-deus/data-engineering_workflow/blob/main/prefect/reasoning_errors/prefect_data_ingestion_flow.py"
    click PP "https://github.com/zeus-deus/data-engineering_workflow/blob/main/prefect/reasoning_errors/prefect_flow.py"
    click PV "https://github.com/zeus-deus/data-engineering_workflow/blob/main/prefect/vectorization/prefect_vectorization_flow.py"
    click VM "https://github.com/zeus-deus/data-engineering_workflow/tree/main/src/vectorization"
    click RI "https://github.com/zeus-deus/data-engineering_workflow/tree/main/src/rag"
    click CS "https://github.com/zeus-deus/data-engineering_workflow/tree/main/docker"
```
