# dalgot4d/ddp_backend architecture

> Architecture diagram of the GitHub repository dalgot4d/ddp_backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dalgot4d/ddp_backend
- Repository: https://github.com/dalgot4d/ddp_backend

## Overview

##### Key Architectural Patterns to Highlight

a. Service-Oriented Architecture:
- Clear separation between services (ddpairbyte, ddpdbt, ddpprefect)
- API boundaries between components

b. Task Processing:
- Celery worker flow for async operations
- WebSocket connections for real-time updates

c. Data Flow Patterns:
- ETL/ELT workflows
- Data transformation pipelines
- Notification systems

##### Component Relationships

- API calls between components
- Data flow directions
- Authentication/Authorization flows
- Task delegation paths

##### Detailed Breakdown

Create sub-sections showing:
a. API Layer:
- REST endpoints structure
- WebSocket connections
- Authentication middleware

b. Core Services:
- Data insights engine
- Transformation service
- Notification system
- Organization management

c. Integration Points:
- Airbyte connectors
- dbt project management
- Prefect workflow orchestration

##### Additional Elements

Include:
- Version information
- Security boundaries
- Scaling components
- Error handling flows

##### Key Features to Emphasize

Highlight:
- Multi-tenant architecture
- Role-based access control
- Async task processing
- Data pipeline management
- Integration with external tools

##### Technical Details

Annotations for:
- Technology stack (Python 3.10, Django, etc.)
- Communication protocols (REST, WebSocket)
- Database schemas
- Configuration management

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph External["External Services Layer"]
        direction TB
        AB["Airbyte"]:::external
        PF["Prefect"]:::external
        DBT["dbt"]:::external
        AWS["AWS Services"]:::external
    end

    subgraph Core["Core Application Layer"]
        direction TB
        Django["Django App"]:::core
        Celery["Celery Workers & Beat"]:::core
        API["API Endpoints"]:::core
        WS["WebSocket Consumers"]:::core
        
        subgraph CoreServices["Core Services"]
            direction TB
            DI["Data Insights Engine"]:::service
            TS["Transformation Service"]:::service
            NS["Notification System"]:::service
            OM["Organization Management"]:::service
        end
        
        subgraph Auth["Authentication & Authorization"]
            direction TB
            AS["Auth System"]:::security
            RBAC["Role-based Access"]:::security
        end
    end

    subgraph Storage["Storage Layer"]
        direction TB
        PG[(PostgreSQL)]:::storage
        RD[(Redis)]:::storage
        FS["File Storage"]:::storage
    end

    %% Relationships
    External --> Core
    Core --> Storage
    Celery --> RD
    Django --> PG
    WS --> RD
    
    %% Click events
    click Django "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/settings.py"
    click Celery "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/celery.py"
    click API "https://github.com/DalgoT4D/DDP_backend/tree/main/ddpui/api"
    click WS "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/websockets/airbyte_consumer.py"
    click AB "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/ddpairbyte/airbyte_service.py"
    click PF "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/ddpprefect/prefect_service.py"
    click DBT "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/ddpdbt/dbt_service.py"
    click AWS "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/utils/awsses.py"
    click DI "https://github.com/DalgoT4D/DDP_backend/tree/main/ddpui/datainsights"
    click TS "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/core/transformfunctions.py"
    click NS "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/core/notifications_service.py"
    click OM "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/core/orgfunctions.py"
    click AS "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/auth.py"
    click RBAC "https://github.com/DalgoT4D/DDP_backend/blob/main/ddpui/models/role_based_access.py"

    %% Styling
    classDef core fill:#2374ab
    classDef external fill:#57a773
    classDef storage fill:#ffc857
    classDef service fill:#7d4e57
    classDef security fill:#d81159

    %% Legend
    subgraph Legend
        direction TB
        C1[Core Components]:::core
        C2[External Services]:::external
        C3[Storage Systems]:::storage
        C4[Internal Services]:::service
        C5[Security Components]:::security
    end
```
