# instill-ai/pipeline-backend architecture

> Architecture diagram of the GitHub repository instill-ai/pipeline-backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/instill-ai/pipeline-backend
- Repository: https://github.com/instill-ai/pipeline-backend

## Overview

##### Main Architectural Components

a) Core Services:
- Pipeline Management Service
- Component Definition Service
- Integration Service
- Secret Management Service
- Worker Service
- Blob Storage Service

b) External Dependencies:
- Database (PostgreSQL)
- Message Queue System
- Blob Storage
- Authentication/Authorization Service

##### Component Interactions

**API Gateway -> Pipeline Backend -> Database**

- > Worker Service
- > Component Execution

- > Blob Storage

- > Component Registry

##### Key Architectural Patterns

a) Service Layer:
- Handler Layer (pkg/handler)
- Service Layer (pkg/service)
- Repository Layer (pkg/repository)

b) Component System:
- AI Components
- Data Components
- Application Components
- Operator Components

##### Data Flow Representation

Pipeline execution flow:
**Trigger -> Pipeline Service -> Worker**
- > Component1 -> Component2 -> Component3 -> Response

##### Detailed Component Structure

- Component Definition Structure
- Recipe Structure
- Pipeline DAG Structure

##### Additional Elements

Supporting elements:

a) Authentication/Authorization flow
b) Error handling paths
c) Monitoring and logging paths
d) Scaling components

##### Technical Details

Specific implementations:

a) Service Interfaces:
- Component execution
- Pipeline management
- Integration handling

b) Data Stores:
- Pipeline definitions
- Component configurations
- Execution logs

c) Worker Pool:
- Task distribution
- Component execution

##### Notes Section

A notes section describing:
- Scaling considerations
- Failure handling
- Security implementations
- Performance optimizations

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "External Services"
        DB[(PostgreSQL)]
        MQ[Message Queue]
        BS[Blob Storage]
        Auth[Auth Service]
    end

    subgraph "Core Services"
        direction TB
        API[API Gateway]
        
        subgraph "Service Layer"
            direction TB
            PMS[Pipeline Management Service]:::service
            CDS[Component Definition Service]:::service
            IS[Integration Service]:::service
            SMS[Secret Management Service]:::service
            WS[Worker Service]:::service
            BSS[Blob Storage Service]:::service
        end

        subgraph "Component System"
            direction TB
            AI[AI Components]:::component
            DATA[Data Components]:::component
            APP[Application Components]:::component
            OP[Operator Components]:::component
        end

        subgraph "Pipeline Execution"
            direction TB
            PRUN[Pipeline Run]:::execution
            COMP1[Component 1]:::execution
            COMP2[Component 2]:::execution
            COMP3[Component 3]:::execution
        end
    end

    API --> PMS
    API --> CDS
    API --> IS
    API --> SMS

    PMS --> DB
    CDS --> DB
    IS --> DB
    SMS --> DB

    PMS --> WS
    WS --> PRUN
    PRUN --> COMP1
    COMP1 --> COMP2
    COMP2 --> COMP3

    WS --> BS
    BSS --> BS

    WS --> AI
    WS --> DATA
    WS --> APP
    WS --> OP

    Auth -.-> API

    classDef service fill:#3498db
    classDef component fill:#9b59b6
    classDef execution fill:#2ecc71
    classDef external fill:#95a5a6

    %% Click events for component mapping
    click PMS "https://github.com/instill-ai/pipeline-backend/blob/main/pkg/service/pipeline.go"
    click CDS "https://github.com/instill-ai/pipeline-backend/blob/main/pkg/service/component_definition.go"
    click IS "https://github.com/instill-ai/pipeline-backend/blob/main/pkg/service/integration.go"
    click SMS "https://github.com/instill-ai/pipeline-backend/blob/main/pkg/service/secret.go"
    click WS "https://github.com/instill-ai/pipeline-backend/blob/main/pkg/worker/main.go"
    click BSS "https://github.com/instill-ai/pipeline-backend/blob/main/pkg/service/blobstorage.go"
    click AI "https://github.com/instill-ai/pipeline-backend/tree/main/pkg/component/ai"
    click DATA "https://github.com/instill-ai/pipeline-backend/tree/main/pkg/component/data"
    click APP "https://github.com/instill-ai/pipeline-backend/tree/main/pkg/component/application"
    click OP "https://github.com/instill-ai/pipeline-backend/tree/main/pkg/component/operator"
    click PRUN "https://github.com/instill-ai/pipeline-backend/blob/main/pkg/service/pipelinerun.go"
```
