# instill-ai/artifact-backend architecture

> Architecture diagram of the GitHub repository instill-ai/artifact-backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/instill-ai/artifact-backend
- Repository: https://github.com/instill-ai/artifact-backend

## Overview

##### Main Components

a. Service Layer Components:
- Public API Service (gRPC/HTTP endpoints)
- Management Service
- Pipeline Service
- Retrieval Service
- Permission Service
- Usage Service

b. Storage Components:
- PostgreSQL Database (evident from migrations)
- Milvus Vector Database (for embeddings)
- MinIO Object Storage (for file storage)

c. Worker Components:
- Persistent Catalog Worker
- Temporary Catalog Worker

d. Supporting Infrastructure:
- OpenTelemetry Integration
- Logging System
- ACL (Access Control Layer)

##### Client Layer
- Interceptor/middleware layer for request handling

##### Service Layer
- Handlers for different functionalities (catalog, chunks, knowledge base, etc.)

##### Processing Layer
- Workers for handling catalog operations
- Pipeline processing flow

##### Storage Layer
- Databases and object storage systems
- Repository interfaces connecting to storage

##### External Services Layer
- External integrations and clients

##### Connection and Flow Representation

- Request flow from clients through services
- Data flow between services and storage
- Worker interaction with storage and services
- Event flows for asynchronous operations

- Blue: Core services
- Green: Storage systems
- Orange: Workers and processing components
- Purple: External services
- Gray: Supporting infrastructure

##### Important Details to Highlight

a. Data Processing Pipeline:
- Flow of unstructured data input
- Transformation processes
- Creation of Instill Catalog format

b. Storage Patterns:
- Demonstrate how files, embeddings, and metadata are stored
- Relationship between different storage systems

c. Security Layer:
- ACL implementation
- Authentication/authorization flow

##### Scaling and Performance

Indicate:
- Horizontal scaling points
- Caching layers if present
- Load balancing possibilities

Remember to include version information and any relevant deployment considerations based on the Dockerfile and configuration files present in the repository.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Style definitions
    classDef client fill:#E1F5FE,stroke:#01579B
    classDef service fill:#E8F5E9,stroke:#1B5E20
    classDef storage fill:#FFF3E0,stroke:#E65100
    classDef worker fill:#F3E5F5,stroke:#4A148C
    classDef infra fill:#ECEFF1,stroke:#263238
    classDef handler fill:#E0F2F1,stroke:#004D40
    classDef repo fill:#FBE9E7,stroke:#BF360C

    %% Client Layer
    subgraph Clients
        GRPC["gRPC Client"]:::client
        HTTP["HTTP Client"]:::client
        INT["Request Interceptors"]:::client
    end

    click GRPC "https://github.com/instill-ai/artifact-backend/tree/main/pkg/client/grpc/"
    click HTTP "https://github.com/instill-ai/artifact-backend/tree/main/pkg/client/http/"
    click INT "https://github.com/instill-ai/artifact-backend/blob/main/pkg/middleware/interceptor.go"

    %% Service Layer
    subgraph Services
        MS["Management Service"]:::service
        PS["Pipeline Service"]:::service
        RS["Retrieval Service"]:::service
        PMS["Permission Service"]:::service
        US["Usage Service"]:::service
    end

    click MS "https://github.com/instill-ai/artifact-backend/blob/main/pkg/service/mgmt.go"
    click PS "https://github.com/instill-ai/artifact-backend/blob/main/pkg/service/pipeline.go"
    click RS "https://github.com/instill-ai/artifact-backend/blob/main/pkg/service/retrieval.go"
    click PMS "https://github.com/instill-ai/artifact-backend/blob/main/pkg/service/permission.go"
    click US "https://github.com/instill-ai/artifact-backend/blob/main/pkg/usage/usage.go"

    %% Handler Layer
    subgraph Handlers
        CH["Catalog Handler"]:::handler
        KH["Knowledge Base Handler"]:::handler
        CKH["Chunks Handler"]:::handler
        RH["Retrieval Handler"]:::handler
    end

    click CH "https://github.com/instill-ai/artifact-backend/blob/main/pkg/handler/catalog.go"
    click KH "https://github.com/instill-ai/artifact-backend/blob/main/pkg/handler/knowledgebase.go"
    click CKH "https://github.com/instill-ai/artifact-backend/blob/main/pkg/handler/chunks.go"
    click RH "https://github.com/instill-ai/artifact-backend/blob/main/pkg/handler/retrieval.go"

    %% Worker Layer
    subgraph Workers
        PCW["Persistent Catalog Worker"]:::worker
        TCW["Temporary Catalog Worker"]:::worker
    end

    click PCW "https://github.com/instill-ai/artifact-backend/blob/main/pkg/worker/persistentcatalogworker.go"
    click TCW "https://github.com/instill-ai/artifact-backend/blob/main/pkg/worker/tempcatalogworker.go"

    %% Repository Layer
    subgraph Repositories
        KR["Knowledge Base Repository"]:::repo
        ER["Embedding Repository"]:::repo
        CR["Chunk Repository"]:::repo
        OR["Object Repository"]:::repo
    end

    click KR "https://github.com/instill-ai/artifact-backend/blob/main/pkg/repository/knowledgebase.go"
    click ER "https://github.com/instill-ai/artifact-backend/blob/main/pkg/repository/embedding.go"
    click CR "https://github.com/instill-ai/artifact-backend/blob/main/pkg/repository/chunk.go"
    click OR "https://github.com/instill-ai/artifact-backend/blob/main/pkg/repository/object.go"

    %% Storage Layer
    subgraph Storage
        PG[(PostgreSQL)]:::storage
        MV[(Milvus)]:::storage
        MO[(MinIO)]:::storage
    end

    click PG "https://github.com/instill-ai/artifact-backend/blob/main/pkg/db/db.go"
    click MV "https://github.com/instill-ai/artifact-backend/blob/main/pkg/milvus/milvus.go"
    click MO "https://github.com/instill-ai/artifact-backend/blob/main/pkg/minio/minio.go"

    %% Infrastructure Layer
    subgraph Infrastructure
        OT["OpenTelemetry"]:::infra
        LOG["Logging System"]:::infra
        ACL["Access Control Layer"]:::infra
        CFG["Configuration"]:::infra
    end

    click OT "https://github.com/instill-ai/artifact-backend/tree/main/pkg/logger/otel/"
    click LOG "https://github.com/instill-ai/artifact-backend/blob/main/pkg/logger/logger.go"
    click ACL "https://github.com/instill-ai/artifact-backend/blob/main/pkg/acl/acl.go"
    click CFG "https://github.com/instill-ai/artifact-backend/blob/main/config/config.go"

    %% Connections
    GRPC & HTTP --> INT
    INT --> MS & PS & RS & PMS & US
    MS & PS & RS & PMS & US --> CH & KH & CKH & RH
    CH & KH & CKH & RH --> PCW & TCW
    CH & KH & CKH & RH --> KR & ER & CR & OR
    KR & ER & CR & OR --> PG & MV & MO
    PCW & TCW --> PG & MV & MO

    %% Infrastructure Connections
    OT -.-> MS & PS & RS & PMS & US
    LOG -.-> MS & PS & RS & PMS & US
    ACL -.-> INT
    CFG -.-> MS & PS & RS & PMS & US

    %% Legend
    subgraph Legend
        C1["Client Components"]:::client
        S1["Service Components"]:::service
        ST1["Storage Components"]:::storage
        W1["Worker Components"]:::worker
        I1["Infrastructure Components"]:::infra
        H1["Handler Components"]:::handler
        R1["Repository Components"]:::repo
    end
```
