# snowflakedb/snowflake-ingest-java architecture

> Architecture diagram of the GitHub repository snowflakedb/snowflake-ingest-java, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/snowflakedb/snowflake-ingest-java
- Repository: https://github.com/snowflakedb/snowflake-ingest-java

## Overview

##### Project Type and Purpose
- It is a Java client SDK (library) that enables applications to ingest data into Snowflake via two APIs:
1. Snowpipe (file-based ingest)
2. Snowpipe Streaming (row-by-row ingest)

##### Main Components
a. Client Application Layer
- Your end user’s Java app that depends on the SDK.
- Imports classes like SimpleIngestManager or SnowflakeStreamingIngestClient.
b. SDK Public API Layer
- SimpleIngestManager (for file-based ingest)
- SnowflakeStreamingIngestClient & SnowflakeStreamingIngestChannel (for streaming)
- Example helpers (IngestExampleHelper, StreamingIngestExample)
c. Connection & Authentication Layer
- JWTManager & OAuthManager – generate & manage tokens
- OAuthClient, OAuthCredential – handle key-pair or OAuth authentication
- SecurityManager – signs requests
- RequestBuilder – builds HTTP requests
d. Telemetry & Response Handling
- TelemetryService – collects usage metrics
- ServiceResponseHandler – parses HTTP responses, throws IngestResponseException
e. Internal Streaming Engine
- ChannelCache, ChannelFlushContext, FlushService, Flusher – manage buffers & flush cycles
- BlobBuilder, ChunkMetadata, ParquetRowBuffer, ParquetFlusher – format data, serialize to Parquet
- InternalStageManager / InternalStage – manage a staging area on Snowflake or external storage
- RegisterService – register staged blobs with Snowflake REST API
f. File Transfer Agent (extensibility point)
- IcebergFileTransferAgent & IStorageManager – abstract over S3, GCS, Azure, Iceberg
- Implementations: IcebergS3Client, IcebergGCSClient, IcebergAzureClient
g. External Integrations
- Snowflake Ingest REST Endpoint (Snowpipe & Streaming ingest APIs)
- Cloud Object Storage (AWS S3, GCS, Azure Blob) used as temporary staging

##### Relationships & Interactions
1. Client App calls SDK public APIs (SimpleIngestManager or SnowflakeStreamingIngestClient).
2. SDK API invokes Authentication Layer to obtain JWT/OAuth tokens.
3. RequestBuilder + SecurityManager + TelemetryService package HTTP requests.
4. For file-based ingest:
– Upload file to a Snowflake stage (via PUT REST call)
– Call “ingest files” REST endpoint
5. For streaming ingest:
– Buffer rows in Channel
– On flush trigger: serialize to Parquet (ParquetFlusher) → BlobBuilder
– FileTransferAgent uploads blob to external storage
– RegisterService calls “register blob” REST endpoint
– ServiceResponseHandler polls for commit via getLatestCommittedOffsetToken
6. All HTTP interactions go to Snowflake’s ingest REST servers; responses update SDK state.

##### Architectural Patterns & Design Principles
- **Layered Architecture:** clear separation between Public API, Connection/Auth, Internal Engine, I/O
- **Factory & Builder Patterns:** SnowflakeStreamingIngestClientFactory; RequestBuilder; BlobBuilder
- **Abstraction & Extension Points:** IStorageManager for any storage provider; pluggable file transfer agents
- **Immutability & Thread-Safety:** Internal buffers and channels are carefully synchronized (RingBuffer, ChannelRuntimeState)
- **Dependency Injection (minimal):** passing ParameterProvider and internal configurations

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Application Layer
    subgraph "Client Application Layer"
        direction TB
        IngestExampleHelper["IngestExampleHelper"] 
        SnowflakeIngestBasicExample["SnowflakeIngestBasicExample"]
        SnowflakeStreamingIngestExample["SnowflakeStreamingIngestExample"]
    end

    %% SDK Public API Layer
    subgraph "SDK Public API Layer"
        direction TB
        SimpleIngestManager["SimpleIngestManager"]:::public
        SnowflakeStreamingIngestClient["SnowflakeStreamingIngestClient"]:::public
        SnowflakeStreamingIngestChannel["SnowflakeStreamingIngestChannel"]:::public
        SnowflakeStreamingIngestClientFactory["SnowflakeStreamingIngestClientFactory"]:::public
    end

    %% Connection & Authentication Layer
    subgraph "Auth & Connection Layer"
        direction TB
        JWTManager["JWTManager"]:::auth
        OAuthManager["OAuthManager"]:::auth
        OAuthClient["OAuthClient"]:::auth
        OAuthCredential["OAuthCredential"]:::auth
        SecurityManager["SecurityManager"]:::auth
        RequestBuilder["RequestBuilder"]:::auth
    end

    %% Telemetry & Response Handling
    subgraph "Telemetry & Response Handling"
        direction TB
        TelemetryService["TelemetryService"]:::auth
        ServiceResponseHandler["ServiceResponseHandler"]:::auth
    end

    %% Internal Streaming Engine
    subgraph "Internal Streaming Engine"
        direction TB
        ChannelCache["ChannelCache"]:::internal
        ChannelFlushContext["ChannelFlushContext"]:::internal
        FlushService["FlushService"]:::internal
        Flusher["Flusher"]:::internal
        ParquetRowBuffer["ParquetRowBuffer"]:::internal
        ParquetFlusher["ParquetFlusher"]:::internal
        BlobBuilder["BlobBuilder"]:::internal
        ChunkMetadata["ChunkMetadata"]:::internal
        InternalStage["InternalStage"]:::internal
        InternalStageManager["InternalStageManager"]:::internal
        RegisterService["RegisterService"]:::internal
        IStorageManager["IStorageManager"]:::internal
        IcebergFileTransferAgent["IcebergFileTransferAgent"]:::internal
        IcebergS3Client["IcebergS3Client"]:::internal
        IcebergGCSClient["IcebergGCSClient"]:::internal
        IcebergAzureClient["IcebergAzureClient"]:::internal
    end

    %% External Systems
    SnowflakeAPI["Snowflake Ingest REST API"]:::external
    CloudStorage["Cloud Storage\n(S3, GCS, Azure)"]:::external

    %% Connections
    IngestExampleHelper -->|uses| SimpleIngestManager
    SnowflakeIngestBasicExample -->|uses| SimpleIngestManager
    SnowflakeStreamingIngestExample -->|uses| SnowflakeStreamingIngestClient

    SimpleIngestManager -->|authenticate| JWTManager
    SnowflakeStreamingIngestClient -->|authenticate| OAuthManager

    JWTManager --> OAuthClient
    OAuthManager --> OAuthCredential
    OAuthClient --> SecurityManager
    OAuthCredential --> SecurityManager

    SecurityManager --> RequestBuilder
    RequestBuilder --> TelemetryService
    TelemetryService --> ServiceResponseHandler

    %% File-based ingest flow
    SimpleIngestManager -->|PUT staged file| ServiceResponseHandler
    ServiceResponseHandler -->|POST /ingestFiles| SnowflakeAPI

    %% Streaming ingest flow
    SnowflakeStreamingIngestClient --> ChannelCache
    ChannelCache --> ChannelFlushContext
    ChannelFlushContext --> FlushService
    FlushService --> Flusher
    Flusher --> ParquetRowBuffer
    ParquetRowBuffer --> ParquetFlusher
    ParquetFlusher --> BlobBuilder
    BlobBuilder --> IcebergFileTransferAgent
    IcebergFileTransferAgent --> IStorageManager
    IStorageManager --> IcebergS3Client
    IStorageManager --> IcebergGCSClient
    IStorageManager --> IcebergAzureClient
    IcebergFileTransferAgent -->|upload blob| CloudStorage
    CloudStorage --> RegisterService
    RegisterService -->|POST /registerBlob| SnowflakeAPI
    SnowflakeAPI -->|GET /getLatestCommittedOffsetToken| ServiceResponseHandler

    %% Styles
    classDef public fill:#cce5ff,stroke:#004085,color:#004085;
    classDef auth fill:#fff3cd,stroke:#856404,color:#856404;
    classDef internal fill:#d4edda,stroke:#155724,color:#155724;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d;

    %% Click Events
    click IngestExampleHelper "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/example/IngestExampleHelper.java"
    click SnowflakeIngestBasicExample "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/example/SnowflakeIngestBasicExample.java"
    click SnowflakeStreamingIngestExample "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/example/SnowflakeStreamingIngestExample.java"
    click SimpleIngestManager "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/SimpleIngestManager.java"
    click SnowflakeStreamingIngestClient "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/SnowflakeStreamingIngestClient.java"
    click SnowflakeStreamingIngestChannel "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/SnowflakeStreamingIngestChannel.java"
    click SnowflakeStreamingIngestClientFactory "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/SnowflakeStreamingIngestClientFactory.java"
    click JWTManager "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/JWTManager.java"
    click OAuthManager "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/OAuthManager.java"
    click OAuthClient "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/OAuthClient.java"
    click OAuthCredential "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/OAuthCredential.java"
    click SecurityManager "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/SecurityManager.java"
    click RequestBuilder "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/RequestBuilder.java"
    click TelemetryService "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/TelemetryService.java"
    click ServiceResponseHandler "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/connection/ServiceResponseHandler.java"
    click ChannelCache "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/ChannelCache.java"
    click ChannelFlushContext "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/ChannelFlushContext.java"
    click FlushService "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/FlushService.java"
    click Flusher "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/Flusher.java"
    click BlobBuilder "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/BlobBuilder.java"
    click ChunkMetadata "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/ChunkMetadata.java"
    click ParquetRowBuffer "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/ParquetRowBuffer.java"
    click ParquetFlusher "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/ParquetFlusher.java"
    click InternalStage "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/InternalStage.java"
    click InternalStageManager "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/InternalStageManager.java"
    click RegisterService "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/RegisterService.java"
    click IStorageManager "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/IStorageManager.java"
    click IcebergFileTransferAgent "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergFileTransferAgent.java"
    click IcebergS3Client "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java"
    click IcebergGCSClient "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergGCSClient.java"
    click IcebergAzureClient "https://github.com/snowflakedb/snowflake-ingest-java/blob/master/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergAzureClient.java"
```
