How snowflakedb/snowflake-ingest-java works
Project Type and Purpose
• It is a Java client SDK (library) that enables applications to ingest data into Snowflake via two APIs:
1. Snowpipe (file-based ingest)
2. Snowpipe Streaming (row-by-row ingest)
Main Components
a. Client Application Layer
• Your end user’s Java app that depends on the SDK.
• Imports classes like SimpleIngestManager or SnowflakeStreamingIngestClient.
b. SDK Public API Layer
• SimpleIngestManager (for file-based ingest)
• SnowflakeStreamingIngestClient & SnowflakeStreamingIngestChannel (for streaming)
• Example helpers (IngestExampleHelper, StreamingIngestExample)
c. Connection & Authentication Layer
• JWTManager & OAuthManager – generate & manage tokens
• OAuthClient, OAuthCredential – handle key-pair or OAuth authentication
• SecurityManager – signs requests
• RequestBuilder – builds HTTP requests
d. Telemetry & Response Handling
• TelemetryService – collects usage metrics
• ServiceResponseHandler – parses HTTP responses, throws IngestResponseException
e. Internal Streaming Engine
• ChannelCache, ChannelFlushContext, FlushService, Flusher – manage buffers & flush cycles
• BlobBuilder, ChunkMetadata, ParquetRowBuffer, ParquetFlusher – format data, serialize to Parquet
• InternalStageManager / InternalStage – manage a staging area on Snowflake or external storage
• RegisterService – register staged blobs with Snowflake REST API
f. File Transfer Agent (extensibility point)
• IcebergFileTransferAgent & IStorageManager – abstract over S3, GCS, Azure, Iceberg
• Implementations: IcebergS3Client, IcebergGCSClient, IcebergAzureClient
g. External Integrations
• Snowflake Ingest REST Endpoint (Snowpipe & Streaming ingest APIs)
• Cloud Object Storage (AWS S3, GCS, Azure Blob) used as temporary staging
Relationships & Interactions
1. Client App calls SDK public APIs (SimpleIngestManager or SnowflakeStreamingIngestClient).
2. SDK API invokes Authentication Layer to obtain JWT/OAuth tokens.
3. RequestBuilder + SecurityManager + TelemetryService package HTTP requests.
4. For file-based ingest:
– Upload file to a Snowflake stage (via PUT REST call)
– Call “ingest files” REST endpoint
5. For streaming ingest:
– Buffer rows in Channel
– On flush trigger: serialize to Parquet (ParquetFlusher) → BlobBuilder
– FileTransferAgent uploads blob to external storage
– RegisterService calls “register blob” REST endpoint
– ServiceResponseHandler polls for commit via getLatestCommittedOffsetToken
6. All HTTP interactions go to Snowflake’s ingest REST servers; responses update SDK state.
Architectural Patterns & Design Principles
• Layered Architecture: clear separation between Public API, Connection/Auth, Internal Engine, I/O
• Factory & Builder Patterns: SnowflakeStreamingIngestClientFactory; RequestBuilder; BlobBuilder
• Abstraction & Extension Points: IStorageManager for any storage provider; pluggable file transfer agents
• Immutability & Thread-Safety: Internal buffers and channels are carefully synchronized (RingBuffer, ChannelRuntimeState)
• Dependency Injection (minimal): passing ParameterProvider and internal configurations