# viveknaskar/google-dataflow-redis-example architecture

> Architecture diagram of the GitHub repository viveknaskar/google-dataflow-redis-example, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/viveknaskar/google-dataflow-redis-example
- Repository: https://github.com/viveknaskar/google-dataflow-redis-example

## Overview

##### Main Components
- Input Source: Google Cloud Storage bucket (where the *.txt files are stored).
- Pipeline Execution: The Dataflow pipeline, executed by the StarterPipeline class. This component includes multiple processing steps (transformation, validation, logging, sorting, etc.) as represented by the functions classes (e.g., ProcessingRecords, TransformingData, RedisHashIO) and constants that configure those steps.
- Data Output: Memorystore (Redis) instance, where the processed data is stored.
- Infrastructure Setup: The pipeline uses Maven (via pom.xml) for build and dependency management.
- External Access: Compute Engine VM used to telnet into the Redis instance for checking keys.

##### Component Interactions
- The data flow starts from reading input files from Cloud Storage.
- Data is fed into the Dataflow pipeline where multiple transformation stages take place. For example:
- "ProcessingRecords" and "ProcessingPPID" handle record-level processing.
- "TransformingData" and "CombiningTransforms" perform necessary data transformations.
- "RedisHashIO" handles the custom HSET implementation to write data into Redis.
- Various logging functions (e.g., LogValidRecords, LogInvalidRecords) monitor pipeline progress.
- Data finally flows into Memorystore (Redis) for storage.
- Separately, the user (or administrator) connects from a Compute Engine VM to confirm that data has been inserted into Redis.

##### Architectural Patterns and Design Principles
- Modular Design: The separation into functions packages indicates that the pipeline is designed in modular stages, each with a single responsibility (transforming, processing, logging, etc.).
- Dataflow Pattern: The architecture reflects a streaming or batch processing pipeline where data is processed in stages from input to output.
- External Integration: Clearly delimited interactions with external systems (Cloud Storage for inputs, Redis for outputs, and Compute Engine for diagnostic access).

##### Relevant Technologies/Frameworks
- Google Cloud Dataflow as the runner for the pipeline.
- Google Cloud Storage as the data source.
- Google Memorystore (Redis 5.0) as the data destination.
- Maven for project dependency management and execution.
- Java as the programming language for pipeline implementation.

##### Elements
- Clear Labels: Name each component such as “Cloud Storage (Input Files)”, “Dataflow Pipeline” (with its sub-stages if required), “Memorystore - Redis”, and “Compute Engine VM (Redis Checker)”.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source Layer
    subgraph "Data Source Layer"
        A["Cloud Storage (Input Files)"]:::cloud
    end

    %% Infrastructure Setup
    subgraph "Infrastructure Setup"
        I1["pom.xml"]:::infra
        I2["Constants & Config"]:::infra
    end

    %% Dataflow Pipeline
    subgraph "Dataflow Pipeline"
        P0["StarterPipeline"]:::pipeline
        subgraph "Processing Functions"
            P1["ProcessingRecords"]:::pipeline
            P2["ProcessingPPID"]:::pipeline
            P3["TransformingData"]:::pipeline
            P4["CombiningTransforms"]:::pipeline
            P5["SortRecords"]:::pipeline
        end
        P6["RedisHashIO"]:::pipeline
        subgraph "Logging Functions"
            L1["LogValidRecords"]:::logging
            L2["LogInvalidRecords"]:::logging
            L3["LogProcessDetails"]:::logging
            L4["LogTotalRecords"]:::logging
        end
    end

    %% Data Output Layer
    subgraph "Data Output Layer"
        D["Memorystore - Redis"]:::cloud
    end

    %% Monitoring Layer
    subgraph "Monitoring"
        M["Compute Engine VM (Redis Checker)"]:::monitor
    end

    %% Data Flow Connections
    A -->|"reads data"| P0

    I1 -->|"build config"| P0
    I2 -->|"run config"| P0

    P0 -->|"processes"| P1
    P1 -->|"further processing"| P2
    P2 -->|"transforms"| P3
    P3 -->|"combines"| P4
    P4 -->|"sorts"| P5
    P5 -->|"prepares for storage"| P6

    P6 -->|"stores data"| D

    %% Logging connections from StarterPipeline
    P0 -->|"logs"| L1
    P0 -->|"logs"| L2
    P0 -->|"logs"| L3
    P0 -->|"logs"| L4

    %% Monitoring connection
    M -->|"checks"| D

    %% Click Events
    click A "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/sampleinput.txt"
    click I1 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/pom.xml"
    click I2 "https://github.com/viveknaskar/google-dataflow-redis-example/tree/master/src/main/java/com/viveknaskar/constants/"
    click P0 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/StarterPipeline.java"
    click P1 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/ProcessingRecords.java"
    click P2 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/ProcessingPPID.java"
    click P3 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/TransformingData.java"
    click P4 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/CombiningTransforms.java"
    click P5 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/SortRecords.java"
    click L1 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/LogValidRecords.java"
    click L2 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/LogInvalidRecords.java"
    click L3 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/LogProcessDetails.java"
    click L4 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/LogTotalRecords.java"
    click P6 "https://github.com/viveknaskar/google-dataflow-redis-example/blob/master/src/main/java/com/viveknaskar/functions/RedisHashIO.java"

    %% Style Definitions
    classDef cloud fill:#f1c232,stroke:#000,stroke-width:1px;
    classDef pipeline fill:#6fa8dc,stroke:#000,stroke-width:1px;
    classDef logging fill:#ffe599,stroke:#000,stroke-width:1px;
    classDef infra fill:#b7e1cd,stroke:#000,stroke-width:1px;
    classDef monitor fill:#d5a6bd,stroke:#000,stroke-width:1px;
```
