How viveknaskar/google-dataflow-redis-example works
Main Components
• Input Source: Google Cloud Storage bucket (where the *.txt files are stored).
• Pipeline Execution: The Dataflow pipeline, executed by the StarterPipeline class. This component includes multiple processing steps (transformation, validation, logging, sorting, etc.) as represented by the functions classes (e.g., ProcessingRecords, TransformingData, RedisHashIO) and constants that configure those steps.
• Data Output: Memorystore (Redis) instance, where the processed data is stored.
• Infrastructure Setup: The pipeline uses Maven (via pom.xml) for build and dependency management.
• External Access: Compute Engine VM used to telnet into the Redis instance for checking keys.
Component Interactions
• The data flow starts from reading input files from Cloud Storage.
• Data is fed into the Dataflow pipeline where multiple transformation stages take place. For example:
• "ProcessingRecords" and "ProcessingPPID" handle record-level processing.
• "TransformingData" and "CombiningTransforms" perform necessary data transformations.
• "RedisHashIO" handles the custom HSET implementation to write data into Redis.
• Various logging functions (e.g., LogValidRecords, LogInvalidRecords) monitor pipeline progress.
• Data finally flows into Memorystore (Redis) for storage.
• Separately, the user (or administrator) connects from a Compute Engine VM to confirm that data has been inserted into Redis.
Architectural Patterns and Design Principles
• Modular Design: The separation into functions packages indicates that the pipeline is designed in modular stages, each with a single responsibility (transforming, processing, logging, etc.).
• Dataflow Pattern: The architecture reflects a streaming or batch processing pipeline where data is processed in stages from input to output.
• External Integration: Clearly delimited interactions with external systems (Cloud Storage for inputs, Redis for outputs, and Compute Engine for diagnostic access).
Relevant Technologies/Frameworks
• Google Cloud Dataflow as the runner for the pipeline.
• Google Cloud Storage as the data source.
• Google Memorystore (Redis 5.0) as the data destination.
• Maven for project dependency management and execution.
• Java as the programming language for pipeline implementation.
Elements
• Clear Labels: Name each component such as “Cloud Storage (Input Files)”, “Dataflow Pipeline” (with its sub-stages if required), “Memorystore - Redis”, and “Compute Engine VM (Redis Checker)”.