How tosun-si/teams-league-python-standard-beam works
Main Components
a) Data Sources:
• Google Cloud Storage (JSON files)
• Cloud Pub/Sub (for streaming input)
b) Processing Layer:
• Apache Beam Pipeline
• Dataflow Runner (GCP serverless execution)
c) Data Sink:
• BigQuery (output destination)
Key Architectural Elements
[Input Layer] → [Processing Layer] → [Output Layer]
Detailed Component Breakdown
a) Input Sources (show as parallel inputs):
• Main Apache Beam pipeline components:
• team_stats_transform.py (core transformation logic)
• Domain objects (team_stats.py, team_best_passer_stats.py, team_top_scorer_stats.py)
• Validation layer (team_stats_validation_exception.py)
• Dataflow runner as a container around the processing components
c) Output Layer:
• BigQuery table with two write methods:
• FILE_LOADS (batch)
• STREAMING_INSERTS (streaming)
Flow Patterns to Highlight
• Data flow direction
• Transform operations
• Error handling paths
• Different paths for batch vs. streaming processing
Additional Elements
A separate section showing:
• Configuration components (team_league_options.py)
• Pipeline options and parameters
• Environment configuration
Technical Details
Annotations for:
• Python version
• Apache Beam version
• GCP service dependencies
• Key configuration parameters (project_id, dataset, table names)
The configuration flexibility
Remember to indicate that this is a serverless architecture running on GCP Dataflow, which handles the infrastructure management automatically.