# tosun-si/teams-league-python-standard-beam architecture

> Architecture diagram of the GitHub repository tosun-si/teams-league-python-standard-beam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tosun-si/teams-league-python-standard-beam
- Repository: https://github.com/tosun-si/teams-league-python-standard-beam

## Overview

##### Main Components

a) Data Sources:
- Google Cloud Storage (JSON files)
- Cloud Pub/Sub (for streaming input)

b) Processing Layer:
- Apache Beam Pipeline
- Dataflow Runner (GCP serverless execution)

c) Data Sink:
- BigQuery (output destination)

##### Key Architectural Elements

**[Input Layer] → [Processing Layer] → [Output Layer]**

##### Detailed Component Breakdown

a) Input Sources (show as parallel inputs):

- Main Apache Beam pipeline components:
- team_stats_transform.py (core transformation logic)
- Domain objects (team_stats.py, team_best_passer_stats.py, team_top_scorer_stats.py)
- Validation layer (team_stats_validation_exception.py)
- Dataflow runner as a container around the processing components

c) Output Layer:
- BigQuery table with two write methods:
- FILE_LOADS (batch)
- STREAMING_INSERTS (streaming)

##### Flow Patterns to Highlight

- Data flow direction
- Transform operations
- Error handling paths
- Different paths for batch vs. streaming processing

##### Additional Elements

A separate section showing:
- Configuration components (team_league_options.py)
- Pipeline options and parameters
- Environment configuration

##### Technical Details

Annotations for:
- Python version
- Apache Beam version
- GCP service dependencies
- Key configuration parameters (project_id, dataset, table names)

##### The configuration flexibility

Remember to indicate that this is a serverless architecture running on GCP Dataflow, which handles the infrastructure management automatically.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        GCS[("Google Cloud Storage")]
        PS[("Cloud Pub/Sub")]
        JSON["Input JSON Files"]:::files
        click JSON "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/scripts/input_teams_stats.json"
    end

    subgraph "Processing Layer"
        direction TB
        DF["Dataflow Runner"]

        subgraph "Pipeline Components"
            direction TB
            TST["Team Stats Transform"]:::core
            click TST "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/domain_ptransform/team_stats_transform.py"
            
            subgraph "Domain Objects"
                direction TB
                TS["Team Stats"]:::domain
                TBS["Team Best Passer Stats"]:::domain
                TTS["Team Top Scorer Stats"]:::domain
                TSR["Team Stats Raw"]:::domain
                click TS "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/domain/team_stats.py"
                click TBS "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/domain/team_best_passer_stats.py"
                click TTS "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/domain/team_top_scorer_stats.py"
                click TSR "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/domain/team_stats_raw.py"
            end

            VAL["Validation Layer"]:::validation
            click VAL "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/domain/exception/team_stats_validation_exception.py"

            MAP["Stats Mapper"]:::mapper
            click MAP "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/application/team_stats_mapper.py"
        end

        subgraph "Configuration"
            direction TB
            OPTS["Pipeline Options"]:::config
            SCHEMA["Table Schema"]:::config
            click OPTS "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/team_league/application/team_league_options.py"
            click SCHEMA "https://github.com/tosun-si/teams-league-python-standard-beam/blob/main/scripts/team_stat_table_schema.json"
        end
    end

    subgraph Output
        BQ[("BigQuery")]
        BATCH["File Loads Mode"]
        STREAM["Streaming Inserts Mode"]
    end

    %% Connections
    GCS --> JSON
    JSON --> TST
    PS --> TST
    TST --> |Transform| TS
    TST --> |Transform| TBS
    TST --> |Transform| TTS
    TST --> |Transform| TSR
    TS & TBS & TTS & TSR --> VAL
    VAL --> MAP
    MAP --> BQ
    OPTS --> DF
    SCHEMA --> BQ
    BQ --> BATCH
    BQ --> STREAM

    %% Styles
    classDef core fill:#90EE90,stroke:#333,stroke-width:2px
    classDef domain fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef validation fill:#FFB6C1,stroke:#333,stroke-width:1px
    classDef config fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef mapper fill:#F0E68C,stroke:#333,stroke-width:1px
    classDef files fill:#D3D3D3,stroke:#333,stroke-width:1px

    %% Legend
    subgraph Legend
        L1["Core Transform"]:::core
        L2["Domain Objects"]:::domain
        L3["Validation"]:::validation
        L4["Configuration"]:::config
        L5["Mapper"]:::mapper
        L6["Files"]:::files
    end
```
