# uowol/chzzk-data-analytics architecture

> Architecture diagram of the GitHub repository uowol/chzzk-data-analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uowol/chzzk-data-analytics
- Repository: https://github.com/uowol/chzzk-data-analytics

## Overview

##### Project Type and Purpose
- This is a data-analytics ETL pipeline tool focused on collecting (crawl) broadcast chat and donation data, then processing it for analysis and future predictive/LLM-driven insights.
- The project is structured as a pipeline orchestration system (YAML-defined pipelines + a Python runner) with modular components for each step, layered domain logic, and external API integration.

##### File Structure
- Top-level orchestrator:
- run_pipeline.py
- pipelines/*.yaml
- Components layer (task implementations):
- components/chzzk_chat_crawl.py
- components/chzzk_streaming_check.py
- Domain logic (client & data models):
- modules/chzzk/api.py (external chat API calls)
- modules/chzzk/chat.py, constants.py, enums.py
- Base abstractions:
- classes/base.py (generic task/interface definitions)
- classes/chzzk.py (shared utilities)
- Environment & tooling:
- Dockerfile, .devcontainer (development containers)
- pyproject.toml (Poetry dependencies)

##### Extract Key Insights from the README
- Pipelines are defined as YAML in the `pipelines/` folder.
- Invocation: `poetry run python run_pipeline.py --pipeline crawl_chat.yaml`.
- Future goals: integrate LLM + RAG for natural-language analytics.

##### Core Components
a. Pipeline Orchestrator
– Reads YAML definitions
– Instantiates and schedules tasks
b. Task Components
– ChatCrawlTask (components/chzzk_chat_crawl.py)
– StreamingCheckTask (components/chzzk_streaming_check.py)
c. Domain/Service Layer
– ChzzkAPIClient (modules/chzzk/api.py)
– ChatParser, Enums, Constants (modules/chzzk/chat.py, …)
d. Base Framework
– AbstractTask, BaseClient (classes/base.py, classes/chzzk.py)
e. External Services
– Broadcast chat & donation API (Twitch/Chzzk endpoints)
f. Data Storage
– Raw data (files or database – assume a CSV/DB sink)
– Processed analytics outputs
g. Future LLM/RAG Analytics Module
– Natural-language interface layer

##### Relationships & Interactions
- Task Components → Domain/Service Layer → External API (data retrieval)
- Task Components → Data Storage (persist raw results)
- Optional: Storage → Analytics Module → User Interface (future)

##### Architectural Patterns & Principles
- Pipeline Pattern (YAML definitions + runner)
- Layered Architecture:
- Presentation/CLI (run_pipeline.py)
- Orchestration (pipeline engine)
- Task Implementation (components)
- Domain Logic (modules)
- Infrastructure (storage, external API)
- Modular & Extensible: add new component *.py under components and reference in YAML.

##### Technologies & Frameworks
- Python 3.x, Poetry (pyproject.toml)
- PyYAML for pipeline config
- Docker/.devcontainer for reproducible environments
- Standard logging & error handling in classes/base.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        PD["Pipeline Definitions\n(pipelines/*.yaml)"]:::config
        ORC["run_pipeline.py\n(CLI Orchestrator)"]:::orchestration
    end
    PD -->|"reads config"| ORC

    %% Component Layer
    subgraph "Component Layer"
        CCT["ChatCrawlTask"]:::component
        SCT["StreamingCheckTask"]:::component
    end
    ORC -->|"initiates"| CCT
    ORC -->|"initiates"| SCT

    %% Base Framework
    subgraph "Base Framework"
        AT["AbstractTask & Interfaces"]:::base
        SU["Shared Utilities & BaseClient"]:::base
    end
    AT -->|extends| CCT
    AT -->|extends| SCT
    SU -->|provides utils| CCT
    SU -->|provides utils| SCT

    %% Domain Layer
    subgraph "Domain Layer"
        API["ChzzkAPIClient"]:::domain
        CP["Chat Parsing & Models"]:::domain
        DC["Domain Constants"]:::domain
        DE["Domain Enums"]:::domain
    end
    CCT -->|"calls"| API
    SCT -->|"calls"| API
    CCT -->|"parses data"| CP
    SCT -->|"uses constants/enums"| DC
    SCT -->|"uses constants/enums"| DE

    %% External Services
    subgraph "External Services"
        EXT["Broadcast Chat & Donation API"]:::external
    end
    API -->|"fetch chat/donations"| EXT

    %% Data Storage
    subgraph "Data Storage"
        RAW["Raw Data Store\n(CSV/DB)"]:::storage
        PROC["Processed Analytics"]:::storage
    end
    CCT -.-|"saves raw"| RAW
    SCT -.-|"saves raw"| RAW
    PROC -->|"reads from RAW"| PROC

    %% Future Analytics
    subgraph "Future LLM/RAG Analytics"
        LLM["Natural-Language Analytics Module"]:::future
    end
    PROC -->|"feeds data"| LLM

    %% Click Events
    click PD "https://github.com/uowol/chzzk-data-analytics/blob/main/pipelines/crawl_chat.yaml"
    click ORC "https://github.com/uowol/chzzk-data-analytics/blob/main/run_pipeline.py"
    click CCT "https://github.com/uowol/chzzk-data-analytics/blob/main/components/chzzk_chat_crawl.py"
    click SCT "https://github.com/uowol/chzzk-data-analytics/blob/main/components/chzzk_streaming_check.py"
    click API "https://github.com/uowol/chzzk-data-analytics/blob/main/modules/chzzk/api.py"
    click CP "https://github.com/uowol/chzzk-data-analytics/blob/main/modules/chzzk/chat.py"
    click DC "https://github.com/uowol/chzzk-data-analytics/blob/main/modules/chzzk/constants.py"
    click DE "https://github.com/uowol/chzzk-data-analytics/blob/main/modules/chzzk/enums.py"
    click AT "https://github.com/uowol/chzzk-data-analytics/blob/main/classes/base.py"
    click SU "https://github.com/uowol/chzzk-data-analytics/blob/main/classes/chzzk.py"
    click RAW "https://github.com/uowol/chzzk-data-analytics/blob/main/pyproject.toml"
    click LLM "https://github.com/uowol/chzzk-data-analytics/blob/main/pyproject.toml"
    click RAW "https://github.com/uowol/chzzk-data-analytics/blob/main/pyproject.toml"
    click EXT "https://github.com/uowol/chzzk-data-analytics/blob/main/pyproject.toml"

    %% Environment & Tooling
    DEV["Docker Environment"]:::config
    CORNER["pyproject.toml"]:::config
    DEV --> CORNER

    click DEV "https://github.com/uowol/chzzk-data-analytics/tree/main/Dockerfile"
    click DEV "https://github.com/uowol/chzzk-data-analytics/blob/main/.devcontainer/devcontainer.json"
    click CORNER "https://github.com/uowol/chzzk-data-analytics/blob/main/pyproject.toml"

    %% Styles
    classDef orchestration fill:#DDDDDD,stroke:#333,stroke-width:1px
    classDef config fill:#CFD8DC,stroke:#333,stroke-width:1px
    classDef component fill:#BBDEFB,stroke:#333,stroke-width:1px
    classDef domain fill:#E1BEE7,stroke:#333,stroke-width:1px
    classDef base fill:#B3E5FC,stroke:#333,stroke-width:1px
    classDef external fill:#FFCC80,stroke:#333,stroke-width:1px
    classDef storage fill:#C8E6C9,stroke:#333,stroke-width:1px
    classDef future fill:#F8BBD0,stroke:#333,stroke-width:1px
```
