How uowol/chzzk-data-analytics works
Project Type and Purpose
• This is a data-analytics ETL pipeline tool focused on collecting (crawl) broadcast chat and donation data, then processing it for analysis and future predictive/LLM-driven insights.
• The project is structured as a pipeline orchestration system (YAML-defined pipelines + a Python runner) with modular components for each step, layered domain logic, and external API integration.
File Structure
• Top-level orchestrator:
• run_pipeline.py
• pipelines/*.yaml
• Components layer (task implementations):
• components/chzzk_chat_crawl.py
• components/chzzk_streaming_check.py
• Domain logic (client & data models):
• modules/chzzk/api.py (external chat API calls)
• modules/chzzk/chat.py, constants.py, enums.py
• Base abstractions:
• classes/base.py (generic task/interface definitions)
• classes/chzzk.py (shared utilities)
• Environment & tooling:
• Dockerfile, .devcontainer (development containers)
• pyproject.toml (Poetry dependencies)
Extract Key Insights from the README
• Pipelines are defined as YAML in the pipelines/ folder.
• Invocation: poetry run python run_pipeline.py --pipeline crawl_chat.yaml.
• Future goals: integrate LLM + RAG for natural-language analytics.
Core Components
a. Pipeline Orchestrator
– Reads YAML definitions
– Instantiates and schedules tasks
b. Task Components
– ChatCrawlTask (components/chzzk_chat_crawl.py)
– StreamingCheckTask (components/chzzk_streaming_check.py)
c. Domain/Service Layer
– ChzzkAPIClient (modules/chzzk/api.py)
– ChatParser, Enums, Constants (modules/chzzk/chat.py, …)
d. Base Framework
– AbstractTask, BaseClient (classes/base.py, classes/chzzk.py)
e. External Services
– Broadcast chat & donation API (Twitch/Chzzk endpoints)
f. Data Storage
– Raw data (files or database – assume a CSV/DB sink)
– Processed analytics outputs
g. Future LLM/RAG Analytics Module
– Natural-language interface layer
Relationships & Interactions
• Task Components → Domain/Service Layer → External API (data retrieval)
• Task Components → Data Storage (persist raw results)
• Optional: Storage → Analytics Module → User Interface (future)
Architectural Patterns & Principles
• Pipeline Pattern (YAML definitions + runner)
• Layered Architecture:
• Presentation/CLI (run_pipeline.py)
• Orchestration (pipeline engine)
• Task Implementation (components)
• Domain Logic (modules)
• Infrastructure (storage, external API)
• Modular & Extensible: add new component *.py under components and reference in YAML.
Technologies & Frameworks
• Python 3.x, Poetry (pyproject.toml)
• PyYAML for pipeline config
• Docker/.devcontainer for reproducible environments
• Standard logging & error handling in classes/base.py