How loinguyen3108/sportify-music-analysis works
Main Components Identification
• Data Ingestion:
• Spotify Crawler: Located under src/crawler, this component interacts with the Spotify Web API to retrieve artist, album, playlist, and track data.
• Kafka: Under both containers/kafka and src/infratructure/kafka, it serves as the message streaming backbone.
• Data Storage:
• PostgreSQL: Scripts in containers/postgres indicate its use as a staging area.
• Clickhouse: Containers/clickhouse represents an analytical database for real‑time metrics.
• Data Lake / GCS: Mentioned in the README for storing raw and intermediate data.
• BigQuery: For star schema analytics as noted in the README.
• Data Processing:
• Apache Spark: Containers/spark and src/infratructure/spark map out usage of Spark for distributed ETL and transformations.
• Orchestration:
• Airflow: Under containers/airflow, responsible for scheduling and managing pipeline workflows.
• Auxiliary Components:
• CLI Utilities: Found in src/cli for tasks such as authentication, ingestion, consumption, and monitoring.
• Pipelines: src/pipelines holds ETL pipeline scripts, queries, and transformations.
• Terraform: Infrastructure code under terraform for provisioning cloud resources.
• Migration scripts: Directories under migration show SQL scripts for schema changes.
Component Relationships & Interactions
• Data Flow:
• Data is first acquired through the Spotify crawler.
• Freshly fetched data is published to Kafka, then staged into PostgreSQL.
• Data is simultaneously stored into a data lake (GCS/MinIO) for further processing.
• Apache Spark processes and transforms the data, loading it into analytical stores (BigQuery and Clickhouse).
• Airflow orchestrates and triggers these pipeline stages, ensuring dependencies are maintained.
• Communication Patterns:
• Messages flow from producers (crawlers or ingestion processes) to Kafka.
• Consumers (Spark jobs or ETL scripts in pipelines) retrieve data from Kafka.
• SQL migrations update the underlying schema in PostgreSQL or Clickhouse.
• Deployment Architecture:
• The entire system is containerized using Docker & managed with Docker Compose.
• Terraform defines infrastructure resources externally for scaling in a cloud environment.
Architectural Patterns & Key Design Principles
• Modularity: Each service (crawling, processing, storage, orchestration) is clearly defined in its own container or module.
• Scalability: Leveraging Kafka and Spark supports both real‑time and batch processing.
• Extensibility: The use of CLI modules and pipelines suggests that future expansion or modification of ingestion and processing modules is considered.
• Infrastructure as Code: Terraform scripts proclaim a commitment to scalable and reproducible cloud resource management.
Technologies & Frameworks to Highlight
• Docker & Docker Compose for containerization.
• Apache Kafka for streaming pipelines.
• Apache Airflow for orchestrating ETL workflows.
• Apache Spark for distributed processing.
• PostgreSQL for initial staging and Clickhouse for real‑time analytics.
• BigQuery for cloud‑based dimensional modeling (star schema) and analytics.
• Terraform for managing provisioning and infrastructure.
• Python with various utility modules in src for ETL and CLI interactions.
a. An outline that categorizes the project into major layers:
• Ingestion (Spotify API, Crawler, Kafka)
• Storage (PostgreSQL, Data Lake, Clickhouse, BigQuery)
• Processing (Spark ETL, Data Pipelines)
• Orchestration & Management (Airflow, Terraform)
b. Sketch how each component connects:
• Flow from the Spotify Web API, processed by the crawler (src/crawler) which sends messages into Kafka.
• Kafka feeding data into both the immediate landing zone (PostgreSQL) and the Spark ETL pipeline.
• Transformation in Spark and then the storage in BigQuery (for batch analytics) and Clickhouse (for real‑time queries).
• Incorporate the orchestration by showing Airflow’s control over these processes.
c. Underlying infrastructure:
• Docker containers (as seen in the containers/ directory) representing each service.
• Terraform-managed resources to achieve cloud scalability.
d. Labels and visual markers:
• Deployment-time aspects (e.g., Docker Compose and configuration flow).