How luisumina/soundwave-analytics works
Main Components
• Data Sources: Clearly indicate the CSV files (users, songs, play logs) and the Spotify API data.
• Ingestion Layer: Represent the multiple Python scripts in the “scripts/ingestion” folder (e.g., extract_users.py, extract_songs.py, extract_play_logs.py, extract_spotify.py). These are responsible for extracting raw data.
• Data Lake: Show the storage location (local file system and/or cloud storage like S3 or Azure Blob) for raw data ingestion.
• Orchestration Layer: Indicate components like Apache Airflow (with the “dags” directory as listed in the README) that manage recurring workflows.
• Streaming Components: Include the real-time event streaming with Kafka or AWS Kinesis.
• Infrastructure & Deployment: Highlight the use of containerization (Docker), infrastructure as code (Terraform), and cloud platform deployment (AWS, Azure).
• Visualization: Add the dashboarding tools (Power BI, Tableau, or Metabase) that represent the final consumption layer.
Diagramming the Relationships and Data Flows
• From the Data Warehouse, indicate the route to visualization/dashboards.
• Also, demonstrate the parallel or integrated streaming pipeline that pushes real-time events into a separate processing or visualization path.
• Be sure to label any cases where the data flows into multiple destinations (for instance, both Data Lake for batch storage and real-time components for streaming).
Architectural Patterns and Design Principles
• Layered architecture — separating raw data ingestion, transformation, structured storage, and presentation.
• Use of a dimensional modeling approach (Star Schema) to optimize analytical queries.
• If applicable, indicate the orchestration pattern (using Apache Airflow’s DAGs for scheduling and dependency management).
• Hints of scalability through the adoption of cloud storage and streaming services.