# kshitij-git24/subscription-churn-prediction architecture

> Architecture diagram of the GitHub repository kshitij-git24/subscription-churn-prediction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kshitij-git24/subscription-churn-prediction
- Repository: https://github.com/kshitij-git24/subscription-churn-prediction

## Overview

- “BigData_Spotify_Data_Fetching.ipynb” – likely responsible for extracting streaming data from a provided URL using memory‐efficient techniques (e.g., generator functions processing JSON records line-by-line).
- “BigData Spotify Main Report Notebook.ipynb” – which probably contains the data transformation, feature engineering (especially on user-artist interactions) and reporting parts of the analysis.
This suggests a pipeline where raw streaming data is ingested, processed/engineered with PySpark, and then used to train ML models that address churn prediction imbalances.

a. Data Source & Ingestion Layer:
- External Streaming Data API: Represents the URL source from where JSON records are fetched.
- Data Fetching Script: Corresponding to the “BigData_Spotify_Data_Fetching.ipynb” that reads and processes data using a generator function.
b. Data Processing & Feature Engineering Layer:
- PySpark Processing Engine: Illustrate the role of distributed processing for handling and transforming the data.
- Feature Engineering Component: Highlight operations such as the analysis of user-artist interactions.
c. Machine Learning & Model Training Layer:
- ML Model Training Module: Indicate where machine learning models are built/trained, including handling weighted classes to address cancellation imbalances.
d. Reporting & Analysis:
- Main Report Notebook: Represent “BigData Spotify Main Report Notebook.ipynb” which compiles the analysis and reporting outputs.
e. Cloud & Scalability Infrastructure:
- Databricks Environment: Mark this as the cloud environment for scalable processing.
- GCP Integration: Note the use of GCP as the underlying cloud infrastructure supporting data storage, compute, or orchestration.

##### Relationships and data flow between these components
- Transition from feature engineering into the ML model training component.
- Integration with Databricks and GCP as the overarching infrastructure that connects and supports the data processing and model training components.

##### Architectural patterns and key technologies
- Use of distributed processing (via PySpark) to handle large volumes of streaming data.
- Use of memory-efficient Python techniques (generator functions) in the ingestion phase.
- Call out cloud-based scaling and management through Databricks and GCP.
- Optionally, annotate any lines indicating where testing and iterative development (e.g., notebooks for exploratory analysis) support the overall pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion Layer"
        API["External Streaming Data API"]:::api
        fetch["Data Fetching Script"]:::notebook
    end

    subgraph "Data Processing & Feature Engineering Layer"
        pyspark["PySpark Processing Engine"]:::processing
        feature["Feature Engineering Component"]:::processing
    end

    subgraph "Machine Learning & Model Training Layer"
        ml["ML Model Training Module"]:::ml
    end

    subgraph "Reporting & Analysis Layer"
        report["Main Report Notebook"]:::notebook
    end

    subgraph "Cloud & Scalability Infrastructure"
        databricks["Databricks Environment"]:::cloud
        gcp["GCP Integration"]:::cloud
    end

    API -->|"fetches"| fetch
    fetch -->|"ingests_data"| pyspark
    pyspark -->|"transforms"| feature
    feature -->|"trains"| ml
    ml -->|"reports_to"| report

    %% Cloud integration connections
    pyspark --- databricks
    ml --- databricks
    ml --- gcp
    report --- gcp

    %% Click Events
    click fetch "https://github.com/kshitij-git24/subscription-churn-prediction/blob/main/BigData_Spotify_Data_Fetching.ipynb"
    click pyspark "https://github.com/kshitij-git24/subscription-churn-prediction/blob/main/BigData Spotify Main Report Notebook.ipynb"
    click report "https://github.com/kshitij-git24/subscription-churn-prediction/blob/main/BigData Spotify Main Report Notebook.ipynb"

    %% Styles
    classDef api fill:#f9c3c3,stroke:#b30000,stroke-width:2px;
    classDef notebook fill:#c3d9f9,stroke:#003399,stroke-width:2px;
    classDef processing fill:#d9f9c3,stroke:#339900,stroke-width:2px;
    classDef ml fill:#f9f1c3,stroke:#cc9900,stroke-width:2px;
    classDef cloud fill:#fdeab4,stroke:#e09f00,stroke-width:2px;
```
