# kalpadas599/real-time-analysis-bi architecture

> Architecture diagram of the GitHub repository kalpadas599/real-time-analysis-bi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kalpadas599/real-time-analysis-bi
- Repository: https://github.com/kalpadas599/real-time-analysis-bi

## Overview

##### Project type and purpose
- This is a data-pipeline/full-stack microservice project that ingests data from external sources (Google Ads, Twitter), processes it with Spark and a spaCy-based sentiment analyzer, stores results in multiple backends (MongoDB, Snowflake, Firebase), and exposes a Flask API. It’s containerized (Dockerfile, docker-compose) with CI in GitHub workflows.

##### File structure
- Top-level:
- app.py / src/api/flask_api.py – Flask API entrypoint
- run_ingestion.py – orchestration script for data ingestion
- Dockerfile + docker-compose.yml – container definitions and service composition
- requirements.txt – Python dependencies
- mock_google_ads_data.json, power_bi_setup.md – auxiliary files
- en_core_web_sm… – bundled spaCy English model
- src/:
- api/ – Flask API code
- data_ingestion/ – modules to pull from Google Ads, Twitter, and push to Confluent Kafka
- data_processing/ – spark_processor for batch/stream processing
- ml/ – sentiment_analyzer running spaCy model
- storage/ – clients for Firebase, MongoDB, Snowflake

##### README
- (Empty in this case; all insights come from file tree.)

a. Main components:
1. External Data Sources (Google Ads API, Twitter API)
2. Data Ingestion Service (confluent_producer)
3. Message Broker (Kafka cluster)
4. Processing Layer (Spark Processor)
5. ML Layer (spaCy Sentiment Analyzer)
6. Storage Layer (MongoDB, Snowflake, Firebase)
7. API Layer (Flask API)
8. Orchestration/Containerization (Docker Compose)
9. CI/CD (GitHub Actions)
b. Relationships & interactions:
- Data Sources → Data Ingestion Service → Kafka
- Kafka → Spark Processor → ML Sentiment Analyzer → Storage Clients
- Storage Clients → Databases (MongoDB, Snowflake, Firebase)
- Flask API ↔ Databases (reads processed data)
- Docker-Compose runs ingest/process/api services in containers
- GitHub Actions triggers builds/tests on push
c. Architectural patterns & principles:
- Separation of concerns (ingest, process, ML, API)
- Event-driven architecture via Kafka
- Containerized microservices
- Polyglot storage (NoSQL, data warehouse, real-time DB)
d. Key technologies:
- Python, Flask, Spark, spaCy (en_core_web_sm)
- Confluent Kafka
- MongoDB, Snowflake, Firebase
- Docker, Docker-Compose
- GitHub Actions

##### Elements
- Blue for ingestion components
- Orange for processing/ML
- Green for storage
- Purple for API
- Gray for orchestration/CI/CD
- Rectangles for microservices

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        direction TB
        GoogleAds["Google Ads Client"]:::external
        TwitterClient["Twitter Client"]:::external
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer" 
        direction TB
        ConfluentProducer["Confluent Producer"]:::ingestion
        runIngestion["Orchestration Script"]:::ingestion
    end

    %% Message Broker
    KafkaBroker["Kafka Cluster"]:::external

    %% Processing & ML Layer
    subgraph "Processing & ML Layer"
        direction TB
        SparkProcessor["Spark Processor"]:::processing
        SentimentAnalyzer["spaCy Sentiment Analyzer"]:::processing
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        MongoDB[(MongoDB)]:::storage
        Snowflake[(Snowflake)]:::storage
        Firebase[(Firebase)]:::storage
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        FlaskEntrypoint["app.py"]:::api
        FlaskModule["Flask API Module"]:::api
    end

    %% Orchestration & CI/CD
    subgraph "Orchestration & CI/CD"
        direction TB
        DockerfileNode["Dockerfile"]:::orchestration
        DockerCompose["docker-compose.yml"]:::orchestration
        CICD[".github/workflows/main.yml"]:::orchestration
    end

    %% Data Flows
    GoogleAds -->|"fetch data"| ConfluentProducer
    TwitterClient -->|"fetch data"| ConfluentProducer
    runIngestion -->|"start ingestion"| ConfluentProducer
    ConfluentProducer -->|"produce events"| KafkaBroker
    KafkaBroker -->|"stream data"| SparkProcessor
    SparkProcessor -->|"batch/stream jobs"| SentimentAnalyzer
    SentimentAnalyzer -->|"store results"| MongoDB
    SentimentAnalyzer -->|"store results"| Snowflake
    SentimentAnalyzer -->|"store results"| Firebase
    FlaskModule -->|"read data"| MongoDB
    FlaskModule -->|"read data"| Snowflake
    FlaskModule -->|"read data"| Firebase
    FlaskEntrypoint -->|"routes to"| FlaskModule

    %% Orchestration relations
    DockerCompose -->|runs| ConfluentProducer
    DockerCompose -->|runs| SparkProcessor
    DockerCompose -->|runs| SentimentAnalyzer
    DockerCompose -->|runs| FlaskEntrypoint
    DockerfileNode -->|builds| DockerCompose
    CICD -->|triggers build/tests| DockerfileNode

    %% Click Events
    click GoogleAds "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/data_ingestion/google_ads_client.py"
    click TwitterClient "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/data_ingestion/twitter_client.py"
    click ConfluentProducer "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/data_ingestion/confluent_producer.py"
    click runIngestion "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/run_ingestion.py"
    click DockerCompose "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/docker-compose.yml"
    click SparkProcessor "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/data_processing/spark_processor.py"
    click SentimentAnalyzer "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/ml/sentiment_analyzer.py"
    click Firebase "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/storage/firebase_client.py"
    click MongoDB "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/storage/mongodb_client.py"
    click Snowflake "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/storage/snowflake_client.py"
    click FlaskEntrypoint "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/app.py"
    click FlaskModule "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/src/api/flask_api.py"
    click DockerfileNode "https://github.com/kalpadas599/real-time-analysis-bi/tree/master/Dockerfile"
    click CICD "https://github.com/kalpadas599/real-time-analysis-bi/blob/master/.github/workflows/main.yml"

    %% Styles
    classDef external fill:#fadeff,stroke:#0066cc,color:#000066
    classDef ingestion fill:#e0f0ff,stroke:#004c99,color:#003366
    classDef processing fill:#ffe5cc,stroke:#e69500,color:#804000
    classDef storage fill:#e6ffe6,stroke:#33cc33,color:#006600
    classDef api fill:#f2e6ff,stroke:#8000b3,color:#33004d
    classDef orchestration fill:#f2f2f2,stroke:#999999,color:#333333
```
