# suhaspalani/big_data_instagram architecture

> Architecture diagram of the GitHub repository suhaspalani/big_data_instagram, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/suhaspalani/big_data_instagram
- Repository: https://github.com/suhaspalani/big_data_instagram

## Overview

##### Project Type and Purpose
- This is a full-stack application specifically built to provide Instagram analytics.
- The README clearly reveals that key capabilities include real-time updates via WebSockets and performance monitoring of database operations.

##### File Structure
- The project file tree reveals three main high-level directories: “backend”, “consumer”, and “frontend”.
- The “backend” directory includes files such as main.py (likely launching the FastAPI service), kafka_producer.py (handling message queuing with Kafka), and configuration files.
- The “consumer” directory covers data processing roles with files like app.py, data_processor.py, and instagram_scraper.py (which integrates with Apify to scrape Instagram profiles). This service listens to Kafka and handles database interactions.
- The “frontend” directory houses a React.js app with components that display analytics, influencer forms, Kafka status, and performance metrics.
- Additionally, Docker-related files (Dockerfile in each component, docker-compose.yml) indicate that containerization is employed to standardize deployments.

##### Extract Architectural Insights from the README
- The README highlights key components like FastAPI, Kafka, MongoDB, DynamoDB, Apify integration, and a React frontend.
- It details the data flow: from scraping Instagram profiles (consumer fetching data via API calls) to processing messages via Kafka, and finally serving data to the frontend using WebSockets.
- The README also underlines the importance of scalability and performance monitoring.

a. Main Components:
- Frontend: React.js application for data visualization.
- Backend: FastAPI service handling API requests and interfacing with Kafka.
- Consumer: Python service that listens to Kafka topics, scrapes Instagram data, and processes/stores it.
- Databases: Two storage systems – MongoDB (local) as well as AWS DynamoDB (cloud).
- External Services: Kafka (and ZooKeeper), Apify for scraping, and possibly AWS services for DynamoDB.
- Docker & Docker Compose: Containerization and orchestration layer ensuring services run in a coordinated fashion.

b. Relationships and Data Flow:
- The Frontend (React app) communicates with the Backend (FastAPI) via API calls and connects via WebSockets for real-time data updates.
- The Backend sends messages to Kafka. These messages include Instagram profile analysis requests.
- The Consumer service listens to Kafka topics. On receiving messages, it uses Apify to scrape Instagram profiles, processes the data, and writes to either MongoDB or DynamoDB.
- API calls from Frontend to Backend.
- Message flow from Backend (Kafka Producer) to Kafka.
- Kafka messages being consumed by the Consumer service.
- Real-time updates flowing back from Backend (via WebSockets) to Frontend.

c. Architectural Patterns and Technologies:
- Microservices: Backend and Consumer are separate services communicating asynchronously via Kafka.
- Containerization: Docker and docker-compose ensure modular deployment.
- Data-driven pipeline: Messaging queue (Kafka) decouples scraping/processing from API layer.
- Scalability: Use of Kafka and cloud-based DynamoDB shows the design’s readiness to scale.
- Use of modern frameworks: FastAPI, React.js, and integration with Apify.

d. Relevant Technologies:
- Backend: FastAPI (Python), Kafka (with a producer), and configuration management via config.py.
- Consumer: Python with modules for data processing, Instagram scraping using Apify, and database handling.
- Frontend: React.js with components handling different analytics UI functions.
- Databases: MongoDB and DynamoDB.
- Orchestration: Docker, docker-compose, and external services (Kafka, ZooKeeper).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        F1["React Frontend"]:::frontend
    end

    %% Application Layer
    subgraph "Application Layer"
        subgraph "FastAPI Backend"
            B1["FastAPI Backend"]:::backend
            B1a["main.py"]:::backendSub
            B1b["kafka_producer.py"]:::backendSub
            B1c["config.py"]:::backendSub
            B1d["models.py"]:::backendSub
            B1 --> B1a
            B1 --> B1b
            B1 --> B1c
            B1 --> B1d
        end
        subgraph "Data Consumer"
            C1["Data Consumer (Instagram Scraper)"]:::consumer
            C1a["app.py"]:::consumerSub
            C1b["data_processor.py"]:::consumerSub
            C1c["instagram_scraper.py"]:::consumerSub
            C1d["db_handler.py"]:::consumerSub
            C1 --> C1a
            C1 --> C1b
            C1 --> C1c
            C1 --> C1d
        end
    end

    %% Messaging Infrastructure
    subgraph "Messaging Infrastructure"
        K["Kafka Broker"]:::kafka
    end

    %% Data Layer
    subgraph "Data Layer"
        M["MongoDB"]:::database
        D["DynamoDB"]:::database
    end

    %% External Services
    subgraph "External Services"
        A["Apify Integration"]:::external
    end

    %% Container Orchestration
    subgraph "Container Orchestration"
        DO["Docker & Docker Compose"]:::orchestration
        DOa["backend/Dockerfile"]:::orchestrationSub
        DOb["consumer/Dockerfile"]:::orchestrationSub
        DOc["frontend/Dockerfile"]:::orchestrationSub
        DOd["docker-compose.yml"]:::orchestrationSub
        DO --> DOa
        DO --> DOb
        DO --> DOc
        DO --> DOd
    end

    %% Data Flow Connections
    F1 -->|"HTTP/WSS"| B1
    B1 -->|"Publishes"| K
    K -->|"ConsumedBy"| C1
    C1 -->|"StoresIn"| M
    C1 -->|"StoresIn"| D
    B1 -->|"RealTimeUpdate"| F1
    C1 -->|"Calls"| A
    DO --> F1
    DO --> B1
    DO --> C1

    %% Click Events
    click F1 "https://github.com/suhaspalani/big_data_instagram/tree/main/frontend/"
    click B1 "https://github.com/suhaspalani/big_data_instagram/blob/main/backend/main.py"
    click B1a "https://github.com/suhaspalani/big_data_instagram/blob/main/backend/main.py"
    click B1b "https://github.com/suhaspalani/big_data_instagram/blob/main/backend/kafka_producer.py"
    click B1c "https://github.com/suhaspalani/big_data_instagram/blob/main/backend/config.py"
    click B1d "https://github.com/suhaspalani/big_data_instagram/blob/main/backend/models.py"
    click C1 "https://github.com/suhaspalani/big_data_instagram/blob/main/consumer/app.py"
    click C1a "https://github.com/suhaspalani/big_data_instagram/blob/main/consumer/app.py"
    click C1b "https://github.com/suhaspalani/big_data_instagram/blob/main/consumer/data_processor.py"
    click C1c "https://github.com/suhaspalani/big_data_instagram/blob/main/consumer/instagram_scraper.py"
    click C1d "https://github.com/suhaspalani/big_data_instagram/blob/main/consumer/db_handler.py"
    click A "https://github.com/suhaspalani/big_data_instagram/blob/main/consumer/instagram_scraper.py"
    click DOa "https://github.com/suhaspalani/big_data_instagram/tree/main/backend/Dockerfile"
    click DOb "https://github.com/suhaspalani/big_data_instagram/tree/main/consumer/Dockerfile"
    click DOc "https://github.com/suhaspalani/big_data_instagram/tree/main/frontend/Dockerfile"
    click DOd "https://github.com/suhaspalani/big_data_instagram/blob/main/docker-compose.yml"

    %% Styles
    classDef frontend fill:#f9c74f,stroke:#000,stroke-width:2px;
    classDef backend fill:#90be6d,stroke:#000,stroke-width:2px;
    classDef backendSub fill:#b7e4c7,stroke:#000,stroke-width:1px;
    classDef consumer fill:#f9844a,stroke:#000,stroke-width:2px;
    classDef consumerSub fill:#ffcccb,stroke:#000,stroke-width:1px;
    classDef kafka fill:#577590,stroke:#000,stroke-width:2px;
    classDef database fill:#277da1,stroke:#000,stroke-width:2px;
    classDef external fill:#f94144,stroke:#000,stroke-width:2px;
    classDef orchestration fill:#9d4edd,stroke:#000,stroke-width:2px;
    classDef orchestrationSub fill:#d3c0eb,stroke:#000,stroke-width:1px;
```
