# nandishns/automated-software-and-product-listing-platform architecture

> Architecture diagram of the GitHub repository nandishns/automated-software-and-product-listing-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nandishns/automated-software-and-product-listing-platform
- Repository: https://github.com/nandishns/automated-software-and-product-listing-platform

## Overview

##### Main Components
- Data Acquisition:
– Web Scraper Component (located in the “Scraper” directory) that uses BeautifulSoup and Selenium to gather product details.
– Crawler (the “crawler.py” file) which may complement scraping.
- Data Ingestion & Streaming:
– Apache Kafka (and Zookeeper) as defined by the “docker-compose.yml” and “kafka-zoopkeeper-deployment.yaml” files.
- Processing Components (Kafka Consumers):
– Three separate Dockerized consumers, each in its own directory under “KafkaConsumer”:
∘ Software consumer
∘ TwitterLLM consumer
∘ NewsLLM consumer
– Each consumer has its own cron job YAML file (e.g., “product-cronjob.yaml”, “twitter-cronjob.yaml”, “news-cronjob.yaml”) for scheduled execution.
- Backend API & Interface:
– Flask Application (in “Flask/app.py”) acting as a backend service.
– Frontend website (in “Frontend/website.py”) likely serving data or visualizations.
- Data Storage:
– MongoDB Atlas (as mentioned in the README, integrating persistent storage for new listings).
- External Services & Additional Processing:
– Integration points for the G2 API.
– Large Language Models (LLMs) used for advanced text analysis on streamed content.
- Containerization & Orchestration:
– Docker containers for each service.
– Kubernetes scheduled jobs (using provided YAML files) for orchestrating tasks.

##### Relationships and Interactions
- Data flows from the Scraper to the Kafka topics –
– The scrapers obtain initial product information and send these events or messages to corresponding Kafka topics.
- Kafka acts as the central data pipeline –
– Different topics (e.g., “software”, “news”, “x-llm”) separate data streams.
- Kafka Consumers subscribe to these topics –
– Each consumer processes the incoming data (e.g., data from Twitter or news sites) before pushing it to a datastore or further analysis.
- Processed data is then sent to MongoDB –
– New listings added or updated in the database trigger updates in the product listing.
- The Flask API and Streamlit Frontend provide user interfaces –
– These components read from MongoDB and/or Kafka to display updated product information.
- All components are containerized and scheduled –
– Docker handles the packaging, while Kubernetes cron jobs manage regular scraping and consumer tasks.

##### Architectural Patterns and Design Principles
- Microservices & Containerization: Each major component (scraper, each Kafka consumer, backend API) is isolated in its own container.
- Event-driven Architecture: Kafka is the backbone ensuring decoupled communication among data producers and consumers.
- Modularity & Scalability: The project is designed to easily add new data sources or processing components by following the established pattern (each with its own consumer and cron job).

##### Relevant Technologies, Frameworks, and Libraries
- Scraping: BeautifulSoup and Selenium.
- Data Streaming: Apache Kafka (with embedded Zookeeper).
- Containerization/Orchestration: Docker and Kubernetes.
- Web Frameworks: Flask (backend) and Streamlit (frontend).
- Data Storage: MongoDB Atlas.
- Integration with External APIs: G2 API and potential LLM services.
- Scheduling & Automated Tasks: Cron jobs defined via YAML files.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Acquisition Layer
    subgraph "Data Acquisition"
        A1["Web Scraper (BeautifulSoup/Selenium)"]:::acquisition
        A2["Crawler (crawler.py)"]:::acquisition
    end

    %% Data Ingestion & Streaming Layer
    subgraph "Data Ingestion & Streaming"
        B1["Kafka Broker & Zookeeper"]:::stream
    end

    %% Processing Components Layer
    subgraph "Processing Components"
        C1["Software Consumer (Docker)"]:::processing
        C2["TwitterLLM Consumer (Docker)"]:::processing
        C3["NewsLLM Consumer (Docker)"]:::processing
    end

    %% Scheduling Layer (Kubernetes Cron Jobs)
    subgraph "Scheduling (K8s Cron Jobs)"
        SC1["product-cronjob.yaml"]:::scheduling
        SC2["twitter-cronjob.yaml"]:::scheduling
        SC3["news-cronjob.yaml"]:::scheduling
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        E1["MongoDB Atlas"]:::storage
    end

    %% Backend & Frontend Layer
    subgraph "Backend & Frontend"
        D1["Flask API"]:::api
        D2["Streamlit Frontend"]:::frontend
    end

    %% External Integrations
    subgraph "External Integrations"
        F1["G2 API"]:::external
        F2["LLM Processing"]:::external
    end

    %% Connections between layers
    A1 -->|"sends_product_data"| B1
    A2 -->|"feeds_data"| B1

    B1 -->|"software_topic"| C1
    B1 -->|"twitter_topic"| C2
    B1 -->|"news_topic"| C3

    SC1 -->|"triggers"| C1
    SC2 -->|"triggers"| C2
    SC3 -->|"triggers"| C3

    C1 -->|"processed_data"| E1
    C2 -->|"processed_data"| E1
    C3 -->|"processed_data"| E1

    E1 -->|"data_for_listing"| D1
    E1 -->|"data_for_listing"| D2

    D1 -->|"integrates_with"| F1
    D2 -->|"integrates_with"| F2

    %% Click Events for Component Mapping
    click A1 "https://github.com/nandishns/automated-software-and-product-listing-platform/blob/main/Scraper/scraper.py"
    click A2 "https://github.com/nandishns/automated-software-and-product-listing-platform/blob/main/crawler.py"
    click B1 "https://github.com/nandishns/automated-software-and-product-listing-platform/blob/main/docker-compose.yml"
    click B1 "https://github.com/nandishns/automated-software-and-product-listing-platform/blob/main/kafka-zoopkeeper-deployment.yaml"
    click C1 "https://github.com/nandishns/automated-software-and-product-listing-platform/tree/main/KafkaConsumer/Software"
    click C2 "https://github.com/nandishns/automated-software-and-product-listing-platform/tree/main/KafkaConsumer/TwitterLLM"
    click C3 "https://github.com/nandishns/automated-software-and-product-listing-platform/tree/main/KafkaConsumer/NewsLLM"
    click D1 "https://github.com/nandishns/automated-software-and-product-listing-platform/blob/main/Flask/app.py"
    click D2 "https://github.com/nandishns/automated-software-and-product-listing-platform/blob/main/Frontend/website.py"
    click E1 "https://github.com/nandishns/automated-software-and-product-listing-platform/blob/main/README.md"

    %% Styles
    classDef acquisition fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef stream fill:#ccf2ff,stroke:#0066cc,stroke-width:2px;
    classDef processing fill:#ccffcc,stroke:#009933,stroke-width:2px;
    classDef api fill:#ffccff,stroke:#cc0099,stroke-width:2px;
    classDef frontend fill:#ccffff,stroke:#009999,stroke-width:2px;
    classDef storage fill:#ffffcc,stroke:#cccc00,stroke-width:2px;
    classDef scheduling fill:#e6e6fa,stroke:#9966cc,stroke-width:2px;
    classDef external fill:#ffe4b5,stroke:#ff8c00,stroke-width:2px;
```
