# omkarcloud/g2-scraper architecture

> Architecture diagram of the GitHub repository omkarcloud/g2-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/omkarcloud/g2-scraper
- Repository: https://github.com/omkarcloud/g2-scraper

## Overview

##### Project Type and Purpose
This is a web scraping application specifically designed to extract product data from G2.com. It's a full-stack application with a Python backend for scraping and a Next.js frontend for user interaction.

##### Main Components

A. Frontend Layer (Next.js):
- User Interface Components
- API Integration Layer
- State Management
- Data Visualization

B. Backend Layer (Python):
- Scraping Engine (Botasaurus Framework)
- Cache Management
- Data Processing
- API Endpoints

C. External Services:
- G2.com (Target Website)
- RapidAPI Integration
- Data Storage (Cache System)

**┌─────────────────────────────────────────────────────────────┐**
- Client Browser                          │
- ┬─────────────────────────────────┘
- ↓
**┌─────────────────────────────────────────────────────────────┐**
- Frontend (Next.js)                       │
- ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐   │
- Pages    │  │  Components  │  │   API Services   │   │
- ┘  └──────────────┘  └──────────────────┘   │
- ┬─────────────────────────────────┘
- ↓
**┌─────────────────────────────────────────────────────────────┐**
- Backend (Python)                         │
- ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐   │
- Scraper    │  │    Cache     │  │   API Handlers   │   │
- Engine     │  │   System     │  │                  │   │
- ┬───────┘  └──────┬───────┘  └──────────┬───────┘   │
- ┼────────────────┼─────────────────────┼────────────┘
- ↓                ↓                      ↓
**┌────────────────┐ ┌─────────────┐    ┌──────────────────┐**
- G2.com      │ │  Local      │    │    RapidAPI      │
- Website     │ │  Storage    │    │    Service       │
- ┘ └─────────────┘    └──────────────────┘

##### Key Relationships to Highlight

a. Data Flow:
- User input → Frontend → Backend → G2.com
- Scraped data → Cache → Frontend
- API requests → RapidAPI → Frontend

b. Component Interactions:
- Frontend-Backend communication
- Cache system integration
- External API integration

##### Important Implementation Details

- Botasaurus framework integration
- Caching mechanism
- Parallel processing capabilities
- Error handling and retry mechanisms
- Data transformation flow

##### Additional Elements

- Component types
- Data flow directions
- Integration points
- Caching mechanisms
- API endpoints

##### Technical Specifications

Key technical aspects:
- Python backend with Botasaurus framework
- Next.js frontend
- JSON-based cache storage
- RapidAPI integration
- Parallel scraping capabilities

This system design should effectively communicate the architecture while maintaining clarity and showing the sophisticated nature of the scraping system, including its caching, parallel processing, and error handling capabilities.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Client
        Browser["Client Browser"]
    end

    subgraph FrontendLayer["Frontend Layer"]
        UI["User Interface"]:::frontend
        Pages["Pages"]:::frontend
        APIInt["API Integration"]:::frontend
        StateMan["State Management"]:::frontend
        Components["Components"]:::frontend
    end

    subgraph BackendLayer["Backend Layer"]
        ScrapingEngine["Scraping Engine"]:::backend
        CacheSystem["Cache System"]:::backend
        APIHandlers["API Handlers"]:::backend
        DataProcess["Data Processing"]:::backend
        HelperUtils["Helper Utilities"]:::backend
    end

    subgraph ExternalServices["External Services"]
        G2["G2.com Website"]:::external
        RapidAPI["RapidAPI Service"]:::external
    end

    subgraph Storage["Storage Layer"]
        LocalCache["Local Cache Storage"]:::storage
        ProductCache["Product Cache"]:::storage
        CategoryCache["Category Cache"]:::storage
    end

    %% Relationships
    Browser --> UI
    UI --> Pages
    UI --> Components
    UI --> APIInt
    Components --> StateMan
    APIInt --> APIHandlers
    APIHandlers --> ScrapingEngine
    ScrapingEngine --> G2
    ScrapingEngine --> CacheSystem
    CacheSystem --> LocalCache
    LocalCache --> ProductCache
    LocalCache --> CategoryCache
    APIHandlers --> RapidAPI
    ScrapingEngine --> DataProcess
    DataProcess --> HelperUtils

    %% Click Events
    click Components "https://github.com/omkarcloud/g2-scraper/tree/master/frontend/src/components/"
    click APIInt "https://github.com/omkarcloud/g2-scraper/blob/master/frontend/src/utils/api.tsx"
    click Pages "https://github.com/omkarcloud/g2-scraper/tree/master/frontend/src/pages/"
    click StateMan "https://github.com/omkarcloud/g2-scraper/blob/master/frontend/src/components/auth/context.tsx"
    click ScrapingEngine "https://github.com/omkarcloud/g2-scraper/blob/master/src/g2/g2_scraper.py"
    click DataProcess "https://github.com/omkarcloud/g2-scraper/blob/master/backend/inputs/get_g2_products.js"
    click ProductCache "https://github.com/omkarcloud/g2-scraper/tree/master/cache/get_products/"
    click CategoryCache "https://github.com/omkarcloud/g2-scraper/tree/master/cache/get_products_by_category/"
    click HelperUtils "https://github.com/omkarcloud/g2-scraper/blob/master/src/g2/helpers.py"
    click APIHandlers "https://github.com/omkarcloud/g2-scraper/tree/master/frontend/src/components/ApiIntegrationComponent/"

    %% Styling
    classDef frontend fill:#3498db,stroke:#2980b9,color:white
    classDef backend fill:#2ecc71,stroke:#27ae60,color:white
    classDef external fill:#e67e22,stroke:#d35400,color:white
    classDef storage fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Legend
    subgraph Legend
        Frontend["Frontend Component"]:::frontend
        Backend["Backend Component"]:::backend
        External["External Service"]:::external
        Storage["Storage Component"]:::storage
    end
```
