# yesbhautik/public-api-scraper architecture

> Architecture diagram of the GitHub repository yesbhautik/public-api-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yesbhautik/public-api-scraper
- Repository: https://github.com/yesbhautik/public-api-scraper

## Overview

##### Frontend
- **Web Interface:** `templates/index.html`
- **User Interaction:** Input fields for model, endpoint, and search keywords; display of search progress and results.
##### Backend
- **Main Application:** `main.go`
- **GitHub Search Module:** `internal/github/search.go`
- **API Key Verifier:** `internal/verifier/api_verifier.go`
##### Real-Time Communication
- **WebSocket Server:** Facilitates live updates between frontend and backend.
##### Database (if any)
- Current file tree does not indicate a database; data may be held in-memory or logs.
##### Docker Services
- **Web Service Container:** Runs the Go application.
- **Docker Compose:** Manages multi-container deployment if extended.
##### CI/CD Pipelines
- **GitHub Actions:** Defined in `.github/workflows/` for testing and Docker image publishing.

**b. Determine Relationships and Interactions:**
- **User ↔ Web Interface:** Users interact with the frontend to initiate searches.
- **Web Interface ↔ WebSocket Server:** Facilitates real-time communication for search progress and results.
- **WebSocket Server ↔ Backend Modules:** Backend processes handle GitHub searches and API verification, sending updates back to the frontend via WebSockets.
- **Backend Modules ↔ GitHub API:** Backend interacts with GitHub using personal access tokens to search repositories.
- **Docker Services ↔ Host System:** Docker containers deploy and run the application in various environments.

**c. Highlight Architectural Patterns or Design Principles:**
- **Layered Architecture:** Separation between presentation (frontend), business logic (backend modules), and communication (WebSockets).
- **Modular Design:** Distinct internal packages (`github`, `verifier`) promote maintainability and scalability.
- **Containerization:** Ensures consistent deployment across different environments.
- **Real-Time Data Handling:** Utilizes WebSockets for dynamic user experience.

**d. Include Relevant Technologies, Frameworks, or Libraries:**
- **Programming Language:** Go
- **Web Server:** Go’s standard `net/http` package
- **WebSockets:** Likely using a Go WebSocket library (e.g., Gorilla WebSocket)
- **Docker:** For containerization
- **GitHub API:** For searching repositories and accessing data
- **CI/CD Tools:** GitHub Actions for automation

##### Elements

- **Clear Labels:**

- Data flow (e.g., User Actions → Frontend → Backend → GitHub API).
- Dependencies (e.g., Backend relies on GitHub API).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global entities
    User("User") 

    %% Subgraphs and modules

    subgraph "Frontend"
        Frontend["Frontend Web Interface"]:::frontend
    end

    subgraph "Communication"
        WS["WebSocket Server"]:::communication
    end

    subgraph "Backend"
        Backend["Backend Server"]:::backend
        GitHubSearch["GitHub Search Module"]:::backend
        APIVerifier["API Key Verifier"]:::backend
    end

    subgraph "External Services"
        GitHubAPI["GitHub API"]:::external
    end

    subgraph "Deployment"
        Dockerfile["Dockerfile"]:::docker
        DockerCompose["docker-compose.yml"]:::docker
    end

    subgraph "CI/CD"
        CI_Test[".github/workflows/docker-image-test.yml"]:::ci_cd
        CI_Publish[".github/workflows/docker-publish.yml.tmp"]:::ci_cd
    end

    %% Connections
    User -->|"Interacts"| Frontend
    Frontend -->|"Uses WebSocket"| WS
    WS -->|"Communicates with"| Backend
    Backend -->|"Uses"| GitHubSearch
    Backend -->|"Uses"| APIVerifier
    GitHubSearch -->|"Calls"| GitHubAPI
    APIVerifier -->|"Validates with"| GitHubAPI

    Backend -->|"Containerized by"| Dockerfile
    DockerCompose -->|"Manages"| Dockerfile
    DockerCompose -->|"Deploys"| Backend

    CI_Test -->|"Triggers on code changes"| Dockerfile
    CI_Publish -->|"Publishes Docker Images"| Dockerfile

    %% Click Events
    click Frontend "https://github.com/yesbhautik/public-api-scraper/blob/main/templates/index.html"
    click Backend "https://github.com/yesbhautik/public-api-scraper/blob/main/main.go"
    click GitHubSearch "https://github.com/yesbhautik/public-api-scraper/blob/main/internal/github/search.go"
    click APIVerifier "https://github.com/yesbhautik/public-api-scraper/blob/main/internal/verifier/api_verifier.go"
    click Dockerfile "https://github.com/yesbhautik/public-api-scraper/tree/main/Dockerfile"
    click DockerCompose "https://github.com/yesbhautik/public-api-scraper/blob/main/docker-compose.yml"
    click CI_Test "https://github.com/yesbhautik/public-api-scraper/blob/main/.github/workflows/docker-image-test.yml"
    click CI_Publish "https://github.com/yesbhautik/public-api-scraper/blob/main/.github/workflows/docker-publish.yml.tmp"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef backend fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef communication stroke-dasharray: 5 5,stroke:#000,stroke-width:1px;
    classDef docker fill:#D3D3D3,stroke:#000,stroke-width:1px;
    classDef ci_cd fill:#FFB6C1,stroke:#000,stroke-width:1px;

    class Frontend frontend
    class Backend,GitHubSearch,APIVerifier backend
    class GitHubAPI external
    class WS communication
    class Dockerfile,DockerCompose docker
    class CI_Test,CI_Publish ci_cd
```
