# rahuletto/goscraper architecture

> Architecture diagram of the GitHub repository rahuletto/goscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rahuletto/goscraper
- Repository: https://github.com/rahuletto/goscraper

## Overview

##### Project type and purpose
- The project is a Golang API focused on web scraping. It fetches HTML content from specific URLs and parses it dynamically using regular expressions. Its design emphasizes performance, adaptability to HTML changes, and secure data handling with encryption.
- The integration with Supabase for storing scraped data is a core part of its functionality. The project also supports deployment via Docker.

##### File structure
- At the top level, you have configuration files (e.g., Dockerfile, compose.yaml, .env.example) and Go module files (go.mod, go.sum) which set up environments, dependency management, and containerization.
- The src directory houses the main application:
- main.go serves as the entry point.
- The handlers directory contains API endpoints for various functionalities such as attendance, calendar, course, login, marks, timetable, and user management.
- The helpers directory encapsulates the business logic. There are domain-specific helpers (for attendance, calendar, course, etc.) as well as specialized helpers interacting with databases.
- The globals directory and configuration settings (e.g., DevMode) provide runtime behaviors.
- The types directory defines the core data structures used throughout the application.
- The utils directory holds utility functions for encoding, error handling, parsing, regex operations, and validation.
- The separation of handlers and helpers indicates a clear design to keep request routing and business logic independent.

##### Evaluate insights from the README
- The README provides an overview of environment variables required for secure operations (SUPABASE URL/KEY, encryption, validation, and CORS settings).
- It details the Supabase setup including database tables (goscrape and gocal) and mentions cron jobs for data maintenance.
- The instructions emphasize both development and production builds as well as Docker-based deployment.

a. Main components to include:
- API Gateway / Router: Representing the entry-point (main.go) that receives requests.
- Handlers: Components that manage specific API endpoints (attendance, calendar, course, login, marks, timetable, user). They are responsible for parsing the incoming requests.
- Business Logic / Helpers: Modules that encapsulate the bulk of the logic for scraping, regex parsing, and domain-specific operations. This layer also includes database helpers.
- Utility Functions: A shared library handling encoding, error management, parsing, regex operations, and input validation.
- Data Types: The defined models representing attendance, calendar, course, marks, timetable, and user data.
- External Services: Supabase as the primary database and, indirectly, any cron job functionalities for maintenance tasks.

b. Relationships and interactions:
- Incoming HTTP requests are received by the API Gateway and routed to the respective handlers.
- Each handler calls the appropriate helper functions to perform tasks such as scraping HTML content, parsing it via regex, and validating inputs.
- The helper functions interact with the Supabase database (using database-specific helper functions) to store and retrieve data.
- Utility functions are leveraged across handlers and helpers for common tasks (e.g., error handling, encoding, etc.).
- Environmental configurations (from the .env file) inform interactions—especially for validating requests and secure data encryption.

c. Architectural patterns and design principles:
- Separation of Concerns: Handlers separate request processing from business logic and data operations.
- Modular Design: The use of distinct directories (handlers, helpers, utils, types) encourages readable and maintainable code.
- Configuration-Driven Behavior: Use of environmental variables for secure credentials and operational mode toggling (DevMode).
- Integration with External Systems: The design leverages Supabase for data persistence and uses cron jobs (configured on the Supabase side) for periodic clean-up.

d. Relevant technology and tools:
- Go as the core programming language.
- Docker for containerization and deployment.
- Supabase as a backend-as-a-service platform for database interactions and cron job scheduling.
- Regex utilities within the utils package for dynamic HTML content scraping.

- Use clear labels for each major component such as “API Gateway”, “Handlers (attendance, calendar, etc.)”, “Business Logic / Helpers”, “Utility Functions”, and “Supabase Database”.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global external client request
    Client["Client Request"]:::external

    %% API Layer
    subgraph "API Layer"
        API["API Gateway / Router"]:::api
        HANDLERS["Handlers (attendance,calendar,course,login,marks,timetable,user)"]:::api
    end

    %% Business Logic Layer
    subgraph "Business Logic Layer"
        LOGIC["Business Logic / Helpers"]:::helpers
        DBHELPERS["DB Helpers (Supabase integration)"]:::helpers
    end

    %% Shared Libraries & Configurations
    subgraph "Shared Library"
        UTILS["Utility Functions"]:::utility
        DATATYPES["Data Types"]:::utility
        GLOBAL["Global Configuration"]:::config
    end

    %% External Database
    SUPABASE["Supabase Database\n(tables: goscrape & gocal, Cron jobs)"]:::external

    %% Infrastructure / Deployment Components
    subgraph "Infrastructure"
        DOCKER["Dockerfile"]:::deployment
        COMPOSE["compose.yaml"]:::deployment
        ENV[".env.example"]:::deployment
    end

    %% Flow of interactions
    Client -->|"HTTP_Request"| API
    API -->|"Routes"| HANDLERS
    HANDLERS -->|"Invokes"| LOGIC
    LOGIC -->|"Calls"| DBHELPERS
    DBHELPERS -->|"Queries"| SUPABASE
    SUPABASE -->|"Returns_Data"| DBHELPERS

    %% Shared library usage
    HANDLERS -->|"Uses"| UTILS
    LOGIC -->|"Uses"| UTILS
    LOGIC -->|"Relies_on"| DATATYPES
    API -->|"Reads"| GLOBAL
    LOGIC -->|"Reads"| GLOBAL

    %% Deployment influence
    DOCKER ---|"Containers"| API
    DOCKER ---|"Containers"| LOGIC
    COMPOSE ---|"Orchestrates"| API
    COMPOSE ---|"Orchestrates"| LOGIC
    ENV ---|"Configures"| API

    %% Class definitions for color coding
    classDef api fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef helpers fill:#ABEBC6,stroke:#229954,stroke-width:2px;
    classDef utility fill:#F9E79F,stroke:#D4AC0D,stroke-width:2px;
    classDef config fill:#F5CBA7,stroke:#CA6F1E,stroke-width:2px;
    classDef deployment fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef external fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;

    %% Click events for components mapped to file paths
    click API "https://github.com/rahuletto/goscraper/blob/main/src/main.go"
    click HANDLERS "https://github.com/rahuletto/goscraper/tree/main/src/handlers/"
    click LOGIC "https://github.com/rahuletto/goscraper/tree/main/src/helpers/"
    click DBHELPERS "https://github.com/rahuletto/goscraper/tree/main/src/helpers/databases/"
    click UTILS "https://github.com/rahuletto/goscraper/tree/main/src/utils/"
    click DATATYPES "https://github.com/rahuletto/goscraper/tree/main/src/types/"
    click GLOBAL "https://github.com/rahuletto/goscraper/blob/main/src/globals/DevMode.go"
    click DOCKER "https://github.com/rahuletto/goscraper/tree/main/dockerfile"
    click COMPOSE "https://github.com/rahuletto/goscraper/blob/main/compose.yaml"
    click ENV "https://github.com/rahuletto/goscraper/blob/main/.env.example"
```
