# root-daemon/goscraper architecture

> Architecture diagram of the GitHub repository root-daemon/goscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/root-daemon/goscraper
- Repository: https://github.com/root-daemon/goscraper

## Overview

##### Project Type and Purpose
- This is a backend service implemented in Golang. Its core purpose is to fetch HTML content from specified URLs, parse that content dynamically using regex, and then process and possibly store or handle the data.
- The README mentions integration with Supabase (via environment variables such as SUPABASE_URL and SUPABASE_KEY), as well as encryption and validation keys – indicating that the API not only parses HTML content but also manages secure data interactions.

##### Analyzing the File Structure
- At the top level, you have configuration files (dockerfile, compose.yaml, .env.example) that indicate containerized deployment.
- The project’s entry point is located in src/main.go.
- The src/handlers directory holds multiple endpoint handlers (attendance.go, calendar.go, course.go, login.go, marks.go, timetable.go, user.go) which represent the various API routes for handling different business functions.
- The src/helpers directory (and its subdirectories) is organized to provide domain-specific functionalities, for example, for each domain (attendance, calendar, course, etc.) there is an associated helper. Moreover, a nested src/helpers/databases directory suggests encapsulated logic for database interactions or Supabase communication.
- The src/types folder defines data types (e.g., AttendanceType, CourseType, etc.), which implies strict typing and structure for the API’s domain objects.
- The src/utils folder contains various utility modules for encoding, error handling, parsing, regex-based extraction, and validation – underlining a design that separates infrastructure code from business logic.

##### Main Components and Their Responsibilities
a. API Server (Entry Point & Routing):
- src/main.go serves as the starting gateway to the application. It likely sets up HTTP routing and middleware.
- The API server receives incoming requests from clients (which may include a front-end or other external clients).

b. Handlers (Controller Layer):
- Each file in src/handlers (e.g., login, course, etc.) corresponds to an endpoint in the API.
- They parse the HTTP request, perform authentication/validation (using utilities and validation keys), and delegate operations to helper functions.

c. Helpers (Business Logic & Domain Operations):
- Domain-specific helpers (attendance, calendar, course, etc.) encapsulate the business logic.
- They may also coordinate with the database-related helpers (located in src/helpers/databases) to fetch or store data securely. This may include encryption (using the provided ENCRYPTION_KEY) and interactions with Supabase.

d. Utility Modules:
- Located in src/utils, these modules provide shared code such as regex parsing, data encoding, error handling, and request validation. They help keep the system DRY and maintainable.

e. Data Types (Models):
- The src/types files define the data models that structure the data processed through the system.
- This ensures that data conforms to expected schemas as it flows between components.

f. Deployment and Containerization:
- The presence of dockerfile, compose.yaml, .env.example, and .dockerignore indicates that the project is containerized.
- These configuration files manage environment variables, external service integrations (Supabase), and deployment automation.

##### Relationships and Data Flow
- Utility modules interact with handlers or helpers when data parsing, validation, or error management is required.

##### Architectural Patterns and Design Principles
- Separation of concerns: Keep business logic separated from request handling, which improves maintainability and testability.
- Containerization: The Docker-related files indicate that the application is designed to be deployed in a containerized environment – show a “Container/Deployment” layer interacting with the API infrastructure.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client"
        C["Client Request"]:::client
    end

    %% Deployment / Containerization Layer
    subgraph "Container/Deployment"
        D1["Dockerfile"]:::deployment
        D2["Compose File"]:::deployment
        D3["Env Sample"]:::deployment
        D4["Docker Ignore"]:::deployment
    end

    %% API Layer (Entry Point)
    subgraph "API Layer"
        API["API Server (main.go)"]:::entry
    end

    %% Handler Layer
    subgraph "Handler Layer"
        H1["Attendance Handler"]:::handler
        H2["Calendar Handler"]:::handler
        H3["Course Handler"]:::handler
        H4["Login Handler"]:::handler
        H5["Marks Handler"]:::handler
        H6["Timetable Handler"]:::handler
        H7["User Handler"]:::handler
    end

    %% Business Logic (Helpers)
    subgraph "Business Logic"
        B1["Attendance Helper"]:::helper
        B2["Calendar Helper"]:::helper
        B3["Course Helper"]:::helper
        B4["Timetable Helper"]:::helper
        B5["User Helper"]:::helper
    end

    %% Database Helpers
    subgraph "Database Helpers"
        DB1["General Database Helper"]:::db
        DB2["Calendar Database Helper"]:::db
    end

    %% Utility Modules
    subgraph "Utility Modules"
        U["Utilities (Regex, Encode, Error, Validate, etc.)"]:::util
    end

    %% Data Models
    subgraph "Data Models"
        M["Data Types (Types)"]:::model
    end

    %% External Service
    subgraph "External Services"
        EX["Supabase"]:::external
    end

    %% Connections and Data Flow
    C -->|"sends_request"| API
    D1 -->|"configures"| API
    API -->|"routes_to"| H1
    API -->|"routes_to"| H2
    API -->|"routes_to"| H3
    API -->|"routes_to"| H4
    API -->|"routes_to"| H5
    API -->|"routes_to"| H6
    API -->|"routes_to"| H7

    H1 -->|"invokes"| B1
    H2 -->|"invokes"| B2
    H3 -->|"invokes"| B3
    H6 -->|"invokes"| B4
    H7 -->|"invokes"| B5
    %% For handlers without explicit helpers, utilize utilities directly
    H4 -->|"utilizes"| U
    H5 -->|"utilizes"| U

    B1 -->|"calls"| DB1
    B3 -->|"calls"| DB1
    B4 -->|"calls"| DB1
    B5 -->|"calls"| DB1
    B2 -->|"calls"| DB2

    %% Database helpers interacting with external service
    DB1 -->|"communicates"| EX
    DB2 -->|"communicates"| EX

    %% Helpers use Data Models for data structuring
    B1 -->|"uses"| M
    B2 -->|"uses"| M
    B3 -->|"uses"| M
    B4 -->|"uses"| M
    B5 -->|"uses"| M

    %% Utility modules support both Handlers and Helpers (dotted connections)
    U ---|"supports"| H1
    U ---|"supports"| H2
    U ---|"supports"| H3
    U ---|"supports"| B1
    U ---|"supports"| B2
    U ---|"supports"| B3
    U ---|"supports"| B4
    U ---|"supports"| B5

    %% Click Events for API Server and Components
    click API "https://github.com/root-daemon/goscraper/blob/main/src/main.go"
    click H1 "https://github.com/root-daemon/goscraper/blob/main/src/handlers/attendance.go"
    click H2 "https://github.com/root-daemon/goscraper/blob/main/src/handlers/calendar.go"
    click H3 "https://github.com/root-daemon/goscraper/blob/main/src/handlers/course.go"
    click H4 "https://github.com/root-daemon/goscraper/blob/main/src/handlers/login.go"
    click H5 "https://github.com/root-daemon/goscraper/blob/main/src/handlers/marks.go"
    click H6 "https://github.com/root-daemon/goscraper/blob/main/src/handlers/timetable.go"
    click H7 "https://github.com/root-daemon/goscraper/blob/main/src/handlers/user.go"
    click B1 "https://github.com/root-daemon/goscraper/blob/main/src/helpers/AttendanceHelper.go"
    click B2 "https://github.com/root-daemon/goscraper/blob/main/src/helpers/CalendarHelper.go"
    click B3 "https://github.com/root-daemon/goscraper/blob/main/src/helpers/CourseHelper.go"
    click B4 "https://github.com/root-daemon/goscraper/blob/main/src/helpers/TimetableHelper.go"
    click B5 "https://github.com/root-daemon/goscraper/blob/main/src/helpers/UserHelper.go"
    click DB1 "https://github.com/root-daemon/goscraper/blob/main/src/helpers/databases/DatabaseHelper.go"
    click DB2 "https://github.com/root-daemon/goscraper/blob/main/src/helpers/databases/CalendarDBHelper.go"
    click M "https://github.com/root-daemon/goscraper/tree/main/src/types"
    click U "https://github.com/root-daemon/goscraper/tree/main/src/utils"
    click D1 "https://github.com/root-daemon/goscraper/tree/main/dockerfile"
    click D2 "https://github.com/root-daemon/goscraper/blob/main/compose.yaml"
    click D3 "https://github.com/root-daemon/goscraper/blob/main/.env.example"
    click D4 "https://github.com/root-daemon/goscraper/blob/main/.dockerignore"

    %% Class Definitions for Color Coding
    classDef client fill:#c7f9cc,stroke:#333,stroke-width:2px
    classDef deployment fill:#e0e0e0,stroke:#333,stroke-width:2px
    classDef entry fill:#ffcccb,stroke:#333,stroke-width:2px
    classDef handler fill:#add8e6,stroke:#333,stroke-width:2px
    classDef helper fill:#ffe4b5,stroke:#333,stroke-width:2px
    classDef db fill:#fddbb0,stroke:#333,stroke-width:2px
    classDef util fill:#fff799,stroke:#333,stroke-width:2px
    classDef model fill:#d3f8e2,stroke:#333,stroke-width:2px
    classDef external fill:#ffc0cb,stroke:#333,stroke-width:2px
```
