How root-daemon/goscraper works
Project Type and Purpose
• This is a backend service implemented in Golang. Its core purpose is to fetch HTML content from specified URLs, parse that content dynamically using regex, and then process and possibly store or handle the data.
• The README mentions integration with Supabase (via environment variables such as SUPABASE_URL and SUPABASE_KEY), as well as encryption and validation keys – indicating that the API not only parses HTML content but also manages secure data interactions.
Analyzing the File Structure
• At the top level, you have configuration files (dockerfile, compose.yaml, .env.example) that indicate containerized deployment.
• The project’s entry point is located in src/main.go.
• The src/handlers directory holds multiple endpoint handlers (attendance.go, calendar.go, course.go, login.go, marks.go, timetable.go, user.go) which represent the various API routes for handling different business functions.
• The src/helpers directory (and its subdirectories) is organized to provide domain-specific functionalities, for example, for each domain (attendance, calendar, course, etc.) there is an associated helper. Moreover, a nested src/helpers/databases directory suggests encapsulated logic for database interactions or Supabase communication.
• The src/types folder defines data types (e.g., AttendanceType, CourseType, etc.), which implies strict typing and structure for the API’s domain objects.
• The src/utils folder contains various utility modules for encoding, error handling, parsing, regex-based extraction, and validation – underlining a design that separates infrastructure code from business logic.
Main Components and Their Responsibilities
a. API Server (Entry Point & Routing):
• src/main.go serves as the starting gateway to the application. It likely sets up HTTP routing and middleware.
• The API server receives incoming requests from clients (which may include a front-end or other external clients).
b. Handlers (Controller Layer):
• Each file in src/handlers (e.g., login, course, etc.) corresponds to an endpoint in the API.
• They parse the HTTP request, perform authentication/validation (using utilities and validation keys), and delegate operations to helper functions.
c. Helpers (Business Logic & Domain Operations):
• Domain-specific helpers (attendance, calendar, course, etc.) encapsulate the business logic.
• They may also coordinate with the database-related helpers (located in src/helpers/databases) to fetch or store data securely. This may include encryption (using the provided ENCRYPTION_KEY) and interactions with Supabase.
d. Utility Modules:
• Located in src/utils, these modules provide shared code such as regex parsing, data encoding, error handling, and request validation. They help keep the system DRY and maintainable.
e. Data Types (Models):
• The src/types files define the data models that structure the data processed through the system.
• This ensures that data conforms to expected schemas as it flows between components.
f. Deployment and Containerization:
• The presence of dockerfile, compose.yaml, .env.example, and .dockerignore indicates that the project is containerized.
• These configuration files manage environment variables, external service integrations (Supabase), and deployment automation.
Relationships and Data Flow
• Utility modules interact with handlers or helpers when data parsing, validation, or error management is required.
Architectural Patterns and Design Principles
• Separation of concerns: Keep business logic separated from request handling, which improves maintainability and testability.
• Containerization: The Docker-related files indicate that the application is designed to be deployed in a containerized environment – show a “Container/Deployment” layer interacting with the API infrastructure.