How rahuletto/goscraper works
Project type and purpose
• The project is a Golang API focused on web scraping. It fetches HTML content from specific URLs and parses it dynamically using regular expressions. Its design emphasizes performance, adaptability to HTML changes, and secure data handling with encryption.
• The integration with Supabase for storing scraped data is a core part of its functionality. The project also supports deployment via Docker.
File structure
• At the top level, you have configuration files (e.g., Dockerfile, compose.yaml, .env.example) and Go module files (go.mod, go.sum) which set up environments, dependency management, and containerization.
• The src directory houses the main application:
• main.go serves as the entry point.
• The handlers directory contains API endpoints for various functionalities such as attendance, calendar, course, login, marks, timetable, and user management.
• The helpers directory encapsulates the business logic. There are domain-specific helpers (for attendance, calendar, course, etc.) as well as specialized helpers interacting with databases.
• The globals directory and configuration settings (e.g., DevMode) provide runtime behaviors.
• The types directory defines the core data structures used throughout the application.
• The utils directory holds utility functions for encoding, error handling, parsing, regex operations, and validation.
• The separation of handlers and helpers indicates a clear design to keep request routing and business logic independent.
Evaluate insights from the README
• The README provides an overview of environment variables required for secure operations (SUPABASE URL/KEY, encryption, validation, and CORS settings).
• It details the Supabase setup including database tables (goscrape and gocal) and mentions cron jobs for data maintenance.
• The instructions emphasize both development and production builds as well as Docker-based deployment.
a. Main components to include:
• API Gateway / Router: Representing the entry-point (main.go) that receives requests.
• Handlers: Components that manage specific API endpoints (attendance, calendar, course, login, marks, timetable, user). They are responsible for parsing the incoming requests.
• Business Logic / Helpers: Modules that encapsulate the bulk of the logic for scraping, regex parsing, and domain-specific operations. This layer also includes database helpers.
• Utility Functions: A shared library handling encoding, error management, parsing, regex operations, and input validation.
• Data Types: The defined models representing attendance, calendar, course, marks, timetable, and user data.
• External Services: Supabase as the primary database and, indirectly, any cron job functionalities for maintenance tasks.
b. Relationships and interactions:
• Incoming HTTP requests are received by the API Gateway and routed to the respective handlers.
• Each handler calls the appropriate helper functions to perform tasks such as scraping HTML content, parsing it via regex, and validating inputs.
• The helper functions interact with the Supabase database (using database-specific helper functions) to store and retrieve data.
• Utility functions are leveraged across handlers and helpers for common tasks (e.g., error handling, encoding, etc.).
• Environmental configurations (from the .env file) inform interactions—especially for validating requests and secure data encryption.
c. Architectural patterns and design principles:
• Separation of Concerns: Handlers separate request processing from business logic and data operations.
• Modular Design: The use of distinct directories (handlers, helpers, utils, types) encourages readable and maintainable code.
• Configuration-Driven Behavior: Use of environmental variables for secure credentials and operational mode toggling (DevMode).
• Integration with External Systems: The design leverages Supabase for data persistence and uses cron jobs (configured on the Supabase side) for periodic clean-up.
d. Relevant technology and tools:
• Go as the core programming language.
• Docker for containerization and deployment.
• Supabase as a backend-as-a-service platform for database interactions and cron job scheduling.
• Regex utilities within the utils package for dynamic HTML content scraping.
• Use clear labels for each major component such as “API Gateway”, “Handlers (attendance, calendar, etc.)”, “Business Logic / Helpers”, “Utility Functions”, and “Supabase Database”.