How daohoangson/go-sitemirror works
Project Type and Overall Purpose
• The project is a website mirroring application written in Go. It is designed to download and cache web assets so that if a source server goes down, an almost exact mirror can be served to users.
• Its core functions include crawling websites, caching the downloaded data, orchestrating these processes, and serving the cached content via an HTTP server.
Main Components
• Entry Point (main.go): This bootstraps the application.
• Crawler Module (crawler/):
– Contains the logic to fetch website assets.
– Files such as crawler.go, download.go, and utils.go indicate that this part handles URL crawling, content downloading, and utilities for processing direct downloads.
• Cacher Module (cacher/):
– Manages HTTP caching for downloaded assets.
– Files like cacher.go, fileop.go, and related test files suggest operations on file I/O and cache management (metadata, content types, etc.).
• Engine Module (engine/):
– Acts as the orchestrator. It loads configurations and likely manages periodic refreshing of cache, along with invoking the crawler/cacher as needed.
• Web Server (web/):
– Implements an HTTP server to serve the mirrored content.
– The presence of web/internal indicates additional layers or helper functions for serving specific information (possibly for monitoring or detailed endpoint configuration).
• Testing/Utilities (testing/):
– Contains modules such as logger, os, and other utilities to support testing across the application.
• External/Dev Infrastructure:
– Files like Dockerfile, .dockerignore, and GitHub workflows indicate Docker-based deployments and CI/CD processing.
– The fly.toml configuration in the README describes deployment specifics to platforms like Fly.io.
Relationships and Data Flow
• Entry Point (main.go) initializes the configuration (from engine), starts the caching routines (cacher) and web server (web), and triggers the crawler as needed.
• The Crawler fetches web assets (following the specified auto-download depth and restrictions as given in the flags) and passes the downloaded data to the Cacher.
• The Cacher handles file I/O operations—storing web assets along with their metadata—and ensures data freshness (cache bump and TTL settings).
• The Engine interconnects these parts by managing configurations (reading from command-line flags), scheduling auto-refresh tasks, and handling periodic cache updates.
• The Web Server then serves the cached content to users and may also provide diagnostic endpoints (as seen in web/internal components).
Architectural Patterns and Design Principles
• Modularity: The clear separation among the crawler, cacher, engine, and web modules ensures high cohesion within modules and loose coupling between them.
• Component Separation: Each module has a defined responsibility (downloading, caching, orchestration, serving) making the application easier to maintain and extend.
• Dependency Management: Use of configuration parameters (flag-driven) and DI (implicit when calling functions across modules) ensures flexibility in behavior.
• Testable Code: The presence of extensive test suites (e.g., suite_test.go files) indicate that the system is designed for unit and integration testing.