# daohoangson/go-sitemirror architecture

> Architecture diagram of the GitHub repository daohoangson/go-sitemirror, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daohoangson/go-sitemirror
- Repository: https://github.com/daohoangson/go-sitemirror

## Overview

##### Project Type and Overall Purpose
- The project is a website mirroring application written in Go. It is designed to download and cache web assets so that if a source server goes down, an almost exact mirror can be served to users.
- Its core functions include crawling websites, caching the downloaded data, orchestrating these processes, and serving the cached content via an HTTP server.

##### Main Components
- Entry Point (main.go): This bootstraps the application.
- Crawler Module (crawler/):
– Contains the logic to fetch website assets.
– Files such as crawler.go, download.go, and utils.go indicate that this part handles URL crawling, content downloading, and utilities for processing direct downloads.
- Cacher Module (cacher/):
– Manages HTTP caching for downloaded assets.
– Files like cacher.go, fileop.go, and related test files suggest operations on file I/O and cache management (metadata, content types, etc.).
- Engine Module (engine/):
– Acts as the orchestrator. It loads configurations and likely manages periodic refreshing of cache, along with invoking the crawler/cacher as needed.
- Web Server (web/):
– Implements an HTTP server to serve the mirrored content.
– The presence of web/internal indicates additional layers or helper functions for serving specific information (possibly for monitoring or detailed endpoint configuration).
- Testing/Utilities (testing/):
– Contains modules such as logger, os, and other utilities to support testing across the application.
- External/Dev Infrastructure:
– Files like Dockerfile, .dockerignore, and GitHub workflows indicate Docker-based deployments and CI/CD processing.
– The fly.toml configuration in the README describes deployment specifics to platforms like Fly.io.

##### Relationships and Data Flow
- Entry Point (main.go) initializes the configuration (from engine), starts the caching routines (cacher) and web server (web), and triggers the crawler as needed.
- The Crawler fetches web assets (following the specified auto-download depth and restrictions as given in the flags) and passes the downloaded data to the Cacher.
- The Cacher handles file I/O operations—storing web assets along with their metadata—and ensures data freshness (cache bump and TTL settings).
- The Engine interconnects these parts by managing configurations (reading from command-line flags), scheduling auto-refresh tasks, and handling periodic cache updates.
- The Web Server then serves the cached content to users and may also provide diagnostic endpoints (as seen in web/internal components).

##### Architectural Patterns and Design Principles
- Modularity: The clear separation among the crawler, cacher, engine, and web modules ensures high cohesion within modules and loose coupling between them.
- Component Separation: Each module has a defined responsibility (downloading, caching, orchestration, serving) making the application easier to maintain and extend.
- Dependency Management: Use of configuration parameters (flag-driven) and DI (implicit when calling functions across modules) ensures flexibility in behavior.
- Testable Code: The presence of extensive test suites (e.g., suite_test.go files) indicate that the system is designed for unit and integration testing.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Application Core
    subgraph "Application Core"
        E["Entry Point (main.go)"]:::core
        EN["Engine Module"]:::core
        CR["Crawler Module"]:::core
        CA["Cacher Module"]:::core
        WS["Web Server"]:::frontend
    end

    E -->|"init"| EN
    EN -->|"ScheduleRefresh"| CR
    EN -->|"ManageCache"| CA
    CR -->|"DownloadAsset"| CA
    CA -->|"ServeContent"| WS
    E -->|"StartServer"| WS

    %% Testing & Utilities
    subgraph "Testing & Utilities"
        T["Testing/Utilities"]:::testing
    end

    %% External & Dev Infrastructure
    subgraph "External & Dev Infrastructure"
        EX["Dev Infrastructure"]:::external
        DF["Dockerfile"]:::external
        DN[".dockerignore"]:::external
        GH[".github/workflows"]:::external
        EX --> DF
        EX --> DN
        EX --> GH
    end

    %% Click Events
    click E "https://github.com/daohoangson/go-sitemirror/blob/master/main.go"
    click CR "https://github.com/daohoangson/go-sitemirror/tree/master/crawler/"
    click CA "https://github.com/daohoangson/go-sitemirror/tree/master/cacher/"
    click EN "https://github.com/daohoangson/go-sitemirror/tree/master/engine/"
    click WS "https://github.com/daohoangson/go-sitemirror/tree/master/web/"
    click T "https://github.com/daohoangson/go-sitemirror/tree/master/testing/"
    click DF "https://github.com/daohoangson/go-sitemirror/tree/master/Dockerfile"
    click DN "https://github.com/daohoangson/go-sitemirror/blob/master/.dockerignore"
    click GH "https://github.com/daohoangson/go-sitemirror/tree/master/.github/workflows/"

    %% Styles
    classDef core fill:#ffe6e6,stroke:#cc0000,stroke-width:2px;
    classDef frontend fill:#e6f7ff,stroke:#0066cc,stroke-width:2px;
    classDef testing fill:#e6ffe6,stroke:#009900,stroke-width:2px;
    classDef external fill:#f9e6ff,stroke:#9900cc,stroke-width:2px;
```
