# xroche/httrack architecture

> Architecture diagram of the GitHub repository xroche/httrack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xroche/httrack
- Repository: https://github.com/xroche/httrack

## Overview

##### Project Type and Purpose
- The project is an offline website copier (an offline browser) that downloads an entire website to a local directory.
- Its key use case is to mirror websites—including HTML pages, images, and other content—and maintain the relative link structure.
- It supports both a command-line version (for Linux/Unix/BSD) and a graphical/web interface (WebHTTrack/WinHTTrack).

##### Analyzing the File Structure
- Top-Level Directories:
- “src”: Contains the main source code for the offline browser including the core downloading logic, cache, filters, network communication, and modules (e.g., htsnet.h, htsparse.c, htscache.c).
- “html”: Provides the static assets, help pages, and documentation for the web interface (WebHTTrack).
- “libtest”: Includes test examples for callbacks and API usage, indicating extensibility and testing of library functions.
- “tests”: Provides multiple automated tests to verify functionalities such as charset conversion, cookie handling, IDNA processing, etc.
- “templates”: Contains HTML templates used for generating index pages and user-facing output.
- “src/proxy”: Indicates that the project integrates or optionally uses proxy functionalities.
- “man”: Contains manual pages for command-line usage.
- Build Tools and Configuration Files: Files like configure.ac, Makefile.am, and others indicate the use of autotools to handle configuration and compilation.

##### Key Components
a. Core Engine / Downloader:
- Main “httrack” executable (found in src/httrack.c) and related libraries (such as libhttrack) that handle the website copying and download logic.
- Modules such as network (htsnet.h), caching (htscache.c), robots/filtering (htsrobots.c, htsfilters.c), and content handling (htsparse.c).

b. User Interfaces:
- Command-Line Interface: Direct interaction through the main executable.
- Web-Based Interface: Represented by “webhttrack” and related files in both src and html directories. This interface is responsible for providing a GUI experience for configuration and monitoring.

c. Proxy Module:
- A module under “src/proxy” which can act as an intermediary, capturing or forwarding requests.

d. Template and HTML Assets:
- Static webpages, templates, and assets under the “html” and “templates” directories that support the web interface.

e. Testing and Examples:
- “tests” and “libtest” directories ensure that each component (especially the core engine and API callbacks) is robust and offer samples for integration.

f. Build and Configuration Infrastructure:
- Autotools configuration files (configure.ac, Makefile.am, etc.) that drive the build process.
- Language support files (in “lang” folder) that indicate multi-language support for end users.

##### Relationships and Interactions
- User Input: A user (via CLI or WebUI) triggers website copying.
- The command is passed to the Core Engine which manages:
- Network communications (handling HTTP/FTP requests, using modules like htsnet).
- Parsing and processing content (html, filtering via htsfilters, caching via htscache).
- Managing the recursive download logic.
- The Template Engine generates local web pages using the files in “templates” and “html” to allow offline browsing.
- Optional components like the Proxy module are used if the user configures the system to route through an intermediary.
- Testing frameworks in “tests” and “libtest” provide feedback cycles for core functionalities.

##### Architectural Patterns and Technologies
- Modular Design: Components (downloader, filters, caching, UI) are clearly separated in different modules (files within src).
- Extensibility: The callbacks and plugin examples in “libtest” indicate a design that supports extending or tweaking functionality.
- Separation of Concerns: There’s a clear separation between user interfaces (CLI, WebUI), download engine, and proxy functionality.
- Build automation with GNU Autotools (configure.ac, Makefile.am) ensures portability across platforms.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Users
    U["User"]:::user

    %% User Interface Layer
    subgraph "User Interface Layer"
        CLI["Command-Line Interface (CLI)"]:::cli
        WEB["Web-Based Interface"]:::webui
    end

    %% Core Engine Layer
    subgraph "Core Engine Layer"
        CORE["Core Downloader / Engine"]:::core
        NET["Network Module"]:::core
        CACHE["Cache Manager"]:::core
        PARSER["Parser Module"]:::core
        FILTERS["Filters & Robots"]:::core
    end

    %% Auxiliary Modules
    subgraph "Auxiliary Modules"
        PROXY["Proxy Module"]:::aux
    end

    %% Static Content Layer
    subgraph "Static Content"
        TEMPLATE["Template & HTML Assets"]:::templ
    end

    %% Support Infrastructure
    subgraph "Support Infrastructure"
        BUILD["Build & Configuration"]:::build
        TEST["Testing & Examples"]:::test
    end

    %% Connections
    U --> CLI
    U --> WEB

    CLI --> CORE
    WEB --> CORE

    CORE --> NET
    CORE --> CACHE
    CORE --> PARSER
    CORE --> FILTERS

    CORE --> TEMPLATE
    CORE --> PROXY
    CORE --> TEST
    BUILD --> CORE
    TEMPLATE --> WEB

    %% Click Events
    click CLI "https://github.com/xroche/httrack/blob/master/src/htscoremain.c"
    click WEB "https://github.com/xroche/httrack/blob/master/src/webhttrack.vcproj"
    click CORE "https://github.com/xroche/httrack/blob/master/src/httrack.c"
    click NET "https://github.com/xroche/httrack/blob/master/src/htsnet.h"
    click CACHE "https://github.com/xroche/httrack/blob/master/src/htscache.c"
    click PARSER "https://github.com/xroche/httrack/blob/master/src/htsparse.c"
    click FILTERS "https://github.com/xroche/httrack/blob/master/src/htsrobots.c"
    click PROXY "https://github.com/xroche/httrack/blob/master/src/proxy/proxytrack.c"
    click TEMPLATE "https://github.com/xroche/httrack/blob/master/templates/index-header.html"
    click TEST "https://github.com/xroche/httrack/tree/master/tests"
    click BUILD "https://github.com/xroche/httrack/blob/master/configure.ac"

    %% Styles
    classDef user fill:#d4d4f7,stroke:#6c757d,stroke-width:2px;
    classDef cli fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef webui fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef core fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef aux fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef templ fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef test fill:#f5c6cb,stroke:#721c24,stroke-width:2px;
    classDef build fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
