# agude/wayback-machine-archiver architecture

> Architecture diagram of the GitHub repository agude/wayback-machine-archiver, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/agude/wayback-machine-archiver
- Repository: https://github.com/agude/wayback-machine-archiver

## Overview

##### Project type and purpose
- This is a standalone, open-source command-line utility written in Python, designed to submit web pages (or entire sitemaps) to the Internet Archive’s Wayback Machine.
- There is no persistent database or web frontend—its sole role is to ingest URLs, optionally parse sitemap XML, and call an external HTTP API.

##### File structure
- Top-level:
- setup.py, requirements.txt, run_test.sh – packaging, dependencies, test runner
- .github/workflows – CI pipelines for tests and releases
- tests/ – unit tests exercising each internal function
- wayback_machine_archiver/ – the main Python package
- Inside wayback_machine_archiver:
- archiver.py – contains the CLI entry point, argument parsing, job orchestration
- utility functions: extracting URLs from sitemaps, formatting Wayback Machine URLs, HTTP calls

##### Key insights from the README
- Command-line interface driven by argparse
- Supports positional URLs, local or remote sitemap.xml, file of URLs
- Concurrency controlled by a “jobs” flag, rate-limiting/backoff built in
- External dependency: requests library
- No database or long-term storage—results flow directly to the Internet Archive

##### Main components
a. User / CLI Layer
- User invokes “archiver” with flags and parameters
- Argparse handles: URLs, --file, --sitemaps, --jobs, --rate-limit-wait
b. Input Resolver
- URLs from CLI args
- URLs from a local file (file://)
- Sitemap loader:
– Remote: HTTP GET to fetch sitemap
– Local: read file from disk
c. Sitemap Parser
- XML parser that extracts <loc> entries into a URL list
d. URL Formatter
- Takes each URL and builds the Wayback Machine submission endpoint
e. HTTP Worker Pool
- Concurrency controlled by “jobs”
- Rate limiter/backoff between requests
- Uses requests.post/requests.get to call the Wayback Machine API
f. Logging & Reporting
- CLI-level logging (to console or file)
- Exit code reporting
g. External Dependency
- Internet Archive API (via HTTP)
h. CI/CD Pipelines (optional to include)
- GitHub Actions: run tests, lint, package, release

##### Relationships and data flow
- User → CLI Layer → Input Resolver → (Sitemap Parser if needed) → URL Formatter → HTTP Worker Pool → Internet Archive API
- Logging taps into each stage to report progress/failures
- CI/CD workflows run on code pushes/tags, invoking tests and releasing via setup.py/pip

##### Architectural patterns & design principles
- Single-process, modular design: each utility function is testable in isolation
- Separation of concerns:
- Argument parsing
- Input I/O (file vs HTTP)
- Data parsing (XML)
- Business logic (URL formatting, rate limiting)
- External API calls
- Concurrency via worker pool—keeps network I/O asynchronous from CLI thread

##### Be detailed but not overly complex
- Concurrency and rate-limit controls as distinct elements
- Don’t add fictitious layers—focus on real modules visible in the file tree and README

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and CLI
    User["User"]:::input
    CLI["CLI Layer\n(archiver.py)"]:::input
    click CLI "https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py"

    %% Input Resolver
    subgraph "Input Resolver"
        direction TB
        LocalLoader["Local File Loader\n(is_sitemap_local, load_local_sitemap)"]:::input
        click LocalLoader "https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py"
        RemoteLoader["Remote Sitemap Downloader\n(download_remote_sitemap)"]:::input
        click RemoteLoader "https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py"
    end

    %% Sitemap Parser
    Parser["Sitemap Parser\n(get_namespace, extract_pages_from_sitemap)"]:::processing
    click Parser "https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py"

    %% URL Formatter
    Formatter["URL Formatter\n(format_archive_url)"]:::processing
    click Formatter "https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py"

    %% HTTP Worker Pool
    WorkerPool["HTTP Worker Pool & Rate Limiter\n(requests.post/get)"]:::processing
    click WorkerPool "https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py"
    Requests["HTTP Client\n(requests)"]:::external
    click Requests "https://github.com/agude/wayback-machine-archiver/blob/master/requirements.txt"

    %% External API
    InternetArchive(("Internet Archive API")):::external

    %% Logging & Reporting
    Logging["Logging & Reporting\n(console logging, exit codes)"]:::processing
    click Logging "https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py"

    %% CI/CD
    subgraph "CI/CD Pipelines" 
        direction TB
        TestsWorkflow["GitHub Actions Tests"]:::infra
        click TestsWorkflow "https://github.com/agude/wayback-machine-archiver/blob/master/.github/workflows/tests.yml"
        ReleaseWorkflow["GitHub Actions Release"]:::infra
        click ReleaseWorkflow "https://github.com/agude/wayback-machine-archiver/blob/master/.github/workflows/release.yml"
    end

    %% Unit Tests
    subgraph "Unit Tests" 
        direction TB
        TestsDir["tests/"]:::infra
        click TestsDir "https://github.com/agude/wayback-machine-archiver/tree/master/tests/"
    end

    %% Main flow
    User -->|"invoke archiver"| CLI
    CLI -->|"resolve inputs"| LocalLoader
    CLI -->|"resolve inputs"| RemoteLoader
    LocalLoader --> Parser
    RemoteLoader --> Parser
    Parser --> Formatter
    Formatter --> WorkerPool
    WorkerPool --> InternetArchive

    %% Logging taps
    CLI -.->|logs| Logging
    LocalLoader -.->|logs| Logging
    RemoteLoader -.->|logs| Logging
    Parser -.->|logs| Logging
    Formatter -.->|logs| Logging
    WorkerPool -.->|logs| Logging

    %% CI/CD triggers tests
    TestsWorkflow -->|runs| TestsDir
    TestsWorkflow -->|validates| CLI
    ReleaseWorkflow -->|packages| CLI

    %% Classes
    classDef input fill:#e3fcec,stroke:#2ecc71;
    classDef processing fill:#e8f1fc,stroke:#3498db;
    classDef external fill:#fdebd0,stroke:#e67e22;
    classDef infra fill:#f2f2f2,stroke:#95a5a6;
```
