# tzhenyu/pastyearfetcher architecture

> Architecture diagram of the GitHub repository tzhenyu/pastyearfetcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tzhenyu/pastyearfetcher
- Repository: https://github.com/tzhenyu/pastyearfetcher

## Overview

##### Project type & purpose (what you’re diagramming)
This repo is a **single-script Python CLI tool** (not a service) meant to **authenticate to TAR UMT ePrints**, **query/search via RSS (XML)**, derive **direct download links**, and then **batch-download past-year exam papers** with **smart filenames** (subject/month/year).

Key signals:
- Top-level `main.py` + `requirements.txt` → standalone Python utility.
- README “How It Works” → RSS feed (XML) parsing + link conversion + downloads.
- No `src/`, no web server, no DB, no Docker/infra → local execution + remote dependency (ePrints).

So your “system” is primarily:
- A **local executable** (Python script)
- Interacting with **external remote systems** (ePrints website endpoints + RSS feed + file download endpoints)

##### What the file tree tells you
Top-level:
- `main.py` — all core logic lives here (auth, RSS query, parsing, downloading, naming).
- `.github/workflows/main.yml` — CI pipeline (lint/test/build). Include as “delivery/quality” plane (optional but good for completeness).

Notably absent:
- No database, no backend API, no frontend → don’t invent components.

##### Extract architecture from the README (runtime behavior)
From README:
- User provides credentials → tool logs in once (session/cookies).
- User provides course code + optional faculty filter → tool queries ePrints RSS feed search endpoint.
- Tool parses XML results → constructs download links.
- Tool downloads all matching papers → organizes/names them.

##### User (Student) — inputs credentials + search criteria.

**Local runtime (your code)**
##### Past Year Fetcher (Python CLI) [main.py]
- **CLI/Input Handler** (prompts/args for username/password/course/faculty)
- **Auth/Session Manager** (login, cookie/session handling)
- **RSS Query Builder** (build RSS feed URL from search parameters)
- **RSS/XML Parser** (parse feed results)
- **Link Resolver** (convert RSS items → direct download URLs)
- **Downloader** (HTTP GET files, handle retries/timeouts)
- **File Naming/Organizer** (subject/month/year naming; output path handling)
- **Logging/Error Handling** (user-visible progress/errors)

**External systems**
##### TAR UMT ePrints System (External)
- **Auth/Login Endpoint** (HTML form / SSO-ish flow)
- **RSS Feed Endpoint** (returns XML search results)
- **Document Download Endpoint(s)** (PDF/zip/whatever paper files are)

**Local environment (optional but helpful)**

##### GitHub Actions CI (`.github/workflows/main.yml`)
- Runs on push/PR; executes lint/tests (whatever is configured).

##### User → Past Year Fetcher
- “Credentials (username/password)”
- “Search params (course code, faculty filter)”

##### Past Year Fetcher → ePrints Auth Endpoint
- “POST login” / “Session initiation”
##### ePrints Auth Endpoint → Past Year Fetcher
- “Session cookie / auth token / redirect chain result”

##### Past Year Fetcher → ePrints RSS Feed Endpoint
- “GET RSS feed (XML) with query params”
##### RSS Feed Endpoint → Past Year Fetcher
- “RSS XML (search results)”

##### Past Year Fetcher → ePrints Download Endpoint(s)
- “GET file content (authenticated) for each paper”
##### Download Endpoint(s) → Past Year Fetcher
- “PDF/binary stream + headers (filename hints, content-type)”

##### Past Year Fetcher → Local File System (if applicable)
- “Write file (smart filename: subject-month-year.pdf)”
- or if not saving: “Provide file stream to user / temporary handling”

Cross-cutting notes:
- “Batch loop over RSS items”
- “Rate limiting / retries” (if present in code; otherwise keep generic)

##### Architectural patterns/design principles that fit
This project is best described as:
- **Integration/automation tool** (client-side)
- **Session-based HTTP scraping + feed consumption** (RSS/XML as structured interface)
- **Pipeline architecture** (Input → Auth → Query → Parse → Resolve → Download → Name/Output)
- **statelessness regarding credentials** (README claims “Credentials are not stored”)

##### Relevant technologies/libraries
From repo:
- Python (explicit)
From typical implementation:
- `requests` (HTTP session)
- XML parsing (`xml.etree.ElementTree`, `feedparser`, etc.)
- Possibly HTML parsing if login flow requires it

- “Python 3.x” in the main component header.
- Auth/Downloader: `requests.Session`
- RSS Parser: `feedparser` or stdlib XML
- Naming: `re`, `datetime` (if used)

##### What “very detailed” looks like here (without overcomplicating)

- User ↔ Past Year Fetcher ↔ TAR UMT ePrints
- RSS + downloads as interactions.

- Inside “Past Year Fetcher”, show Auth, RSS Query, Parser, Link Resolver, Downloader, Naming, Logging.
- Which subcomponent talks to which external endpoint.

That’s enough detail to capture essentially all architectural elements present in the repo, while staying faithful to the project’s actual structure.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% =========================
    %% Actors
    %% =========================
    User["User (Student)"]:::actor

    %% =========================
    %% Local Machine (Runtime)
    %% =========================
    subgraph "Local Machine (CLI Runtime)"
        direction TB

        CLI["Past Year Fetcher (Python CLI)"]:::cli

        subgraph "main.py (Single-script Pipeline)"
            direction TB

            Input["CLI / Input Handler"]:::module
            Auth["Auth / Session Manager"]:::module
            Query["RSS Query Builder"]:::module
            Parse["RSS / XML Parser"]:::module
            Resolve["Link Resolver"]:::module
            Download["Downloader"]:::module
            Naming["File Naming / Organizer"]:::module
            Log["Logging / Error Handling"]:::module

            NoteNoStore["Note: No credential storage (ephemeral session cookies)"]:::note
        end

        Deps["Runtime Dependencies (Python libs)"]:::deps
        FS[("Local File System / Output")]:::storage

        RSSXML["RSS Feed (XML)"]:::doc
        FileBin["Paper File (PDF/binary stream)"]:::doc
    end

    %% =========================
    %% External System
    %% =========================
    subgraph "TAR UMT ePrints (External)"
        direction TB

        EPrints["TAR UMT ePrints System"]:::external
        Login["Auth / Login Endpoint"]:::endpoint
        RSS["RSS Feed Endpoint (XML Search)"]:::endpoint
        DLEndpoint["Document Download Endpoint(s)"]:::endpoint
    end

    %% =========================
    %% Dev / CI
    %% =========================
    subgraph "Dev / Quality"
        direction TB
        README["Project Usage / Flow Documentation"]:::doc
        CI["GitHub Actions CI Workflow"]:::ci
    end

    %% =========================
    %% Internal wiring (pipeline)
    %% =========================
    CLI -->|"orchestrates"| Input
    CLI -->|"orchestrates"| Auth
    CLI -->|"orchestrates"| Query
    CLI -->|"orchestrates"| Parse
    CLI -->|"orchestrates"| Resolve
    CLI -->|"orchestrates"| Download
    CLI -->|"orchestrates"| Naming
    CLI -->|"orchestrates"| Log
    NoteNoStore -.->|"applies_to"| Auth

    Deps -.->|"HTTP+session(e.g.,requests.Session)"| Auth
    Deps -.->|"HTTP_get+retries/timeouts"| Download
    Deps -.->|"XML_parsing(feedparser/stdlib)"| Parse
    Deps -.->|"datetime/re/formatting"| Naming

    %% =========================
    %% Runtime interactions (numbered)
    %% =========================
    User -->|"1:provide_credentials+search_params"| Input
    Input -->|"2:params"| Auth
    Input -->|"2:params"| Query

    Auth -->|"3:POST_login(credentials)"| Login
    Login -->|"4:session_cookie/token"| Auth

    Query -->|"5:build_RSS_URL+query_params"| RSS
    RSS -->|"6:RSS_XML_results"| RSSXML
    RSSXML -->|"7:parse_items"| Parse
    Parse -->|"8:parsed_records"| Resolve

    Resolve -->|"9:derive_direct_download_URLs"| Download
    Download -->|"10:GET_files(authenticated)"| DLEndpoint
    DLEndpoint -->|"11:binary_stream+headers"| FileBin
    FileBin -->|"12:stream_to_writer"| Download

    Download -->|"13:pass_metadata"| Naming
    Naming -->|"14:write_named_files(subject-month-year)"| FS

    %% Cross-cutting logs
    Input -.->|"log_prompts+validation"| Log
    Auth -.->|"log_auth_progress+errors"| Log
    Query -.->|"log_query"| Log
    Parse -.->|"log_parse_issues"| Log
    Resolve -.->|"log_link_resolution"| Log
    Download -.->|"log_download_progress+retries"| Log
    Naming -.->|"log_output_paths"| Log

    %% External grouping
    EPrints -->|"contains"| Login
    EPrints -->|"contains"| RSS
    EPrints -->|"contains"| DLEndpoint

    %% CI / Docs relations
    README -.->|"describes_usage+how_it_works"| CLI
    CI -.->|"checks_on_push/PR"| CLI
    CI -.->|"reads_deps"| Deps

    %% =========================
    %% Click events (component mapping)
    %% =========================
    click CLI "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Input "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Auth "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Query "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Parse "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Resolve "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Download "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Naming "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Log "https://github.com/tzhenyu/pastyearfetcher/blob/main/main.py"
    click Deps "https://github.com/tzhenyu/pastyearfetcher/blob/main/requirements.txt"
    click CI "https://github.com/tzhenyu/pastyearfetcher/blob/main/.github/workflows/main.yml"
    click README "https://github.com/tzhenyu/pastyearfetcher/blob/main/readme.md"

    %% =========================
    %% Styles
    %% =========================
    classDef actor fill:#FFE8A3,stroke:#B78103,stroke-width:2px,color:#2B1A00;
    classDef cli fill:#C7E6FF,stroke:#1C6AA6,stroke-width:2px,color:#06233A;
    classDef module fill:#E7F3FF,stroke:#2D7DBA,stroke-width:1.5px,color:#06233A;
    classDef external fill:#E9D5FF,stroke:#6B21A8,stroke-width:2px,color:#2B0B45;
    classDef endpoint fill:#F3E8FF,stroke:#7C3AED,stroke-width:1.5px,color:#2B0B45;
    classDef storage fill:#D1FAE5,stroke:#047857,stroke-width:2px,color:#052E1B;
    classDef doc fill:#FFF7ED,stroke:#C2410C,stroke-width:1.5px,color:#3B1606;
    classDef ci fill:#FEE2E2,stroke:#B91C1C,stroke-width:2px,color:#3A0A0A;
    classDef deps fill:#E5E7EB,stroke:#374151,stroke-width:1.5px,color:#111827;
    classDef note fill:#FAF5FF,stroke:#A855F7,stroke-dasharray:4 3,color:#3B0764;
```
