How tzhenyu/pastyearfetcher works
Project type & purpose (what you’re diagramming)
This repo is a single-script Python CLI tool (not a service) meant to authenticate to TAR UMT ePrints, query/search via RSS (XML), derive direct download links, and then batch-download past-year exam papers with smart filenames (subject/month/year).
Key signals:
• Top-level main.py + requirements.txt → standalone Python utility.
• README “How It Works” → RSS feed (XML) parsing + link conversion + downloads.
• No src/, no web server, no DB, no Docker/infra → local execution + remote dependency (ePrints).
So your “system” is primarily:
• A local executable (Python script)
• Interacting with external remote systems (ePrints website endpoints + RSS feed + file download endpoints)
What the file tree tells you
Top-level:
• main.py — all core logic lives here (auth, RSS query, parsing, downloading, naming).
• .github/workflows/main.yml — CI pipeline (lint/test/build). Include as “delivery/quality” plane (optional but good for completeness).
Notably absent:
• No database, no backend API, no frontend → don’t invent components.
Extract architecture from the README (runtime behavior)
From README:
• User provides credentials → tool logs in once (session/cookies).
• User provides course code + optional faculty filter → tool queries ePrints RSS feed search endpoint.
• Tool parses XML results → constructs download links.
• Tool downloads all matching papers → organizes/names them.
User (Student) — inputs credentials + search criteria.
Local runtime (your code)
Past Year Fetcher (Python CLI) [main.py]
• CLI/Input Handler (prompts/args for username/password/course/faculty)
• Auth/Session Manager (login, cookie/session handling)
• RSS Query Builder (build RSS feed URL from search parameters)
• RSS/XML Parser (parse feed results)
• Link Resolver (convert RSS items → direct download URLs)
• Downloader (HTTP GET files, handle retries/timeouts)
• File Naming/Organizer (subject/month/year naming; output path handling)
• Logging/Error Handling (user-visible progress/errors)
External systems
TAR UMT ePrints System (External)
• Auth/Login Endpoint (HTML form / SSO-ish flow)
• RSS Feed Endpoint (returns XML search results)
• Document Download Endpoint(s) (PDF/zip/whatever paper files are)
Local environment (optional but helpful)
GitHub Actions CI (.github/workflows/main.yml)
• Runs on push/PR; executes lint/tests (whatever is configured).
User → Past Year Fetcher
• “Credentials (username/password)”
• “Search params (course code, faculty filter)”
Past Year Fetcher → ePrints Auth Endpoint
• “POST login” / “Session initiation”
ePrints Auth Endpoint → Past Year Fetcher
• “Session cookie / auth token / redirect chain result”
Past Year Fetcher → ePrints RSS Feed Endpoint
• “GET RSS feed (XML) with query params”
RSS Feed Endpoint → Past Year Fetcher
• “RSS XML (search results)”
Past Year Fetcher → ePrints Download Endpoint(s)
• “GET file content (authenticated) for each paper”
Download Endpoint(s) → Past Year Fetcher
• “PDF/binary stream + headers (filename hints, content-type)”
Past Year Fetcher → Local File System (if applicable)
• “Write file (smart filename: subject-month-year.pdf)”
• or if not saving: “Provide file stream to user / temporary handling”
Cross-cutting notes:
• “Batch loop over RSS items”
• “Rate limiting / retries” (if present in code; otherwise keep generic)
Architectural patterns/design principles that fit
This project is best described as:
• Integration/automation tool (client-side)
• Session-based HTTP scraping + feed consumption (RSS/XML as structured interface)
• Pipeline architecture (Input → Auth → Query → Parse → Resolve → Download → Name/Output)
• statelessness regarding credentials (README claims “Credentials are not stored”)
Relevant technologies/libraries
From repo:
• Python (explicit)
From typical implementation:
• requests (HTTP session)
• XML parsing (xml.etree.ElementTree, feedparser, etc.)
• Possibly HTML parsing if login flow requires it
• “Python 3.x” in the main component header.
• Auth/Downloader: requests.Session
• RSS Parser: feedparser or stdlib XML
• Naming: re, datetime (if used)
What “very detailed” looks like here (without overcomplicating)
• User ↔ Past Year Fetcher ↔ TAR UMT ePrints
• RSS + downloads as interactions.
• Inside “Past Year Fetcher”, show Auth, RSS Query, Parser, Link Resolver, Downloader, Naming, Logging.
• Which subcomponent talks to which external endpoint.
That’s enough detail to capture essentially all architectural elements present in the repo, while staying faithful to the project’s actual structure.