How xixu-me/xget works
Project Type and Purpose
• Xget is a backend-only, high-performance HTTP proxy/accelerator service implemented in Node.js.
• It runs as a serverless API (Cloudflare Workers or Vercel serverless functions) or as a containerized Node.js app (Docker).
• Its core job is to accept incoming download/Git/API requests, transform URLs according to a platform config (GitHub, npm, PyPI, container registries, AI providers, etc.), fetch the resource from the upstream host with retries, caching and compression, then stream it back to the client with security headers and performance metrics.
File Structure Highlights
• Root: server.js (entrypoint for Node), Dockerfile, wrangler.toml (Cloudflare), vercel.json (Vercel).
• api/[…path].js & api/health.js: serverless handlers routing all incoming requests.
• src/index.js, src/config/{index.js,platforms.js}: main logic – environment/config loading, platform map, request transformation.
• PerformanceMonitor, retry & timeout logic live under src (imported by api handlers).
• Tests in test/, scripts in scripts/, CI definitions under .github/workflows.
Key Architectural Patterns
• API Gateway / Router: Single HTTP entry, dynamic path mapping via catch-all route.
• Strategy / Plugin Pattern: Platform definitions in src/config/platforms.js, each with its own base URL and transform function.
• Proxy + Streaming: Upstream fetch with HTTP/3, gzip/deflate/brotli, range requests, then pipe response.
• Caching Layer: Cloudflare Cache API at the edge, 30-minute TTL by default, with bypass rules for Git operations.
• Observability: PerformanceMonitor class instruments each stage (DNS, connect, first byte, end) and returns X-Performance-Metrics.
• Security Middleware: Request validation (method whitelist, path length limit), injection of security headers (CSP, HSTS, X-Frame, X-XSS, Referrer-Policy).
A. Clients
• Git clients (git clone/pull/push)
• Download tools (wget, curl, aria2, IDM)
• Browsers (via extensions or direct HTTP)
• CI/CD runners (GitHub Actions, GitLab CI)
B. CDN / Edge Cache (Cloudflare Workers + Cache API)
• Performs routing at nearest PoP, applies caching and security headers.
C. Serverless/API Layer
• api/[…path].js: universal catch-all for HTTP requests
• api/health.js: health-check endpoint
• server.js: Express-style wrapper for self-hosted mode
D. Core Service Modules
1. Router/Controller
• Parses the incoming path, extracts platform prefix
2. Config & Platform Strategy
• src/config/platforms.js maps prefixes → base URLs + transform logic
3. Fetcher / Proxy Engine
• Handles HTTP/3, multi-range, parallel download, retry logic
4. Cache Manager
• Integrates with Cloudflare Cache API for reading/writing edge cache
5. Performance Monitor
• Instruments timing and emits X-Performance-Metrics
6. Security Layer
• Validates methods/paths, injects security headers
E. External Upstream Services
• Git endpoints (GitHub, GitLab, Gitea, AOSP)
• Package registries (npm, PyPI, Maven, etc.)
• Container registries (docker, ghcr, gcr, etc.)
• AI inference APIs (OpenAI, Anthropic, HuggingFace, etc.)
• Other downloads (Apache, CTAN, arXiv, F-Droid, Linux distros)
Relationships & Data Flow
1. Client → DNS → CDN (Cloudflare PoP)
2. CDN Worker (api/[…path].js) → Security Layer → Router
3. Router determines platform config → Fetcher
4. Fetcher calls Upstream Service (with retry, timeout)
5. Response streamed back to Cache Manager → Edge Cache & Performance Monitor
6. CDN returns response to Client with security headers and X-Performance-Metrics