How jordyvandijken/vlrggapi works
Project type and purpose
• Project type: A thin RESTful microservice built with FastAPI, designed to scrape and re-expose content from vlr.gg.
• Purpose: Provide cached JSON endpoints for Valorant esports news, match results, rankings, stats, and streams by scraping vlr.gg.
File structure
• Top-level files:
• main.py – entrypoint launching the FastAPI app via uvicorn
• Dockerfile, docker-compose.yml, Procfile, app.json, runtime.txt, vercel.json – containerization & deployment configs
• requirements.txt, pip/ – Python dependencies
• Directories:
• api/ – contains scraping logic
– base_scraper.py: abstract scraper class handles HTTP requests, parsing
– scrape.py: endpoint-specific scraper subclasses
• models/ – Pydantic response schemas (models/responses.py)
• utils/ – helper functions and constants used across scrapers and endpoints
• tests/ – pytest suite for API endpoints
• CI/CD workflows:
• .github/workflows/docker*.yml – automated build & publish pipelines
README for architecture clues
• Technology stack: FastAPI, uvicorn, Requests, Selectolax (HTML parser)
• No persistent datastore: data is fetched & cached in memory (FastAPI’s caching or reverse proxy).
• Endpoints follow a uniform pattern: GET, 5-minute cache, JSON ‘data’ wrapper.
Main components
a. Client Layer
– Web or mobile clients calling the REST API
b. API Layer
– FastAPI application served by uvicorn/Gunicorn
– Endpoint routers (defined implicitly in main.py)
– Pydantic models for request/response validation
c. Scraper Layer
– BaseScraper (HTTP client + HTML parser)
– Scraper subclasses in scrape.py for each endpoint
– Utility functions/constants supporting scraping logic
d. External System
– vlr.gg website (source of truth)
e. Caching Layer
– In-memory cache (FastAPI cache middleware or similar) for 5-minute TTL
f. Infrastructure & Deployment
– Docker container
– docker-compose for local dev
– Vercel/GitHub Actions for CI/CD & hosting
Relationships and data flows
1) Client issues HTTP GET → API gateway (uvicorn)
2) FastAPI routes request → validate via Pydantic model → check cache
3a) Cache hit: return cached JSON
3b) Cache miss: invoke Scraper Layer
• BaseScraper fetches HTML from vlr.gg
• Selectolax parses DOM; scrape.py extracts fields
• utils/helpers apply transformations
• models/responses wrap segments into JSON
4) API returns JSON response → Client
5) CI/CD pipelines build Docker image → deploy to Vercel/AWS/GCP
Architectural patterns & design principles
• Layered architecture: clear separation between HTTP interface (FastAPI), business logic (scrapers), data modeling (Pydantic), and utilities.
• Single responsibility: each scraper class handles exactly one endpoint.
• Caching strategy: 5-minute TTL to reduce load on external site.
• Containerization: uniform runtime across dev/staging/prod.
Capturing essential details
• Parser library (Selectolax) next to the scraping component.
• Pydantic models under “Data Modeling.”
• Test suite as an auxiliary component (optional) to illustrate quality gate in CI.