How niklastiede/github-trending-api works
Project type and purpose
• This is a backend web‐service (“Github‐Trending‐API”) written in Python that scrapes GitHub’s Trending pages and exposes the results as JSON via two REST endpoints (/repositories and /developers).
• It uses FastAPI to implement the API, aiohttp for asynchronous HTTP fetching, BeautifulSoup for parsing, and Pydantic schemas for validation.
File structure
• Top level:
• Dockerfile, heroku.yml, Makefile, CI configs (GitHub Actions), requirements-*.txt
• app/ – contains the FastAPI application:
– main.py defines the API routes and FastAPI app
– scraping.py implements the logic to fetch and parse GitHub pages
– allowed_parameters.py holds valid query parameters
• data/ – conversion scripts and schemas:
– schemas.py defines data models
– helper scripts (*_html_to_json.py, scrape_allowed_languages.py) for offline data processing
• tests/ – unit and benchmark tests
README for additional insights
• Endpoints accept path/query parameters (language, since, spoken_lang)
• Demonstrates the core flow: Client → FastAPI route → Scraper → GitHub page → parse → JSON response
• Mentions deployment targets (Heroku, Docker)
• Client: curl, web browser, Swagger UI
• API Layer: FastAPI/uvicorn server (app/main.py)
• Service Layer: Scraper service (app/scraping.py)
• HTTP Client: aiohttp module
• HTML Parser: BeautifulSoup inside scraping.py
• Data Models: Pydantic schemas (data/schemas.py)
• External Service: GitHub Trending web pages
• Offline Tools (optional): conversion scripts in data/
• CI/CD & Deployment: GitHub Actions, Docker container, Heroku
1. Client ──HTTP request──> FastAPI Router
2. Router invokes Scraper Service
3. Scraper Service uses aiohttp to fetch GitHub HTML
4. HTML passed to BeautifulSoup parser
5. Parsed results mapped to Pydantic models
6. JSON response returned to Client
7. CI/CD pipeline builds Docker image and deploys to Heroku
c. Architectural patterns and principles:
• Layered architecture (API layer → service layer → external)
• Asynchronous I/O (aiohttp + uvicorn event loop)
• Separation of concerns (routing, scraping logic, parameter validation, data models)
• Use of Pydantic for input/output validation
d. Key technologies:
• Python 3.9
• FastAPI + uvicorn (ASGI server)
• aiohttp (async HTTP client)
• BeautifulSoup4 (HTML parsing)
• Pydantic (data validation)
• Docker / Heroku / GitHub Actions