# niklastiede/github-trending-api architecture

> Architecture diagram of the GitHub repository niklastiede/github-trending-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/niklastiede/github-trending-api
- Repository: https://github.com/niklastiede/github-trending-api

## Overview

##### Project type and purpose
- This is a backend web‐service (“Github‐Trending‐API”) written in Python that scrapes GitHub’s Trending pages and exposes the results as JSON via two REST endpoints (`/repositories` and `/developers`).
- It uses FastAPI to implement the API, aiohttp for asynchronous HTTP fetching, BeautifulSoup for parsing, and Pydantic schemas for validation.

##### File structure
- Top level:
- Dockerfile, heroku.yml, Makefile, CI configs (GitHub Actions), requirements-*.txt
- `app/` – contains the FastAPI application:
– `main.py` defines the API routes and FastAPI app
– `scraping.py` implements the logic to fetch and parse GitHub pages
– `allowed_parameters.py` holds valid query parameters
- `data/` – conversion scripts and schemas:
– `schemas.py` defines data models
– helper scripts (`*_html_to_json.py`, `scrape_allowed_languages.py`) for offline data processing
- `tests/` – unit and benchmark tests

##### README for additional insights
- Endpoints accept path/query parameters (`language`, `since`, `spoken_lang`)
- Demonstrates the core flow: Client → FastAPI route → Scraper → GitHub page → parse → JSON response
- Mentions deployment targets (Heroku, Docker)

- Client: curl, web browser, Swagger UI
- API Layer: FastAPI/uvicorn server (`app/main.py`)
- Service Layer: Scraper service (`app/scraping.py`)
- HTTP Client: aiohttp module
- HTML Parser: BeautifulSoup inside `scraping.py`
- Data Models: Pydantic schemas (`data/schemas.py`)
- External Service: GitHub Trending web pages
- Offline Tools (optional): conversion scripts in `data/`
- CI/CD & Deployment: GitHub Actions, Docker container, Heroku
1. Client ──HTTP request──> FastAPI Router
2. Router invokes Scraper Service
3. Scraper Service uses aiohttp to fetch GitHub HTML
4. HTML passed to BeautifulSoup parser
5. Parsed results mapped to Pydantic models
6. JSON response returned to Client
7. CI/CD pipeline builds Docker image and deploys to Heroku
c. Architectural patterns and principles:
- Layered architecture (API layer → service layer → external)
- Asynchronous I/O (aiohttp + uvicorn event loop)
- Separation of concerns (routing, scraping logic, parameter validation, data models)
- Use of Pydantic for input/output validation
d. Key technologies:
- Python 3.9
- FastAPI + uvicorn (ASGI server)
- aiohttp (async HTTP client)
- BeautifulSoup4 (HTML parsing)
- Pydantic (data validation)
- Docker / Heroku / GitHub Actions

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Clients
    subgraph "Clients" 
        direction TB
        Client["Client (curl, browser, Swagger UI)"]:::external
    end

    %% Public API Layer
    subgraph "Public API Layer"
        direction TB
        APIRouter["FastAPI Router\n(app/main.py)"]:::api
        Params["Allowed Query Parameters\n(app/allowed_parameters.py)"]:::api
    end

    %% Service Layer
    subgraph "Service Layer"
        direction TB
        Scraper["Scraper Service\n(app/scraping.py)"]:::service
        HTTPClient["aiohttp HTTP Client"]:::service
        Parser["BeautifulSoup Parser"]:::service
    end

    %% Data Models
    subgraph "Data Models"
        direction TB
        Schemas["Pydantic Data Models\n(data/schemas.py)"]:::models
    end

    %% External Dependency
    subgraph "External Systems"
        direction TB
        GitHub["GitHub Trending Pages"]:::external
    end

    %% Offline Processing Utilities
    subgraph "Data Processing Utilities"
        direction TB
        RepoConv["HTML-to-JSON (repos)\n(data/repo_html_to_json.py)"]:::offline
        DevConv["HTML-to-JSON (devs)\n(data/dev_html_to_json.py)"]:::offline
        LangScrape["Allowed Languages Scraper\n(data/scrape_allowed_languages.py)"]:::offline
    end

    %% Sample Data Files
    subgraph "Sample Data Files"
        direction TB
        Rep1HTML["repodata1.html"]:::offline
        Rep1JSON["repodata1.json"]:::offline
        Rep2HTML["repodata2.html"]:::offline
        Rep2JSON["repodata2.json"]:::offline
        Rep3HTML["repodata3.html"]:::offline
        Rep3JSON["repodata3.json"]:::offline
        Rep4HTML["repodata4.html"]:::offline
        Rep4JSON["repodata4.json"]:::offline
        Dev1HTML["devdata1.html"]:::offline
        Dev1JSON["devdata1.json"]:::offline
        Dev2HTML["devdata2.html"]:::offline
        Dev2JSON["devdata2.json"]:::offline
        Dev3HTML["devdata3.html"]:::offline
        Dev3JSON["devdata3.json"]:::offline
    end

    %% CI/CD & Deployment
    subgraph "CI/CD & Deployment"
        direction TB
        CI["GitHub Actions CI\n(.github/workflows/continuous_integration.yaml)"]:::ci
        CD["GitHub Actions Release\n(.github/workflows/release_and_deploy.yaml)"]:::ci
        DockerCfg["Dockerfile"]:::ci
        HerokuCfg["heroku.yml"]:::ci
        MakefileNode["Makefile"]:::ci
    end

    %% Tests
    subgraph "Tests"
        direction TB
        TestMain["main_test.py"]:::test
        TestScraper["scraper_test.py"]:::test
        Benchmark["requests_benchmark.py"]:::test
    end

    %% Project Metadata
    subgraph "Project Metadata"
        direction TB
        ReqProd["requirements-prod.txt"]:::ci
        ReqDev["requirements-dev.txt"]:::ci
        SetupCfg["setup.cfg"]:::ci
        README["README.md"]:::ci
        Changelog["CHANGELOG.md"]:::ci
        License["LICENSE"]:::ci
    end

    %% Connections
    Client -->|"HTTP request"| APIRouter
    APIRouter -->|"validate params"| Params
    APIRouter -->|"invoke scrape()"| Scraper
    Scraper -->|"fetch HTML"| HTTPClient
    HTTPClient -->|"GET request"| GitHub
    GitHub -->|"HTML response"| Parser
    Parser -->|"parsed data"| Schemas
    Schemas -->|"response JSON"| APIRouter
    APIRouter -->|"HTTP response"| Client

    CI -->|build image| DockerCfg
    CD -->|deploy| HerokuCfg
    CD -->|via| MakefileNode

    %% Click Events
    click APIRouter "https://github.com/niklastiede/github-trending-api/blob/main/app/main.py"
    click Scraper "https://github.com/niklastiede/github-trending-api/blob/main/app/scraping.py"
    click Params "https://github.com/niklastiede/github-trending-api/blob/main/app/allowed_parameters.py"
    click Schemas "https://github.com/niklastiede/github-trending-api/blob/main/data/schemas.py"
    click RepoConv "https://github.com/niklastiede/github-trending-api/blob/main/data/repo_html_to_json.py"
    click DevConv "https://github.com/niklastiede/github-trending-api/blob/main/data/dev_html_to_json.py"
    click LangScrape "https://github.com/niklastiede/github-trending-api/blob/main/data/scrape_allowed_languages.py"
    click Rep1HTML "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata1.html"
    click Rep1JSON "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata1.json"
    click Rep2HTML "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata2.html"
    click Rep2JSON "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata2.json"
    click Rep3HTML "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata3.html"
    click Rep3JSON "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata3.json"
    click Rep4HTML "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata4.html"
    click Rep4JSON "https://github.com/niklastiede/github-trending-api/blob/main/data/repodata4.json"
    click Dev1HTML "https://github.com/niklastiede/github-trending-api/blob/main/data/devdata1.html"
    click Dev1JSON "https://github.com/niklastiede/github-trending-api/blob/main/data/devdata1.json"
    click Dev2HTML "https://github.com/niklastiede/github-trending-api/blob/main/data/devdata2.html"
    click Dev2JSON "https://github.com/niklastiede/github-trending-api/blob/main/data/devdata2.json"
    click Dev3HTML "https://github.com/niklastiede/github-trending-api/blob/main/data/devdata3.html"
    click Dev3JSON "https://github.com/niklastiede/github-trending-api/blob/main/data/devdata3.json"
    click CI "https://github.com/niklastiede/github-trending-api/blob/main/.github/workflows/continuous_integration.yaml"
    click CD "https://github.com/niklastiede/github-trending-api/blob/main/.github/workflows/release_and_deploy.yaml"
    click DockerCfg "https://github.com/niklastiede/github-trending-api/tree/main/Dockerfile"
    click HerokuCfg "https://github.com/niklastiede/github-trending-api/blob/main/heroku.yml"
    click MakefileNode "https://github.com/niklastiede/github-trending-api/tree/main/Makefile"
    click TestMain "https://github.com/niklastiede/github-trending-api/blob/main/tests/main_test.py"
    click TestScraper "https://github.com/niklastiede/github-trending-api/blob/main/tests/scraper_test.py"
    click Benchmark "https://github.com/niklastiede/github-trending-api/blob/main/tests/requests_benchmark.py"
    click ReqProd "https://github.com/niklastiede/github-trending-api/blob/main/requirements-prod.txt"
    click ReqDev "https://github.com/niklastiede/github-trending-api/blob/main/requirements-dev.txt"
    click SetupCfg "https://github.com/niklastiede/github-trending-api/blob/main/setup.cfg"
    click README "https://github.com/niklastiede/github-trending-api/blob/main/README.md"
    click Changelog "https://github.com/niklastiede/github-trending-api/blob/main/CHANGELOG.md"
    click License "https://github.com/niklastiede/github-trending-api/tree/main/LICENSE"

    %% Styles
    classDef external fill:#c8e6c9,stroke:#2e7d32
    classDef api fill:#bbdefb,stroke:#0d47a1
    classDef service fill:#90caf9,stroke:#1565c0
    classDef models fill:#fff59d,stroke:#f9a825
    classDef offline fill:#ffe082,stroke:#ff6f00
    classDef ci fill:#e0e0e0,stroke:#424242
    classDef test fill:#f8bbd0,stroke:#880e4f
```
