# itsmerockyfiles/websitemetadatascraper architecture

> Architecture diagram of the GitHub repository itsmerockyfiles/websitemetadatascraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/itsmerockyfiles/websitemetadatascraper
- Repository: https://github.com/itsmerockyfiles/websitemetadatascraper

## Overview

This is a website-metadata scraping API with an interactive browser demo and two runtime deployments: an Express server and Vercel functions. The main workflow validates a requested URL, fetches the target page, extracts metadata and structured content, and returns a normalized result. The README also documents Redis caching on the Vercel scrape path. The extraction implementation is sampled only in part; the graph therefore groups extraction capabilities by their documented responsibilities without implying separate internal modules.

Main entry points: [`src/services/scraperService.js`](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js) (Scraper Service), [`src/server.js`](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/server.js) (Express Server), [`src/routes/apiRoutes.js`](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/routes/apiRoutes.js) (Scrape and Health Routes), [`api/v1/scrape.js`](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/api/v1/scrape.js) (Vercel Scrape Function)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Clients"]
  node_api_client(("API Client"))
  node_demo["Demo UI<br/>[app.js]"]
end

subgraph group_api["API Runtime"]
  node_express_server["Express Server<br/>[server.js]"]
  node_api_routes["Scrape and Health Routes<br/>[apiRoutes.js]"]
  node_auth["RapidAPI Auth<br/>[rapidApiAuth.js]"]
  node_rate_limit["Rate Limiter<br/>[rateLimiter.js]"]
  node_error_handler["Error Handler<br/>[errorHandler.js]"]
  node_vercel_scrape["Vercel Scrape Function<br/>[scrape.js]"]
  node_vercel_health["Vercel Health Function<br/>[health.js]"]
end

subgraph group_scraping["Scraping"]
  node_scraper["Scraper Service<br/>[scraperService.js]"]
  node_url_validator["URL and SSRF Validation<br/>[ssrfValidator.js]"]
  node_dns["DNS Resolver"]
  node_metadata_extract["Metadata Extraction<br/>[scraperService.js]"]
  node_structured_data["JSON-LD Parsing<br/>[scraperService.js]"]
  node_headings["Clean Headings<br/>[scraperService.js]"]
  node_result["Metadata Response<br/>[scraperService.js]"]
end

subgraph group_cache["Serverless Cache"]
  node_redis_client["Redis Client<br/>[scrape.js]"]
  node_redis[("Redis Cache")]
end

node_user(("Browser User"))
node_target_site(("Target Website"))

node_user -->|"uses"| node_demo
node_demo -->|"requests page and API"| node_express_server
node_api_client -->|"calls API"| node_express_server
node_api_client -.->|"calls function"| node_vercel_scrape
node_express_server -->|"mounts routes"| node_api_routes
node_express_server -->|"limits API"| node_rate_limit
node_express_server -->|"serves UI"| node_demo
node_api_routes -->|"checks access"| node_auth
node_api_routes -->|"requests scrape"| node_scraper
node_api_routes -->|"forwards errors"| node_error_handler
node_vercel_scrape -->|"requests scrape"| node_scraper
node_vercel_scrape -->|"checks and stores cache"| node_redis_client
node_redis_client -->|"reads and writes"| node_redis
node_scraper -->|"validates URL"| node_url_validator
node_url_validator -->|"resolves hostname"| node_dns
node_scraper -->|"fetches page"| node_target_site
node_scraper -->|"extracts page metadata"| node_metadata_extract
node_scraper -->|"parses JSON-LD"| node_structured_data
node_scraper -->|"extracts headings"| node_headings
node_metadata_extract -->|"contributes fields"| node_result
node_structured_data -->|"contributes schemas"| node_result
node_headings -->|"contributes headings"| node_result
node_scraper -->|"returns result"| node_result
node_vercel_health -->|"returns health status"| node_api_client

click node_demo "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/public/js/app.js"
click node_express_server "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/server.js"
click node_api_routes "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/routes/apiRoutes.js"
click node_auth "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/middleware/rapidApiAuth.js"
click node_rate_limit "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/middleware/rateLimiter.js"
click node_error_handler "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/middleware/errorHandler.js"
click node_vercel_scrape "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/api/v1/scrape.js"
click node_vercel_health "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/api/v1/health.js"
click node_scraper "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js"
click node_url_validator "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/ssrfValidator.js"
click node_metadata_extract "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js"
click node_structured_data "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js"
click node_headings "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js"
click node_result "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js"
click node_redis_client "https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/api/v1/scrape.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_client,node_demo,node_user toneBlue
class node_express_server,node_api_routes,node_auth,node_rate_limit,node_error_handler,node_vercel_scrape,node_vercel_health toneAmber
class node_scraper,node_url_validator,node_dns,node_metadata_extract,node_structured_data,node_headings,node_result toneMint
class node_redis_client,node_redis toneRose
class node_target_site toneIndigo
```

## Components

### Clients

- **API Client**
- [Demo UI](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/public/js/app.js): `public/js/app.js`

### API Runtime

- [Express Server](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/server.js): `src/server.js`
- [Scrape and Health Routes](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/routes/apiRoutes.js): `src/routes/apiRoutes.js`
- [RapidAPI Auth](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/middleware/rapidApiAuth.js): `src/middleware/rapidApiAuth.js`
- [Rate Limiter](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/middleware/rateLimiter.js): `src/middleware/rateLimiter.js`
- [Error Handler](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/middleware/errorHandler.js): `src/middleware/errorHandler.js`
- [Vercel Scrape Function](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/api/v1/scrape.js): `api/v1/scrape.js`
- [Vercel Health Function](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/api/v1/health.js): `api/v1/health.js`

### Scraping

- [Scraper Service](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js): `src/services/scraperService.js`
- [URL and SSRF Validation](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/ssrfValidator.js): `src/services/ssrfValidator.js`
- **DNS Resolver**
- [Metadata Extraction](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js): `src/services/scraperService.js`
- [JSON-LD Parsing](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js): `src/services/scraperService.js`
- [Clean Headings](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js): `src/services/scraperService.js`
- [Metadata Response](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/src/services/scraperService.js): `src/services/scraperService.js`

### Serverless Cache

- [Redis Client](https://github.com/itsmerockyfiles/websitemetadatascraper/blob/main/api/v1/scrape.js): `api/v1/scrape.js`
- **Redis Cache**

### Other components

- **Browser User**
- **Target Website**

## Connections

- Browser User → Demo UI: uses
- Demo UI → Express Server: requests page and API
- API Client → Express Server: calls API
- API Client → Vercel Scrape Function: calls function
- Express Server → Scrape and Health Routes: mounts routes
- Express Server → Rate Limiter: limits API
- Express Server → Demo UI: serves UI
- Scrape and Health Routes → RapidAPI Auth: checks access
- Scrape and Health Routes → Scraper Service: requests scrape
- Scrape and Health Routes → Error Handler: forwards errors
- Vercel Scrape Function → Scraper Service: requests scrape
- Vercel Scrape Function → Redis Client: checks and stores cache
- Redis Client → Redis Cache: reads and writes
- Scraper Service → URL and SSRF Validation: validates URL
- URL and SSRF Validation → DNS Resolver: resolves hostname
- Scraper Service → Target Website: fetches page
- Scraper Service → Metadata Extraction: extracts page metadata
- Scraper Service → JSON-LD Parsing: parses JSON-LD
- Scraper Service → Clean Headings: extracts headings
- Metadata Extraction → Metadata Response: contributes fields
- JSON-LD Parsing → Metadata Response: contributes schemas
- Clean Headings → Metadata Response: contributes headings
- Scraper Service → Metadata Response: returns result
- Vercel Health Function → API Client: returns health status