How itsmerockyfiles/websitemetadatascraper works
This is a website-metadata scraping API with an interactive browser demo and two runtime deployments: an Express server and Vercel functions. The main workflow validates a requested URL, fetches the target page, extracts metadata and structured content, and returns a normalized result. The README also documents Redis caching on the Vercel scrape path. The extraction implementation is sampled only in part; the graph therefore groups extraction capabilities by their documented responsibilities without implying separate internal modules.
Main entry points: scraperService.js Scraper Service · server.js Express Server · apiRoutes.js Scrape and Health Routes · scrape.js Vercel Scrape Function
Components 20
Clients
- API Client
- Demo UI
public/js/app.js
API Runtime
- Express Server
src/server.js - Scrape and Health Routes
src/routes/apiRoutes.js - RapidAPI Auth
src/middleware/rapidApiAuth.js - Rate Limiter
src/middleware/rateLimiter.js - Error Handler
src/middleware/errorHandler.js - Vercel Scrape Function
api/v1/scrape.js - Vercel Health Function
api/v1/health.js
Scraping
- Scraper Service
src/services/scraperService.js - URL and SSRF Validation
src/services/ssrfValidator.js - DNS Resolver
- Metadata Extraction
src/services/scraperService.js - JSON-LD Parsing
src/services/scraperService.js - Clean Headings
src/services/scraperService.js - Metadata Response
src/services/scraperService.js
Serverless Cache
- Redis Client
api/v1/scrape.js - Redis Cache
Other components
- Browser User
- Target Website
Connections 24
- Browser User → Demo UI: uses
- Demo UI → Express Server: requests page and API
- API Client → Express Server: calls API
- API Client → Vercel Scrape Function: calls function
- Express Server → Scrape and Health Routes: mounts routes
- Express Server → Rate Limiter: limits API
- Express Server → Demo UI: serves UI
- Scrape and Health Routes → RapidAPI Auth: checks access
- Scrape and Health Routes → Scraper Service: requests scrape
- Scrape and Health Routes → Error Handler: forwards errors
- Vercel Scrape Function → Scraper Service: requests scrape
- Vercel Scrape Function → Redis Client: checks and stores cache
- Redis Client → Redis Cache: reads and writes
- Scraper Service → URL and SSRF Validation: validates URL
- URL and SSRF Validation → DNS Resolver: resolves hostname
- Scraper Service → Target Website: fetches page
- Scraper Service → Metadata Extraction: extracts page metadata
- Scraper Service → JSON-LD Parsing: parses JSON-LD
- Scraper Service → Clean Headings: extracts headings
- Metadata Extraction → Metadata Response: contributes fields
- JSON-LD Parsing → Metadata Response: contributes schemas
- Clean Headings → Metadata Response: contributes headings
- Scraper Service → Metadata Response: returns result
- Vercel Health Function → API Client: returns health status