How microlinkhq/metascraper works
Main Components Architecture
┌─────────────────────────────────────────────────────────┐
• Client Application │
• ┬─────────────────────────────────┘
┌───────────────────────▼─────────────────────────────────┐
• Metascraper Core │
• ┤
• Rule Engine │
• HTML Parser │
• URL Validator │
• ┬───────────────────────┬─────────────────┘
┌───────────────▼──────────┐ ┌────────▼────────────────┐
• Core Rule Bundles │ │ Specialized Providers │
• ┐ │ ├─────────────────────────┤
• Author │ │ │ - Amazon │
• Date │ │ │ - YouTube │
• Description │ │ │ - Instagram │
• Image │ │ │ - Spotify │
• Title │ │ │ - Telegram │
• ┘ │ └─────────────────────────┘
┌────────────────────────┐│
• Media Handlers ││
• ┤│
• Audio ││
• Video ││
• Image Processing ││
• ┘│
• ┘
Data Flow Representation
Client Request → Metascraper Core → Rule Engine → Rule Bundles → Metadata Response
Key Components to Highlight
a. Core Engine:
• Rule Engine (handles rule prioritization and execution)
• HTML Parser
• URL Validator
• Metadata Aggregator
b. Rule Bundles:
• Core Rules (author, date, description, etc.)
• Media Rules (audio, video, image)
• Provider-Specific Rules (Amazon, YouTube, etc.)
Integration Points
Connections with:
• External HTML Sources
• Media Provider APIs
• Clearbit Logo API
• Browser Contexts (for rendered content)
Additional Elements
Include:
• Version information
• Error handling paths
• Fallback mechanisms
• Cache layer (if applicable)
Technical Details to Note
Annotations for:
• Node.js Runtime
• Supported Metadata Formats (Open Graph, JSON-LD, etc.)
• Browser Integration Points
• API Endpoints