How wetbikeboy2500/web_content_parser works
Main Components
a) Core Components:
• Scraper System
• Parser System
• WQL (Web Query Language) Interpreter
• Headless Browser Interface
• Utility Layer
b) External Components:
• Mobile Headless Browser (flutter_inappwebview)
• Desktop Headless Browser (puppeteer)
• Target Websites
┌─ Client Applications ─────────────────────────────────────┐
• Uses web_content_parser_full.dart │
• ┤
• Parser System │
• ┤
• Scraper System │
• ┤
• WQL Interpreter │ Headless Browser Interface │
• ┘
Component Details
a) Parser System (Purple):
• JSON Models (Author, CatalogEntry, Chapter, etc.)
• Source Templates
• Compute Decorator
• ID Management System
b) Scraper System (Green):
• Subcomponents:
• Generic Scraper Functions
• WQL Functions
• Scraper Source Management
• Request Handling
c) WQL Interpreter (Blue):
• Following elements:
• Tokenizer
• Parser
• Statement Handlers
• Operators
• Logical Selectors
d) Headless Browser Interface (Orange):
• Include:
• Mobile Implementation
• Desktop Implementation
• Request Queue Management
Additional Elements
Special features:
• Result<T> wrapper flow
• Source YAML configuration flow
• ID/ChapterID management
• Compute decoration pattern
The extensibility points
Keep the abstraction level high enough to show the architectural decisions while including enough detail to understand the unique features like WQL and the Result pattern.
Remember to show the relationship between the three main imports (util.dart, scraper.dart, and parser.dart) and how they can be used independently or together through web_content_parser_full.dart.