How shinrenpan/webparser works
Project Type and Purpose
• This is a Swift Package (“WebParser”) that provides a WKWebView-based web crawler/parser for iOS (13+), plus a Demo iOS app that exercises the library.
• The library’s goal is to:
• Load a URL in an off-screen WKWebView
• Execute custom JavaScript
• And decode the result into a Swift Decodable model
• Handle retry logic and cookie management
File Structure Analysis
• Top‐level:
• Package.swift – defines the Swift Package (WebParser)
• Sources/… – the core library modules:
– CookiePool.swift
– Parser.swift
– ParserConfiguration.swift
– ParserError.swift
• Demo/… – an Xcode iOS app project demonstrating use of the library
• Demo app follows an MVVM-style pattern (Main folder: Model, ViewModel, ViewController, Router, ViewOutlet).
• WebParser.xcodeproj – header for potential Obj-C bridging (WebParser.h).
README Insights
• Public API revolves around ParserConfiguration and Parser:
1. Create ParserConfiguration (URL request, windowSize, user agent, retry settings, JavaScript)
2. Init Parser(parserConfiguration:)
3. Call try await parser.decodeResult(SomeData.self) or anyResult()
• No external services beyond iOS frameworks (WKWebKit, Foundation).
Main Components & Interactions
a. Components
• Demo App
– UI Layer: ViewController, ViewModel, Router
– Model Layer: Decodable models for parsed data
• WebParser Library
– Parser API (public interface)
– Parser Core (Parser.swift)
– ParserConfiguration (holds settings)
– CookiePool (manages cookies across loads)
– ParserError (error handling)
– Hidden WKWebView instance
– JavaScript execution engine
• External Dependencies
– WKWebKit (WKWebView)
– Foundation (URLSession under the hood, JSONDecoder)
b. Interactions / Data Flow
1. Demo ViewModel → builds ParserConfiguration
2. ViewModel calls Parser(parserConfiguration:) → public Parser API
3. Parser
– configures an off-screen WKWebView (windowSize, userAgent)
– loads the URL request
– injects & runs the provided JavaScript
– collects JS result (e.g., JSON or HTML string)
– uses JSONDecoder or fallback to Any
– manages retryCount/retryDuration on failures
– uses CookiePool to preserve session cookies across loads
4. Parsed data → returned asynchronously to ViewModel → UI
c. Architectural Patterns
• Demo: MVVM (ViewController ↔ ViewModel ↔ Model)
• Library: Modular separation of concerns (configuration, execution, error handling, cookie management)
Essential Elements
• Clear component names
• Annotations for retry logic and cookie management loops
• Error-handling paths back to the caller (ParserError)