How verdgard/crawlertool works
Crawler Tool is an Android app for extracting data from web pages. The main workflow is user-supplied URL and selector → HTTP or JavaScript-rendered HTML retrieval → XPath/CSS/regex matching → paged results, history, or export. Separate capabilities include batch pagination, HTML-tree inspection, configurable headers/proxy/theme, and an offline XPath guide. The graph follows documented capabilities where source excerpts are unavailable; unsampled internals and uncertain UI wiring are represented conservatively.
Main entry points: MainViewModel.kt Main state · MainActivity.kt Main screen · CrawlerRepository.kt Crawler repository · CrawlerEngine.kt Crawler engine
Components 22
App interaction
- Main screen
app/src/main/kotlin/com/SoloSu/Crawler_tool/MainActivity.kt - Main state
app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/MainViewModel.kt - Settings screen
app/src/main/kotlin/com/SoloSu/Crawler_tool/SettingsActivity.kt - Settings state
app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/SettingsViewModel.kt - Notification helper
app/src/main/kotlin/com/SoloSu/Crawler_tool/NotificationHelper.kt
Crawling and matching
- Crawler repository
app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/CrawlerRepository.kt - Crawler engine
app/src/main/kotlin/com/SoloSu/Crawler_tool/CrawlerEngine.kt - XPath evaluator
app/src/main/kotlin/com/SoloSu/Crawler_tool/JsoupXPathEngine.kt - JS renderer
app/src/main/kotlin/com/SoloSu/Crawler_tool/JsRenderEngine.kt
Preferences and history
- Settings repository
app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/SettingsRepository.kt - Settings storage
- Header manager
app/src/main/kotlin/com/SoloSu/Crawler_tool/HeaderManager.kt - History repository
app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/HistoryRepository.kt - History manager
app/src/main/kotlin/com/SoloSu/Crawler_tool/HistoryManager.kt - History storage
Results and inspection
- Result list
app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultAdapter.kt - Result item
app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultItem.kt - HTML tree viewer
app/src/main/kotlin/com/SoloSu/Crawler_tool/HtmlTreeDialog.kt - XPath guide
app/src/main/kotlin/com/SoloSu/Crawler_tool/XPathRulesActivity.kt - Export output
Other components
- User
- Web pages
Connections 21
- User → Main screen: enters query
- Main screen → Main state: forwards actions
- Main state → Crawler repository: requests extraction
- Crawler repository → Crawler engine: fetches and matches
- Crawler engine → Web pages: requests HTML
- Crawler engine → XPath evaluator: evaluates XPath
- Crawler repository → JS renderer: renders when selected
- JS renderer → Web pages: loads page
- Main state → Result list: publishes results
- Result list → Result item: displays items
- Main state → History repository: records and loads
- History repository → History manager: delegates history
- History manager → History storage: reads and writes
- User → Settings screen: configures app
- Settings screen → Settings state: forwards changes
- Settings state → Settings repository: updates preferences
- Settings repository → Settings storage: reads and writes
- Settings repository → Header manager: manages custom headers
- Main state → Export output: writes result formats
- Main screen → HTML tree viewer: opens DOM viewer
- Main screen → XPath guide: opens offline guide