# verdgard/crawlertool architecture

> Architecture diagram of the GitHub repository verdgard/crawlertool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/verdgard/crawlertool
- Repository: https://github.com/verdgard/crawlertool

## Overview

Crawler Tool is an Android app for extracting data from web pages. The main workflow is user-supplied URL and selector → HTTP or JavaScript-rendered HTML retrieval → XPath/CSS/regex matching → paged results, history, or export. Separate capabilities include batch pagination, HTML-tree inspection, configurable headers/proxy/theme, and an offline XPath guide. The graph follows documented capabilities where source excerpts are unavailable; unsampled internals and uncertain UI wiring are represented conservatively.

Main entry points: [`app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/MainViewModel.kt`](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/MainViewModel.kt) (Main state), [`app/src/main/kotlin/com/SoloSu/Crawler_tool/MainActivity.kt`](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/MainActivity.kt) (Main screen), [`app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/CrawlerRepository.kt`](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/CrawlerRepository.kt) (Crawler repository), [`app/src/main/kotlin/com/SoloSu/Crawler_tool/CrawlerEngine.kt`](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/CrawlerEngine.kt) (Crawler engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interaction["App interaction"]
  node_main["Main screen<br/>[MainActivity.kt]"]
  node_mainvm["Main state<br/>[MainViewModel.kt]"]
  node_settings["Settings screen"]
  node_settingsvm["Settings state"]
  node_notifications["Notification helper"]
end

subgraph group_crawl["Crawling and matching"]
  node_crawlerrepo["Crawler repository"]
  node_engine["Crawler engine<br/>[CrawlerEngine.kt]"]
  node_xpath["XPath evaluator"]
  node_jsrender["JS renderer<br/>[JsRenderEngine.kt]"]
end

subgraph group_preferences["Preferences and history"]
  node_settingsrepo["Settings repository"]
  node_settingsstore[("Settings storage")]
  node_headers["Header manager<br/>[HeaderManager.kt]"]
  node_historyrepo["History repository"]
  node_history["History manager<br/>[HistoryManager.kt]"]
  node_historystore[("History storage")]
end

subgraph group_inspection["Results and inspection"]
  node_results["Result list<br/>[ResultAdapter.kt]"]
  node_resultmodel["Result item<br/>[ResultItem.kt]"]
  node_tree["HTML tree viewer<br/>[HtmlTreeDialog.kt]"]
  node_guide["XPath guide"]
  node_export[("Export output")]
end

node_user(("User"))
node_web(("Web pages"))

node_user -->|"enters query"| node_main
node_main -->|"forwards actions"| node_mainvm
node_mainvm -->|"requests extraction"| node_crawlerrepo
node_crawlerrepo -->|"fetches and matches"| node_engine
node_engine -->|"requests HTML"| node_web
node_engine -->|"evaluates XPath"| node_xpath
node_crawlerrepo -.->|"renders when selected"| node_jsrender
node_jsrender -->|"loads page"| node_web
node_mainvm -->|"publishes results"| node_results
node_results -->|"displays items"| node_resultmodel
node_mainvm -->|"records and loads"| node_historyrepo
node_historyrepo -->|"delegates history"| node_history
node_history -->|"reads and writes"| node_historystore
node_user -->|"configures app"| node_settings
node_settings -->|"forwards changes"| node_settingsvm
node_settingsvm -->|"updates preferences"| node_settingsrepo
node_settingsrepo -->|"reads and writes"| node_settingsstore
node_settingsrepo -->|"manages custom headers"| node_headers
node_mainvm -->|"writes result formats"| node_export
node_main -->|"opens DOM viewer"| node_tree
node_main -->|"opens offline guide"| node_guide

click node_main "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/MainActivity.kt"
click node_mainvm "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/MainViewModel.kt"
click node_settings "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/SettingsActivity.kt"
click node_settingsvm "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/SettingsViewModel.kt"
click node_crawlerrepo "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/CrawlerRepository.kt"
click node_engine "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/CrawlerEngine.kt"
click node_xpath "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/JsoupXPathEngine.kt"
click node_jsrender "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/JsRenderEngine.kt"
click node_settingsrepo "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/SettingsRepository.kt"
click node_headers "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/HeaderManager.kt"
click node_historyrepo "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/HistoryRepository.kt"
click node_history "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/HistoryManager.kt"
click node_results "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultAdapter.kt"
click node_resultmodel "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultItem.kt"
click node_tree "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/HtmlTreeDialog.kt"
click node_guide "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/XPathRulesActivity.kt"
click node_notifications "https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/NotificationHelper.kt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_mainvm,node_settings,node_settingsvm,node_notifications,node_user toneBlue
class node_crawlerrepo,node_engine,node_xpath,node_jsrender toneAmber
class node_settingsrepo,node_settingsstore,node_headers,node_historyrepo,node_history,node_historystore toneMint
class node_results,node_resultmodel,node_tree,node_guide,node_export toneRose
class node_web toneIndigo
```

## Components

### App interaction

- [Main screen](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/MainActivity.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/MainActivity.kt`
- [Main state](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/MainViewModel.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/MainViewModel.kt`
- [Settings screen](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/SettingsActivity.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/SettingsActivity.kt`
- [Settings state](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/SettingsViewModel.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/viewmodel/SettingsViewModel.kt`
- [Notification helper](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/NotificationHelper.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/NotificationHelper.kt`

### Crawling and matching

- [Crawler repository](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/CrawlerRepository.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/CrawlerRepository.kt`
- [Crawler engine](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/CrawlerEngine.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/CrawlerEngine.kt`
- [XPath evaluator](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/JsoupXPathEngine.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/JsoupXPathEngine.kt`
- [JS renderer](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/JsRenderEngine.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/JsRenderEngine.kt`

### Preferences and history

- [Settings repository](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/SettingsRepository.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/SettingsRepository.kt`
- **Settings storage**
- [Header manager](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/HeaderManager.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/HeaderManager.kt`
- [History repository](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/HistoryRepository.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/repository/HistoryRepository.kt`
- [History manager](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/HistoryManager.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/HistoryManager.kt`
- **History storage**

### Results and inspection

- [Result list](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultAdapter.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultAdapter.kt`
- [Result item](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultItem.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/ResultItem.kt`
- [HTML tree viewer](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/HtmlTreeDialog.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/HtmlTreeDialog.kt`
- [XPath guide](https://github.com/verdgard/crawlertool/blob/main/app/src/main/kotlin/com/SoloSu/Crawler_tool/XPathRulesActivity.kt): `app/src/main/kotlin/com/SoloSu/Crawler_tool/XPathRulesActivity.kt`
- **Export output**

### Other components

- **User**
- **Web pages**

## Connections

- User → Main screen: enters query
- Main screen → Main state: forwards actions
- Main state → Crawler repository: requests extraction
- Crawler repository → Crawler engine: fetches and matches
- Crawler engine → Web pages: requests HTML
- Crawler engine → XPath evaluator: evaluates XPath
- Crawler repository → JS renderer: renders when selected
- JS renderer → Web pages: loads page
- Main state → Result list: publishes results
- Result list → Result item: displays items
- Main state → History repository: records and loads
- History repository → History manager: delegates history
- History manager → History storage: reads and writes
- User → Settings screen: configures app
- Settings screen → Settings state: forwards changes
- Settings state → Settings repository: updates preferences
- Settings repository → Settings storage: reads and writes
- Settings repository → Header manager: manages custom headers
- Main state → Export output: writes result formats
- Main screen → HTML tree viewer: opens DOM viewer
- Main screen → XPath guide: opens offline guide