# nftercel/appcrawler architecture

> Architecture diagram of the GitHub repository nftercel/appcrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nftercel/appcrawler
- Repository: https://github.com/nftercel/appcrawler

## Overview

##### Project type and purpose
- This is a command-line, full-stack tool (but not web-stack) that implements an automated app crawler for Android/iOS.
- Purpose: parse CLI args and YAML/JSON configs → drive a traversal engine (“Crawler”) → use various drivers (Appium, ADB, Selenium) → explore UI elements → capture screenshots/mind-maps/logs → generate test reports (Allure, JUnit5, HTML/XML, diffs).

##### File structure
- Top level:
- pom.xml → Maven build, dependency management
- src/main/java & src/main/scala → mixed Java/Scala code
- src/main/resources → config templates (mustache), log4j2.yaml
- src/main/scala/com/ceshiren/appcrawler/core → core engine: AppCrawler, Crawler, AutomationSuite
- src/main/scala/com/ceshiren/appcrawler/driver → driver abstractions & implementations (AdbDriver, AppiumClient, SeleniumDriver, ReactWebDriver, plus MockDriver for testing)
- src/main/scala/com/ceshiren/appcrawler/model → domain models (Step, TreeNode, PageSource, DataRecord)
- src/main/scala/com/ceshiren/appcrawler/plugin → plugin interfaces & implementations (ReportPlugin, DiffSuite, FreeMind, JUnit5, ScalaTest)
- src/main/java/com/ceshiren/appcrawler/report & diff → report helpers, YAML/Allure/JUnit5 utilities
- src/test → comprehensive unit and integration tests

##### README for additional insights
- CLI options define interaction points: app path, platform, config, capabilities, output directory, report options, diff options, demo mode.
- Describes configuration model (capability, testcase, selectedList, triggerActions, blackList, pluginList, etc.).
- Highlights plugins (report generation, diff, mind-map).

##### Main components & interactions
a. Components
1. CLI Interface & Argument Parser
2. Configuration Loader (JSON/YAML parser + LoadYaml/ReadYaml → builds CrawlerConf)
3. Core Crawler Engine
- AppCrawler (entrypoint)
- AutomationSuite / AppiumSuite / Crawler
4. Driver Manager (Strategy pattern)
- AppiumClient → communicates with Appium Server
- AdbDriver → uses ADB directly
- SeleniumDriver / ReactWebDriver → for WebView or React Native
5. Domain Model
- TreeNode, DataRecord, Step, PageSource
6. Plugin Manager & Plugins
- ReportPlugin → Junit5Report, AllureReport, HTML/XML
- DiffPlugin → computes master vs candidate diffs
- FreeMind Plugin → generates mind-map
7. Output Sink
- Screenshots directory
- Logs (Log4j2)
- Mind-map files
- Report artifacts (HTML, XML)
b. Relationships & data flow
1. User runs CLI → CLI Interface parses args → invokes Configuration Loader
2. Crawler Engine initializes with CrawlerConf → Plugin Manager registers plugins
3. Core loop:
– Query Driver Manager for UI tree → Domain Model builds TreeNode graph
– Crawler traverses TreeNode according to selectedList/firstList/blackList rules
– For each element: send commands to Driver → device/emulator → capture PageSource and Screenshot → update DataRecord
– Plugin hooks (triggerActions) executed at runtime
4. After traversal or on timeout: Plugin Manager invokes report plugins → writes HTML/XML, diffs, mind-maps, logs
c. Architectural patterns & principles
- Strategy pattern for drivers (pluggable: Appium, ADB, Selenium, Mock).
- Plugin architecture for extensibility (ReportPlugin, DiffPlugin, FreeMind).
- Separation of concerns: Configuration, Core Engine, Drivers, Plugins, Reporting.
- Model–View–Controller style inside CLI: CLI (View/Input), Core Engine (Controller), Domain Model (Model).
d. Key technologies/frameworks
- Java + Scala (mixed codebase)
- Maven build (pom.xml)
- Log4j2 for logging
- Mustache templates for report generation
- Jackson or SnakeYAML for JSON/YAML parsing
- Appium Java client

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI Layer"
        CLI["CLI Interface & Argument Parser"]:::cli
    end

    subgraph "Configuration"
        ConfLoader["Configuration Loader"]:::config
    end

    subgraph "Core Engine"
        Crawler["Core Crawler Engine"]:::core
        Automation["Automation Suites"]:::core
    end

    subgraph "Driver Layer"
        DriverMgr["Driver Manager"]:::driver
        BaseDDT["BaseDDT"]:::driver
        Adb["AdbDriver"]:::driver
        Appium["AppiumClient"]:::driver
        Selenium["SeleniumDriver & DDT"]:::driver
        React["ReactWebDriver"]:::driver
        Mock["MockDriver & MockDDT"]:::driver
    end

    subgraph "Domain Model"
        TreeNode["TreeNode"]:::model
        DataRec["DataRecord"]:::model
        Step["Step"]:::model
        PageSrc["PageSource"]:::model
        Screenshot["ScreenShot"]:::model
    end

    subgraph "Plugin Manager"
        PluginMgr["Plugin Manager"]:::plugin
        PluginIF["Plugin Interfaces"]:::plugin
        ReportFactory["ReportFactory & DiffSuite"]:::plugin
    end

    subgraph "Plugins"
        FreeMind["FreeMind Plugin"]:::plugin
        FlowDiff["FlowDiff Plugin"]:::plugin
        LogPlug["LogPlugin"]:::plugin
        TagLimit["TagLimitPlugin"]:::plugin
        JUnitRun["JUnit5 Plugin"]:::plugin
        ScalaTestRun["ScalaTest Plugin"]:::plugin
    end

    subgraph "Report & Diff Helpers"
        Junit5Rep["Junit5Report.java"]:::report
        AllureTest["AllureTest.java"]:::report
        AllureDiff["AllureDiffTest.java"]:::report
        ReportUtil["Report.java"]:::report
        XPath["XPathUtil.java"]:::report
    end

    subgraph "Output Sink"
        Screenshots["Screenshots Directory"]:::output
        Logs["Log4j2 Logs"]:::output
        MindMaps["Mind-map Files"]:::output
        Reports["Report Artifacts (HTML/XML)"]:::output
        Templates["Mustache Templates"]:::output
    end

    subgraph "External Systems"
        AppiumServer["Appium Server"]:::external
        Device["Device/Emulator"]:::external
    end

    CLI -->|"parses args"| ConfLoader
    ConfLoader -->|"builds CrawlerConf"| Crawler
    Crawler -->|"initializes"| Automation
    Crawler -->|"uses"| DriverMgr
    DriverMgr --> Adb
    DriverMgr --> Appium
    DriverMgr --> Selenium
    DriverMgr --> React
    DriverMgr --> Mock
    DriverMgr --> Device
    Appium --> AppiumServer
    Crawler --> TreeNode
    Crawler --> DataRec
    Crawler --> Step
    Crawler --> PageSrc
    Crawler --> Screenshot
    Crawler -->|"registers"| PluginMgr
    PluginMgr --> PluginIF
    PluginMgr --> ReportFactory
    PluginIF --> FreeMind
    PluginIF --> FlowDiff
    PluginIF --> LogPlug
    PluginIF --> TagLimit
    ReportFactory --> JUnitRun
    ReportFactory --> ScalaTestRun
    JUnitRun --> Junit5Rep
    JUnitRun --> AllureTest
    FlowDiff --> XPath
    FreeMind --> Templates
    LogPlug --> Logs
    TagLimit --> Reports
    JUnitRun --> Reports
    ScalaTestRun --> Reports
    FlowDiff --> Reports
    FreeMind --> MindMaps
    Crawler --> Screenshots

    classDef cli fill:#E3F2FD,stroke:#90CAF9;
    classDef config fill:#FFF3E0,stroke:#FFB74D;
    classDef core fill:#E8F5E9,stroke:#66BB6A;
    classDef driver fill:#F3E5F5,stroke:#AB47BC;
    classDef model fill:#E1F5FE,stroke:#29B6F6;
    classDef plugin fill:#FFFDE7,stroke:#FDD835;
    classDef report fill:#EFEBE9,stroke:#8D6E63;
    classDef output fill:#FBE9E7,stroke:#FF7043;
    classDef external fill:#ECEFF1,stroke:#B0BEC5;

    click CLI "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/AppCrawler.scala"
    click CLI "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/Banner.scala"
    click ConfLoader "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/core/CrawlerConf.scala"
    click ConfLoader "https://github.com/nftercel/appcrawler/blob/master/src/main/java/com/ceshiren/appcrawler/report/LoadYaml.java"
    click ConfLoader "https://github.com/nftercel/appcrawler/blob/master/src/main/java/com/ceshiren/appcrawler/report/ReadYaml.java"
    click Crawler "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/core/Crawler.scala"
    click Automation "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/core/AutomationSuite.scala"
    click Automation "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/core/AppiumSuite.scala"
    click Adb "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/AdbDriver.scala"
    click Appium "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/AppiumClient.scala"
    click Selenium "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/SeleniumDriver.scala"
    click Selenium "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/SeleniumDDT.java"
    click React "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/ReactWebDriver.scala"
    click Mock "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/MockDriver.scala"
    click Mock "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/MockDDT.java"
    click BaseDDT "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/driver/BaseDDT.java"
    click TreeNode "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/model/TreeNode.scala"
    click DataRec "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/model/DataRecord.scala"
    click Step "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/model/Step.scala"
    click PageSrc "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/model/PageSource.java"
    click Screenshot "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/model/ScreenShot.scala"
    click PluginIF "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/Plugin.scala"
    click PluginIF "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/ReportPlugin.scala"
    click FreeMind "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/FreeMind.scala"
    click FlowDiff "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/FlowDiff.scala"
    click LogPlug "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/LogPlugin.scala"
    click TagLimit "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/TagLimitPlugin.scala"
    click JUnitRun "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/junit5/JUnit5Runtime.scala"
    click JUnitRun "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/junit5/AllureTemplate.java"
    click ScalaTestRun "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/scalatest/ScalaTestRuntime.scala"
    click ScalaTestRun "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/scalatest/ScalaTestTemplate.scala"
    click ReportFactory "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/report/ReportFactory.scala"
    click ReportFactory "https://github.com/nftercel/appcrawler/blob/master/src/main/scala/com/ceshiren/appcrawler/plugin/report/DiffSuite.scala"
    click Junit5Rep "https://github.com/nftercel/appcrawler/blob/master/src/main/java/com/ceshiren/appcrawler/report/Junit5Report.java"
    click AllureTest "https://github.com/nftercel/appcrawler/blob/master/src/main/java/com/ceshiren/appcrawler/report/AllureTest.java"
    click AllureDiff "https://github.com/nftercel/appcrawler/blob/master/src/main/java/com/ceshiren/appcrawler/report/AllureDiffTest.java"
    click ReportUtil "https://github.com/nftercel/appcrawler/blob/master/src/main/java/com/ceshiren/appcrawler/report/Report.java"
    click XPath "https://github.com/nftercel/appcrawler/blob/master/src/main/java/com/ceshiren/appcrawler/diff/XPathUtil.java"
    click Templates "https://github.com/nftercel/appcrawler/blob/master/src/main/resources/template.mustache"
    click Logs "https://github.com/nftercel/appcrawler/blob/master/src/main/resources/log4j2.yaml"
```
