How nftercel/appcrawler works
Project type and purpose
• This is a command-line, full-stack tool (but not web-stack) that implements an automated app crawler for Android/iOS.
• Purpose: parse CLI args and YAML/JSON configs → drive a traversal engine (“Crawler”) → use various drivers (Appium, ADB, Selenium) → explore UI elements → capture screenshots/mind-maps/logs → generate test reports (Allure, JUnit5, HTML/XML, diffs).
File structure
• Top level:
• pom.xml → Maven build, dependency management
• src/main/java & src/main/scala → mixed Java/Scala code
• src/main/resources → config templates (mustache), log4j2.yaml
• src/main/scala/com/ceshiren/appcrawler/core → core engine: AppCrawler, Crawler, AutomationSuite
• src/main/scala/com/ceshiren/appcrawler/driver → driver abstractions & implementations (AdbDriver, AppiumClient, SeleniumDriver, ReactWebDriver, plus MockDriver for testing)
• src/main/scala/com/ceshiren/appcrawler/model → domain models (Step, TreeNode, PageSource, DataRecord)
• src/main/scala/com/ceshiren/appcrawler/plugin → plugin interfaces & implementations (ReportPlugin, DiffSuite, FreeMind, JUnit5, ScalaTest)
• src/main/java/com/ceshiren/appcrawler/report & diff → report helpers, YAML/Allure/JUnit5 utilities
• src/test → comprehensive unit and integration tests
README for additional insights
• CLI options define interaction points: app path, platform, config, capabilities, output directory, report options, diff options, demo mode.
• Describes configuration model (capability, testcase, selectedList, triggerActions, blackList, pluginList, etc.).
• Highlights plugins (report generation, diff, mind-map).
Main components & interactions
a. Components
1. CLI Interface & Argument Parser
2. Configuration Loader (JSON/YAML parser + LoadYaml/ReadYaml → builds CrawlerConf)
3. Core Crawler Engine
• AppCrawler (entrypoint)
• AutomationSuite / AppiumSuite / Crawler
4. Driver Manager (Strategy pattern)
• AppiumClient → communicates with Appium Server
• AdbDriver → uses ADB directly
• SeleniumDriver / ReactWebDriver → for WebView or React Native
5. Domain Model
• TreeNode, DataRecord, Step, PageSource
6. Plugin Manager & Plugins
• ReportPlugin → Junit5Report, AllureReport, HTML/XML
• DiffPlugin → computes master vs candidate diffs
• FreeMind Plugin → generates mind-map
7. Output Sink
• Screenshots directory
• Logs (Log4j2)
• Mind-map files
• Report artifacts (HTML, XML)
b. Relationships & data flow
1. User runs CLI → CLI Interface parses args → invokes Configuration Loader
2. Crawler Engine initializes with CrawlerConf → Plugin Manager registers plugins
3. Core loop:
– Query Driver Manager for UI tree → Domain Model builds TreeNode graph
– Crawler traverses TreeNode according to selectedList/firstList/blackList rules
– For each element: send commands to Driver → device/emulator → capture PageSource and Screenshot → update DataRecord
– Plugin hooks (triggerActions) executed at runtime
4. After traversal or on timeout: Plugin Manager invokes report plugins → writes HTML/XML, diffs, mind-maps, logs
c. Architectural patterns & principles
• Strategy pattern for drivers (pluggable: Appium, ADB, Selenium, Mock).
• Plugin architecture for extensibility (ReportPlugin, DiffPlugin, FreeMind).
• Separation of concerns: Configuration, Core Engine, Drivers, Plugins, Reporting.
• Model–View–Controller style inside CLI: CLI (View/Input), Core Engine (Controller), Domain Model (Model).
d. Key technologies/frameworks
• Java + Scala (mixed codebase)
• Maven build (pom.xml)
• Log4j2 for logging
• Mustache templates for report generation
• Jackson or SnakeYAML for JSON/YAML parsing
• Appium Java client