How letterligo/agent-webcloak works
WebCloak is a research artifact for defending webpages against LLM-driven scraping and evaluating scraper behavior on LLMCrawlBench. Its main workflow applies Stage 1 structural obfuscation or Stage 2 semantic defenses to HTML, then evaluates LLM scraper implementations against original or defended pages. The tree also contains Crawl4AI and browser-agent experiment implementations. Stage 2 source references identify its orchestration dependencies; several experiment internals are unsampled, so their detailed runtime links are omitted.
Main entry points: async_webcrawler.py Async Crawler · main_stage2.py Semantic Defense · defend.py Structural Defense · run_browser_use.py Browser-Use Runner
Components 30
WebCloak Defense
- Structural Defense
artifact/source_code/stage1/defend.py - Page Iteration
artifact/source_code/stage1/iterate.py - Semantic Defense
artifact/source_code/stage2/main_stage2.py - Defense Prompts
artifact/source_code/stage2/prompts.py - LLM Providers
artifact/source_code/stage2/language_models.py - Defense Evaluators
artifact/source_code/stage2/evaluators.py - Defense Config
artifact/source_code/stage2/config.py - Defense Utilities
artifact/source_code/stage2/utils.py
Scraper Evaluation
- Browser-Use Runner
artifact/experiments/run_browser_use.py - Browser-Use Adapter
artifact/experiments/util_browser_use.py - Crawl4AI Runner
artifact/experiments/run_crawl4ai.py - Crawl4AI Adapter
artifact/experiments/util_crawl4ai.py - LLM-to-Script Runner
artifact/experiments/llm2script.py - Script Variants
artifact/experiments/LLM-to-Script/l2s_3.py
Crawl4AI Runtime
- Async Crawler
artifact/experiments/crawl4ai_package/async_webcrawler.py - Browser Manager
artifact/experiments/crawl4ai_package/browser_manager.py - Crawler Configuration
artifact/experiments/crawl4ai_package/async_configs.py - Content Scraping
artifact/experiments/crawl4ai_package/content_scraping_strategy.py - Content Extraction
artifact/experiments/crawl4ai_package/extraction_strategy.py - Markdown Generation
artifact/experiments/crawl4ai_package/markdown_generation_strategy.py - HTML-to-Text
artifact/experiments/crawl4ai_package/html2text/__init__.py - Crawl Results
artifact/experiments/crawl4ai_package/models.py - Crawl Cache
artifact/experiments/crawl4ai_package/async_database.py
Shared Support
- Crawl Logging
artifact/experiments/crawl4ai_package/async_logger.py - Crawl Utilities
artifact/experiments/crawl4ai_package/utils.py - Extraction Prompts
artifact/experiments/crawl4ai_package/prompts.py
Other components
- Researcher
- HTML Pages
- LLMCrawlBench
- Scraper Results
Connections 34
- Researcher → Structural Defense: runs defense
- HTML Pages → Structural Defense: provides pages
- Structural Defense → Page Iteration: delegates iteration
artifact/source_code/stage1/defend.py - Structural Defense → HTML Pages: writes defended pages
- Researcher → Semantic Defense: runs defense
- HTML Pages → Semantic Defense: provides input
- Semantic Defense → Defense Prompts: uses prompts
artifact/source_code/stage2/main_stage2.py - Semantic Defense → LLM Providers: calls models
artifact/source_code/stage2/main_stage2.py - Semantic Defense → Defense Evaluators: evaluates variations
artifact/source_code/stage2/main_stage2.py - Semantic Defense → Defense Config: reads settings
artifact/source_code/stage2/main_stage2.py - Semantic Defense → Defense Utilities: uses utilities
artifact/source_code/stage2/main_stage2.py - Researcher → Browser-Use Runner: runs experiment
- Browser-Use Runner → Browser-Use Adapter: uses adapter
artifact/experiments/util_browser_use.py - Researcher → Crawl4AI Runner: runs experiment
- Crawl4AI Runner → Crawl4AI Adapter: uses adapter
- Researcher → LLM-to-Script Runner: runs experiment
- LLM-to-Script Runner → Script Variants: dispatches scripts
- LLMCrawlBench → Browser-Use Runner: supplies benchmark pages
- LLMCrawlBench → Crawl4AI Runner: supplies benchmark pages
- LLMCrawlBench → LLM-to-Script Runner: supplies benchmark pages
- Browser-Use Runner → Scraper Results: produces results
- Crawl4AI Runner → Scraper Results: produces results
- LLM-to-Script Runner → Scraper Results: produces results
- Async Crawler → Crawler Configuration: accepts settings
- Async Crawler → Browser Manager: drives browser
- Async Crawler → Content Scraping: scrapes content
- Async Crawler → Content Extraction: extracts content
- Async Crawler → Markdown Generation: generates markdown
- Markdown Generation → HTML-to-Text: converts HTML
- Async Crawler → Crawl Cache: reads and writes cache
- Crawl Cache → Crawl Results: serializes results
artifact/experiments/crawl4ai_package/async_database.py - Async Crawler → Crawl Logging: logs activity
- Async Crawler → Crawl Utilities: uses helpers
- Content Extraction → Extraction Prompts: uses prompts