How trex-fze/open-defender-icap works
Open Defender is an AI-assisted ICAP web-filtering stack. A device’s proxy request reaches the ICAP adaptor, which consults the policy engine and cache; decisions may persist classifications or enqueue asynchronous classification and page-fetch work. Workers enrich and classify content, while the admin API and React console support policy, taxonomy, investigation, and operations workflows. The README documents the principal runtime links; unsampled service internals are represented at their exact source directories, so finer call-level relationships are omitted.
Main entry points: src/main.rs ICAP Adaptor · src/main.rs LLM Worker · src/main.rs Admin API · src/main.rs Reclassification Worker
Components 24
Proxy Ingress
- HAProxy / Squid
- ICAP Adaptor
services/icap-adaptor/src/main.rs
Decision & Governance
- Policy Engine
services/policy-engine/src/main.rs - Redis Cache
- Classifications & Overrides
services/policy-engine/src/store.rs - Taxonomy Governance
crates/taxonomy/src/lib.rs - Activation Profiles
crates/taxonomy/src/activation.rs
Async Enrichment
- Classification Stream
- Page-Fetch Stream
- LLM Worker
workers/llm-worker/src/main.rs - Reclassification Worker
workers/reclass-worker/src/main.rs - Page Fetcher
workers/page-fetcher/src/main.rs - Crawl4AI Service
services/crawl4ai-service/app/main.py - Classification Requests
services/admin-api/src/classification_requests.rs - Page Contents
services/admin-api/src/page_contents.rs
Administration
- Admin API
services/admin-api/src/main.rs - React Admin Console
web-admin/src/App.tsx - odctl CLI
cli/odctl/src/main.rs
Operations & Observability
- Filebeat
- Event Ingester
services/event-ingester/src/main.rs - Elasticsearch
- Kibana
- Prometheus
Other components
- User / Device
Connections 37
- User / Device → HAProxy / Squid: sends proxy traffic
README.md - HAProxy / Squid → ICAP Adaptor: forwards ICAP requests
README.md - ICAP Adaptor → Policy Engine: requests decision
README.md - ICAP Adaptor → Redis Cache: looks up
README.md - Policy Engine → ICAP Adaptor: returns verdict
README.md - Policy Engine → Classifications & Overrides: persists decisions
README.md - ICAP Adaptor → Classification Stream: enqueues classification
README.md - ICAP Adaptor → Page-Fetch Stream: enqueues page fetch
README.md - Classification Stream → LLM Worker: dispatches jobs
README.md - Classification Stream → Reclassification Worker: dispatches jobs
README.md - Reclassification Worker → Classification Stream: queues refresh
README.md - Reclassification Worker → Page-Fetch Stream: queues recrawl
README.md - LLM Worker → Classification Requests: records pending review
README.md - LLM Worker → Classifications & Overrides: writes verdicts
README.md - LLM Worker → Redis Cache: updates cache
README.md - Reclassification Worker → Classifications & Overrides: updates overrides
README.md - Page-Fetch Stream → Page Fetcher: dispatches fetch jobs
README.md - Page Fetcher → Crawl4AI Service: requests crawl
README.md - Page Fetcher → Page Contents: stores page context
README.md - Taxonomy Governance → Policy Engine: supplies taxonomy
README.md - Taxonomy Governance → LLM Worker: supplies taxonomy
README.md - Taxonomy Governance → Reclassification Worker: supplies taxonomy
README.md - Admin API → Activation Profiles: loads and saves profiles
README.md - Activation Profiles → Policy Engine: refreshes activation
README.md - Activation Profiles → LLM Worker: refreshes activation
README.md - Classifications & Overrides → Admin API: serves management data
README.md - Page Contents → Admin API: serves page context
README.md - Classification Requests → Admin API: serves review queue
README.md - React Admin Console → Admin API: calls management API
README.md - odctl CLI → Admin API: calls management API
README.md - Filebeat → Event Ingester: ships logs
README.md - Event Ingester → Elasticsearch: writes events
README.md - Elasticsearch → Kibana: feeds dashboards
README.md - Admin API → Prometheus: exports metrics
README.md - ICAP Adaptor → Prometheus: exports metrics
README.md - LLM Worker → Prometheus: exports metrics
README.md - Event Ingester → Prometheus: exports metrics
README.md