# trex-fze/open-defender-icap architecture

> Architecture diagram of the GitHub repository trex-fze/open-defender-icap, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trex-fze/open-defender-icap
- Repository: https://github.com/trex-fze/open-defender-icap

## Overview

Open Defender is an AI-assisted ICAP web-filtering stack. A device’s proxy request reaches the ICAP adaptor, which consults the policy engine and cache; decisions may persist classifications or enqueue asynchronous classification and page-fetch work. Workers enrich and classify content, while the admin API and React console support policy, taxonomy, investigation, and operations workflows. The README documents the principal runtime links; unsampled service internals are represented at their exact source directories, so finer call-level relationships are omitted.

Main entry points: [`services/icap-adaptor/src/main.rs`](https://github.com/trex-fze/open-defender-icap/blob/main/services/icap-adaptor/src/main.rs) (ICAP Adaptor), [`workers/llm-worker/src/main.rs`](https://github.com/trex-fze/open-defender-icap/blob/main/workers/llm-worker/src/main.rs) (LLM Worker), [`services/admin-api/src/main.rs`](https://github.com/trex-fze/open-defender-icap/blob/main/services/admin-api/src/main.rs) (Admin API), [`workers/reclass-worker/src/main.rs`](https://github.com/trex-fze/open-defender-icap/blob/main/workers/reclass-worker/src/main.rs) (Reclassification Worker)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingress["Proxy Ingress"]
  node_proxy["HAProxy / Squid"]
  node_icap["ICAP Adaptor<br/>[main.rs]"]
end

subgraph group_decision["Decision &amp; Governance"]
  node_policy["Policy Engine<br/>[main.rs]"]
  node_cache[("Redis Cache")]
  node_classifications[("Classifications &amp; Overrides<br/>[store.rs]")]
  node_taxonomy["Taxonomy Governance<br/>[lib.rs]"]
  node_activation[("Activation Profiles<br/>[activation.rs]")]
end

subgraph group_async["Async Enrichment"]
  node_class_jobs["Classification Stream"]
  node_fetch_jobs["Page-Fetch Stream"]
  node_llm["LLM Worker<br/>[main.rs]"]
  node_reclass["Reclassification Worker<br/>[main.rs]"]
  node_fetcher["Page Fetcher<br/>[main.rs]"]
  node_crawl["Crawl4AI Service<br/>[main.py]"]
  node_pending[("Classification Requests")]
  node_pages[("Page Contents<br/>[page_contents.rs]")]
end

subgraph group_admin["Administration"]
  node_admin_api["Admin API<br/>[main.rs]"]
  node_console["React Admin Console<br/>[App.tsx]"]
  node_cli["odctl CLI<br/>[main.rs]"]
end

subgraph group_ops["Operations &amp; Observability"]
  node_filebeat["Filebeat"]
  node_ingester["Event Ingester<br/>[main.rs]"]
  node_elastic[("Elasticsearch")]
  node_kibana["Kibana"]
  node_prometheus["Prometheus"]
end

node_device(("User / Device"))

node_device -->|"sends proxy traffic"| node_proxy
node_proxy -->|"forwards ICAP requests"| node_icap
node_icap -->|"requests decision"| node_policy
node_icap -->|"looks up"| node_cache
node_policy -->|"returns verdict"| node_icap
node_policy -->|"persists decisions"| node_classifications
node_icap -->|"enqueues classification"| node_class_jobs
node_icap -->|"enqueues page fetch"| node_fetch_jobs
node_class_jobs -->|"dispatches jobs"| node_llm
node_class_jobs -->|"dispatches jobs"| node_reclass
node_reclass -->|"queues refresh"| node_class_jobs
node_reclass -->|"queues recrawl"| node_fetch_jobs
node_llm -->|"records pending review"| node_pending
node_llm -->|"writes verdicts"| node_classifications
node_llm -->|"updates cache"| node_cache
node_reclass -->|"updates overrides"| node_classifications
node_fetch_jobs -->|"dispatches fetch jobs"| node_fetcher
node_fetcher -->|"requests crawl"| node_crawl
node_fetcher -->|"stores page context"| node_pages
node_taxonomy -->|"supplies taxonomy"| node_policy
node_taxonomy -->|"supplies taxonomy"| node_llm
node_taxonomy -->|"supplies taxonomy"| node_reclass
node_admin_api -->|"loads and saves profiles"| node_activation
node_activation -->|"refreshes activation"| node_policy
node_activation -->|"refreshes activation"| node_llm
node_classifications -->|"serves management data"| node_admin_api
node_pages -->|"serves page context"| node_admin_api
node_pending -->|"serves review queue"| node_admin_api
node_console -->|"calls management API"| node_admin_api
node_cli -->|"calls management API"| node_admin_api
node_filebeat -->|"ships logs"| node_ingester
node_ingester -->|"writes events"| node_elastic
node_elastic -->|"feeds dashboards"| node_kibana
node_admin_api -->|"exports metrics"| node_prometheus
node_icap -->|"exports metrics"| node_prometheus
node_llm -->|"exports metrics"| node_prometheus
node_ingester -->|"exports metrics"| node_prometheus

click node_icap "https://github.com/trex-fze/open-defender-icap/blob/main/services/icap-adaptor/src/main.rs"
click node_policy "https://github.com/trex-fze/open-defender-icap/blob/main/services/policy-engine/src/main.rs"
click node_classifications "https://github.com/trex-fze/open-defender-icap/blob/main/services/policy-engine/src/store.rs"
click node_taxonomy "https://github.com/trex-fze/open-defender-icap/blob/main/crates/taxonomy/src/lib.rs"
click node_activation "https://github.com/trex-fze/open-defender-icap/blob/main/crates/taxonomy/src/activation.rs"
click node_llm "https://github.com/trex-fze/open-defender-icap/blob/main/workers/llm-worker/src/main.rs"
click node_reclass "https://github.com/trex-fze/open-defender-icap/blob/main/workers/reclass-worker/src/main.rs"
click node_fetcher "https://github.com/trex-fze/open-defender-icap/blob/main/workers/page-fetcher/src/main.rs"
click node_crawl "https://github.com/trex-fze/open-defender-icap/blob/main/services/crawl4ai-service/app/main.py"
click node_pending "https://github.com/trex-fze/open-defender-icap/blob/main/services/admin-api/src/classification_requests.rs"
click node_pages "https://github.com/trex-fze/open-defender-icap/blob/main/services/admin-api/src/page_contents.rs"
click node_admin_api "https://github.com/trex-fze/open-defender-icap/blob/main/services/admin-api/src/main.rs"
click node_console "https://github.com/trex-fze/open-defender-icap/blob/main/web-admin/src/App.tsx"
click node_cli "https://github.com/trex-fze/open-defender-icap/blob/main/cli/odctl/src/main.rs"
click node_ingester "https://github.com/trex-fze/open-defender-icap/blob/main/services/event-ingester/src/main.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_proxy,node_icap,node_device toneBlue
class node_policy,node_cache,node_classifications,node_taxonomy,node_activation toneAmber
class node_class_jobs,node_fetch_jobs,node_llm,node_reclass,node_fetcher,node_crawl,node_pending,node_pages toneMint
class node_admin_api,node_console,node_cli toneRose
class node_filebeat,node_ingester,node_elastic,node_kibana,node_prometheus toneIndigo
```

## Components

### Proxy Ingress

- **HAProxy / Squid**
- [ICAP Adaptor](https://github.com/trex-fze/open-defender-icap/blob/main/services/icap-adaptor/src/main.rs): `services/icap-adaptor/src/main.rs`

### Decision & Governance

- [Policy Engine](https://github.com/trex-fze/open-defender-icap/blob/main/services/policy-engine/src/main.rs): `services/policy-engine/src/main.rs`
- **Redis Cache**
- [Classifications & Overrides](https://github.com/trex-fze/open-defender-icap/blob/main/services/policy-engine/src/store.rs): `services/policy-engine/src/store.rs`
- [Taxonomy Governance](https://github.com/trex-fze/open-defender-icap/blob/main/crates/taxonomy/src/lib.rs): `crates/taxonomy/src/lib.rs`
- [Activation Profiles](https://github.com/trex-fze/open-defender-icap/blob/main/crates/taxonomy/src/activation.rs): `crates/taxonomy/src/activation.rs`

### Async Enrichment

- **Classification Stream**
- **Page-Fetch Stream**
- [LLM Worker](https://github.com/trex-fze/open-defender-icap/blob/main/workers/llm-worker/src/main.rs): `workers/llm-worker/src/main.rs`
- [Reclassification Worker](https://github.com/trex-fze/open-defender-icap/blob/main/workers/reclass-worker/src/main.rs): `workers/reclass-worker/src/main.rs`
- [Page Fetcher](https://github.com/trex-fze/open-defender-icap/blob/main/workers/page-fetcher/src/main.rs): `workers/page-fetcher/src/main.rs`
- [Crawl4AI Service](https://github.com/trex-fze/open-defender-icap/blob/main/services/crawl4ai-service/app/main.py): `services/crawl4ai-service/app/main.py`
- [Classification Requests](https://github.com/trex-fze/open-defender-icap/blob/main/services/admin-api/src/classification_requests.rs): `services/admin-api/src/classification_requests.rs`
- [Page Contents](https://github.com/trex-fze/open-defender-icap/blob/main/services/admin-api/src/page_contents.rs): `services/admin-api/src/page_contents.rs`

### Administration

- [Admin API](https://github.com/trex-fze/open-defender-icap/blob/main/services/admin-api/src/main.rs): `services/admin-api/src/main.rs`
- [React Admin Console](https://github.com/trex-fze/open-defender-icap/blob/main/web-admin/src/App.tsx): `web-admin/src/App.tsx`
- [odctl CLI](https://github.com/trex-fze/open-defender-icap/blob/main/cli/odctl/src/main.rs): `cli/odctl/src/main.rs`

### Operations & Observability

- **Filebeat**
- [Event Ingester](https://github.com/trex-fze/open-defender-icap/blob/main/services/event-ingester/src/main.rs): `services/event-ingester/src/main.rs`
- **Elasticsearch**
- **Kibana**
- **Prometheus**

### Other components

- **User / Device**

## Connections

- User / Device → HAProxy / Squid: sends proxy traffic (evidence: `README.md`)
- HAProxy / Squid → ICAP Adaptor: forwards ICAP requests (evidence: `README.md`)
- ICAP Adaptor → Policy Engine: requests decision (evidence: `README.md`)
- ICAP Adaptor → Redis Cache: looks up (evidence: `README.md`)
- Policy Engine → ICAP Adaptor: returns verdict (evidence: `README.md`)
- Policy Engine → Classifications & Overrides: persists decisions (evidence: `README.md`)
- ICAP Adaptor → Classification Stream: enqueues classification (evidence: `README.md`)
- ICAP Adaptor → Page-Fetch Stream: enqueues page fetch (evidence: `README.md`)
- Classification Stream → LLM Worker: dispatches jobs (evidence: `README.md`)
- Classification Stream → Reclassification Worker: dispatches jobs (evidence: `README.md`)
- Reclassification Worker → Classification Stream: queues refresh (evidence: `README.md`)
- Reclassification Worker → Page-Fetch Stream: queues recrawl (evidence: `README.md`)
- LLM Worker → Classification Requests: records pending review (evidence: `README.md`)
- LLM Worker → Classifications & Overrides: writes verdicts (evidence: `README.md`)
- LLM Worker → Redis Cache: updates cache (evidence: `README.md`)
- Reclassification Worker → Classifications & Overrides: updates overrides (evidence: `README.md`)
- Page-Fetch Stream → Page Fetcher: dispatches fetch jobs (evidence: `README.md`)
- Page Fetcher → Crawl4AI Service: requests crawl (evidence: `README.md`)
- Page Fetcher → Page Contents: stores page context (evidence: `README.md`)
- Taxonomy Governance → Policy Engine: supplies taxonomy (evidence: `README.md`)
- Taxonomy Governance → LLM Worker: supplies taxonomy (evidence: `README.md`)
- Taxonomy Governance → Reclassification Worker: supplies taxonomy (evidence: `README.md`)
- Admin API → Activation Profiles: loads and saves profiles (evidence: `README.md`)
- Activation Profiles → Policy Engine: refreshes activation (evidence: `README.md`)
- Activation Profiles → LLM Worker: refreshes activation (evidence: `README.md`)
- Classifications & Overrides → Admin API: serves management data (evidence: `README.md`)
- Page Contents → Admin API: serves page context (evidence: `README.md`)
- Classification Requests → Admin API: serves review queue (evidence: `README.md`)
- React Admin Console → Admin API: calls management API (evidence: `README.md`)
- odctl CLI → Admin API: calls management API (evidence: `README.md`)
- Filebeat → Event Ingester: ships logs (evidence: `README.md`)
- Event Ingester → Elasticsearch: writes events (evidence: `README.md`)
- Elasticsearch → Kibana: feeds dashboards (evidence: `README.md`)
- Admin API → Prometheus: exports metrics (evidence: `README.md`)
- ICAP Adaptor → Prometheus: exports metrics (evidence: `README.md`)
- LLM Worker → Prometheus: exports metrics (evidence: `README.md`)
- Event Ingester → Prometheus: exports metrics (evidence: `README.md`)
