# blessedrebus/krawl architecture

> Architecture diagram of the GitHub repository blessedrebus/krawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blessedrebus/krawl
- Repository: https://github.com/blessedrebus/krawl

## Overview

#### Purpose and Runtime
Krawl is a Python/ASGI web honeypot that serves deceptive pages, captures attacker behavior and credentials, classifies IP reputation, and exposes a hidden operational dashboard. Primary ASGI entry point: `src/app.py`; container startup: `entrypoint.sh`, `Dockerfile`.

#### Request Pipeline
Incoming requests are routed through the application and request middleware. Ban checks can stop known/rate-limited sources; deception middleware records and shapes requests before handlers serve honeypot content. Core boundaries: `src/app.py`, `src/middleware/ban_check.py`, `src/middleware/deception.py`.

#### Honeypot and Deception Serving
Honeypot routes deliver fake applications, spider-trap pages, credential traps, canary content, delays, randomized headers, and error behavior. Static deception inventory is path-oriented and extensive; reusable HTML page types are separate from dashboard templates. Key paths: `src/routes/honeypot.py`, `src/generators.py`, `src/templates/deception/`, `src/templates/html/`.

#### Capture, Persistence, and Reputation
Captured access events, credentials, generated pages, IP statistics, and analytics are persisted behind database modules. Periodic analysis converts recent behavioral signals into IP categories used for dashboard views, bans, and export. Key paths: `src/database/core.py`, `src/database/access_logs.py`, `src/database/ip_stats.py`, `src/tasks/analyze_ips.py`.

#### Dashboard, API, and Firewall Boundary
A secret-path dashboard provides operational views and protected management functions; HTMX/Jinja templates render panels. API routes expose data and IP exports, while firewall format implementations isolate raw, iptables, and nftables output. Key paths: `src/routes/dashboard.py`, `src/routes/api.py`, `src/routes/htmx.py`, `src/firewall/`.

#### AI and External Integrations
Optional AI generation creates and caches unique deception pages, with fallback to ordinary honeypot serving. Configured providers are OpenRouter or OpenAI-compatible APIs; a canary-token URL can trigger external alerts. Key paths: `src/generative_ai.py`, `src/deception_responses.py`, `src/database/generated_pages.py`.

#### Deployment and Operations
Standalone mode uses SQLite and in-process caching; scalable mode uses PostgreSQL plus Redis to support multiple replicas and cache dashboard/hot-path data. Background task orchestration handles analysis, cache warming, retention, log flushing, metrics, and database dumps. Kubernetes/Helm defaults support production infrastructure. Key paths: `src/config.py`, `src/tasks_master.py`, `docker/docker-compose.scalable.yaml`, `helm/templates/deployment.yaml`, `helm/templates/postgres.yaml`, `helm/templates/redis.yaml`.

Main entry points: [`src/app.py`](https://github.com/blessedrebus/krawl/blob/main/src/app.py) (ASGI application), [`src/database/core.py`](https://github.com/blessedrebus/krawl/blob/main/src/database/core.py) (Database core), [`src/database/ip_stats.py`](https://github.com/blessedrebus/krawl/blob/main/src/database/ip_stats.py) (IP statistics), [`src/database/access_logs.py`](https://github.com/blessedrebus/krawl/blob/main/src/database/access_logs.py) (Captured access logs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["ASGI runtime"]
  node_container["Container startup<br/>deployment<br/>[entrypoint.sh]"]
  node_app{{"ASGI application<br/>Python ASGI<br/>[app.py]"}}
  node_ban_check["Ban check middleware<br/>[ban_check.py]"]
  node_deception_mw["Deception middleware<br/>[deception.py]"]
end

subgraph group_deception["Deception serving"]
  node_honeypot_routes["Honeypot routes<br/>route handlers<br/>[honeypot.py]"]
  node_page_generators["Page generators<br/>content generator<br/>[generators.py]"]
  node_deception_templates["Deception inventory<br/>static templates"]
  node_html_templates["Reusable HTML pages<br/>templates"]
  node_ai_generation["AI page generation<br/>optional integration<br/>[generative_ai.py]"]
  node_response_fallback["Deception responses<br/>fallback responses"]
end

subgraph group_data["Capture and intelligence"]
  node_database_core[("Database core<br/>persistence boundary<br/>[core.py]")]
  node_access_logs[("Captured access logs<br/>event store<br/>[access_logs.py]")]
  node_ip_stats[("IP statistics<br/>reputation store<br/>[ip_stats.py]")]
  node_generated_pages[("Generated page cache<br/>page store<br/>[generated_pages.py]")]
  node_ip_analysis["IP analysis task<br/>periodic task<br/>[analyze_ips.py]"]
end

subgraph group_operations["Operations and deployment"]
  node_dashboard["Secret dashboard<br/>operational UI<br/>[dashboard.py]"]
  node_api["Management API<br/>API routes<br/>[api.py]"]
  node_htmx["Dashboard panels<br/>HTMX routes<br/>[htmx.py]"]
  node_firewall["Firewall exporters<br/>export adapters"]
  node_task_orchestrator["Background orchestration<br/>task scheduler<br/>[tasks_master.py]"]
  node_runtime_config["Runtime configuration<br/>[config.py]"]
  node_scalable_stack["Scalable stack<br/>Compose deployment"]
  node_kubernetes["Kubernetes deployment<br/>Helm deployment<br/>[deployment.yaml]"]
end

node_container -->|"starts"| node_app
node_app -->|"request pipeline"| node_ban_check
node_ban_check -->|"allowed requests"| node_deception_mw
node_deception_mw -->|"serves"| node_honeypot_routes
node_deception_mw -->|"captures events"| node_access_logs
node_honeypot_routes -->|"builds pages"| node_page_generators
node_page_generators -->|"uses inventory"| node_deception_templates
node_page_generators -->|"renders"| node_html_templates
node_honeypot_routes -.->|"optional generation"| node_ai_generation
node_ai_generation -->|"caches pages"| node_generated_pages
node_ai_generation -.->|"on unavailable provider"| node_response_fallback
node_access_logs -->|"persists through"| node_database_core
node_ip_stats -->|"persists through"| node_database_core
node_generated_pages -->|"persists through"| node_database_core
node_task_orchestrator -->|"schedules"| node_ip_analysis
node_ip_analysis -->|"reads signals"| node_access_logs
node_ip_analysis -->|"writes categories"| node_ip_stats
node_ip_stats -->|"supplies reputation"| node_ban_check
node_app -->|"routes hidden UI"| node_dashboard
node_app -->|"routes API"| node_api
node_app -->|"routes panels"| node_htmx
node_dashboard -->|"views intelligence"| node_ip_stats
node_htmx -->|"renders activity"| node_access_logs
node_api -->|"exports IP rules"| node_firewall
node_runtime_config -->|"selects backend"| node_database_core
node_scalable_stack -->|"provides PostgreSQL"| node_database_core
node_kubernetes -->|"deploys replicas"| node_app

click node_container "https://github.com/blessedrebus/krawl/blob/main/entrypoint.sh"
click node_app "https://github.com/blessedrebus/krawl/blob/main/src/app.py"
click node_ban_check "https://github.com/blessedrebus/krawl/blob/main/src/middleware/ban_check.py"
click node_deception_mw "https://github.com/blessedrebus/krawl/blob/main/src/middleware/deception.py"
click node_honeypot_routes "https://github.com/blessedrebus/krawl/blob/main/src/routes/honeypot.py"
click node_page_generators "https://github.com/blessedrebus/krawl/blob/main/src/generators.py"
click node_ai_generation "https://github.com/blessedrebus/krawl/blob/main/src/generative_ai.py"
click node_response_fallback "https://github.com/blessedrebus/krawl/blob/main/src/deception_responses.py"
click node_database_core "https://github.com/blessedrebus/krawl/blob/main/src/database/core.py"
click node_access_logs "https://github.com/blessedrebus/krawl/blob/main/src/database/access_logs.py"
click node_ip_stats "https://github.com/blessedrebus/krawl/blob/main/src/database/ip_stats.py"
click node_generated_pages "https://github.com/blessedrebus/krawl/blob/main/src/database/generated_pages.py"
click node_ip_analysis "https://github.com/blessedrebus/krawl/blob/main/src/tasks/analyze_ips.py"
click node_dashboard "https://github.com/blessedrebus/krawl/blob/main/src/routes/dashboard.py"
click node_api "https://github.com/blessedrebus/krawl/blob/main/src/routes/api.py"
click node_htmx "https://github.com/blessedrebus/krawl/blob/main/src/routes/htmx.py"
click node_task_orchestrator "https://github.com/blessedrebus/krawl/blob/main/src/tasks_master.py"
click node_runtime_config "https://github.com/blessedrebus/krawl/blob/main/src/config.py"
click node_scalable_stack "https://github.com/blessedrebus/krawl/blob/main/docker/docker-compose.scalable.yaml"
click node_kubernetes "https://github.com/blessedrebus/krawl/blob/main/helm/templates/deployment.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_container,node_app,node_ban_check,node_deception_mw toneBlue
class node_honeypot_routes,node_page_generators,node_deception_templates,node_html_templates,node_ai_generation,node_response_fallback toneAmber
class node_database_core,node_access_logs,node_ip_stats,node_generated_pages,node_ip_analysis toneMint
class node_dashboard,node_api,node_htmx,node_firewall,node_task_orchestrator,node_runtime_config,node_scalable_stack,node_kubernetes toneRose
```

## Components

### ASGI runtime

- [Container startup](https://github.com/blessedrebus/krawl/blob/main/entrypoint.sh): deployment · `entrypoint.sh`
- [ASGI application](https://github.com/blessedrebus/krawl/blob/main/src/app.py): Python ASGI · `src/app.py`
- [Ban check middleware](https://github.com/blessedrebus/krawl/blob/main/src/middleware/ban_check.py): middleware · Stops banned or rate-limited sources. · `src/middleware/ban_check.py`
- [Deception middleware](https://github.com/blessedrebus/krawl/blob/main/src/middleware/deception.py): middleware · Records and shapes inbound requests. · `src/middleware/deception.py`

### Deception serving

- [Honeypot routes](https://github.com/blessedrebus/krawl/blob/main/src/routes/honeypot.py): route handlers · `src/routes/honeypot.py`
- [Page generators](https://github.com/blessedrebus/krawl/blob/main/src/generators.py): content generator · `src/generators.py`
- **Deception inventory**: static templates
- **Reusable HTML pages**: templates
- [AI page generation](https://github.com/blessedrebus/krawl/blob/main/src/generative_ai.py): optional integration · Uses configured OpenRouter or OpenAI-compatible providers. · `src/generative_ai.py`
- [Deception responses](https://github.com/blessedrebus/krawl/blob/main/src/deception_responses.py): fallback responses · `src/deception_responses.py`

### Capture and intelligence

- [Database core](https://github.com/blessedrebus/krawl/blob/main/src/database/core.py): persistence boundary · SQLite standalone or PostgreSQL at scale. · `src/database/core.py`
- [Captured access logs](https://github.com/blessedrebus/krawl/blob/main/src/database/access_logs.py): event store · `src/database/access_logs.py`
- [IP statistics](https://github.com/blessedrebus/krawl/blob/main/src/database/ip_stats.py): reputation store · `src/database/ip_stats.py`
- [Generated page cache](https://github.com/blessedrebus/krawl/blob/main/src/database/generated_pages.py): page store · `src/database/generated_pages.py`
- [IP analysis task](https://github.com/blessedrebus/krawl/blob/main/src/tasks/analyze_ips.py): periodic task · Classifies recent behavior into IP categories. · `src/tasks/analyze_ips.py`

### Operations and deployment

- [Secret dashboard](https://github.com/blessedrebus/krawl/blob/main/src/routes/dashboard.py): operational UI · `src/routes/dashboard.py`
- [Management API](https://github.com/blessedrebus/krawl/blob/main/src/routes/api.py): API routes · Exposes data and IP exports. · `src/routes/api.py`
- [Dashboard panels](https://github.com/blessedrebus/krawl/blob/main/src/routes/htmx.py): HTMX routes · `src/routes/htmx.py`
- **Firewall exporters**: export adapters · Produces raw, iptables, and nftables formats.
- [Background orchestration](https://github.com/blessedrebus/krawl/blob/main/src/tasks_master.py): task scheduler · Runs analysis, warming, retention, flushing, metrics, and dumps. · `src/tasks_master.py`
- [Runtime configuration](https://github.com/blessedrebus/krawl/blob/main/src/config.py): configuration · Selects standalone or scalable infrastructure. · `src/config.py`
- [Scalable stack](https://github.com/blessedrebus/krawl/blob/main/docker/docker-compose.scalable.yaml): Compose deployment · PostgreSQL and Redis for multi-replica operation. · `docker/docker-compose.scalable.yaml`
- [Kubernetes deployment](https://github.com/blessedrebus/krawl/blob/main/helm/templates/deployment.yaml): Helm deployment · `helm/templates/deployment.yaml`

## Connections

- Container startup → ASGI application: starts
- ASGI application → Ban check middleware: request pipeline
- Ban check middleware → Deception middleware: allowed requests
- Deception middleware → Honeypot routes: serves
- Deception middleware → Captured access logs: captures events
- Honeypot routes → Page generators: builds pages
- Page generators → Deception inventory: uses inventory
- Page generators → Reusable HTML pages: renders
- Honeypot routes → AI page generation: optional generation
- AI page generation → Generated page cache: caches pages
- AI page generation → Deception responses: on unavailable provider
- Captured access logs → Database core: persists through
- IP statistics → Database core: persists through
- Generated page cache → Database core: persists through
- Background orchestration → IP analysis task: schedules
- IP analysis task → Captured access logs: reads signals
- IP analysis task → IP statistics: writes categories
- IP statistics → Ban check middleware: supplies reputation
- ASGI application → Secret dashboard: routes hidden UI
- ASGI application → Management API: routes API
- ASGI application → Dashboard panels: routes panels
- Secret dashboard → IP statistics: views intelligence
- Dashboard panels → Captured access logs: renders activity
- Management API → Firewall exporters: exports IP rules
- Runtime configuration → Database core: selects backend
- Scalable stack → Database core: provides PostgreSQL
- Kubernetes deployment → ASGI application: deploys replicas