How blessedrebus/krawl works
Purpose and Runtime
Krawl is a Python/ASGI web honeypot that serves deceptive pages, captures attacker behavior and credentials, classifies IP reputation, and exposes a hidden operational dashboard. Primary ASGI entry point: src/app.py; container startup: entrypoint.sh, Dockerfile.
Request Pipeline
Incoming requests are routed through the application and request middleware. Ban checks can stop known/rate-limited sources; deception middleware records and shapes requests before handlers serve honeypot content. Core boundaries: src/app.py, src/middleware/ban_check.py, src/middleware/deception.py.
Honeypot and Deception Serving
Honeypot routes deliver fake applications, spider-trap pages, credential traps, canary content, delays, randomized headers, and error behavior. Static deception inventory is path-oriented and extensive; reusable HTML page types are separate from dashboard templates. Key paths: src/routes/honeypot.py, src/generators.py, src/templates/deception/, src/templates/html/.
Capture, Persistence, and Reputation
Captured access events, credentials, generated pages, IP statistics, and analytics are persisted behind database modules. Periodic analysis converts recent behavioral signals into IP categories used for dashboard views, bans, and export. Key paths: src/database/core.py, src/database/access_logs.py, src/database/ip_stats.py, src/tasks/analyze_ips.py.
Dashboard, API, and Firewall Boundary
A secret-path dashboard provides operational views and protected management functions; HTMX/Jinja templates render panels. API routes expose data and IP exports, while firewall format implementations isolate raw, iptables, and nftables output. Key paths: src/routes/dashboard.py, src/routes/api.py, src/routes/htmx.py, src/firewall/.
AI and External Integrations
Optional AI generation creates and caches unique deception pages, with fallback to ordinary honeypot serving. Configured providers are OpenRouter or OpenAI-compatible APIs; a canary-token URL can trigger external alerts. Key paths: src/generative_ai.py, src/deception_responses.py, src/database/generated_pages.py.
Deployment and Operations
Standalone mode uses SQLite and in-process caching; scalable mode uses PostgreSQL plus Redis to support multiple replicas and cache dashboard/hot-path data. Background task orchestration handles analysis, cache warming, retention, log flushing, metrics, and database dumps. Kubernetes/Helm defaults support production infrastructure. Key paths: src/config.py, src/tasks_master.py, docker/docker-compose.scalable.yaml, helm/templates/deployment.yaml, helm/templates/postgres.yaml, helm/templates/redis.yaml.
Main entry points: app.py ASGI application · core.py Database core · ip_stats.py IP statistics · access_logs.py Captured access logs
Components 23
ASGI runtime
- Container startup
entrypoint.sh - ASGI application
src/app.py - Ban check middleware · Stops banned or rate-limited sources.
src/middleware/ban_check.py - Deception middleware · Records and shapes inbound requests.
src/middleware/deception.py
Deception serving
- Honeypot routes
src/routes/honeypot.py - Page generators
src/generators.py - Deception inventory
- Reusable HTML pages
- AI page generation · Uses configured OpenRouter or OpenAI-compatible providers.
src/generative_ai.py - Deception responses
src/deception_responses.py
Capture and intelligence
- Database core · SQLite standalone or PostgreSQL at scale.
src/database/core.py - Captured access logs
src/database/access_logs.py - IP statistics
src/database/ip_stats.py - Generated page cache
src/database/generated_pages.py - IP analysis task · Classifies recent behavior into IP categories.
src/tasks/analyze_ips.py
Operations and deployment
- Secret dashboard
src/routes/dashboard.py - Management API · Exposes data and IP exports.
src/routes/api.py - Dashboard panels
src/routes/htmx.py - Firewall exporters · Produces raw, iptables, and nftables formats.
- Background orchestration · Runs analysis, warming, retention, flushing, metrics, and dumps.
src/tasks_master.py - Runtime configuration · Selects standalone or scalable infrastructure.
src/config.py - Scalable stack · PostgreSQL and Redis for multi-replica operation.
docker/docker-compose.scalable.yaml - Kubernetes deployment
helm/templates/deployment.yaml
Connections 27
- Container startup → ASGI application: starts
- ASGI application → Ban check middleware: request pipeline
- Ban check middleware → Deception middleware: allowed requests
- Deception middleware → Honeypot routes: serves
- Deception middleware → Captured access logs: captures events
- Honeypot routes → Page generators: builds pages
- Page generators → Deception inventory: uses inventory
- Page generators → Reusable HTML pages: renders
- Honeypot routes → AI page generation: optional generation
- AI page generation → Generated page cache: caches pages
- AI page generation → Deception responses: on unavailable provider
- Captured access logs → Database core: persists through
- IP statistics → Database core: persists through
- Generated page cache → Database core: persists through
- Background orchestration → IP analysis task: schedules
- IP analysis task → Captured access logs: reads signals
- IP analysis task → IP statistics: writes categories
- IP statistics → Ban check middleware: supplies reputation
- ASGI application → Secret dashboard: routes hidden UI
- ASGI application → Management API: routes API
- ASGI application → Dashboard panels: routes panels
- Secret dashboard → IP statistics: views intelligence
- Dashboard panels → Captured access logs: renders activity
- Management API → Firewall exporters: exports IP rules
- Runtime configuration → Database core: selects backend
- Scalable stack → Database core: provides PostgreSQL
- Kubernetes deployment → ASGI application: deploys replicas