How projectdiscovery/katana works
Katana is a configurable web crawler with standard, hybrid, and headless modes. Its main workflow is CLI input and configuration, runner setup and engine selection, page fetching and link/endpoint discovery, then filtered and formatted crawl output. Distinct capabilities include scope and similarity controls, JavaScript crawling, automatic form filling, knowledge-base extraction, browser authentication, and response export. The graph preserves these documented branches; some subsystem links are dashed where the sampled source does not show the runtime call directly.
Main entry points: runner.go Runner · crawler_options.go Crawler options · headless.go Headless engine · crawl.go Hybrid engine
Components 21
Entry and setup
- CLI and flags
cmd/katana/main.go - Runner
internal/runner/runner.go - Crawler options
pkg/types/crawler_options.go
Crawl engines
- Standard engine
pkg/engine/standard/crawl.go - Hybrid engine
pkg/engine/hybrid/crawl.go - Headless engine
pkg/engine/headless/headless.go
Browser capabilities
- Browser automation
pkg/engine/headless/browser/browser.go - Auth replay
pkg/engine/headless/auth/replay.go - Browser crawler
pkg/engine/headless/crawler/crawler.go
Discovery and controls
- Response parser
pkg/engine/parser/parser.go - Scope controls
pkg/utils/scope/scope.go - Similarity filtering
pkg/similarity/index.go - Form filling
pkg/utils/formfill.go - Knowledge extraction
pkg/knowledgebase/extractor.go - Endpoint extraction
pkg/knowledgebase/extractors/endpoints/endpoints.go - AI crawl decisions
pkg/decider/decider.go
Results and export
- Output writer
pkg/output/output.go - Traffic export
pkg/output/export/export.go
Other components
- Crawler operator
- Target websites
- Output destination
Connections 26
- Crawler operator → CLI and flags: sets crawl flags
cmd/katana/main.go - CLI and flags → Runner: starts crawl
cmd/katana/main.go - Runner → Crawler options: builds options
internal/runner/runner.go - Runner → Standard engine: selects engine
internal/runner/runner.go - Runner → Hybrid engine: selects engine
internal/runner/runner.go - Runner → Headless engine: selects engine
internal/runner/runner.go - Standard engine → Target websites: fetches pages
- Hybrid engine → Target websites: fetches pages
pkg/engine/hybrid/crawl.go - Headless engine → Browser automation: drives browser
- Browser automation → Target websites: loads pages
- Headless engine → Browser crawler: uses browser crawler
- Headless engine → Auth replay: replays login
- Response parser → Target websites: discovers URLs
- Hybrid engine → Response parser: parses responses
- Headless engine → Response parser: parses responses
- Crawler options → Scope controls: configures scope
- Crawler options → Similarity filtering: configures deduplication
pkg/types/crawler_options.go - Browser crawler → Form filling: fills forms
- Crawler options → Form filling: configures form filling
pkg/types/crawler_options.go - Crawler options → Knowledge extraction: enables classification
pkg/types/crawler_options.go - Knowledge extraction → Endpoint extraction: uses extractor
- Form filling → AI crawl decisions: requests field guidance
pkg/utils/formfill.go - Crawler options → Output writer: configures output
pkg/types/crawler_options.go - Runner → Traffic export: wraps writer
internal/runner/runner.go - Output writer → Output destination: writes results
- Traffic export → Output destination: exports traffic