# projectdiscovery/katana architecture

> Architecture diagram of the GitHub repository projectdiscovery/katana, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/projectdiscovery/katana
- Repository: https://github.com/projectdiscovery/katana

## Overview

Katana is a configurable web crawler with standard, hybrid, and headless modes. Its main workflow is CLI input and configuration, runner setup and engine selection, page fetching and link/endpoint discovery, then filtered and formatted crawl output. Distinct capabilities include scope and similarity controls, JavaScript crawling, automatic form filling, knowledge-base extraction, browser authentication, and response export. The graph preserves these documented branches; some subsystem links are dashed where the sampled source does not show the runtime call directly.

Main entry points: [`internal/runner/runner.go`](https://github.com/projectdiscovery/katana/blob/dev/internal/runner/runner.go) (Runner), [`pkg/types/crawler_options.go`](https://github.com/projectdiscovery/katana/blob/dev/pkg/types/crawler_options.go) (Crawler options), [`pkg/engine/headless/headless.go`](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/headless.go) (Headless engine), [`pkg/engine/hybrid/crawl.go`](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/hybrid/crawl.go) (Hybrid engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and setup"]
  node_cli["CLI and flags<br/>[main.go]"]
  node_runner["Runner<br/>[runner.go]"]
  node_options["Crawler options<br/>[crawler_options.go]"]
end

subgraph group_crawl["Crawl engines"]
  node_standard["Standard engine<br/>[crawl.go]"]
  node_hybrid["Hybrid engine<br/>[crawl.go]"]
  node_headless["Headless engine<br/>[headless.go]"]
end

subgraph group_browser["Browser capabilities"]
  node_browser["Browser automation<br/>[browser.go]"]
  node_auth["Auth replay<br/>[replay.go]"]
  node_crawler["Browser crawler<br/>[crawler.go]"]
end

subgraph group_discovery["Discovery and controls"]
  node_parser["Response parser<br/>[parser.go]"]
  node_scope["Scope controls<br/>[scope.go]"]
  node_similarity["Similarity filtering<br/>[index.go]"]
  node_formfill["Form filling<br/>[formfill.go]"]
  node_knowledge["Knowledge extraction<br/>[extractor.go]"]
  node_endpoints["Endpoint extraction<br/>[endpoints.go]"]
  node_decider["AI crawl decisions<br/>[decider.go]"]
end

subgraph group_results["Results and export"]
  node_output["Output writer<br/>[output.go]"]
  node_export["Traffic export<br/>[export.go]"]
end

node_user(("Crawler operator"))
node_site(("Target websites"))
node_destination(("Output destination"))

node_user -->|"sets crawl flags"| node_cli
node_cli -->|"starts crawl"| node_runner
node_runner -->|"builds options"| node_options
node_runner -->|"selects engine"| node_standard
node_runner -->|"selects engine"| node_hybrid
node_runner -->|"selects engine"| node_headless
node_standard -.->|"fetches pages"| node_site
node_hybrid -->|"fetches pages"| node_site
node_headless -.->|"drives browser"| node_browser
node_browser -.->|"loads pages"| node_site
node_headless -.->|"uses browser crawler"| node_crawler
node_headless -.->|"replays login"| node_auth
node_parser -->|"discovers URLs"| node_site
node_hybrid -.->|"parses responses"| node_parser
node_headless -.->|"parses responses"| node_parser
node_options -.->|"configures scope"| node_scope
node_options -->|"configures deduplication"| node_similarity
node_crawler -.->|"fills forms"| node_formfill
node_options -->|"configures form filling"| node_formfill
node_options -->|"enables classification"| node_knowledge
node_knowledge -.->|"uses extractor"| node_endpoints
node_formfill -->|"requests field guidance"| node_decider
node_options -->|"configures output"| node_output
node_runner -->|"wraps writer"| node_export
node_output -.->|"writes results"| node_destination
node_export -.->|"exports traffic"| node_destination

click node_cli "https://github.com/projectdiscovery/katana/blob/dev/cmd/katana/main.go"
click node_runner "https://github.com/projectdiscovery/katana/blob/dev/internal/runner/runner.go"
click node_options "https://github.com/projectdiscovery/katana/blob/dev/pkg/types/crawler_options.go"
click node_standard "https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/standard/crawl.go"
click node_hybrid "https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/hybrid/crawl.go"
click node_headless "https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/headless.go"
click node_browser "https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/browser/browser.go"
click node_auth "https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/auth/replay.go"
click node_crawler "https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/crawler/crawler.go"
click node_parser "https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/parser/parser.go"
click node_scope "https://github.com/projectdiscovery/katana/blob/dev/pkg/utils/scope/scope.go"
click node_similarity "https://github.com/projectdiscovery/katana/blob/dev/pkg/similarity/index.go"
click node_formfill "https://github.com/projectdiscovery/katana/blob/dev/pkg/utils/formfill.go"
click node_knowledge "https://github.com/projectdiscovery/katana/blob/dev/pkg/knowledgebase/extractor.go"
click node_endpoints "https://github.com/projectdiscovery/katana/blob/dev/pkg/knowledgebase/extractors/endpoints/endpoints.go"
click node_decider "https://github.com/projectdiscovery/katana/blob/dev/pkg/decider/decider.go"
click node_output "https://github.com/projectdiscovery/katana/blob/dev/pkg/output/output.go"
click node_export "https://github.com/projectdiscovery/katana/blob/dev/pkg/output/export/export.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_runner,node_options toneBlue
class node_standard,node_hybrid,node_headless toneAmber
class node_browser,node_auth,node_crawler toneMint
class node_parser,node_scope,node_similarity,node_formfill,node_knowledge,node_endpoints,node_decider toneRose
class node_output,node_export,node_user,node_site,node_destination toneIndigo
```

## Components

### Entry and setup

- [CLI and flags](https://github.com/projectdiscovery/katana/blob/dev/cmd/katana/main.go): `cmd/katana/main.go`
- [Runner](https://github.com/projectdiscovery/katana/blob/dev/internal/runner/runner.go): `internal/runner/runner.go`
- [Crawler options](https://github.com/projectdiscovery/katana/blob/dev/pkg/types/crawler_options.go): `pkg/types/crawler_options.go`

### Crawl engines

- [Standard engine](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/standard/crawl.go): `pkg/engine/standard/crawl.go`
- [Hybrid engine](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/hybrid/crawl.go): `pkg/engine/hybrid/crawl.go`
- [Headless engine](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/headless.go): `pkg/engine/headless/headless.go`

### Browser capabilities

- [Browser automation](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/browser/browser.go): `pkg/engine/headless/browser/browser.go`
- [Auth replay](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/auth/replay.go): `pkg/engine/headless/auth/replay.go`
- [Browser crawler](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/headless/crawler/crawler.go): `pkg/engine/headless/crawler/crawler.go`

### Discovery and controls

- [Response parser](https://github.com/projectdiscovery/katana/blob/dev/pkg/engine/parser/parser.go): `pkg/engine/parser/parser.go`
- [Scope controls](https://github.com/projectdiscovery/katana/blob/dev/pkg/utils/scope/scope.go): `pkg/utils/scope/scope.go`
- [Similarity filtering](https://github.com/projectdiscovery/katana/blob/dev/pkg/similarity/index.go): `pkg/similarity/index.go`
- [Form filling](https://github.com/projectdiscovery/katana/blob/dev/pkg/utils/formfill.go): `pkg/utils/formfill.go`
- [Knowledge extraction](https://github.com/projectdiscovery/katana/blob/dev/pkg/knowledgebase/extractor.go): `pkg/knowledgebase/extractor.go`
- [Endpoint extraction](https://github.com/projectdiscovery/katana/blob/dev/pkg/knowledgebase/extractors/endpoints/endpoints.go): `pkg/knowledgebase/extractors/endpoints/endpoints.go`
- [AI crawl decisions](https://github.com/projectdiscovery/katana/blob/dev/pkg/decider/decider.go): `pkg/decider/decider.go`

### Results and export

- [Output writer](https://github.com/projectdiscovery/katana/blob/dev/pkg/output/output.go): `pkg/output/output.go`
- [Traffic export](https://github.com/projectdiscovery/katana/blob/dev/pkg/output/export/export.go): `pkg/output/export/export.go`

### Other components

- **Crawler operator**
- **Target websites**
- **Output destination**

## Connections

- Crawler operator → CLI and flags: sets crawl flags (evidence: `cmd/katana/main.go`)
- CLI and flags → Runner: starts crawl (evidence: `cmd/katana/main.go`)
- Runner → Crawler options: builds options (evidence: `internal/runner/runner.go`)
- Runner → Standard engine: selects engine (evidence: `internal/runner/runner.go`)
- Runner → Hybrid engine: selects engine (evidence: `internal/runner/runner.go`)
- Runner → Headless engine: selects engine (evidence: `internal/runner/runner.go`)
- Standard engine → Target websites: fetches pages
- Hybrid engine → Target websites: fetches pages (evidence: `pkg/engine/hybrid/crawl.go`)
- Headless engine → Browser automation: drives browser
- Browser automation → Target websites: loads pages
- Headless engine → Browser crawler: uses browser crawler
- Headless engine → Auth replay: replays login
- Response parser → Target websites: discovers URLs
- Hybrid engine → Response parser: parses responses
- Headless engine → Response parser: parses responses
- Crawler options → Scope controls: configures scope
- Crawler options → Similarity filtering: configures deduplication (evidence: `pkg/types/crawler_options.go`)
- Browser crawler → Form filling: fills forms
- Crawler options → Form filling: configures form filling (evidence: `pkg/types/crawler_options.go`)
- Crawler options → Knowledge extraction: enables classification (evidence: `pkg/types/crawler_options.go`)
- Knowledge extraction → Endpoint extraction: uses extractor
- Form filling → AI crawl decisions: requests field guidance (evidence: `pkg/utils/formfill.go`)
- Crawler options → Output writer: configures output (evidence: `pkg/types/crawler_options.go`)
- Runner → Traffic export: wraps writer (evidence: `internal/runner/runner.go`)
- Output writer → Output destination: writes results
- Traffic export → Output destination: exports traffic
