# kant2002/ncrawler architecture

> Architecture diagram of the GitHub repository kant2002/ncrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kant2002/ncrawler
- Repository: https://github.com/kant2002/ncrawler

## Overview

NCrawler is a multithreaded, pipeline-based web crawler. A caller starts a crawl at a URI; the crawler checks rules and history, schedules downloads, and passes downloaded content through pipeline steps. The HTML processor extracts page data and queues discovered links for further crawling. Documented extensions include text, PDF, IFilter, and language processing, plus interchangeable storage backends. Several of those implementations are unsampled, so their internal links are not asserted; optional integrations are shown as unverified.

Main entry points: [`src/NCrawler/Crawler.cs`](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Crawler.cs) (Crawler orchestration), [`src/NCrawler/Interfaces/ICrawlerQueue.cs`](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Interfaces/ICrawlerQueue.cs) (Queue contract), [`src/NCrawler/Interfaces/ICrawlerHistory.cs`](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Interfaces/ICrawlerHistory.cs) (History contract), [`src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs`](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs) (HTML processing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_engine["Crawl engine"]
  node_crawler["Crawler orchestration<br/>[Crawler.cs]"]
  node_downloader["Web downloader<br/>[WebDownloaderV2.cs]"]
  node_rules["Crawl rules"]
  node_robots["Robots policy<br/>[RobotService.cs]"]
  node_crawlstep["Crawl step<br/>[CrawlStep.cs]"]
  node_propertybag["Response properties<br/>[PropertyBag.cs]"]
end

subgraph group_processing["Content processing"]
  node_html["HTML processing"]
  node_textprocessor["Text processing"]
  node_linkextract["Link extraction"]
end

subgraph group_storage["Queue and history"]
  node_queue["Queue contract<br/>[ICrawlerQueue.cs]"]
  node_history["History contract<br/>[ICrawlerHistory.cs]"]
  node_memoryservices["In-memory services"]
  node_fileadapter[("File storage")]
  node_efadapter[("Entity Framework storage")]
  node_esentadapter[("ESENT storage")]
  node_isolatedadapter[("Isolated storage")]
  node_redisadapter[("Redis storage")]
end

subgraph group_extensions["Processing extensions"]
  node_pdf["PDF processing"]
  node_ifilter["IFilter processing"]
  node_language["Language detection"]
  node_mp3["MP3 processing"]
  node_sitemap["Sitemap processing"]
end

node_caller(("Crawl caller"))
node_externalweb(("Web servers"))
node_google(("Google detection service"))

node_caller -.->|"starts crawl"| node_crawler
node_crawler -->|"checks URLs"| node_rules
node_rules -->|"checks policy"| node_robots
node_crawler -->|"registers URLs"| node_history
node_crawler -->|"schedules steps"| node_queue
node_crawler -->|"downloads URLs"| node_downloader
node_downloader -.->|"requests pages"| node_externalweb
node_crawler -->|"passes response"| node_propertybag
node_crawler -->|"runs pipeline"| node_html
node_html -->|"reads and enriches"| node_propertybag
node_html -->|"queues links"| node_crawler
node_crawler -->|"queues crawl steps"| node_crawlstep
node_memoryservices -.->|"implements"| node_queue
node_fileadapter -->|"registers services"| node_queue
node_fileadapter -->|"registers services"| node_history
node_efadapter -->|"registers services"| node_queue
node_efadapter -->|"registers services"| node_history
node_esentadapter -.->|"registers services"| node_queue
node_esentadapter -.->|"registers services"| node_history
node_isolatedadapter -->|"registers services"| node_queue
node_isolatedadapter -->|"registers services"| node_history
node_redisadapter -.->|"provides backend"| node_queue
node_redisadapter -.->|"provides backend"| node_history
node_crawler -.->|"runs pipeline"| node_textprocessor
node_crawler -.->|"runs pipeline"| node_linkextract
node_crawler -.->|"runs pipeline"| node_pdf
node_crawler -.->|"runs pipeline"| node_ifilter
node_crawler -.->|"runs pipeline"| node_language
node_language -->|"detects language"| node_google
node_crawler -.->|"runs pipeline"| node_mp3
node_crawler -.->|"runs pipeline"| node_sitemap

click node_crawler "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Crawler.cs"
click node_downloader "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/WebDownloaderV2.cs"
click node_rules "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/CrawlerRulesService.cs"
click node_robots "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/RobotService.cs"
click node_crawlstep "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/CrawlStep.cs"
click node_propertybag "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/PropertyBag.cs"
click node_queue "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Interfaces/ICrawlerQueue.cs"
click node_history "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Interfaces/ICrawlerHistory.cs"
click node_memoryservices "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/InMemoryCrawlerQueueService.cs"
click node_fileadapter "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.FileStorageServices/FileStorageModule.cs"
click node_efadapter "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.EntityFramework/EfServicesModule.cs"
click node_esentadapter "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.EsentServices/EsentServicesModule.cs"
click node_isolatedadapter "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs"
click node_redisadapter "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.RedisServices/RedisServicesModule.cs"
click node_html "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs"
click node_textprocessor "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.HtmlProcessor/TextDocumentProcessor.cs"
click node_linkextract "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.HtmlProcessor/LinkExtractionProcessor.cs"
click node_pdf "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.iTextSharpPdfProcessor/iTextSharpPdfProcessor.cs"
click node_ifilter "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.IFilterProcessor/IFilterProcessor.cs"
click node_language "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.LanguageDetection.Google/GoogleLanguageDetection.cs"
click node_mp3 "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.MP3Processor/MP3FileProcessor.cs"
click node_sitemap "https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.SitemapProcessor/SitemapProcessor.cs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawler,node_downloader,node_rules,node_robots,node_crawlstep,node_propertybag toneBlue
class node_html,node_textprocessor,node_linkextract toneAmber
class node_queue,node_history,node_memoryservices,node_fileadapter,node_efadapter,node_esentadapter,node_isolatedadapter,node_redisadapter,node_externalweb toneMint
class node_pdf,node_ifilter,node_language,node_mp3,node_sitemap toneRose
class node_caller,node_google toneIndigo
```

## Components

### Crawl engine

- [Crawler orchestration](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Crawler.cs): `src/NCrawler/Crawler.cs`
- [Web downloader](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/WebDownloaderV2.cs): `src/NCrawler/Services/WebDownloaderV2.cs`
- [Crawl rules](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/CrawlerRulesService.cs): `src/NCrawler/Services/CrawlerRulesService.cs`
- [Robots policy](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/RobotService.cs): `src/NCrawler/Services/RobotService.cs`
- [Crawl step](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/CrawlStep.cs): `src/NCrawler/CrawlStep.cs`
- [Response properties](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/PropertyBag.cs): `src/NCrawler/PropertyBag.cs`

### Content processing

- [HTML processing](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs): `src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs`
- [Text processing](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.HtmlProcessor/TextDocumentProcessor.cs): `src/NCrawler.HtmlProcessor/TextDocumentProcessor.cs`
- [Link extraction](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.HtmlProcessor/LinkExtractionProcessor.cs): `src/NCrawler.HtmlProcessor/LinkExtractionProcessor.cs`

### Queue and history

- [Queue contract](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Interfaces/ICrawlerQueue.cs): `src/NCrawler/Interfaces/ICrawlerQueue.cs`
- [History contract](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Interfaces/ICrawlerHistory.cs): `src/NCrawler/Interfaces/ICrawlerHistory.cs`
- [In-memory services](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler/Services/InMemoryCrawlerQueueService.cs): `src/NCrawler/Services/InMemoryCrawlerQueueService.cs`
- [File storage](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.FileStorageServices/FileStorageModule.cs): `src/NCrawler.FileStorageServices/FileStorageModule.cs`
- [Entity Framework storage](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.EntityFramework/EfServicesModule.cs): `src/NCrawler.EntityFramework/EfServicesModule.cs`
- [ESENT storage](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.EsentServices/EsentServicesModule.cs): `src/NCrawler.EsentServices/EsentServicesModule.cs`
- [Isolated storage](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs): `src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs`
- [Redis storage](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.RedisServices/RedisServicesModule.cs): `src/NCrawler.RedisServices/RedisServicesModule.cs`

### Processing extensions

- [PDF processing](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.iTextSharpPdfProcessor/iTextSharpPdfProcessor.cs): `src/NCrawler.iTextSharpPdfProcessor/iTextSharpPdfProcessor.cs`
- [IFilter processing](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.IFilterProcessor/IFilterProcessor.cs): `src/NCrawler.IFilterProcessor/IFilterProcessor.cs`
- [Language detection](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.LanguageDetection.Google/GoogleLanguageDetection.cs): `src/NCrawler.LanguageDetection.Google/GoogleLanguageDetection.cs`
- [MP3 processing](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.MP3Processor/MP3FileProcessor.cs): `src/NCrawler.MP3Processor/MP3FileProcessor.cs`
- [Sitemap processing](https://github.com/kant2002/ncrawler/blob/master/src/NCrawler.SitemapProcessor/SitemapProcessor.cs): `src/NCrawler.SitemapProcessor/SitemapProcessor.cs`

### Other components

- **Crawl caller**
- **Web servers**
- **Google detection service**

## Connections

- Crawl caller → Crawler orchestration: starts crawl
- Crawler orchestration → Crawl rules: checks URLs (evidence: `src/NCrawler/Crawler.cs`)
- Crawl rules → Robots policy: checks policy (evidence: `src/NCrawler/Services/CrawlerRulesService.cs`)
- Crawler orchestration → History contract: registers URLs (evidence: `src/NCrawler/Crawler.cs`)
- Crawler orchestration → Queue contract: schedules steps (evidence: `src/NCrawler/Crawler.cs`)
- Crawler orchestration → Web downloader: downloads URLs (evidence: `src/NCrawler/Crawler.cs`)
- Web downloader → Web servers: requests pages
- Crawler orchestration → Response properties: passes response (evidence: `src/NCrawler/Crawler.cs`)
- Crawler orchestration → HTML processing: runs pipeline (evidence: `src/NCrawler/Crawler.cs`)
- HTML processing → Response properties: reads and enriches (evidence: `src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs`)
- HTML processing → Crawler orchestration: queues links (evidence: `src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs`)
- Crawler orchestration → Crawl step: queues crawl steps (evidence: `src/NCrawler/Crawler.cs`)
- In-memory services → Queue contract: implements
- File storage → Queue contract: registers services (evidence: `src/NCrawler.FileStorageServices/FileStorageModule.cs`)
- File storage → History contract: registers services (evidence: `src/NCrawler.FileStorageServices/FileStorageModule.cs`)
- Entity Framework storage → Queue contract: registers services (evidence: `src/NCrawler.EntityFramework/EfServicesModule.cs`)
- Entity Framework storage → History contract: registers services (evidence: `src/NCrawler.EntityFramework/EfServicesModule.cs`)
- ESENT storage → Queue contract: registers services
- ESENT storage → History contract: registers services
- Isolated storage → Queue contract: registers services (evidence: `src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs`)
- Isolated storage → History contract: registers services (evidence: `src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs`)
- Redis storage → Queue contract: provides backend
- Redis storage → History contract: provides backend
- Crawler orchestration → Text processing: runs pipeline
- Crawler orchestration → Link extraction: runs pipeline
- Crawler orchestration → PDF processing: runs pipeline
- Crawler orchestration → IFilter processing: runs pipeline
- Crawler orchestration → Language detection: runs pipeline
- Language detection → Google detection service: detects language (evidence: `src/NCrawler.LanguageDetection.Google/GoogleLanguageDetection.cs`)
- Crawler orchestration → MP3 processing: runs pipeline
- Crawler orchestration → Sitemap processing: runs pipeline
