How kant2002/ncrawler works
NCrawler is a multithreaded, pipeline-based web crawler. A caller starts a crawl at a URI; the crawler checks rules and history, schedules downloads, and passes downloaded content through pipeline steps. The HTML processor extracts page data and queues discovered links for further crawling. Documented extensions include text, PDF, IFilter, and language processing, plus interchangeable storage backends. Several of those implementations are unsampled, so their internal links are not asserted; optional integrations are shown as unverified.
Main entry points: Crawler.cs Crawler orchestration · ICrawlerQueue.cs Queue contract · ICrawlerHistory.cs History contract · HtmlDocumentProcessor.cs HTML processing
Components 25
Crawl engine
- Crawler orchestration
src/NCrawler/Crawler.cs - Web downloader
src/NCrawler/Services/WebDownloaderV2.cs - Crawl rules
src/NCrawler/Services/CrawlerRulesService.cs - Robots policy
src/NCrawler/Services/RobotService.cs - Crawl step
src/NCrawler/CrawlStep.cs - Response properties
src/NCrawler/PropertyBag.cs
Content processing
- HTML processing
src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs - Text processing
src/NCrawler.HtmlProcessor/TextDocumentProcessor.cs - Link extraction
src/NCrawler.HtmlProcessor/LinkExtractionProcessor.cs
Queue and history
- Queue contract
src/NCrawler/Interfaces/ICrawlerQueue.cs - History contract
src/NCrawler/Interfaces/ICrawlerHistory.cs - In-memory services
src/NCrawler/Services/InMemoryCrawlerQueueService.cs - File storage
src/NCrawler.FileStorageServices/FileStorageModule.cs - Entity Framework storage
src/NCrawler.EntityFramework/EfServicesModule.cs - ESENT storage
src/NCrawler.EsentServices/EsentServicesModule.cs - Isolated storage
src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs - Redis storage
src/NCrawler.RedisServices/RedisServicesModule.cs
Processing extensions
- PDF processing
src/NCrawler.iTextSharpPdfProcessor/iTextSharpPdfProcessor.cs - IFilter processing
src/NCrawler.IFilterProcessor/IFilterProcessor.cs - Language detection
src/NCrawler.LanguageDetection.Google/GoogleLanguageDetection.cs - MP3 processing
src/NCrawler.MP3Processor/MP3FileProcessor.cs - Sitemap processing
src/NCrawler.SitemapProcessor/SitemapProcessor.cs
Other components
- Crawl caller
- Web servers
- Google detection service
Connections 31
- Crawl caller → Crawler orchestration: starts crawl
- Crawler orchestration → Crawl rules: checks URLs
src/NCrawler/Crawler.cs - Crawl rules → Robots policy: checks policy
src/NCrawler/Services/CrawlerRulesService.cs - Crawler orchestration → History contract: registers URLs
src/NCrawler/Crawler.cs - Crawler orchestration → Queue contract: schedules steps
src/NCrawler/Crawler.cs - Crawler orchestration → Web downloader: downloads URLs
src/NCrawler/Crawler.cs - Web downloader → Web servers: requests pages
- Crawler orchestration → Response properties: passes response
src/NCrawler/Crawler.cs - Crawler orchestration → HTML processing: runs pipeline
src/NCrawler/Crawler.cs - HTML processing → Response properties: reads and enriches
src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs - HTML processing → Crawler orchestration: queues links
src/NCrawler.HtmlProcessor/HtmlDocumentProcessor.cs - Crawler orchestration → Crawl step: queues crawl steps
src/NCrawler/Crawler.cs - In-memory services → Queue contract: implements
- File storage → Queue contract: registers services
src/NCrawler.FileStorageServices/FileStorageModule.cs - File storage → History contract: registers services
src/NCrawler.FileStorageServices/FileStorageModule.cs - Entity Framework storage → Queue contract: registers services
src/NCrawler.EntityFramework/EfServicesModule.cs - Entity Framework storage → History contract: registers services
src/NCrawler.EntityFramework/EfServicesModule.cs - ESENT storage → Queue contract: registers services
- ESENT storage → History contract: registers services
- Isolated storage → Queue contract: registers services
src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs - Isolated storage → History contract: registers services
src/NCrawler.IsolatedStorageServices/IsolatedStorageModule.cs - Redis storage → Queue contract: provides backend
- Redis storage → History contract: provides backend
- Crawler orchestration → Text processing: runs pipeline
- Crawler orchestration → Link extraction: runs pipeline
- Crawler orchestration → PDF processing: runs pipeline
- Crawler orchestration → IFilter processing: runs pipeline
- Crawler orchestration → Language detection: runs pipeline
- Language detection → Google detection service: detects language
src/NCrawler.LanguageDetection.Google/GoogleLanguageDetection.cs - Crawler orchestration → MP3 processing: runs pipeline
- Crawler orchestration → Sitemap processing: runs pipeline