# rodrask/doc_searcher architecture

> Architecture diagram of the GitHub repository rodrask/doc_searcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rodrask/doc_searcher
- Repository: https://github.com/rodrask/doc_searcher

## Overview

##### Main Components Identification

a) Crawler Component
- Located in doc_searcher/crawler/
- Handles web crawling functionality
- Uses Scrapy framework (evident from scrapy.cfg)
- Produces compressed crawl files (domain.crawl.gz)

b) Indexer Component
- Located in doc_searcher/indexer/
- Processes crawled data
- Uses Tantivy for indexing (mentioned in README)
- Generates index-stats.json

c) Web Service Component
- Located in doc_searcher/web_service/
- Provides search interface
- Includes template-based UI
- Handles query parsing and search functions

**[Web Pages/Domains] → [Crawler] → [Crawl Files] → [Indexer] → [Search Index] → [Web Service] → [User Interface]**

##### Detailed Component Breakdown

For each major component, create a sub-section showing:

Crawler Component:
- Input: domains.txt file
- Process: domain_spider.py functionality
- Output: domain.crawl.gz files
- Parallel processing capability

Indexer Component:
- Input: crawl.gz files
- Processing: index_functions.py
- Output: Tantivy index and index-stats.json
- Batch processing capability (10000 ops)

Web Service Component:
- Query parsing (query_parser.py)
- Search functions (search_functions.py)
- Statistics tracking (stat.py)
- Template rendering system

##### Technical Details

Technical specifications:
- Python 3.11 runtime
- Parallel processing support
- Port configuration for web service
- Memory limits (15000 pages per domain)
- Batch size (10000 operations)

##### Additional Elements

Supporting elements:
- Configuration files location
- Script execution flows
- Error handling paths
- Scaling capabilities

##### Deployment Flow

A separate section showing:
- Build process with UV
- Virtual environment setup
- Package installation flow
- Service startup sequence

Remember to:
- Keep component relationships clear
- Both synchronous and asynchronous operations
- Scaling points
- Configuration parameters
- Monitoring/statistics collection points

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        DomainList["Domain List"]:::input
        click DomainList "https://github.com/rodrask/doc_searcher/blob/master/input/domains.txt"
        TestDomains["Test Domains"]:::input
        click TestDomains "https://github.com/rodrask/doc_searcher/blob/master/input/test_domains.txt"
    end

    subgraph "Crawler Layer"
        CrawlerMain["Crawler Main"]:::crawler
        click CrawlerMain "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/crawl.py"
        DomainSpider["Domain Spider"]:::crawler
        click DomainSpider "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/crawler/domain_spider.py"
        URLUtils["URL Utilities"]:::crawler
        click URLUtils "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/crawler/url_utils.py"
        CrawlerSettings["Crawler Settings"]:::crawler
        click CrawlerSettings "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/crawler/settings.py"
        ScrapyConfig["Scrapy Config"]:::crawler
        click ScrapyConfig "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/crawler/scrapy.cfg"
    end

    subgraph "Storage Layer"
        CrawlFiles[("Crawl Files\n(.crawl.gz)")]:::storage
        TantivyIndex[("Tantivy Index")]:::storage
        IndexStats["Index Stats JSON"]:::storage
    end

    subgraph "Indexer Layer"
        IndexerMain["Indexer Main"]:::indexer
        click IndexerMain "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/index.py"
        IndexFunctions["Index Functions"]:::indexer
        click IndexFunctions "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/indexer/index_functions.py"
    end

    subgraph "Web Service Layer"
        WebMain["Web Service Main"]:::webservice
        click WebMain "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web.py"
        QueryParser["Query Parser"]:::webservice
        click QueryParser "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web_service/query_parser.py"
        SearchFunctions["Search Functions"]:::webservice
        click SearchFunctions "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web_service/search_functions.py"
        Statistics["Statistics Tracker"]:::webservice
        click Statistics "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web_service/stat.py"
        QueryDicts["Query Dictionaries"]:::webservice
        click QueryDicts "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web_service/query_dicts.py"
    end

    subgraph "UI Templates"
        MainTemplate["Main Template"]:::template
        click MainTemplate "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web_service/templates/index.html"
        SearchTemplate["Search Results"]:::template
        click SearchTemplate "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web_service/templates/search.html"
        StatTemplate["Statistics Page"]:::template
        click StatTemplate "https://github.com/rodrask/doc_searcher/blob/master/doc_searcher/web_service/templates/stat.html"
    end

    subgraph "Execution Scripts"
        ParallelCrawl["Parallel Crawl"]:::script
        click ParallelCrawl "https://github.com/rodrask/doc_searcher/blob/master/scripts/parallel_crawl.sh"
        SingleCrawl["Single Crawl"]:::script
        click SingleCrawl "https://github.com/rodrask/doc_searcher/blob/master/scripts/single_crawl.sh"
        IndexDir["Index Directory"]:::script
        click IndexDir "https://github.com/rodrask/doc_searcher/blob/master/scripts/index_dir.sh"
        IndexList["Index List"]:::script
        click IndexList "https://github.com/rodrask/doc_searcher/blob/master/scripts/index_list.sh"
        WebScript["Web Service"]:::script
        click WebScript "https://github.com/rodrask/doc_searcher/blob/master/scripts/web.sh"
    end

    %% Relationships
    DomainList --> CrawlerMain
    TestDomains --> CrawlerMain
    CrawlerMain --> DomainSpider
    DomainSpider --> URLUtils
    DomainSpider --> CrawlerSettings
    DomainSpider --> ScrapyConfig
    DomainSpider --> CrawlFiles
    CrawlFiles --> IndexerMain
    IndexerMain --> IndexFunctions
    IndexFunctions --> TantivyIndex
    IndexFunctions --> IndexStats
    TantivyIndex --> WebMain
    WebMain --> QueryParser
    WebMain --> SearchFunctions
    WebMain --> Statistics
    WebMain --> QueryDicts
    Statistics --> StatTemplate
    WebMain --> MainTemplate
    WebMain --> SearchTemplate
    ParallelCrawl --> CrawlerMain
    SingleCrawl --> CrawlerMain
    IndexDir --> IndexerMain
    IndexList --> IndexerMain
    WebScript --> WebMain

    %% Styles
    classDef input fill:#b3e0ff,stroke:#333,stroke-width:2px
    classDef crawler fill:#90EE90,stroke:#333,stroke-width:2px
    classDef storage fill:#FFE5B4,stroke:#333,stroke-width:2px
    classDef indexer fill:#98FB98,stroke:#333,stroke-width:2px
    classDef webservice fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef template fill:#E6E6FA,stroke:#333,stroke-width:2px
    classDef script fill:#F0E68C,stroke:#333,stroke-width:2px
```
