# rharish1/quarry architecture

> Architecture diagram of the GitHub repository rharish1/quarry, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rharish1/quarry
- Repository: https://github.com/rharish1/quarry

## Overview

Quarry exposes a REST search endpoint that validates requests, searches a fallback chain of providers, optionally crawls and quality-ranks pages, then cleans and optionally compresses the content before returning a structured response. The graph preserves documented cache, rate-limit, metrics, and extraction capabilities; several subsystems are represented at their exact enclosing paths because their source was not sampled.

Main entry points: [`pipeline/pipeline.py`](https://github.com/rharish1/quarry/blob/main/pipeline/pipeline.py) (Pipeline orchestrator), [`api/routes/search.py`](https://github.com/rharish1/quarry/blob/main/api/routes/search.py) (Search route), [`pipeline/retrieval/searxng.py`](https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/searxng.py) (SearXNG search), [`api/app.py`](https://github.com/rharish1/quarry/blob/main/api/app.py) (FastAPI app)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and operations"]
  node_search_route["Search route<br/>[search.py]"]
  node_app["FastAPI app<br/>[app.py]"]
  node_rate_limit["Rate limiting<br/>[rate_limit.py]"]
  node_metrics["Prometheus metrics<br/>[metrics.py]"]
end

subgraph group_orchestration["Search orchestration"]
  node_pipeline["Pipeline orchestrator<br/>[pipeline.py]"]
  node_query_norm["Query normalization<br/>[normalizer.py]"]
  node_response["Search response<br/>[search.py]"]
end

subgraph group_retrieval["Retrieval and crawling"]
  node_tavily["Tavily search<br/>[tavily.py]"]
  node_searxng["SearXNG search<br/>[searxng.py]"]
  node_brave["Brave search<br/>[brave.py]"]
  node_ddg["DuckDuckGo search<br/>[duckduckgo.py]"]
  node_crawl["Page crawler"]
  node_rank["Ranked crawling<br/>[manager.py]"]
  node_extractors["Extraction waterfall<br/>[manager.py]"]
end

subgraph group_content["Content processing"]
  node_cleaning["Document cleaning<br/>[cleaner.py]"]
  node_compression["Markdown compression<br/>[compressor.py]"]
  node_formatter["Output formatting<br/>[pipeline.py]"]
end

subgraph group_state["State and resilience"]
  node_cache["Response cache<br/>[cache.py]"]
  node_redis[("Redis")]
  node_resilience["Retry and breakers"]
  node_http_client["Shared HTTP client<br/>[client.py]"]
end

node_client(("HTTP client"))
node_search_services["Search providers"]
node_prometheus["Prometheus"]

node_client -->|"POST /search"| node_search_route
node_search_route -->|"checks limit"| node_rate_limit
node_search_route -->|"executes search"| node_pipeline
node_search_route -->|"records metrics"| node_metrics
node_pipeline -.->|"normalizes query"| node_query_norm
node_pipeline -->|"checks and writes"| node_cache
node_cache -->|"reads and writes"| node_redis
node_pipeline -->|"searches first"| node_tavily
node_pipeline -->|"fallback search"| node_searxng
node_pipeline -->|"fallback search"| node_brave
node_pipeline -->|"final fallback"| node_ddg
node_tavily -->|"queries"| node_search_services
node_searxng -->|"queries"| node_search_services
node_brave -->|"queries"| node_search_services
node_ddg -->|"queries"| node_search_services
node_searxng -->|"uses shared client"| node_http_client
node_pipeline -.->|"ranks when enabled"| node_rank
node_pipeline -.->|"crawls when enabled"| node_crawl
node_rank -->|"dispatches page crawls"| node_crawl
node_crawl -->|"extracts page content"| node_extractors
node_pipeline -->|"cleans documents"| node_cleaning
node_pipeline -.->|"compresses optionally"| node_compression
node_pipeline -->|"formats output"| node_formatter
node_formatter -->|"builds response"| node_response
node_app -->|"registers route"| node_search_route
node_app -->|"initializes and closes"| node_http_client
node_app -->|"initializes limiter"| node_redis
node_rate_limit -->|"checks request quota"| node_redis
node_prometheus -->|"scrapes metrics"| node_metrics
node_searxng -->|"uses retry and breaker"| node_resilience

click node_search_route "https://github.com/rharish1/quarry/blob/main/api/routes/search.py"
click node_app "https://github.com/rharish1/quarry/blob/main/api/app.py"
click node_rate_limit "https://github.com/rharish1/quarry/blob/main/api/middleware/rate_limit.py"
click node_metrics "https://github.com/rharish1/quarry/blob/main/api/metrics.py"
click node_pipeline "https://github.com/rharish1/quarry/blob/main/pipeline/pipeline.py"
click node_query_norm "https://github.com/rharish1/quarry/blob/main/pipeline/query/normalizer.py"
click node_response "https://github.com/rharish1/quarry/blob/main/models/search.py"
click node_cache "https://github.com/rharish1/quarry/blob/main/pipeline/cache/cache.py"
click node_resilience "https://github.com/rharish1/quarry/tree/main/pipeline/resilience"
click node_tavily "https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/tavily.py"
click node_searxng "https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/searxng.py"
click node_brave "https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/brave.py"
click node_ddg "https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/duckduckgo.py"
click node_crawl "https://github.com/rharish1/quarry/tree/main/pipeline/crawler"
click node_rank "https://github.com/rharish1/quarry/blob/main/pipeline/ranking/manager.py"
click node_extractors "https://github.com/rharish1/quarry/blob/main/pipeline/crawler/manager.py"
click node_cleaning "https://github.com/rharish1/quarry/blob/main/pipeline/cleaning/cleaner.py"
click node_compression "https://github.com/rharish1/quarry/blob/main/pipeline/compression/compressor.py"
click node_formatter "https://github.com/rharish1/quarry/blob/main/pipeline/pipeline.py"
click node_http_client "https://github.com/rharish1/quarry/blob/main/pipeline/http/client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_search_route,node_app,node_rate_limit,node_metrics,node_client toneBlue
class node_pipeline,node_query_norm,node_response toneAmber
class node_tavily,node_searxng,node_brave,node_ddg,node_crawl,node_rank,node_extractors toneMint
class node_cleaning,node_compression,node_formatter toneRose
class node_cache,node_redis,node_resilience,node_http_client,node_search_services,node_prometheus toneIndigo
```

## Components

### API and operations

- [Search route](https://github.com/rharish1/quarry/blob/main/api/routes/search.py): `api/routes/search.py`
- [FastAPI app](https://github.com/rharish1/quarry/blob/main/api/app.py): `api/app.py`
- [Rate limiting](https://github.com/rharish1/quarry/blob/main/api/middleware/rate_limit.py): `api/middleware/rate_limit.py`
- [Prometheus metrics](https://github.com/rharish1/quarry/blob/main/api/metrics.py): `api/metrics.py`

### Search orchestration

- [Pipeline orchestrator](https://github.com/rharish1/quarry/blob/main/pipeline/pipeline.py): `pipeline/pipeline.py`
- [Query normalization](https://github.com/rharish1/quarry/blob/main/pipeline/query/normalizer.py): `pipeline/query/normalizer.py`
- [Search response](https://github.com/rharish1/quarry/blob/main/models/search.py): `models/search.py`

### Retrieval and crawling

- [Tavily search](https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/tavily.py): `pipeline/retrieval/tavily.py`
- [SearXNG search](https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/searxng.py): `pipeline/retrieval/searxng.py`
- [Brave search](https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/brave.py): `pipeline/retrieval/brave.py`
- [DuckDuckGo search](https://github.com/rharish1/quarry/blob/main/pipeline/retrieval/duckduckgo.py): `pipeline/retrieval/duckduckgo.py`
- [Page crawler](https://github.com/rharish1/quarry/tree/main/pipeline/crawler): `pipeline/crawler`
- [Ranked crawling](https://github.com/rharish1/quarry/blob/main/pipeline/ranking/manager.py): `pipeline/ranking/manager.py`
- [Extraction waterfall](https://github.com/rharish1/quarry/blob/main/pipeline/crawler/manager.py): `pipeline/crawler/manager.py`

### Content processing

- [Document cleaning](https://github.com/rharish1/quarry/blob/main/pipeline/cleaning/cleaner.py): `pipeline/cleaning/cleaner.py`
- [Markdown compression](https://github.com/rharish1/quarry/blob/main/pipeline/compression/compressor.py): `pipeline/compression/compressor.py`
- [Output formatting](https://github.com/rharish1/quarry/blob/main/pipeline/pipeline.py): `pipeline/pipeline.py`

### State and resilience

- [Response cache](https://github.com/rharish1/quarry/blob/main/pipeline/cache/cache.py): `pipeline/cache/cache.py`
- **Redis**
- [Retry and breakers](https://github.com/rharish1/quarry/tree/main/pipeline/resilience): `pipeline/resilience`
- [Shared HTTP client](https://github.com/rharish1/quarry/blob/main/pipeline/http/client.py): `pipeline/http/client.py`

### Other components

- **HTTP client**
- **Search providers**
- **Prometheus**

## Connections

- HTTP client → Search route: POST /search
- Search route → Rate limiting: checks limit
- Search route → Pipeline orchestrator: executes search
- Search route → Prometheus metrics: records metrics
- Pipeline orchestrator → Query normalization: normalizes query
- Pipeline orchestrator → Response cache: checks and writes
- Response cache → Redis: reads and writes
- Pipeline orchestrator → Tavily search: searches first
- Pipeline orchestrator → SearXNG search: fallback search
- Pipeline orchestrator → Brave search: fallback search
- Pipeline orchestrator → DuckDuckGo search: final fallback
- Tavily search → Search providers: queries
- SearXNG search → Search providers: queries
- Brave search → Search providers: queries
- DuckDuckGo search → Search providers: queries
- SearXNG search → Shared HTTP client: uses shared client
- Pipeline orchestrator → Ranked crawling: ranks when enabled
- Pipeline orchestrator → Page crawler: crawls when enabled
- Ranked crawling → Page crawler: dispatches page crawls
- Page crawler → Extraction waterfall: extracts page content
- Pipeline orchestrator → Document cleaning: cleans documents
- Pipeline orchestrator → Markdown compression: compresses optionally
- Pipeline orchestrator → Output formatting: formats output
- Output formatting → Search response: builds response
- FastAPI app → Search route: registers route
- FastAPI app → Shared HTTP client: initializes and closes
- FastAPI app → Redis: initializes limiter
- Rate limiting → Redis: checks request quota
- Prometheus → Prometheus metrics: scrapes metrics
- SearXNG search → Retry and breakers: uses retry and breaker