How rharish1/quarry works
Quarry exposes a REST search endpoint that validates requests, searches a fallback chain of providers, optionally crawls and quality-ranks pages, then cleans and optionally compresses the content before returning a structured response. The graph preserves documented cache, rate-limit, metrics, and extraction capabilities; several subsystems are represented at their exact enclosing paths because their source was not sampled.
Main entry points: pipeline.py Pipeline orchestrator · search.py Search route · searxng.py SearXNG search · app.py FastAPI app
Components 24
API and operations
- Search route
api/routes/search.py - FastAPI app
api/app.py - Rate limiting
api/middleware/rate_limit.py - Prometheus metrics
api/metrics.py
Search orchestration
- Pipeline orchestrator
pipeline/pipeline.py - Query normalization
pipeline/query/normalizer.py - Search response
models/search.py
Retrieval and crawling
- Tavily search
pipeline/retrieval/tavily.py - SearXNG search
pipeline/retrieval/searxng.py - Brave search
pipeline/retrieval/brave.py - DuckDuckGo search
pipeline/retrieval/duckduckgo.py - Page crawler
pipeline/crawler - Ranked crawling
pipeline/ranking/manager.py - Extraction waterfall
pipeline/crawler/manager.py
Content processing
- Document cleaning
pipeline/cleaning/cleaner.py - Markdown compression
pipeline/compression/compressor.py - Output formatting
pipeline/pipeline.py
State and resilience
- Response cache
pipeline/cache/cache.py - Redis
- Retry and breakers
pipeline/resilience - Shared HTTP client
pipeline/http/client.py
Other components
- HTTP client
- Search providers
- Prometheus
Connections 30
- HTTP client → Search route: POST /search
- Search route → Rate limiting: checks limit
- Search route → Pipeline orchestrator: executes search
- Search route → Prometheus metrics: records metrics
- Pipeline orchestrator → Query normalization: normalizes query
- Pipeline orchestrator → Response cache: checks and writes
- Response cache → Redis: reads and writes
- Pipeline orchestrator → Tavily search: searches first
- Pipeline orchestrator → SearXNG search: fallback search
- Pipeline orchestrator → Brave search: fallback search
- Pipeline orchestrator → DuckDuckGo search: final fallback
- Tavily search → Search providers: queries
- SearXNG search → Search providers: queries
- Brave search → Search providers: queries
- DuckDuckGo search → Search providers: queries
- SearXNG search → Shared HTTP client: uses shared client
- Pipeline orchestrator → Ranked crawling: ranks when enabled
- Pipeline orchestrator → Page crawler: crawls when enabled
- Ranked crawling → Page crawler: dispatches page crawls
- Page crawler → Extraction waterfall: extracts page content
- Pipeline orchestrator → Document cleaning: cleans documents
- Pipeline orchestrator → Markdown compression: compresses optionally
- Pipeline orchestrator → Output formatting: formats output
- Output formatting → Search response: builds response
- FastAPI app → Search route: registers route
- FastAPI app → Shared HTTP client: initializes and closes
- FastAPI app → Redis: initializes limiter
- Rate limiting → Redis: checks request quota
- Prometheus → Prometheus metrics: scrapes metrics
- SearXNG search → Retry and breakers: uses retry and breaker