# qvarcy/spriditis architecture

> Architecture diagram of the GitHub repository qvarcy/spriditis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qvarcy/spriditis
- Repository: https://github.com/qvarcy/spriditis

## Overview

Sprīdītis is a configurable market-research crawler that turns project criteria and seeds into source-backed market entities, then preserves run history and discovery memory for future research. The primary path is CLI/API initiation, search and web discovery, crawling and structured extraction, identity/target checks, persistence, and HTML reporting. Monitoring, change detection, feeds, and optional AI enrichment extend that path. The README documents the product stages; sampled source and the SOURCE INDEX support the shown module relationships, but unsampled implementation details are not inferred.

Main entry points: [`spriditis/crawler/engine.py`](https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/engine.py) (Crawler engine), [`spriditis/api/service.py`](https://github.com/qvarcy/spriditis/blob/main/spriditis/api/service.py) (Run service), [`spriditis/storage/database.py`](https://github.com/qvarcy/spriditis/blob/main/spriditis/storage/database.py) (Research database), [`spriditis/cli.py`](https://github.com/qvarcy/spriditis/blob/main/spriditis/cli.py) (CLI workflows)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Research orchestration"]
  node_cli["CLI workflows<br/>[cli.py]"]
  node_api["Run service<br/>[service.py]"]
  node_project["Project criteria<br/>[projects.py]"]
end

subgraph group_discovery["Search and crawling"]
  node_search_query["Query generation<br/>[query.py]"]
  node_search_provider["Search provider<br/>[searxng.py]"]
  node_discovery_engine["Domain discovery<br/>[discovery.py]"]
  node_feed_discovery["Feed discovery<br/>[feed_discovery.py]"]
  node_crawler["Crawler engine<br/>[engine.py]"]
  node_crawl_policy["Crawl policy<br/>[policy.py]"]
end

subgraph group_evidence["Evidence and analysis"]
  node_extraction["Extraction pipeline<br/>[engine.py]"]
  node_entity["Market entities<br/>[entities.py]"]
  node_target_identity["Target identity<br/>[target.py]"]
  node_resolution["Entity resolution<br/>[identity.py]"]
  node_ai["AI enrichment<br/>[base.py]"]
end

subgraph group_memory["Research memory"]
  node_database[("Research database<br/>[database.py]")]
  node_domain_memory["Domain registry<br/>[domains.py]"]
  node_feed_state["Feed state<br/>[feeds.py]"]
  node_change_detection["Change detection<br/>[changes.py]"]
  node_watch["Watch mode<br/>[watch.py]"]
end

subgraph group_outputs["Interfaces and results"]
  node_report["HTML report<br/>[html.py]"]
end

node_researcher(("Researcher"))
node_web(("Web sources"))

node_researcher -->|"starts research"| node_cli
node_cli -->|"runs service"| node_api
node_api -->|"uses criteria"| node_project
node_api -->|"loads and saves"| node_database
node_api -->|"starts crawl"| node_crawler
node_api -->|"generates report"| node_report
node_crawler -->|"builds queries"| node_search_query
node_search_query -->|"reads criteria"| node_project
node_crawler -->|"searches web"| node_search_provider
node_crawler -->|"discovers domains"| node_discovery_engine
node_crawler -->|"finds feeds"| node_feed_discovery
node_crawler -->|"applies policy"| node_crawl_policy
node_crawler -->|"fetches pages"| node_web
node_crawler -->|"extracts pages"| node_extraction
node_extraction -->|"creates entities"| node_entity
node_crawler -->|"checks target"| node_target_identity
node_target_identity -->|"uses identity signals"| node_resolution
node_crawler -->|"enriches entities"| node_ai
node_api -->|"persists results"| node_entity
node_database -->|"stores registry"| node_domain_memory
node_database -->|"stores feed state"| node_feed_state
node_database -->|"records history"| node_change_detection
node_watch -.->|"reuses run state"| node_database
node_report -->|"presents findings"| node_researcher
node_database -->|"stores identity clusters"| node_resolution

click node_cli "https://github.com/qvarcy/spriditis/blob/main/spriditis/cli.py"
click node_api "https://github.com/qvarcy/spriditis/blob/main/spriditis/api/service.py"
click node_project "https://github.com/qvarcy/spriditis/blob/main/spriditis/core/projects.py"
click node_search_query "https://github.com/qvarcy/spriditis/blob/main/spriditis/search/query.py"
click node_search_provider "https://github.com/qvarcy/spriditis/blob/main/spriditis/search/searxng.py"
click node_discovery_engine "https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/discovery.py"
click node_feed_discovery "https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/feed_discovery.py"
click node_crawler "https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/engine.py"
click node_crawl_policy "https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/policy.py"
click node_extraction "https://github.com/qvarcy/spriditis/blob/main/spriditis/extraction/engine.py"
click node_entity "https://github.com/qvarcy/spriditis/blob/main/spriditis/core/entities.py"
click node_target_identity "https://github.com/qvarcy/spriditis/blob/main/spriditis/resolution/target.py"
click node_resolution "https://github.com/qvarcy/spriditis/blob/main/spriditis/resolution/identity.py"
click node_ai "https://github.com/qvarcy/spriditis/blob/main/spriditis/ai/base.py"
click node_database "https://github.com/qvarcy/spriditis/blob/main/spriditis/storage/database.py"
click node_domain_memory "https://github.com/qvarcy/spriditis/blob/main/spriditis/core/domains.py"
click node_feed_state "https://github.com/qvarcy/spriditis/blob/main/spriditis/core/feeds.py"
click node_change_detection "https://github.com/qvarcy/spriditis/blob/main/spriditis/core/changes.py"
click node_watch "https://github.com/qvarcy/spriditis/blob/main/spriditis/core/watch.py"
click node_report "https://github.com/qvarcy/spriditis/blob/main/spriditis/reports/html.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_api,node_project toneBlue
class node_search_query,node_search_provider,node_discovery_engine,node_feed_discovery,node_crawler,node_crawl_policy toneAmber
class node_extraction,node_entity,node_target_identity,node_resolution,node_ai toneMint
class node_database,node_domain_memory,node_feed_state,node_change_detection,node_watch toneRose
class node_report,node_researcher,node_web toneIndigo
```

## Components

### Research orchestration

- [CLI workflows](https://github.com/qvarcy/spriditis/blob/main/spriditis/cli.py): `spriditis/cli.py`
- [Run service](https://github.com/qvarcy/spriditis/blob/main/spriditis/api/service.py): `spriditis/api/service.py`
- [Project criteria](https://github.com/qvarcy/spriditis/blob/main/spriditis/core/projects.py): `spriditis/core/projects.py`

### Search and crawling

- [Query generation](https://github.com/qvarcy/spriditis/blob/main/spriditis/search/query.py): `spriditis/search/query.py`
- [Search provider](https://github.com/qvarcy/spriditis/blob/main/spriditis/search/searxng.py): `spriditis/search/searxng.py`
- [Domain discovery](https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/discovery.py): `spriditis/crawler/discovery.py`
- [Feed discovery](https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/feed_discovery.py): `spriditis/crawler/feed_discovery.py`
- [Crawler engine](https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/engine.py): `spriditis/crawler/engine.py`
- [Crawl policy](https://github.com/qvarcy/spriditis/blob/main/spriditis/crawler/policy.py): `spriditis/crawler/policy.py`

### Evidence and analysis

- [Extraction pipeline](https://github.com/qvarcy/spriditis/blob/main/spriditis/extraction/engine.py): `spriditis/extraction/engine.py`
- [Market entities](https://github.com/qvarcy/spriditis/blob/main/spriditis/core/entities.py): `spriditis/core/entities.py`
- [Target identity](https://github.com/qvarcy/spriditis/blob/main/spriditis/resolution/target.py): `spriditis/resolution/target.py`
- [Entity resolution](https://github.com/qvarcy/spriditis/blob/main/spriditis/resolution/identity.py): `spriditis/resolution/identity.py`
- [AI enrichment](https://github.com/qvarcy/spriditis/blob/main/spriditis/ai/base.py): `spriditis/ai/base.py`

### Research memory

- [Research database](https://github.com/qvarcy/spriditis/blob/main/spriditis/storage/database.py): `spriditis/storage/database.py`
- [Domain registry](https://github.com/qvarcy/spriditis/blob/main/spriditis/core/domains.py): `spriditis/core/domains.py`
- [Feed state](https://github.com/qvarcy/spriditis/blob/main/spriditis/core/feeds.py): `spriditis/core/feeds.py`
- [Change detection](https://github.com/qvarcy/spriditis/blob/main/spriditis/core/changes.py): `spriditis/core/changes.py`
- [Watch mode](https://github.com/qvarcy/spriditis/blob/main/spriditis/core/watch.py): `spriditis/core/watch.py`

### Interfaces and results

- [HTML report](https://github.com/qvarcy/spriditis/blob/main/spriditis/reports/html.py): `spriditis/reports/html.py`

### Other components

- **Researcher**
- **Web sources**

## Connections

- Researcher → CLI workflows: starts research
- CLI workflows → Run service: runs service (evidence: `spriditis/cli.py`)
- Run service → Project criteria: uses criteria (evidence: `spriditis/api/service.py`)
- Run service → Research database: loads and saves (evidence: `spriditis/api/service.py`)
- Run service → Crawler engine: starts crawl (evidence: `spriditis/api/service.py`)
- Run service → HTML report: generates report (evidence: `spriditis/api/service.py`)
- Crawler engine → Query generation: builds queries (evidence: `spriditis/crawler/engine.py`)
- Query generation → Project criteria: reads criteria (evidence: `spriditis/search/query.py`)
- Crawler engine → Search provider: searches web
- Crawler engine → Domain discovery: discovers domains (evidence: `spriditis/crawler/engine.py`)
- Crawler engine → Feed discovery: finds feeds (evidence: `spriditis/crawler/engine.py`)
- Crawler engine → Crawl policy: applies policy (evidence: `spriditis/crawler/engine.py`)
- Crawler engine → Web sources: fetches pages
- Crawler engine → Extraction pipeline: extracts pages (evidence: `spriditis/crawler/engine.py`)
- Extraction pipeline → Market entities: creates entities (evidence: `spriditis/extraction/engine.py`)
- Crawler engine → Target identity: checks target (evidence: `spriditis/crawler/engine.py`)
- Target identity → Entity resolution: uses identity signals (evidence: `spriditis/resolution/target.py`)
- Crawler engine → AI enrichment: enriches entities (evidence: `spriditis/crawler/engine.py`)
- Run service → Market entities: persists results (evidence: `spriditis/api/service.py`)
- Research database → Domain registry: stores registry (evidence: `spriditis/storage/database.py`)
- Research database → Feed state: stores feed state (evidence: `spriditis/storage/database.py`)
- Research database → Change detection: records history (evidence: `spriditis/storage/database.py`)
- Watch mode → Research database: reuses run state
- HTML report → Researcher: presents findings
- Research database → Entity resolution: stores identity clusters (evidence: `spriditis/storage/database.py`)
