How qvarcy/spriditis works
Sprīdītis is a configurable market-research crawler that turns project criteria and seeds into source-backed market entities, then preserves run history and discovery memory for future research. The primary path is CLI/API initiation, search and web discovery, crawling and structured extraction, identity/target checks, persistence, and HTML reporting. Monitoring, change detection, feeds, and optional AI enrichment extend that path. The README documents the product stages; sampled source and the SOURCE INDEX support the shown module relationships, but unsampled implementation details are not inferred.
Main entry points: engine.py Crawler engine · service.py Run service · database.py Research database · cli.py CLI workflows
Components 22
Research orchestration
- CLI workflows
spriditis/cli.py - Run service
spriditis/api/service.py - Project criteria
spriditis/core/projects.py
Search and crawling
- Query generation
spriditis/search/query.py - Search provider
spriditis/search/searxng.py - Domain discovery
spriditis/crawler/discovery.py - Feed discovery
spriditis/crawler/feed_discovery.py - Crawler engine
spriditis/crawler/engine.py - Crawl policy
spriditis/crawler/policy.py
Evidence and analysis
- Extraction pipeline
spriditis/extraction/engine.py - Market entities
spriditis/core/entities.py - Target identity
spriditis/resolution/target.py - Entity resolution
spriditis/resolution/identity.py - AI enrichment
spriditis/ai/base.py
Research memory
- Research database
spriditis/storage/database.py - Domain registry
spriditis/core/domains.py - Feed state
spriditis/core/feeds.py - Change detection
spriditis/core/changes.py - Watch mode
spriditis/core/watch.py
Interfaces and results
- HTML report
spriditis/reports/html.py
Other components
- Researcher
- Web sources
Connections 25
- Researcher → CLI workflows: starts research
- CLI workflows → Run service: runs service
spriditis/cli.py - Run service → Project criteria: uses criteria
spriditis/api/service.py - Run service → Research database: loads and saves
spriditis/api/service.py - Run service → Crawler engine: starts crawl
spriditis/api/service.py - Run service → HTML report: generates report
spriditis/api/service.py - Crawler engine → Query generation: builds queries
spriditis/crawler/engine.py - Query generation → Project criteria: reads criteria
spriditis/search/query.py - Crawler engine → Search provider: searches web
- Crawler engine → Domain discovery: discovers domains
spriditis/crawler/engine.py - Crawler engine → Feed discovery: finds feeds
spriditis/crawler/engine.py - Crawler engine → Crawl policy: applies policy
spriditis/crawler/engine.py - Crawler engine → Web sources: fetches pages
- Crawler engine → Extraction pipeline: extracts pages
spriditis/crawler/engine.py - Extraction pipeline → Market entities: creates entities
spriditis/extraction/engine.py - Crawler engine → Target identity: checks target
spriditis/crawler/engine.py - Target identity → Entity resolution: uses identity signals
spriditis/resolution/target.py - Crawler engine → AI enrichment: enriches entities
spriditis/crawler/engine.py - Run service → Market entities: persists results
spriditis/api/service.py - Research database → Domain registry: stores registry
spriditis/storage/database.py - Research database → Feed state: stores feed state
spriditis/storage/database.py - Research database → Change detection: records history
spriditis/storage/database.py - Watch mode → Research database: reuses run state
- HTML report → Researcher: presents findings
- Research database → Entity resolution: stores identity clusters
spriditis/storage/database.py