How tigdav/linktracker-scrapper works
LinkTracker Scrapper is a background service for managing tracked links and periodically detecting GitHub and StackOverflow updates, then notifying the Bot service. Its main user workflow is chat/link management through REST or gRPC, with subscription state stored behind configurable repositories and optional cached link reads. A separate Kafka consumer pipeline filters, summarizes, prioritizes, groups, and republishes updates. The graph reflects documented capabilities and sampled call ownership; scheduler wiring and some unsampled routes are shown only where supported by the README or composition entry point.
Main entry points: update_checker.py Update checker · link_service.py Link service · db Repository backends · router_links.py REST API
Components 28
Tracking management
- Chat service
src/application/chat_service.py - Link service
src/application/link_service.py - Cached link service
src/application/cached_link_service.py - Tag service
src/application/tag_service.py - URL classifier
src/application/link_parser.py
Update checking
- Scheduler
src/infrastructure/scheduler.py - Update checker
src/application/update_checker.py - GitHub client
src/infrastructure/github_client.py - StackOverflow client
src/infrastructure/stackoverflow_client.py
Notification pipeline
- Notification sender
src/infrastructure/kafka_notification_sender.py - Bot HTTP client
src/infrastructure/bot_client.py - Kafka consumer
src/update_pipeline/infrastructure/kafka_consumer.py - Update processor
src/update_pipeline/application/processing.py - Update filter
src/update_pipeline/application/filtering.py - Summarizer
src/update_pipeline/application/summarizer.py - Prioritizer
src/update_pipeline/application/prioritizer.py - Update grouper
src/update_pipeline/application/grouping.py - Kafka publisher
src/update_pipeline/infrastructure/kafka_publisher.py
State and integrations
- Repository backends
src/infrastructure/db - PostgreSQL
- Links cache
src/infrastructure/valkey_links_cache.py - Valkey
- Kafka
- Schema Registry
src/infrastructure/schema_registry_client.py
Other components
- Bot / API client
- Bot service
Connections 32
- Bot / API client → REST API: calls REST
- Bot / API client → gRPC API: calls gRPC
- REST API → Chat service: delegates chat
- REST API → Link service: delegates links
- REST API → Tag service: delegates tags
- gRPC API → Chat service: delegates chat
- gRPC API → Link service: delegates links
- Link service → URL classifier: validates URLs
- Link service → Repository backends: reads/writes
- Chat service → Repository backends: reads/writes
- Tag service → Repository backends: reads/writes
- Cached link service → Links cache: reads/writes
- Cached link service → Link service: delegates misses
- Scheduler → Update checker: runs checks
- Update checker → Repository backends: loads tracked links
- Update checker → GitHub client: checks updates
- Update checker → StackOverflow client: checks updates
- Update checker → Notification sender: sends notifications
- Notification sender → Kafka: publishes updates
- Notification sender → Bot HTTP client: sends HTTP
- Bot HTTP client → Bot service: notifies
- Kafka consumer → Kafka: consumes updates
- Kafka consumer → Update processor: processes messages
- Update processor → Update filter: filters
- Update processor → Summarizer: summarizes
- Update processor → Prioritizer: prioritizes
- Kafka consumer → Update grouper: groups updates
- Update grouper → Kafka publisher: dispatches groups
- Kafka publisher → Kafka: publishes results
- Repository backends → PostgreSQL: persists state
- Links cache → Valkey: reads/writes
- Notification sender → Schema Registry: registers Avro schema