# tigdav/linktracker-scrapper architecture

> Architecture diagram of the GitHub repository tigdav/linktracker-scrapper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tigdav/linktracker-scrapper
- Repository: https://github.com/tigdav/linktracker-scrapper

## Overview

LinkTracker Scrapper is a background service for managing tracked links and periodically detecting GitHub and StackOverflow updates, then notifying the Bot service. Its main user workflow is chat/link management through REST or gRPC, with subscription state stored behind configurable repositories and optional cached link reads. A separate Kafka consumer pipeline filters, summarizes, prioritizes, groups, and republishes updates. The graph reflects documented capabilities and sampled call ownership; scheduler wiring and some unsampled routes are shown only where supported by the README or composition entry point.

Main entry points: [`src/application/update_checker.py`](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/update_checker.py) (Update checker), [`src/application/link_service.py`](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/link_service.py) (Link service), [`src/infrastructure/db`](https://github.com/tigdav/linktracker-scrapper/tree/main/src/infrastructure/db) (Repository backends), [`src/api/router_links.py`](https://github.com/tigdav/linktracker-scrapper/blob/main/src/api/router_links.py) (REST API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["Client interfaces"]
  node_rest["REST API<br/>[router_links.py]"]
  node_grpc["gRPC API<br/>[grpc_servicer.py]"]
end

subgraph group_management["Tracking management"]
  node_chat["Chat service<br/>[chat_service.py]"]
  node_links["Link service<br/>[link_service.py]"]
  node_cached_links["Cached link service"]
  node_tags["Tag service<br/>[tag_service.py]"]
  node_classifier["URL classifier<br/>[link_parser.py]"]
end

subgraph group_checking["Update checking"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_checker["Update checker<br/>[update_checker.py]"]
  node_github["GitHub client<br/>[github_client.py]"]
  node_stackoverflow["StackOverflow client"]
end

subgraph group_delivery["Notification pipeline"]
  node_sender["Notification sender"]
  node_bot_http["Bot HTTP client<br/>[bot_client.py]"]
  node_consumer["Kafka consumer<br/>[kafka_consumer.py]"]
  node_processor["Update processor<br/>[processing.py]"]
  node_filter["Update filter<br/>[filtering.py]"]
  node_summarizer["Summarizer<br/>[summarizer.py]"]
  node_prioritizer["Prioritizer<br/>[prioritizer.py]"]
  node_grouper["Update grouper<br/>[grouping.py]"]
  node_publisher["Kafka publisher<br/>[kafka_publisher.py]"]
end

subgraph group_infrastructure["State and integrations"]
  node_repositories["Repository backends"]
  node_postgres[("PostgreSQL")]
  node_cache["Links cache"]
  node_valkey[("Valkey")]
  node_kafka["Kafka"]
  node_schema_registry["Schema Registry"]
end

node_client(("Bot / API client"))
node_update_bot(("Bot service"))

node_client -->|"calls REST"| node_rest
node_client -->|"calls gRPC"| node_grpc
node_rest -->|"delegates chat"| node_chat
node_rest -->|"delegates links"| node_links
node_rest -->|"delegates tags"| node_tags
node_grpc -->|"delegates chat"| node_chat
node_grpc -->|"delegates links"| node_links
node_links -->|"validates URLs"| node_classifier
node_links -->|"reads/writes"| node_repositories
node_chat -->|"reads/writes"| node_repositories
node_tags -->|"reads/writes"| node_repositories
node_cached_links -->|"reads/writes"| node_cache
node_cached_links -->|"delegates misses"| node_links
node_scheduler -->|"runs checks"| node_checker
node_checker -->|"loads tracked links"| node_repositories
node_checker -->|"checks updates"| node_github
node_checker -->|"checks updates"| node_stackoverflow
node_checker -->|"sends notifications"| node_sender
node_sender -->|"publishes updates"| node_kafka
node_sender -.->|"sends HTTP"| node_bot_http
node_bot_http -.->|"notifies"| node_update_bot
node_consumer -->|"consumes updates"| node_kafka
node_consumer -->|"processes messages"| node_processor
node_processor -->|"filters"| node_filter
node_processor -->|"summarizes"| node_summarizer
node_processor -->|"prioritizes"| node_prioritizer
node_consumer -->|"groups updates"| node_grouper
node_grouper -->|"dispatches groups"| node_publisher
node_publisher -->|"publishes results"| node_kafka
node_repositories -->|"persists state"| node_postgres
node_cache -->|"reads/writes"| node_valkey
node_sender -.->|"registers Avro schema"| node_schema_registry

click node_rest "https://github.com/tigdav/linktracker-scrapper/blob/main/src/api/router_links.py"
click node_grpc "https://github.com/tigdav/linktracker-scrapper/blob/main/src/api/grpc_servicer.py"
click node_chat "https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/chat_service.py"
click node_links "https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/link_service.py"
click node_cached_links "https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/cached_link_service.py"
click node_tags "https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/tag_service.py"
click node_classifier "https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/link_parser.py"
click node_scheduler "https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/scheduler.py"
click node_checker "https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/update_checker.py"
click node_github "https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/github_client.py"
click node_stackoverflow "https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/stackoverflow_client.py"
click node_sender "https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/kafka_notification_sender.py"
click node_bot_http "https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/bot_client.py"
click node_consumer "https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/infrastructure/kafka_consumer.py"
click node_processor "https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/processing.py"
click node_filter "https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/filtering.py"
click node_summarizer "https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/summarizer.py"
click node_prioritizer "https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/prioritizer.py"
click node_grouper "https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/grouping.py"
click node_publisher "https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/infrastructure/kafka_publisher.py"
click node_repositories "https://github.com/tigdav/linktracker-scrapper/tree/main/src/infrastructure/db"
click node_cache "https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/valkey_links_cache.py"
click node_schema_registry "https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/schema_registry_client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_rest,node_grpc,node_client toneBlue
class node_chat,node_links,node_cached_links,node_tags,node_classifier toneAmber
class node_scheduler,node_checker,node_github,node_stackoverflow toneMint
class node_sender,node_bot_http,node_consumer,node_processor,node_filter,node_summarizer,node_prioritizer,node_grouper,node_publisher toneRose
class node_repositories,node_postgres,node_cache,node_valkey,node_kafka,node_schema_registry,node_update_bot toneIndigo
```

## Components

### Client interfaces

- [REST API](https://github.com/tigdav/linktracker-scrapper/blob/main/src/api/router_links.py): `src/api/router_links.py`
- [gRPC API](https://github.com/tigdav/linktracker-scrapper/blob/main/src/api/grpc_servicer.py): `src/api/grpc_servicer.py`

### Tracking management

- [Chat service](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/chat_service.py): `src/application/chat_service.py`
- [Link service](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/link_service.py): `src/application/link_service.py`
- [Cached link service](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/cached_link_service.py): `src/application/cached_link_service.py`
- [Tag service](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/tag_service.py): `src/application/tag_service.py`
- [URL classifier](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/link_parser.py): `src/application/link_parser.py`

### Update checking

- [Scheduler](https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/scheduler.py): `src/infrastructure/scheduler.py`
- [Update checker](https://github.com/tigdav/linktracker-scrapper/blob/main/src/application/update_checker.py): `src/application/update_checker.py`
- [GitHub client](https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/github_client.py): `src/infrastructure/github_client.py`
- [StackOverflow client](https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/stackoverflow_client.py): `src/infrastructure/stackoverflow_client.py`

### Notification pipeline

- [Notification sender](https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/kafka_notification_sender.py): `src/infrastructure/kafka_notification_sender.py`
- [Bot HTTP client](https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/bot_client.py): `src/infrastructure/bot_client.py`
- [Kafka consumer](https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/infrastructure/kafka_consumer.py): `src/update_pipeline/infrastructure/kafka_consumer.py`
- [Update processor](https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/processing.py): `src/update_pipeline/application/processing.py`
- [Update filter](https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/filtering.py): `src/update_pipeline/application/filtering.py`
- [Summarizer](https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/summarizer.py): `src/update_pipeline/application/summarizer.py`
- [Prioritizer](https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/prioritizer.py): `src/update_pipeline/application/prioritizer.py`
- [Update grouper](https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/application/grouping.py): `src/update_pipeline/application/grouping.py`
- [Kafka publisher](https://github.com/tigdav/linktracker-scrapper/blob/main/src/update_pipeline/infrastructure/kafka_publisher.py): `src/update_pipeline/infrastructure/kafka_publisher.py`

### State and integrations

- [Repository backends](https://github.com/tigdav/linktracker-scrapper/tree/main/src/infrastructure/db): `src/infrastructure/db`
- **PostgreSQL**
- [Links cache](https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/valkey_links_cache.py): `src/infrastructure/valkey_links_cache.py`
- **Valkey**
- **Kafka**
- [Schema Registry](https://github.com/tigdav/linktracker-scrapper/blob/main/src/infrastructure/schema_registry_client.py): `src/infrastructure/schema_registry_client.py`

### Other components

- **Bot / API client**
- **Bot service**

## Connections

- Bot / API client → REST API: calls REST
- Bot / API client → gRPC API: calls gRPC
- REST API → Chat service: delegates chat
- REST API → Link service: delegates links
- REST API → Tag service: delegates tags
- gRPC API → Chat service: delegates chat
- gRPC API → Link service: delegates links
- Link service → URL classifier: validates URLs
- Link service → Repository backends: reads/writes
- Chat service → Repository backends: reads/writes
- Tag service → Repository backends: reads/writes
- Cached link service → Links cache: reads/writes
- Cached link service → Link service: delegates misses
- Scheduler → Update checker: runs checks
- Update checker → Repository backends: loads tracked links
- Update checker → GitHub client: checks updates
- Update checker → StackOverflow client: checks updates
- Update checker → Notification sender: sends notifications
- Notification sender → Kafka: publishes updates
- Notification sender → Bot HTTP client: sends HTTP
- Bot HTTP client → Bot service: notifies
- Kafka consumer → Kafka: consumes updates
- Kafka consumer → Update processor: processes messages
- Update processor → Update filter: filters
- Update processor → Summarizer: summarizes
- Update processor → Prioritizer: prioritizes
- Kafka consumer → Update grouper: groups updates
- Update grouper → Kafka publisher: dispatches groups
- Kafka publisher → Kafka: publishes results
- Repository backends → PostgreSQL: persists state
- Links cache → Valkey: reads/writes
- Notification sender → Schema Registry: registers Avro schema