# sauravdhakal12/kafka architecture

> Architecture diagram of the GitHub repository sauravdhakal12/kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sauravdhakal12/kafka
- Repository: https://github.com/sauravdhakal12/kafka

## Overview

1. **High-level shape**
   - This repository is a small event-driven system split into **three independent TypeScript task services**: `transaction-service`, `notification-service`, and `listing-service`.
   - The core runtime pattern is **Kafka consumers** processing messages and interacting with a **PostgreSQL** database.

2. **Shared infrastructure**
   - The repo root contains `docker-compose.yml` for the infrastructure layer and `db/` for shared database setup.
   - `docker-compose` starts the two main dependencies: **Kafka** and **Postgres**.
   - `db/schema.sql` and `db/pool.ts` indicate a shared SQL schema plus a reusable DB connection pool used by the services.

3. **Database boundary**
   - The database is a central persistence boundary across tasks.
   - `pnpm run migrate` initializes the schema, so DB setup is treated as a first-class repo concern rather than an external manual step.
   - The presence of `transaction-service/sample-db-schema.prisma` suggests at least one task was designed with a Prisma-style schema reference, but the shared runtime DB layer is still the `db/` folder.

4. **Transaction service**
   - `transaction-service/transaction-consumer.ts` is the entry point for Task 1.
   - This service is a Kafka consumer focused on transaction events, likely validating, transforming, and persisting transaction-related data into Postgres.
   - It has its own `test.ts`, so the task is isolated and testable without running the whole repo.

5. **Notification service**
   - `notification-service/notification-consumer.ts` is Task 2’s consumer.
   - This subsystem is responsible for turning incoming events into notifications, and it has a stronger operational story than the others because it also includes `dlq-replay.ts`.
   - The DLQ replay script implies a **dead-letter queue recovery path**, meaning failed messages are not simply discarded; they can be reprocessed later.

6. **Listing service**
   - `listing-service/listing-consumer.ts` is Task 3’s consumer.
   - Based on the naming, this service likely builds or updates listing records from Kafka events and writes to the shared database.
   - Like the other tasks, it is self-contained and has its own `test.ts`.

7. **Operational scripts**
   - `admin.ts` at the repo root suggests a top-level administrative or maintenance entry point, likely for support tasks beyond the main consumers.
   - The repo README indicates scripts in `package.json` are the main way to run consumers, tests, migrations, and replay utilities, so the operational surface is intentionally command-driven.

8. **Execution model**
   - The services are not a single monolith; they are separate process entry points that can be run independently.
   - Each consumer can be started in “dev” mode without bringing up the others, which is useful for focused debugging and task-by-task validation.
   - This makes the architecture modular even though the services share infrastructure and a database.

9. **Message flow**
   - The primary flow is: **Kafka topic/event → consumer service → database and/or downstream side effect**.
   - The system appears centered on asynchronous processing rather than HTTP request handling.
   - The DLQ replay script in notification service adds a secondary flow: **dead-lettered event → replay tool → consumer reprocessing**.

10. **Testing boundary**
   - Each task has a dedicated test file under its own folder, and the README references matching npm/pnpm scripts.
   - That suggests the repo is structured so each service can be verified independently at the task level, which is important for diagramming the boundaries.

11. **Tech stack that materially affects architecture**
   - **Node.js/TypeScript** is the application runtime.
   - **pnpm** is the package manager and primary script runner.
   - **Docker Compose** is used for infrastructure orchestration.
   - **Kafka** provides asynchronous messaging.
   - **PostgreSQL** is the shared persistent store.

12. **How to draw the architecture**
   - Draw three separate consumer boxes: transaction, notification, and listing.
   - Put Kafka upstream of all three and Postgres downstream as the shared data store.
   - Show `db/` as the shared schema/connection layer supporting all services.
   - Add a special path for notification replay from DLQ back into the notification consumer.
   - Keep `admin.ts` and the test files as supporting operational/test artifacts rather than core runtime services.

Main entry points: [`package.json`](https://github.com/sauravdhakal12/kafka/blob/master/package.json) (Scripts), [`notification-service/notification-consumer.ts`](https://github.com/sauravdhakal12/kafka/blob/master/notification-service/notification-consumer.ts) (Notify consumer), [`transaction-service/transaction-consumer.ts`](https://github.com/sauravdhakal12/kafka/blob/master/transaction-service/transaction-consumer.ts) (Tx consumer), [`listing-service/listing-consumer.ts`](https://github.com/sauravdhakal12/kafka/blob/master/listing-service/listing-consumer.ts) (List consumer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_infra["Infrastructure"]
  node_docker_compose["Compose<br/>[docker-compose.yml]"]
  node_kafka[("Kafka<br/>broker")]
  node_postgres[("Postgres<br/>database")]
end

subgraph group_shared["Shared DB"]
  node_db_schema["Schema<br/>[schema.sql]"]
  node_db_pool["Pool<br/>db connection<br/>[pool.ts]"]
end

subgraph group_tx["Transaction"]
  node_tx_consumer["Tx consumer<br/>kafka consumer"]
  node_tx_test["Tx test<br/>service test<br/>[test.ts]"]
end

subgraph group_notify["Notification"]
  node_notify_consumer["Notify consumer<br/>kafka consumer"]
  node_notify_replay["DLQ replay<br/>replay tool<br/>[dlq-replay.ts]"]
  node_notify_test["Notify test<br/>service test<br/>[test.ts]"]
end

subgraph group_listing["Listing"]
  node_list_consumer["List consumer<br/>kafka consumer"]
  node_list_test["List test<br/>service test<br/>[test.ts]"]
end

subgraph group_ops["Operations"]
  node_package_json["Scripts<br/>[package.json]"]
  node_admin_ts["Admin<br/>maintenance<br/>[admin.ts]"]
end

node_docker_compose -->|"starts"| node_kafka
node_docker_compose -->|"starts"| node_postgres
node_db_schema -->|"migrates"| node_postgres
node_db_pool -->|"connects"| node_postgres
node_package_json -->|"runs"| node_tx_consumer
node_package_json -->|"runs"| node_notify_consumer
node_package_json -->|"runs"| node_list_consumer
node_package_json -->|"runs"| node_notify_replay
node_package_json -->|"runs"| node_admin_ts
node_tx_consumer -->|"consumes"| node_kafka
node_tx_consumer -->|"writes"| node_db_pool
node_notify_consumer -->|"consumes"| node_kafka
node_notify_consumer -->|"writes"| node_db_pool
node_notify_replay -.->|"replays"| node_notify_consumer
node_list_consumer -->|"consumes"| node_kafka
node_list_consumer -->|"writes"| node_db_pool
node_admin_ts -.->|"maintains"| node_postgres
node_tx_test -.->|"verifies"| node_tx_consumer
node_notify_test -.->|"verifies"| node_notify_consumer
node_list_test -.->|"verifies"| node_list_consumer

click node_docker_compose "https://github.com/sauravdhakal12/kafka/blob/master/docker-compose.yml"
click node_db_schema "https://github.com/sauravdhakal12/kafka/blob/master/db/schema.sql"
click node_db_pool "https://github.com/sauravdhakal12/kafka/blob/master/db/pool.ts"
click node_package_json "https://github.com/sauravdhakal12/kafka/blob/master/package.json"
click node_admin_ts "https://github.com/sauravdhakal12/kafka/blob/master/admin.ts"
click node_tx_consumer "https://github.com/sauravdhakal12/kafka/blob/master/transaction-service/transaction-consumer.ts"
click node_tx_test "https://github.com/sauravdhakal12/kafka/blob/master/transaction-service/test.ts"
click node_notify_consumer "https://github.com/sauravdhakal12/kafka/blob/master/notification-service/notification-consumer.ts"
click node_notify_replay "https://github.com/sauravdhakal12/kafka/blob/master/notification-service/dlq-replay.ts"
click node_notify_test "https://github.com/sauravdhakal12/kafka/blob/master/notification-service/test.ts"
click node_list_consumer "https://github.com/sauravdhakal12/kafka/blob/master/listing-service/listing-consumer.ts"
click node_list_test "https://github.com/sauravdhakal12/kafka/blob/master/listing-service/test.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_docker_compose,node_kafka,node_postgres toneBlue
class node_db_schema,node_db_pool toneAmber
class node_tx_consumer,node_tx_test toneMint
class node_notify_consumer,node_notify_replay,node_notify_test toneRose
class node_list_consumer,node_list_test toneIndigo
class node_package_json,node_admin_ts toneTeal
```

## Components

### Infrastructure

Local runtime dependencies and compose orchestration

- [Compose](https://github.com/sauravdhakal12/kafka/blob/master/docker-compose.yml): docker-compose · Starts Kafka and Postgres for local development · `docker-compose.yml`
- **Kafka**: broker · Event bus consumed by all task services
- **Postgres**: database · Shared persistence boundary for all services

### Shared DB

Reusable database schema and connection layer

- [Schema](https://github.com/sauravdhakal12/kafka/blob/master/db/schema.sql): sql schema · Shared database schema definition · `db/schema.sql`
- [Pool](https://github.com/sauravdhakal12/kafka/blob/master/db/pool.ts): db connection · Reusable PostgreSQL connection pool · `db/pool.ts`

### Transaction

Transaction task service

- [Tx consumer](https://github.com/sauravdhakal12/kafka/blob/master/transaction-service/transaction-consumer.ts): kafka consumer · Processes transaction events and persists results · `transaction-service/transaction-consumer.ts`
- [Tx test](https://github.com/sauravdhakal12/kafka/blob/master/transaction-service/test.ts): service test · Task-level verification for the transaction service · `transaction-service/test.ts`

### Notification

Notification task service

- [Notify consumer](https://github.com/sauravdhakal12/kafka/blob/master/notification-service/notification-consumer.ts): kafka consumer · Transforms incoming events into notifications · `notification-service/notification-consumer.ts`
- [DLQ replay](https://github.com/sauravdhakal12/kafka/blob/master/notification-service/dlq-replay.ts): replay tool · Reprocesses dead-lettered notification events · `notification-service/dlq-replay.ts`
- [Notify test](https://github.com/sauravdhakal12/kafka/blob/master/notification-service/test.ts): service test · Task-level verification for the notification service · `notification-service/test.ts`

### Listing

Listing task service

- [List consumer](https://github.com/sauravdhakal12/kafka/blob/master/listing-service/listing-consumer.ts): kafka consumer · Builds or updates listings from Kafka events · `listing-service/listing-consumer.ts`
- [List test](https://github.com/sauravdhakal12/kafka/blob/master/listing-service/test.ts): service test · Task-level verification for the listing service · `listing-service/test.ts`

### Operations

Repo-level scripts and maintenance entry points

- [Scripts](https://github.com/sauravdhakal12/kafka/blob/master/package.json): pnpm scripts · Primary command surface for running tasks, tests, and migrations · `package.json`
- [Admin](https://github.com/sauravdhakal12/kafka/blob/master/admin.ts): maintenance · Top-level administrative entry point · `admin.ts`

## Connections

- Compose → Kafka: starts · Compose brings up the broker
- Compose → Postgres: starts · Compose brings up the database
- Schema → Postgres: migrates · Schema initializes the shared database
- Pool → Postgres: connects · Services use the shared connection pool
- Scripts → Tx consumer: runs · Script entry point for the transaction service
- Scripts → Notify consumer: runs · Script entry point for the notification service
- Scripts → List consumer: runs · Script entry point for the listing service
- Scripts → DLQ replay: runs · Script entry point for DLQ replay
- Scripts → Admin: runs · Script entry point for maintenance tasks
- Tx consumer → Kafka: consumes · Reads transaction events from Kafka
- Tx consumer → Pool: writes · Persists transaction data through the shared pool
- Notify consumer → Kafka: consumes · Reads notification-related events from Kafka
- Notify consumer → Pool: writes · Stores notification state through the shared pool
- DLQ replay → Notify consumer: replays · Feeds dead-lettered messages back into the consumer
- List consumer → Kafka: consumes · Reads listing events from Kafka
- List consumer → Pool: writes · Persists listing data through the shared pool
- Admin → Postgres: maintains · Administrative operations may touch the database
- Tx test → Tx consumer: verifies · Service-specific testing boundary
- Notify test → Notify consumer: verifies · Service-specific testing boundary
- List test → List consumer: verifies · Service-specific testing boundary