How sauravdhakal12/kafka works
1. High-level shape
- This repository is a small event-driven system split into three independent TypeScript task services: transaction-service, notification-service, and listing-service.
- The core runtime pattern is Kafka consumers processing messages and interacting with a PostgreSQL database.
2. Shared infrastructure
- The repo root contains docker-compose.yml for the infrastructure layer and db/ for shared database setup.
- docker-compose starts the two main dependencies: Kafka and Postgres.
- db/schema.sql and db/pool.ts indicate a shared SQL schema plus a reusable DB connection pool used by the services.
3. Database boundary
- The database is a central persistence boundary across tasks.
- pnpm run migrate initializes the schema, so DB setup is treated as a first-class repo concern rather than an external manual step.
- The presence of transaction-service/sample-db-schema.prisma suggests at least one task was designed with a Prisma-style schema reference, but the shared runtime DB layer is still the db/ folder.
4. Transaction service
- transaction-service/transaction-consumer.ts is the entry point for Task 1.
- This service is a Kafka consumer focused on transaction events, likely validating, transforming, and persisting transaction-related data into Postgres.
- It has its own test.ts, so the task is isolated and testable without running the whole repo.
5. Notification service
- notification-service/notification-consumer.ts is Task 2’s consumer.
- This subsystem is responsible for turning incoming events into notifications, and it has a stronger operational story than the others because it also includes dlq-replay.ts.
- The DLQ replay script implies a dead-letter queue recovery path, meaning failed messages are not simply discarded; they can be reprocessed later.
6. Listing service
- listing-service/listing-consumer.ts is Task 3’s consumer.
- Based on the naming, this service likely builds or updates listing records from Kafka events and writes to the shared database.
- Like the other tasks, it is self-contained and has its own test.ts.
7. Operational scripts
- admin.ts at the repo root suggests a top-level administrative or maintenance entry point, likely for support tasks beyond the main consumers.
- The repo README indicates scripts in package.json are the main way to run consumers, tests, migrations, and replay utilities, so the operational surface is intentionally command-driven.
8. Execution model
- The services are not a single monolith; they are separate process entry points that can be run independently.
- Each consumer can be started in “dev” mode without bringing up the others, which is useful for focused debugging and task-by-task validation.
- This makes the architecture modular even though the services share infrastructure and a database.
9. Message flow
- The primary flow is: Kafka topic/event → consumer service → database and/or downstream side effect.
- The system appears centered on asynchronous processing rather than HTTP request handling.
- The DLQ replay script in notification service adds a secondary flow: dead-lettered event → replay tool → consumer reprocessing.
10. Testing boundary
- Each task has a dedicated test file under its own folder, and the README references matching npm/pnpm scripts.
- That suggests the repo is structured so each service can be verified independently at the task level, which is important for diagramming the boundaries.
11. Tech stack that materially affects architecture
- Node.js/TypeScript is the application runtime.
- pnpm is the package manager and primary script runner.
- Docker Compose is used for infrastructure orchestration.
- Kafka provides asynchronous messaging.
- PostgreSQL is the shared persistent store.
12. How to draw the architecture
- Draw three separate consumer boxes: transaction, notification, and listing.
- Put Kafka upstream of all three and Postgres downstream as the shared data store.
- Show db/ as the shared schema/connection layer supporting all services.
- Add a special path for notification replay from DLQ back into the notification consumer.
- Keep admin.ts and the test files as supporting operational/test artifacts rather than core runtime services.
Main entry points: package.json Scripts · notification-consumer.ts Notify consumer · transaction-consumer.ts Tx consumer · listing-consumer.ts List consumer
Components 14
Infrastructure
Local runtime dependencies and compose orchestration
- Compose · Starts Kafka and Postgres for local development
docker-compose.yml - Kafka · Event bus consumed by all task services
- Postgres · Shared persistence boundary for all services
Shared DB
Reusable database schema and connection layer
Transaction
Transaction task service
- Tx consumer · Processes transaction events and persists results
transaction-service/transaction-consumer.ts - Tx test · Task-level verification for the transaction service
transaction-service/test.ts
Notification
Notification task service
- Notify consumer · Transforms incoming events into notifications
notification-service/notification-consumer.ts - DLQ replay · Reprocesses dead-lettered notification events
notification-service/dlq-replay.ts - Notify test · Task-level verification for the notification service
notification-service/test.ts
Listing
Listing task service
- List consumer · Builds or updates listings from Kafka events
listing-service/listing-consumer.ts - List test · Task-level verification for the listing service
listing-service/test.ts
Connections 20
- Compose → Kafka: starts · Compose brings up the broker
- Compose → Postgres: starts · Compose brings up the database
- Schema → Postgres: migrates · Schema initializes the shared database
- Pool → Postgres: connects · Services use the shared connection pool
- Scripts → Tx consumer: runs · Script entry point for the transaction service
- Scripts → Notify consumer: runs · Script entry point for the notification service
- Scripts → List consumer: runs · Script entry point for the listing service
- Scripts → DLQ replay: runs · Script entry point for DLQ replay
- Scripts → Admin: runs · Script entry point for maintenance tasks
- Tx consumer → Kafka: consumes · Reads transaction events from Kafka
- Tx consumer → Pool: writes · Persists transaction data through the shared pool
- Notify consumer → Kafka: consumes · Reads notification-related events from Kafka
- Notify consumer → Pool: writes · Stores notification state through the shared pool
- DLQ replay → Notify consumer: replays · Feeds dead-lettered messages back into the consumer
- List consumer → Kafka: consumes · Reads listing events from Kafka
- List consumer → Pool: writes · Persists listing data through the shared pool
- Admin → Postgres: maintains · Administrative operations may touch the database
- Tx test → Tx consumer: verifies · Service-specific testing boundary
- Notify test → Notify consumer: verifies · Service-specific testing boundary
- List test → List consumer: verifies · Service-specific testing boundary