# pragatigs/log-ingestion-processing architecture

> Architecture diagram of the GitHub repository pragatigs/log-ingestion-processing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pragatigs/log-ingestion-processing
- Repository: https://github.com/pragatigs/log-ingestion-processing

## Overview

This is an end-to-end log-ingestion demonstration: a sample simulator sends JSON lines over TCP; the ingestor assigns partitions and publishes to Kafka; a consumer polls and processes batches, using Redis to track processed offsets and writing partition-specific files. A scheduled aggregator snapshots and copies those files, then uploads processed outputs to S3. The README documents the broad pipeline; excerpts show the TCP dispatch and consumer control flow. The shared-file handoff between worker and aggregator is not verified, since their shown paths differ.

Main entry points: [`CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java`](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java) (Worker coordinator), [`CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java`](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java) (Scheduled aggregator), [`TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java`](https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java) (Connection handler), [`CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java`](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java) (Log processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_traffic["Traffic and ingestion"]
  node_simulator["Log simulator<br/>[TcpPusher.java]"]
  node_tcp_server["TCP listener"]
  node_connection_handler["Connection handler"]
  node_log_router["Partition router"]
  node_kafka_producer["Kafka producer"]
end

subgraph group_messaging["Message transport"]
  node_kafka["Kafka topic"]
end

subgraph group_worker["Consumer processing"]
  node_consumer_config["Consumer configuration"]
  node_log_poller["Log poller<br/>[LogPoller.java]"]
  node_worker_app["Worker coordinator"]
  node_batch_executor["Batch executor<br/>[BatchExecutor.java]"]
  node_message_processor["Log processor"]
  node_offset_guard["Offset guard<br/>[OffsetGuard.java]"]
  node_redis[("Redis state")]
end

subgraph group_outputs["Aggregation and storage"]
  node_staging_files["Worker log files"]
  node_aggregator["Scheduled aggregator"]
  node_processed_files["Processed files"]
  node_s3[("S3 object storage")]
end

node_simulator -->|"sends TCP logs"| node_tcp_server
node_tcp_server -->|"dispatches sockets"| node_connection_handler
node_connection_handler -->|"routes each line"| node_log_router
node_connection_handler -->|"sends partitioned line"| node_kafka_producer
node_kafka_producer -->|"publishes records"| node_kafka
node_worker_app -->|"builds properties"| node_consumer_config
node_worker_app -->|"polls records"| node_log_poller
node_log_poller -->|"reads topic"| node_kafka
node_worker_app -->|"processes batch"| node_batch_executor
node_batch_executor -->|"processes records"| node_message_processor
node_message_processor -->|"checks and marks"| node_offset_guard
node_offset_guard -->|"reads and writes"| node_redis
node_message_processor -->|"appends log line"| node_staging_files
node_worker_app -->|"receives records"| node_log_poller
node_worker_app -.->|"scheduled separately"| node_aggregator
node_staging_files -.->|"possible file handoff"| node_aggregator
node_aggregator -->|"copies and snapshots"| node_processed_files
node_aggregator -->|"uploads outputs"| node_s3
node_worker_app -->|"clears processed offsets"| node_offset_guard

click node_simulator "https://github.com/pragatigs/log-ingestion-processing/blob/main/simulator/src/main/java/com/tcp/simulator/TcpPusher.java"
click node_tcp_server "https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/TcpServerService.java"
click node_connection_handler "https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java"
click node_log_router "https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/LogRouterService.java"
click node_kafka_producer "https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/KafkaProducerService.java"
click node_consumer_config "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/config/KafkaConsumerConfig.java"
click node_log_poller "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/consumer/LogPoller.java"
click node_worker_app "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java"
click node_batch_executor "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/executor/BatchExecutor.java"
click node_message_processor "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java"
click node_offset_guard "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/OffsetGuard.java"
click node_staging_files "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java"
click node_aggregator "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java"
click node_processed_files "https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_simulator,node_tcp_server,node_connection_handler,node_log_router,node_kafka_producer toneBlue
class node_kafka toneAmber
class node_consumer_config,node_log_poller,node_worker_app,node_batch_executor,node_message_processor,node_offset_guard,node_redis toneMint
class node_staging_files,node_aggregator,node_processed_files,node_s3 toneRose
```

## Components

### Traffic and ingestion

- [Log simulator](https://github.com/pragatigs/log-ingestion-processing/blob/main/simulator/src/main/java/com/tcp/simulator/TcpPusher.java): `simulator/src/main/java/com/tcp/simulator/TcpPusher.java`
- [TCP listener](https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/TcpServerService.java): `TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/TcpServerService.java`
- [Connection handler](https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java): `TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java`
- [Partition router](https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/LogRouterService.java): `TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/LogRouterService.java`
- [Kafka producer](https://github.com/pragatigs/log-ingestion-processing/blob/main/TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/KafkaProducerService.java): `TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/KafkaProducerService.java`

### Message transport

- **Kafka topic**

### Consumer processing

- [Consumer configuration](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/config/KafkaConsumerConfig.java): `CONSUMER/logs-worker/src/main/java/com/processing/logs/config/KafkaConsumerConfig.java`
- [Log poller](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/consumer/LogPoller.java): `CONSUMER/logs-worker/src/main/java/com/processing/logs/consumer/LogPoller.java`
- [Worker coordinator](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java): `CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java`
- [Batch executor](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/executor/BatchExecutor.java): `CONSUMER/logs-worker/src/main/java/com/processing/logs/executor/BatchExecutor.java`
- [Log processor](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java): `CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java`
- [Offset guard](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/OffsetGuard.java): `CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/OffsetGuard.java`
- **Redis state**

### Aggregation and storage

- [Worker log files](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java): `CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java`
- [Scheduled aggregator](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java): `CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java`
- [Processed files](https://github.com/pragatigs/log-ingestion-processing/blob/main/CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java): `CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java`
- **S3 object storage**

## Connections

- Log simulator → TCP listener: sends TCP logs (evidence: `README.md`)
- TCP listener → Connection handler: dispatches sockets (evidence: `TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/TcpServerService.java`)
- Connection handler → Partition router: routes each line (evidence: `TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java`)
- Connection handler → Kafka producer: sends partitioned line (evidence: `TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java`)
- Kafka producer → Kafka topic: publishes records (evidence: `README.md`)
- Worker coordinator → Consumer configuration: builds properties (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java`)
- Worker coordinator → Log poller: polls records (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java`)
- Log poller → Kafka topic: reads topic (evidence: `README.md`)
- Worker coordinator → Batch executor: processes batch (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java`)
- Batch executor → Log processor: processes records (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/executor/BatchExecutor.java`)
- Log processor → Offset guard: checks and marks (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java`)
- Offset guard → Redis state: reads and writes (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/OffsetGuard.java`)
- Log processor → Worker log files: appends log line (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java`)
- Worker coordinator → Log poller: receives records (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java`)
- Worker coordinator → Scheduled aggregator: scheduled separately
- Worker log files → Scheduled aggregator: possible file handoff
- Scheduled aggregator → Processed files: copies and snapshots (evidence: `CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java`)
- Scheduled aggregator → S3 object storage: uploads outputs (evidence: `CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java`)
- Worker coordinator → Offset guard: clears processed offsets (evidence: `CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java`)
