How pragatigs/log-ingestion-processing works
This is an end-to-end log-ingestion demonstration: a sample simulator sends JSON lines over TCP; the ingestor assigns partitions and publishes to Kafka; a consumer polls and processes batches, using Redis to track processed offsets and writing partition-specific files. A scheduled aggregator snapshots and copies those files, then uploads processed outputs to S3. The README documents the broad pipeline; excerpts show the TCP dispatch and consumer control flow. The shared-file handoff between worker and aggregator is not verified, since their shown paths differ.
Main entry points: LogsApplication.java Worker coordinator · ScheduledAggregator.java Scheduled aggregator · ConnectionHandler.java Connection handler · LogMessageProcessor.java Log processor
Components 17
Traffic and ingestion
- Log simulator
simulator/src/main/java/com/tcp/simulator/TcpPusher.java - TCP listener
TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/TcpServerService.java - Connection handler
TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java - Partition router
TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/LogRouterService.java - Kafka producer
TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/KafkaProducerService.java
Message transport
- Kafka topic
Consumer processing
- Consumer configuration
CONSUMER/logs-worker/src/main/java/com/processing/logs/config/KafkaConsumerConfig.java - Log poller
CONSUMER/logs-worker/src/main/java/com/processing/logs/consumer/LogPoller.java - Worker coordinator
CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java - Batch executor
CONSUMER/logs-worker/src/main/java/com/processing/logs/executor/BatchExecutor.java - Log processor
CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java - Offset guard
CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/OffsetGuard.java - Redis state
Aggregation and storage
- Worker log files
CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java - Scheduled aggregator
CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java - Processed files
CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java - S3 object storage
Connections 19
- Log simulator → TCP listener: sends TCP logs
README.md - TCP listener → Connection handler: dispatches sockets
TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/TcpServerService.java - Connection handler → Partition router: routes each line
TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java - Connection handler → Kafka producer: sends partitioned line
TCP-INGESTOR-MW/ingestor/src/main/java/com/tcp/ingestor/service/ConnectionHandler.java - Kafka producer → Kafka topic: publishes records
README.md - Worker coordinator → Consumer configuration: builds properties
CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java - Worker coordinator → Log poller: polls records
CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java - Log poller → Kafka topic: reads topic
README.md - Worker coordinator → Batch executor: processes batch
CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java - Batch executor → Log processor: processes records
CONSUMER/logs-worker/src/main/java/com/processing/logs/executor/BatchExecutor.java - Log processor → Offset guard: checks and marks
CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java - Offset guard → Redis state: reads and writes
CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/OffsetGuard.java - Log processor → Worker log files: appends log line
CONSUMER/logs-worker/src/main/java/com/processing/logs/processor/LogMessageProcessor.java - Worker coordinator → Log poller: receives records
CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java - Worker coordinator → Scheduled aggregator: scheduled separately
- Worker log files → Scheduled aggregator: possible file handoff
- Scheduled aggregator → Processed files: copies and snapshots
CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java - Scheduled aggregator → S3 object storage: uploads outputs
CONSUMER/aggregator/src/main/java/com/processedlogs/aggregator/cron/ScheduledAggregator.java - Worker coordinator → Offset guard: clears processed offsets
CONSUMER/logs-worker/src/main/java/com/processing/logs/LogsApplication.java