How anuj690/customer-behavior-platform works
The Customer Behavior Platform generates customer journey events, publishes them to Kafka, and processes them with Spark Structured Streaming into lakehouse layers and analytical outputs. The main path is producer generation and publication, stream parsing and validation, deduplication, then Bronze/Silver/Gold and analytics writes; invalid records go to a dead-letter store. The graph follows the documented architecture and sampled orchestration. Some streaming-service wiring is omitted from the excerpt, so uncertain internal calls are not shown.
Main entry points: streaming_service/service.py Batch orchestrator · tables.py Iceberg tables · producer_service/service.py Journey publisher · app.py Streaming entry
Components 30
Event Generation
- Producer entry
producer-service/src/producer_service/app.py - Journey publisher
producer-service/src/producer_service/service.py - Customer generator
producer-service/src/producer_service/generators/customer.py - Session generator
producer-service/src/producer_service/generators/session.py - Journey generator
producer-service/src/producer_service/generators/journey.py - Event factory
producer-service/src/producer_service/factories/event_factory.py - Event corruptor
producer-service/src/producer_service/utils/event_corruptor.py - Kafka publisher
producer-service/src/producer_service/kafka/producer.py
Stream Ingestion
- Streaming entry
streaming-service/src/streaming_service/app.py - Batch orchestrator
streaming-service/src/streaming_service/service.py - Kafka consumer
streaming-service/src/streaming_service/kafka/consumer.py - JSON parser
streaming-service/src/streaming_service/transforms/event_parser.py - Event transformer
streaming-service/src/streaming_service/transforms/event_transformer.py - Event validator
streaming-service/src/streaming_service/transforms/validator.py
Lakehouse Processing
- Deduplication
streaming-service/src/streaming_service/deduplication/writer.py - Bronze writer
streaming-service/src/streaming_service/bronze/writer.py - Silver transform
streaming-service/src/streaming_service/silver/transformer.py - Silver writer
streaming-service/src/streaming_service/silver/writer.py - Gold aggregation
streaming-service/src/streaming_service/gold/aggregator.py - Gold writer
streaming-service/src/streaming_service/gold/writer.py - Dead-letter writer
streaming-service/src/streaming_service/dead_letter/writer.py
Analytics
- Dimension outputs
streaming-service/src/streaming_service/analytics/dimension_writer.py - Fact events
streaming-service/src/streaming_service/analytics/fact_events.py - Daily metrics
streaming-service/src/streaming_service/analytics/daily_user_metrics.py - Funnel metrics
streaming-service/src/streaming_service/analytics/funnel_metrics.py
Runtime Platform
- Iceberg tables
streaming-service/src/streaming_service/iceberg/tables.py - Spark session
streaming-service/src/streaming_service/spark/spark_session.py
Other components
- Event source
- Kafka topic
- Amazon S3
Connections 36
- Event source → Producer entry: initiates
- Producer entry → Journey publisher: starts
- Journey publisher → Customer generator: generates customer
- Journey publisher → Session generator: creates session
- Journey publisher → Journey generator: generates journey
- Journey generator → Event factory: builds events
- Journey publisher → Event corruptor: corrupts events
- Journey publisher → Kafka publisher: publishes events
- Kafka publisher → Kafka topic: sends events
- Streaming entry → Spark session: creates session
- Streaming entry → Batch orchestrator: starts
- Batch orchestrator → Kafka consumer: reads stream
- Kafka consumer → Kafka topic: consumes events
- Batch orchestrator → JSON parser: parses stream
- JSON parser → Event transformer: supplies records
- Batch orchestrator → Event validator: validates records
- Batch orchestrator → Deduplication: deduplicates valid
- Batch orchestrator → Dead-letter writer: writes invalid
- Batch orchestrator → Bronze writer: writes batch
- Batch orchestrator → Silver transform: transforms batch
- Silver transform → Silver writer: writes records
- Batch orchestrator → Gold aggregation: aggregates records
- Gold aggregation → Gold writer: writes aggregates
- Batch orchestrator → Dimension outputs: writes dimensions
- Batch orchestrator → Fact events: writes facts
- Batch orchestrator → Daily metrics: writes metrics
- Batch orchestrator → Funnel metrics: writes funnel
- Bronze writer → Iceberg tables: merges records
- Silver writer → Iceberg tables: writes layer
- Gold writer → Iceberg tables: writes layer
- Dead-letter writer → Iceberg tables: stores invalid
- Dimension outputs → Iceberg tables: writes tables
- Fact events → Iceberg tables: writes fact table
- Daily metrics → Iceberg tables: writes metrics
- Funnel metrics → Iceberg tables: merges metrics
- Iceberg tables → Amazon S3: stores data