# pranav-8660/website-clickstream-tracker architecture

> Architecture diagram of the GitHub repository pranav-8660/website-clickstream-tracker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pranav-8660/website-clickstream-tracker
- Repository: https://github.com/pranav-8660/website-clickstream-tracker

## Overview

##### Project Type and Purpose
- This is a microservices-based event-tracking pipeline.
- Purpose: collect frontend events, stream them through Kafka, enrich & process them, persist to multiple datastores, and surface analytics via an API.

##### File Structure
- Top-level directories each host a Spring Boot microservice:
- event-collector-service (ECS)
- event-processor-service (EPS)
- storage-service
- analytics-api-service
- You also have a docker-compose.yaml to orchestrate containers.
- Each service has its own pom.xml, src/main/java with controllers, services, DTOs, configs, and src/main/resources for application.properties.
- No frontend code here, but README calls out a separate JS SDK.

##### README for Insights
- It explicitly names your services and their responsibilities.
- Confirms use of Kafka topics (“clickstream-events”, “processed-events”), and three datastores (Postgres, MongoDB, Elasticsearch).
- Mentions an optional configuration service for the future.

##### Main Components and Data Flows
a. Frontend Tracker SDK
- A lightweight JS library (external to this repo) embedded in web pages.
- Sends events (beacon/batch) to ECS over HTTP.
b. Event Collector Service (ECS)
- Spring Boot REST API endpoint (/events).
- Produces raw events into Kafka topic “clickstream-events”.
c. Event Processor Service (EPS)
- Kafka consumer on “clickstream-events”.
- Enriches records (GeoIP, device parse, session stitching).
- Produces enriched events onto “processed-events” topic.
d. Storage Service
- Kafka consumer on “processed-events”.
- Persists to multiple datastores:
- PostgreSQL for relational/session querying
- MongoDB for flexible JSON storage
- Elasticsearch for search/dashboards
e. Analytics API Service (AAS)
- Reads from databases, provides ML, reporting & visualization endpoints.
f. (Future) Configuration Service
- REST API to supply tracking rules/config to the Frontend SDK.

##### Architectural Patterns & Design Principles
- Microservices, each packaged & deployed independently.
- Event-driven, publish/subscribe pipeline with Kafka.
- Single responsibility per service.
- Separation of ingestion, enrichment, storage, and analytics.
- Stateless services behind containers for easy horizontal scaling.
- Pluggable datastore layer.

##### Technologies & Frameworks
- Spring Boot (all Java services)
- Apache Kafka (messaging)
- PostgreSQL, MongoDB, Elasticsearch (storage)
- Docker Compose for local orchestration
- JS SDK (frontend, external)
- (Future) Config Server (Spring Cloud Config?)

## Diagram (Mermaid)

```mermaid
flowchart TB
    classDef service fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef messaging fill:#FFA500,stroke:#000,stroke-width:1px
    classDef db fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px

    subgraph "Docker Network" 
        direction TB

        subgraph "Ingestion Layer"
            direction TB
            SDK["Frontend Tracker SDK"]:::external
            subgraph "Event Collector Service"
                direction TB
                ECSController["ECSController\n(REST Controller)"]:::service
                EventProducerService["EventProducerService\n(Kafka Producer)"]:::service
                DTOs["DTOs"]:::service
                ECSConfig["application.properties"]:::service
            end
        end

        subgraph "Processing Layer"
            direction TB
            Kafka1["Kafka Cluster\n(clickstream-events,\nprocessed-events)"]:::messaging
            subgraph "Event Processor Service"
                direction TB
                EventConsumer["EventConsumer\n(Kafka Consumer)"]:::service
                EventProcessorService["EventProcessorService\n(Enrichment)"]:::service
                EventProducer2["EventProducer\n(Kafka Producer)"]:::service
                GeoIpConfig["GeoIpConfig"]:::service
                DataCleaners["Data Cleaner Modules"]:::service
            end
        end

        subgraph "Persistence Layer"
            direction TB
            subgraph "Storage Service"
                direction TB
                ProcessedEventConsumer["ProcessedEventConsumer\n(Kafka Consumer)"]:::service
                ProcessedEventModel["ProcessedEventModel"]:::service
                Repositories["Repositories\n(Postgres/MongoDB/ES)"]:::service
                StorageConfig["application.properties"]:::service
            end
            subgraph "Datastores"
                direction TB
                PG["PostgreSQL\n(Relational)"]:::db
                MG["MongoDB\n(JSON Store)"]:::db
                ES["Elasticsearch\n(Search)"]:::db
            end
        end

        subgraph "Analytics Layer"
            direction TB
            AAS["Analytics API Service"]:::service
        end
    end

    SDK -->|"HTTP POST /events"| ECSController
    ECSController -->|calls| EventProducerService
    DTOs -->|used by| ECSController
    ECSConfig -->|configures| ECSController
    EventProducerService -->|"Kafka: clickstream-events"| Kafka1

    Kafka1 -->|"Kafka Consumer"| EventConsumer
    EventConsumer -->|passes to| EventProcessorService
    EventProcessorService -->|calls Data Cleaners| DataCleaners
    DataCleaners -->|uses| GeoIpConfig
    EventProcessorService -->|sends to| EventProducer2
    EventProducer2 -->|"Kafka: processed-events"| Kafka1

    Kafka1 -->|"Kafka Consumer"| ProcessedEventConsumer
    ProcessedEventConsumer -->|maps to| ProcessedEventModel
    ProcessedEventConsumer -->|uses| Repositories
    Repositories --> PG
    Repositories --> MG
    Repositories --> ES
    StorageConfig -->|configures| ProcessedEventConsumer

    PG -->|SQL queries| AAS
    MG -->|NoSQL queries| AAS
    ES -->|Search queries| AAS

    click ECSController "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/event-collector-service/src/main/java/com/pranavv51/event_collector_service/controller/ECSController.java"
    click EventProducerService "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/event-collector-service/src/main/java/com/pranavv51/event_collector_service/service/EventProducerService.java"
    click DTOs "https://github.com/pranav-8660/website-clickstream-tracker/tree/develop/event-collector-service/src/main/java/com/pranavv51/event_collector_service/DTO"
    click ECSConfig "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/event-collector-service/src/main/resources/application.properties"
    click EventConsumer "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/event-processor-service/src/main/java/com/pranavv51/event_processor_service/service/EventConsumer.java"
    click EventProcessorService "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/event-processor-service/src/main/java/com/pranavv51/event_processor_service/service/EventProcessorService.java"
    click EventProducer2 "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/event-processor-service/src/main/java/com/pranavv51/event_processor_service/service/EventProducer.java"
    click GeoIpConfig "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/event-processor-service/src/main/java/com/pranavv51/event_processor_service/config/GeoIpConfig.java"
    click DataCleaners "https://github.com/pranav-8660/website-clickstream-tracker/tree/develop/event-processor-service/src/main/java/com/pranavv51/event_processor_service/service/datacleaner"
    click ProcessedEventConsumer "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/storage-service/src/main/java/com/pranavv51/storage_service/service/ProcessedEventConsumer.java"
    click ProcessedEventModel "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/storage-service/src/main/java/com/pranavv51/storage_service/model/ProcessedEventModel.java"
    click Repositories "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/storage-service/src/main/java/com/pranavv51/storage_service/repository/ProcessedEventRepository.java"
    click StorageConfig "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/storage-service/src/main/resources/application.properties"
    click AAS "https://github.com/pranav-8660/website-clickstream-tracker/tree/develop/analytics-api-service"
    click docker-compose "https://github.com/pranav-8660/website-clickstream-tracker/blob/develop/docker-compose.yaml"
```
