How saenthanp/pricey works
Main Components
• Authentication Service – Manages logins, registrations, and OAuth 2.0 (Google Strategy) authentication.
• Link Service – Handles user-submitted links, sending messages to the Scrape Service via Kafka Streams, and also has GRPC endpoints.
• Jobs Service – Monitors and triggers scraping jobs; uses GRPC calls to notify the Link Service when a job is activated.
• Scrape Service – Implements a worker pool to perform scraping tasks concurrently, receiving work via Kafka streams.
• External/Supporting Components – Include Postgres databases for users, links, and job tracking; Kafka for asynchronous messaging; Nginx Ingress Controller for load-balancing and routing; and Kubernetes for container orchestration.
Relationships and Interactions
• Users interact with the system (likely via an API gateway exposed through Nginx) and authenticate using the Authentication Service.
• Once authenticated, users can submit links via the Link Service.
• Submitted links are pushed to the Scrape Service using Kafka streams for processing.
• The Jobs Service periodically checks the jobs database and, when a scraping trigger is due, sends a GRPC request to the Link Service.
• The Scrape Service uses a worker pool and Go concurrency (channels, go routines) to process the received jobs.
• All services are containerized (Docker) and orchestrated with Kubernetes, and configuration/deployment is managed via k8s YAML files.
Architectural Patterns and Principles
• Microservices architecture with clearly separated services, enabling scalability and independent deployment.
• Messaging-driven communication using Kafka for asynchronous tasks and GRPC for synchronous job triggering.
• Use of containerization (Docker) with orchestration through Kubernetes.
• A clear separation of concerns in the codebase (with distinct directories for configuration, service logic, models, repositories, etc.).
Technologies, Frameworks, and Libraries
• Language: Golang
• Messaging: Kafka Streams, GRPC
• Containerization: Docker and Kubernetes (with specific deployment YAMLs shown in the k8s directory)
• Authentication: OAuth 2.0 (Google Strategy)
• Database: Postgres
• Concurrency: Go built-in concurrency mechanisms (go routines, channels, wait groups)
Elements
• From users through the Ingress Controller to the Authentication Service.
• Between Authentication Service and other services (if applicable, e.g., secure token propagation).
• From the Link Service to Kafka and, in turn, from Kafka to the Scrape Service.
• From the Jobs Service, showing GRPC calls to the Link Service when jobs are triggered.
• Additional labels or legends:
• Annotations to call out elements like “OAuth 2.0,” “GRPC,” “Kafka Streams,” etc., to ensure clarity on the communication mechanisms.