How googlecloudplatform/khi works
Main Components
• Backend Application – Written in Go (located under the “cmd”, “pkg”, “internal” directories). The backend handles log processing, inspection tasks, caching, token management, and API endpoints.
• Log Sources and External Services – The system integrates with Google Cloud Logging (and eventually other logging backends). It also supports connecting to Kubernetes clusters (GKE, Cloud Composer, etc.) and uses the Google Cloud SDK (“gcloud”) for authentication.
• Supporting Services – Modules such as caching (under pkg/common/cache), worker pool and task runners (pkg/common/worker, pkg/task), and various parsers (pkg/parser and pkg/source/*) for transforming raw logs into visualizable data.
• API Gateway/Server – Provided via “pkg/server” which exposes the backend’s business logic and API endpoints to the frontend.
• Docker/Deployment Side – The project includes Dockerfiles, a .devcontainer folder, and GitHub workflows to support containerization, CI/CD, and local development.
Relationships and Interactions
• The Frontend (Angular) communicates with the Backend through a RESTful API or websocket interface that is provided by the server component.
• The Backend aggregates logs by connecting to external sources (Google Cloud Logging, Kubernetes audit logs) and processes them using domain-specific packages (for example, log parsing, inspection tasks, history generation).
• Caching mechanisms (lru, sharding_cache) and in-memory state handling are used to handle the large volume of logs loaded into memory for interactive filtering.
• Worker pools and asynchronous task runners manage background processing for tasks such as log querying and metadata extraction.
• The containerized deployment (Dockerfiles, docker-compose, and devcontainer configuration) encapsulates both the backend and the frontend to run in consistent environments, enabling integrations with continuous integration pipelines.
Architectural Patterns and Design Principles
• Separation of Frontend and Backend: Clear separation is visible in the file tree with a dedicated “web” folder (for UI code) and multiple Go modules for backend logic.
• Modular Design: The “pkg” directory shows modular categories (document handling, inspections, parsing, logging, model definitions, etc.), which enable easy testing and maintenance.
• Use of Dependency Injection & Abstraction: Interfaces for token resolving, HTTP clients, and log parsers allow for decoupling and easier extension or switching of modules.
• Containerization and CI/CD: Dockerfiles and GitHub workflows reflect a modern design approach to deployment and testing.
• Rich in-memory processing: To support interactive filtering and fast visualization, logs are loaded into memory and manipulated with efficient caching and worker routines.
Relevant Technologies, Frameworks, and Libraries
• Programming Languages: Go for backend and TypeScript/Angular for frontend.
• Container Technologies: Docker, Docker Compose.
• CI/CD and Code Quality: GitHub Actions workflows, Makefile tasks, and linting tools (golangci-lint, markdownlint, etc.).
• External Services: Google Cloud Logging, Kubernetes API (for GKE, Cloud Composer, etc.), and possibly other cloud providers as mentioned.
Additional Detail and Professional Precision
• Be as detailed as possible – capture each domain (e.g., “inspection”, “model”, “parser”) as a sub-component or module within the backend.
• That the major flow starts at external log ingestion, then passes through data processing, and finally reaches the frontend UI.
• Optionally, include the internal test and development tools (like scripts, Makefile, test folders) to show how code quality and continuous integration reinforce the development process.