# googlecloudplatform/khi architecture

> Architecture diagram of the GitHub repository googlecloudplatform/khi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/khi
- Repository: https://github.com/googlecloudplatform/khi

## Overview

##### Main Components
- Backend Application – Written in Go (located under the “cmd”, “pkg”, “internal” directories). The backend handles log processing, inspection tasks, caching, token management, and API endpoints.
- Log Sources and External Services – The system integrates with Google Cloud Logging (and eventually other logging backends). It also supports connecting to Kubernetes clusters (GKE, Cloud Composer, etc.) and uses the Google Cloud SDK (“gcloud”) for authentication.
- Supporting Services – Modules such as caching (under pkg/common/cache), worker pool and task runners (pkg/common/worker, pkg/task), and various parsers (pkg/parser and pkg/source/*) for transforming raw logs into visualizable data.
- API Gateway/Server – Provided via “pkg/server” which exposes the backend’s business logic and API endpoints to the frontend.
- Docker/Deployment Side – The project includes Dockerfiles, a .devcontainer folder, and GitHub workflows to support containerization, CI/CD, and local development.

##### Relationships and Interactions
- The Frontend (Angular) communicates with the Backend through a RESTful API or websocket interface that is provided by the server component.
- The Backend aggregates logs by connecting to external sources (Google Cloud Logging, Kubernetes audit logs) and processes them using domain-specific packages (for example, log parsing, inspection tasks, history generation).
- Caching mechanisms (lru, sharding_cache) and in-memory state handling are used to handle the large volume of logs loaded into memory for interactive filtering.
- Worker pools and asynchronous task runners manage background processing for tasks such as log querying and metadata extraction.
- The containerized deployment (Dockerfiles, docker-compose, and devcontainer configuration) encapsulates both the backend and the frontend to run in consistent environments, enabling integrations with continuous integration pipelines.

##### Architectural Patterns and Design Principles
- Separation of Frontend and Backend: Clear separation is visible in the file tree with a dedicated “web” folder (for UI code) and multiple Go modules for backend logic.
- Modular Design: The “pkg” directory shows modular categories (document handling, inspections, parsing, logging, model definitions, etc.), which enable easy testing and maintenance.
- Use of Dependency Injection & Abstraction: Interfaces for token resolving, HTTP clients, and log parsers allow for decoupling and easier extension or switching of modules.
- Containerization and CI/CD: Dockerfiles and GitHub workflows reflect a modern design approach to deployment and testing.
- Rich in-memory processing: To support interactive filtering and fast visualization, logs are loaded into memory and manipulated with efficient caching and worker routines.

##### Relevant Technologies, Frameworks, and Libraries
- Programming Languages: Go for backend and TypeScript/Angular for frontend.
- Container Technologies: Docker, Docker Compose.
- CI/CD and Code Quality: GitHub Actions workflows, Makefile tasks, and linting tools (golangci-lint, markdownlint, etc.).
- External Services: Google Cloud Logging, Kubernetes API (for GKE, Cloud Composer, etc.), and possibly other cloud providers as mentioned.

##### Additional Detail and Professional Precision
- Be as detailed as possible – capture each domain (e.g., “inspection”, “model”, “parser”) as a sub-component or module within the backend.
- That the major flow starts at external log ingestion, then passes through data processing, and finally reaches the frontend UI.
- Optionally, include the internal test and development tools (like scripts, Makefile, test folders) to show how code quality and continuous integration reinforce the development process.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    externalCloud("Google Cloud Logging"):::external
    externalKube("Kubernetes API"):::external

    %% Backend Layer
    subgraph "Backend Layer"
        %% Log Processing & Parsing
        subgraph "Log Processing & Parsing"
            BP_parser("Log Parser (pkg/parser)"):::backend
            BP_source("Log Source Handler (pkg/source)"):::backend
            BP_rawlogs("Raw Logs (pkg/rawlogs)"):::backend
        end
        %% Inspection and Analysis
        subgraph "Inspection and Analysis"
            IN_inspect("Inspection Module (pkg/inspection)"):::backend
            IN_document("Document Handler (pkg/document)"):::backend
            IN_history("History & Model (pkg/model/history)"):::backend
        end
        %% Service/API Layer
        subgraph "Service/API Layer"
            API_server("API Server (pkg/server)"):::backend
            API_params("Parameters Module (pkg/parameters)"):::backend
            API_cmd("Main Command (cmd)"):::backend
        end
        %% Supporting Services
        subgraph "Supporting Services"
            SS_cache("Caching (pkg/common/cache)"):::backend
            SS_worker("Worker Pool (pkg/common/worker)"):::backend
            SS_token("Token Management (pkg/common/token)"):::backend
        end
    end

    %% Frontend Layer
    subgraph "Frontend Layer"
        FE("Frontend Application (web)"):::frontend
    end

    %% Deployment & CI/CD
    subgraph "Deployment & CI/CD"
        DEP_docker("Docker Configuration (Dockerfile)"):::deployment
        DEP_devcontainer("Dev Container (.devcontainer)"):::deployment
        DEP_cicd("CI/CD Pipeline (.github/workflows)"):::deployment
    end

    %% Data Flow Connections
    externalCloud -->|"feeds"| BP_parser
    externalKube -->|"feeds"| BP_source
    
    BP_parser -->|"sends data"| IN_inspect
    BP_source -->|"sends data"| IN_inspect
    BP_rawlogs -->|"sends raw logs"| IN_inspect

    IN_inspect -->|"analyzes"| API_server
    IN_document -->|"provides docs"| API_server
    IN_history -->|"supplies history"| API_server

    API_server -->|"exposes API"| FE

    API_server -->|"uses"| SS_cache
    API_server -->|"uses"| SS_worker
    API_server -->|"uses"| SS_token

    %% Deployment influences both Frontend and Backend
    DEP_docker ---|"builds"| API_server
    DEP_docker --- FE
    DEP_devcontainer ---|"configures"| API_server
    DEP_devcontainer --- FE
    DEP_cicd ---|"tests & deploys"| API_server
    DEP_cicd --- FE

    %% Styling Classes
    classDef backend fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef frontend fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef external fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef deployment fill:#fff3cd,stroke:#856404,stroke-width:2px;

    %% Click Events
    click BP_parser "https://github.com/googlecloudplatform/khi/tree/main/pkg/parser"
    click BP_source "https://github.com/googlecloudplatform/khi/tree/main/pkg/source"
    click BP_rawlogs "https://github.com/googlecloudplatform/khi/tree/main/pkg/rawlogs"
    click IN_inspect "https://github.com/googlecloudplatform/khi/tree/main/pkg/inspection"
    click IN_document "https://github.com/googlecloudplatform/khi/tree/main/pkg/document"
    click IN_history "https://github.com/googlecloudplatform/khi/tree/main/pkg/model/history"
    click API_server "https://github.com/googlecloudplatform/khi/tree/main/pkg/server"
    click API_params "https://github.com/googlecloudplatform/khi/tree/main/pkg/parameters"
    click API_cmd "https://github.com/googlecloudplatform/khi/tree/main/cmd"
    click SS_cache "https://github.com/googlecloudplatform/khi/tree/main/pkg/common/cache"
    click SS_worker "https://github.com/googlecloudplatform/khi/tree/main/pkg/common/worker"
    click SS_token "https://github.com/googlecloudplatform/khi/tree/main/pkg/common/token"
    click FE "https://github.com/googlecloudplatform/khi/tree/main/web"
    click DEP_docker "https://github.com/googlecloudplatform/khi/tree/main/Dockerfile"
    click DEP_devcontainer "https://github.com/googlecloudplatform/khi/blob/main/.devcontainer"
    click DEP_cicd "https://github.com/googlecloudplatform/khi/tree/main/.github/workflows"
```
