# cubefs/compass architecture

> Architecture diagram of the GitHub repository cubefs/compass, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cubefs/compass
- Repository: https://github.com/cubefs/compass

## Overview

##### Main Components
- Diagnostic Engine Modules: Several “task-” modules provide the core functionality.
- Task Application: Likely orchestrates diagnostic tasks and central application logic.
- Task Detect: Implements the detection of anomalies and failures (e.g., abnormal elapsed time, failed tasks) in schedulers or engines.
- Task Parser: Handles log parsing and application of the drain algorithm to aggregate log templates.
- Task Flink and Task Canal (and their corresponding “common” modules): Provide specialized logic for different technologies (Spark, Flink, MapReduce).
- Task GPT: Integrates ChatGPT to generate diagnostic suggestions from abnormal logs.
- Task Metadata, Task MBG, Task Syncer: Likely handle metadata management, code generation for mappers, and data synchronization between systems.
- User Interface (UI) Module:
- Task UI: The frontend for displaying dashboards, detailed reports, and realtime diagnosis for both scheduler and engine issues.
- Core Shared/Utility Components:
- Task Common: Captures shared domain models, utilities, and API definitions.
- Docker Playgrounds: Provide containerized configurations for big data ecosystems (Hadoop, DolphinScheduler, Spark, etc.) to emulate production environments.
- External Integrations & Supporting Services:
- External services like OpenSearch (for log/indexing purposes), Kafka (for message consumption/production), and databases via MyBatis configuration.
- ChatGPT integration for generating diagnostic advice.

##### Relationships and Interactions
- Data Flow and Communication:
- Log Collection: Logs are collected and fed into the parser (using the drain algorithm) to structure them into templates.
- Anomaly Detection: Various detection modules (in task-detect and task-flink) use heuristics and rules to analyze logs and performance metrics.
- Messaging and Integration: Modules use Kafka, Redis, and other configurations to exchange messages. KafkaConfig classes appear in many modules.
- Diagnostic Advice: The task-gpt module calls ChatGPT (via its ChatGPTClient and configuration) to receive suggestions based on parsed abnormal logs.
- Metadata and Sync: Task-syncer and task-metadata modules ensure that meta-information and log data are synchronized between systems.
- UI Dashboard: The task-portal (and task-ui) modules present dashboards and reporting views based on aggregated data from backend services.
- Data ingestion from external big data platforms into the log collection/parsing modules.
- Interactions between detection, parsing, and GPT integration layers.
- Data supply to the UI/portal modules for real-time and offline reporting.

##### Important Architectural Patterns
- Modular / Microservices Based:
- The project’s multi-module Maven structure suggests a decoupled design where different modules can be updated independently.
- Layered Architecture:
- Separate concerns among log data ingestion, parsing, anomaly detection, advice generation, and front-end presentation.
- Integration of External Services:
- Utilization of external systems (Kafka, Redis, OpenSearch, ChatGPT) shows an integration-heavy design.
- Use of Heuristic Rules and AI:
- The drain algorithm is implemented for log aggregation, with additional AI-driven advice from ChatGPT.

##### Relevant Technologies/Frameworks
- Java (Maven projects, Spring Boot likely used in backend modules)
- Kafka (messaging communication)
- Redis (for caching and inter-module state)
- MyBatis (ORM and database configuration)
- Docker (deployment and local playground environment)
- OpenSearch (log management/search)
- ChatGPT integration (external API calls for diagnostic suggestions)
- Vue.js (via task-ui for frontend interface) along with Vite, Pinia, etc.

##### Final Reminders
- Be very detailed – aim to map out as many components as possible, isolating each task module as a separate, interacting entity.
- Use the documentation provided (especially the Architecture and Deployment documents in “document/manual/”) as guides to include any additional nuances.
- Maintain clarity by grouping related modules (diagnosis, log parsing, UI, synchronization) into bounded contexts if possible.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data Sources
    subgraph "External Data Sources"
        E1["Hadoop, Spark, DolphinScheduler"]:::external
    end

    %% External Services
    subgraph "External Services"
        ES1["Kafka Cluster"]:::external
        ES2["Redis Cache"]:::external
        ES3["OpenSearch DB"]:::external
        ES4["ChatGPT API"]:::external
        ES5["MyBatis Config"]:::external
    end

    %% Backend Diagnostic Engine
    subgraph "Backend Diagnostic Engine"
        TA["Task Application"]:::backend
        TP["Task Parser"]:::backend
        TD["Task Detect"]:::backend
        TF["Task Flink"]:::backend
        TC["Task Canal"]:::backend
        TG["Task GPT"]:::backend
        TM["Task Metadata"]:::backend
        TMBG["Task MBG"]:::backend
        TS["Task Syncer"]:::backend
        TCOM["Task Common"]:::backend
    end

    %% UI Layer
    subgraph "UI Layer"
        TUI["Task UI"]:::frontend
        TPort["Task Portal"]:::frontend
    end

    %% Deployment and Documentation
    DP["Docker Playgrounds"]:::container
    Doc["Documentation & Manual"]:::documentation

    %% Data Flow Connections
    E1 -->|"logCollection"| TP

    TP -->|"parsedLogs"| TD
    TP -->|"parsedLogs"| TF
    TP -->|"parsedLogs"| TC

    TD -->|"anomalyData"| TG
    TF -->|"anomalyData"| TG
    TC -->|"anomalyData"| TG

    TD -->|"alerts"| TA
    TF -->|"alerts"| TA
    TC -->|"alerts"| TA

    TG -->|"suggestions"| TA

    TA -->|"sync/update"| TM
    TA -->|"sync/update"| TMBG
    TA -->|"sync/update"| TS

    %% Interaction with Shared Component
    TA --- TCOM
    TP --- TCOM
    TD --- TCOM
    TF --- TCOM
    TC --- TCOM
    TG --- TCOM

    %% UI Connections
    TA -->|"dataFeed"| TUI
    TA -->|"dataFeed"| TPort

    %% External Service Integrations
    TP -->|"queueLogs"| ES1
    TS -->|"cacheSync"| ES2
    TM -->|"storeMeta"| ES3
    TG -->|"callChatGPT"| ES4
    TA ---|"dbConfig"| ES5

    %% Deployment Connections
    DP -->|"deploys"| TA
    DP -->|"deploys"| TUI

    %% Documentation Reference
    TA ---|"guidance"| Doc

    %% Click Events
    click TA "https://github.com/cubefs/compass/tree/main/task-application"
    click TD "https://github.com/cubefs/compass/tree/main/task-detect"
    click TP "https://github.com/cubefs/compass/tree/main/task-parser"
    click TF "https://github.com/cubefs/compass/tree/main/task-flink"
    click TC "https://github.com/cubefs/compass/tree/main/task-canal"
    click TG "https://github.com/cubefs/compass/tree/main/task-gpt"
    click TM "https://github.com/cubefs/compass/tree/main/task-metadata"
    click TMBG "https://github.com/cubefs/compass/tree/main/task-mbg"
    click TS "https://github.com/cubefs/compass/tree/main/task-syncer"
    click TUI "https://github.com/cubefs/compass/tree/main/task-ui"
    click TPort "https://github.com/cubefs/compass/tree/main/task-portal"
    click TCOM "https://github.com/cubefs/compass/tree/main/task-common"
    click DP "https://github.com/cubefs/compass/tree/main/docker/playground"
    click Doc "https://github.com/cubefs/compass/tree/main/document"

    %% Styles
    classDef backend fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef frontend fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef external fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef container fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef documentation fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
