How cubefs/compass works
Main Components
• Diagnostic Engine Modules: Several “task-” modules provide the core functionality.
• Task Application: Likely orchestrates diagnostic tasks and central application logic.
• Task Detect: Implements the detection of anomalies and failures (e.g., abnormal elapsed time, failed tasks) in schedulers or engines.
• Task Parser: Handles log parsing and application of the drain algorithm to aggregate log templates.
• Task Flink and Task Canal (and their corresponding “common” modules): Provide specialized logic for different technologies (Spark, Flink, MapReduce).
• Task GPT: Integrates ChatGPT to generate diagnostic suggestions from abnormal logs.
• Task Metadata, Task MBG, Task Syncer: Likely handle metadata management, code generation for mappers, and data synchronization between systems.
• User Interface (UI) Module:
• Task UI: The frontend for displaying dashboards, detailed reports, and realtime diagnosis for both scheduler and engine issues.
• Core Shared/Utility Components:
• Task Common: Captures shared domain models, utilities, and API definitions.
• Docker Playgrounds: Provide containerized configurations for big data ecosystems (Hadoop, DolphinScheduler, Spark, etc.) to emulate production environments.
• External Integrations & Supporting Services:
• External services like OpenSearch (for log/indexing purposes), Kafka (for message consumption/production), and databases via MyBatis configuration.
• ChatGPT integration for generating diagnostic advice.
Relationships and Interactions
• Data Flow and Communication:
• Log Collection: Logs are collected and fed into the parser (using the drain algorithm) to structure them into templates.
• Anomaly Detection: Various detection modules (in task-detect and task-flink) use heuristics and rules to analyze logs and performance metrics.
• Messaging and Integration: Modules use Kafka, Redis, and other configurations to exchange messages. KafkaConfig classes appear in many modules.
• Diagnostic Advice: The task-gpt module calls ChatGPT (via its ChatGPTClient and configuration) to receive suggestions based on parsed abnormal logs.
• Metadata and Sync: Task-syncer and task-metadata modules ensure that meta-information and log data are synchronized between systems.
• UI Dashboard: The task-portal (and task-ui) modules present dashboards and reporting views based on aggregated data from backend services.
• Data ingestion from external big data platforms into the log collection/parsing modules.
• Interactions between detection, parsing, and GPT integration layers.
• Data supply to the UI/portal modules for real-time and offline reporting.
Important Architectural Patterns
• Modular / Microservices Based:
• The project’s multi-module Maven structure suggests a decoupled design where different modules can be updated independently.
• Layered Architecture:
• Separate concerns among log data ingestion, parsing, anomaly detection, advice generation, and front-end presentation.
• Integration of External Services:
• Utilization of external systems (Kafka, Redis, OpenSearch, ChatGPT) shows an integration-heavy design.
• Use of Heuristic Rules and AI:
• The drain algorithm is implemented for log aggregation, with additional AI-driven advice from ChatGPT.
Relevant Technologies/Frameworks
• Java (Maven projects, Spring Boot likely used in backend modules)
• Kafka (messaging communication)
• Redis (for caching and inter-module state)
• MyBatis (ORM and database configuration)
• Docker (deployment and local playground environment)
• OpenSearch (log management/search)
• ChatGPT integration (external API calls for diagnostic suggestions)
• Vue.js (via task-ui for frontend interface) along with Vite, Pinia, etc.
Final Reminders
• Be very detailed – aim to map out as many components as possible, isolating each task module as a separate, interacting entity.
• Use the documentation provided (especially the Architecture and Deployment documents in “document/manual/”) as guides to include any additional nuances.
• Maintain clarity by grouping related modules (diagnosis, log parsing, UI, synchronization) into bounded contexts if possible.