# chainbase-labs/manuscript-core architecture

> Architecture diagram of the GitHub repository chainbase-labs/manuscript-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chainbase-labs/manuscript-core
- Repository: https://github.com/chainbase-labs/manuscript-core

## Overview

##### Project Type and Purpose
- Manuscript is a multi-component data processing and transformation toolkit, designed to standardize access and processing of blockchain data across various formats and services.
- It provides a Command-Line Interface (CLI) written in Go for script-based interactions and orchestration and a Graphical User Interface (GUI) written in Rust for a more visual and interactive experience.
- On the backend, there is a Flink-based processing engine (in Java) that runs defined ETL operations using plugins for data sources, sinks, and transformations.
- The README emphasizes interoperability, programmability, and monetization within the Chainbase ecosystem, making it both an integration layer and a data processing framework.

##### Analyzing the File Structure
- Top-level directories include:
- .github: CI/CD configurations along with issue and pull request templates.
- cli: Contains all CLI-related Go code, subdivided into client code (API clients for integrations like chatGPT, Flink, Gemini etc.), commands implementation, and supporting packages (for configuration, Docker commands, parsing, etc.). There are clear modular boundaries (e.g., “client”, “commands”, “pkg”) hinting at separation of concerns in the CLI layer.
- examples: Provides multiple docker-compose and manuscript.yaml configuration examples that illustrate different use cases or deployment scenarios.
- flink: Contains a complete Apache Flink runner implementation, including a Maven-based Java project with ETL source, sink, transform plugins – which represents the actual stream processing backend.
- gui: A Rust-based project (with Cargo.toml and Rust source files) that implements the graphical user interface.
- Additional files like Makefile, install scripts, and static assets (manuscript templates, Docker Compose templates) support building and deployment.
- The presence of test files, configuration templates, and deployment-related scripts further validates a well-structured layered architecture (client interfaces, processing engine, orchestration/deployment).

##### Key Architectural Components
a. User Interfaces:
- CLI Client (Go): Entry-point for users via terminal, executing commands (commands folder) and handling configuration and manuscript scripts.
- GUI Client (Rust): Provides a visual interface for data selection, task monitoring, and orchestration.
b. Workflow & Configuration:
- Manuscript Script (manuscript.yaml): Defines the data processing workflow, outlining sources, transformation logic, and sinks.
- Docker Compose Files: Present in examples and within the project to simulate or deploy the processing environment.
c. Processing Engine:
- Flink Cluster/Runner (Java): Implements the actual streaming ETL pipeline. Shows components like source plugins, transformation modules, sink plugins, and utility libraries.
- External integrations with various third-party data systems (Postgres, S3, etc.) as mentioned in the manuscript’s value description.
d. Supporting Components:
- Configuration and State Handling: Common utilities within cli/pkg for manifest configuration, document state and parsing.
- CI/CD Pipelines (GitHub workflows) that ensure continuous integration and release management.

##### Relationships and Interactions
- Users interact with the system primarily through the CLI and GUI.
- CLI commands trigger manuscript processing tasks, such as deployment, initialization, or version management.
- The manuscript.yaml scripts and Docker Compose files drive how data sources are ingested and processed.
- The CLI/GUI interfaces call into:
- Local container orchestration (via Docker/Docker Compose) to spin up test or production Flink clusters.
- The Flink processing engine, which handles data ingestion from defined sources, applies transformation pipelines (with plugins), and forwards results to sinks (files, databases, etc.).
- The modular structure in the CLI (distinct packages for Docker operations, configuration parsing, command definitions) ensures separation of duties and easier maintenance.

##### Final Recommendations
- Every component is clearly labeled with its technology stack (e.g., “CLI (Go)”, “GUI (Rust)”, “Flink Runner (Java/Maven)”).
- From user input via the CLI/GUI to configuration parsing.
- From configuration to deployment steps.
- Through the pipeline: data ingestion → data processing (ETL via Flink) → data sinking (databases, files, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Actor
    User("User"):::actor

    %% User Interfaces Subgraph
    subgraph "User Interfaces"
        CLI("CLI (Go)"):::ui
        GUI("GUI (Rust)"):::ui
    end

    %% Configuration & Deployment Subgraph
    subgraph "Configuration & Deployment"
        Manuscript("Manuscript Script (YAML)"):::config
        Docker("Docker Compose Environment"):::config
    end

    %% Processing Engine Subgraph
    subgraph "Processing Engine"
        Flink("Flink Processing Engine (Java)"):::backend
    end

    %% Supporting Components
    CICD("CI/CD Pipelines"):::cicd
    External("External Data Sources/Services"):::external

    %% Connections from User to Interfaces
    User -->|"initiates"| CLI
    User -->|"initiates"| GUI

    %% Interfaces to Configuration
    CLI -->|"sends_workflow"| Manuscript
    GUI -->|"sends_workflow"| Manuscript
    CLI -->|"orchestrates"| Docker

    %% Configuration to Deployment and Processing Engine
    Manuscript -->|"triggers"| Docker
    Docker -->|"deploys"| Flink

    %% Processing Engine to External Data Sources
    Flink -->|"communicates_with"| External

    %% CI/CD Pipelines triggering builds
    CICD -->|"triggers_build"| CLI
    CICD -->|"triggers_build"| GUI
    CICD -->|"triggers_build"| Flink

    %% Click Events for CLI components
    click CLI "https://github.com/chainbase-labs/manuscript-core/blob/main/cli/main.go"
    click CLI "https://github.com/chainbase-labs/manuscript-core/tree/main/cli/commands"
    click CLI "https://github.com/chainbase-labs/manuscript-core/tree/main/cli/pkg"

    %% Click Event for GUI client
    click GUI "https://github.com/chainbase-labs/manuscript-core/tree/main/gui/"

    %% Click Events for Manuscript Script
    click Manuscript "https://github.com/chainbase-labs/manuscript-core/blob/main/examples/*/manuscript.yaml"
    click Manuscript "https://github.com/chainbase-labs/manuscript-core/blob/main/cli/static/manuscript_template.go"

    %% Click Events for Docker Compose Environment
    click Docker "https://github.com/chainbase-labs/manuscript-core/blob/main/examples/*/docker-compose.yaml"
    click Docker "https://github.com/chainbase-labs/manuscript-core/blob/main/flink/docker-compose.yaml"

    %% Click Event for Flink Processing Engine
    click Flink "https://github.com/chainbase-labs/manuscript-core/tree/main/flink/runner"

    %% Click Event for CI/CD Pipelines
    click CICD "https://github.com/chainbase-labs/manuscript-core/tree/main/.github/workflows"

    %% Styles
    classDef actor fill:#ffccbc,stroke:#bf360c,stroke-width:2px;
    classDef ui fill:#d1e8ff,stroke:#0277bd,stroke-width:2px;
    classDef config fill:#fff2cc,stroke:#d6a700,stroke-width:2px;
    classDef backend fill:#dcedc8,stroke:#558b2f,stroke-width:2px;
    classDef external fill:#f8bbd0,stroke:#c2185b,stroke-width:2px;
    classDef cicd fill:#e1bee7,stroke:#8e24aa,stroke-width:2px;
```
