# medialab/xan architecture

> Architecture diagram of the GitHub repository medialab/xan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/medialab/xan
- Repository: https://github.com/medialab/xan

## Overview

##### Project Type and Purpose
- This is a command-line CSV processing tool written in Rust. Its purpose is to efficiently manipulate CSV files—it can preview, filter, slice, aggregate, sort, join files, and even generate visualizations. In addition, it embeds its own expression language (handled in the “moonblade” submodule) that enables complex on-the-fly data transformations.
- The README emphasizes performance, low-memory usage, multithreading (parallelism), and support for very large files. It also shows that many commands have a consistent interface for CSV processing.

##### File Structure to Identify Architectural Layers
- Top-Level Project Files: Cargo.toml, Cargo.lock, and CI-related files (in .github/workflows) indicate a modern Rust project with automated testing and release management.
- Source Code Structure (src/):
– The src/main.rs file represents the entry point (CLI interface) where command-line arguments are parsed.
– The src/cmd folder contains individual modules for each command (for example: count.rs, view.rs, sort.rs, groupby.rs, etc.). This indicates a modular command-dispatching system.
– The src/moonblade directory holds the core of the custom expression language including the parser (grammar.pest), interpreter, functions, and aggregation logic. It’s the DSL engine behind complex filtering and mappings.
– Other core modules in src (collections, config, graph, json, etc.) suggest support libraries and helper components for data manipulation and possibly visualization (e.g., ratatui, scales).
- Documentation (docs/) and tests (tests/) directories add context on usage and robust testing practices.

##### Key Components
a. CLI / Application Entry Point:
– Represent the main executable (src/main.rs) which parses user inputs and routes commands.
b. Command Dispatch Layer:
c. Core CSV Processing & Transformation Engine:
– Show the “Core Logic” modules that handle CSV ingestion, stream processing, and output formatting. This includes modules like src/util.rs, src/json.rs, src/dates.rs, src/graph.rs, etc.
d. DSL (Moonblade) Engine:
- Parsing (using grammar.pest)
- Expression Evaluation/Interpreter (src/moonblade/interpreter.rs & parser.rs)
- Aggregators and function libraries (src/moonblade/agg, src/moonblade/functions.rs, etc.)
- Documentation generation for DSL (src/moonblade/doc/)
– Indicate that this component is integrated into many command modules (especially those such as filter, map, transform, and agg).
e. Supporting Data Structures / Collections:
f. Testing & CI:
– Show that there is a separate Testing module (tests directory) that runs integration tests for almost every command.
– Mention CI/CD pipelines (files in .github/workflows) that ensure code quality and deployment reliability.
g. External Integrations and I/O:

##### Defining the Relationships and Data Flow
- Each command module then communicates with the “Core CSV Processing Engine” for reading, processing, and writing CSV data.
- Both the “Core CSV Processing Engine” and “Moonblade Engine” might share and update data in “Supporting Data Structures.”
- Also, add a section showing “Testing & CI” interacting with the code base to auto-run tests and builds.

##### Architectural Patterns and Design Principles
- Modularity: Every CSV command is a separate, independent module (Microkernel-like plugin approach) that leverages shared libraries.
- Layered Architecture: Separation between user interface (CLI), business logic (command and CSV processing), DSL evaluation (moonblade), and supporting libraries.
- Efficient Pipeline Processing: Emphasis on stream processing, low-memory footprint, and multithreading can be noted.
- Extensibility: The DSL engine (Moonblade) is designed to be composable and extensible, thus encouraging future enhancements.

## Diagram (Mermaid)

```mermaid
graph TB
    %% CLI / Application Entry Point
    CLI["CLI/Entry Point"]:::cli

    %% Command Dispatch Layer
    subgraph "Command Dispatch Layer"
        CMD_COUNT["Count Command"]:::cmd
        CMD_FILTER["Filter Command"]:::cmd
        CMD_SORT["Sort Command"]:::cmd
        CMD_AGG["Aggregate Command"]:::cmd
    end

    %% Core CSV Processing & Transformation Engine
    subgraph "CSV Processing Core"
        UTIL["Util Module"]:::core
        JSON["JSON Module"]:::core
        DATES["Dates Module"]:::core
        GRAPH["Graph Module"]:::core
    end

    %% DSL (Moonblade) Engine
    subgraph "Moonblade Engine"
        DSL_GRAMMAR["DSL Parser (grammar.pest)"]:::dsl
        DSL_INTERPRETER["Interpreter"]:::dsl
        DSL_PARSER["Parser"]:::dsl
        DSL_FUNCTIONS["Functions"]:::dsl
        DSL_AGG["Aggregators"]:::dsl
    end

    %% Supporting Data Structures / Collections
    subgraph "Collections"
        COLL_UNION["Union-Find"]:::collection
        COLL_COUNTER["Counter"]:::collection
    end

    %% I/O Layer
    IO["I/O Layer: File Ingestion & Output"]:::io

    %% Testing & CI
    subgraph "Testing and CI"
        TESTS["Tests Suite"]:::testing
        CI["CI/CD Pipelines"]:::testing
    end

    %% Data Flow Connections
    CLI -->|"parses_command"| CMD_COUNT
    CLI -->|"parses_command"| CMD_FILTER
    CLI -->|"parses_command"| CMD_SORT
    CLI -->|"parses_command"| CMD_AGG

    CMD_COUNT -->|"processes_CSV"| UTIL
    CMD_FILTER -->|"processes_CSV"| JSON
    CMD_SORT -->|"processes_CSV"| DATES
    CMD_AGG -->|"processes_CSV"| GRAPH

    CMD_FILTER -->|"invokes_DSL"| DSL_INTERPRETER
    CMD_AGG -->|"invokes_DSL"| DSL_FUNCTIONS

    UTIL -->|"updates"| COLL_UNION
    JSON -->|"updates"| COLL_COUNTER
    DATES -->|"updates"| COLL_COUNTER
    GRAPH -->|"updates"| COLL_UNION

    UTIL -->|"outputs"| IO
    JSON -->|"outputs"| IO
    DATES -->|"outputs"| IO
    GRAPH -->|"outputs"| IO

    DSL_INTERPRETER -->|"modifies"| COLL_COUNTER

    TESTS -->|"tests"| CLI
    TESTS -->|"tests"| CMD_COUNT
    TESTS -->|"tests"| UTIL
    CI -->|"builds"| CLI

    %% Click Events
    click CLI "https://github.com/medialab/xan/blob/master/src/main.rs"
    click CMD_COUNT "https://github.com/medialab/xan/blob/master/src/cmd/count.rs"
    click CMD_FILTER "https://github.com/medialab/xan/blob/master/src/cmd/filter.rs"
    click CMD_SORT "https://github.com/medialab/xan/blob/master/src/cmd/sort.rs"
    click CMD_AGG "https://github.com/medialab/xan/blob/master/src/cmd/agg.rs"
    click UTIL "https://github.com/medialab/xan/blob/master/src/util.rs"
    click JSON "https://github.com/medialab/xan/blob/master/src/json.rs"
    click DATES "https://github.com/medialab/xan/blob/master/src/dates.rs"
    click GRAPH "https://github.com/medialab/xan/blob/master/src/graph.rs"
    click DSL_GRAMMAR "https://github.com/medialab/xan/blob/master/src/moonblade/grammar.pest"
    click DSL_INTERPRETER "https://github.com/medialab/xan/blob/master/src/moonblade/interpreter.rs"
    click DSL_PARSER "https://github.com/medialab/xan/blob/master/src/moonblade/parser.rs"
    click DSL_FUNCTIONS "https://github.com/medialab/xan/blob/master/src/moonblade/functions.rs"
    click DSL_AGG "https://github.com/medialab/xan/tree/master/src/moonblade/agg/"
    click COLL_UNION "https://github.com/medialab/xan/blob/master/src/collections/union_find.rs"
    click COLL_COUNTER "https://github.com/medialab/xan/blob/master/src/collections/counter.rs"
    click TESTS "https://github.com/medialab/xan/tree/master/tests/"
    click CI "https://github.com/medialab/xan/tree/master/.github/workflows/"

    %% Styling
    classDef cli fill:#ffcccc,stroke:#aa0000,stroke-width:2px;
    classDef cmd fill:#ccffcc,stroke:#008800,stroke-width:2px;
    classDef core fill:#cce5ff,stroke:#0044cc,stroke-width:2px;
    classDef dsl fill:#fff0b3,stroke:#cc9900,stroke-width:2px;
    classDef collection fill:#e6ccff,stroke:#6600cc,stroke-width:2px;
    classDef io fill:#d9ead3,stroke:#38761d,stroke-width:2px;
    classDef testing fill:#fce5cd,stroke:#b45f06,stroke-width:2px;
```
