How medialab/xan works
Project Type and Purpose
• This is a command-line CSV processing tool written in Rust. Its purpose is to efficiently manipulate CSV files—it can preview, filter, slice, aggregate, sort, join files, and even generate visualizations. In addition, it embeds its own expression language (handled in the “moonblade” submodule) that enables complex on-the-fly data transformations.
• The README emphasizes performance, low-memory usage, multithreading (parallelism), and support for very large files. It also shows that many commands have a consistent interface for CSV processing.
File Structure to Identify Architectural Layers
• Top-Level Project Files: Cargo.toml, Cargo.lock, and CI-related files (in .github/workflows) indicate a modern Rust project with automated testing and release management.
• Source Code Structure (src/):
– The src/main.rs file represents the entry point (CLI interface) where command-line arguments are parsed.
– The src/cmd folder contains individual modules for each command (for example: count.rs, view.rs, sort.rs, groupby.rs, etc.). This indicates a modular command-dispatching system.
– The src/moonblade directory holds the core of the custom expression language including the parser (grammar.pest), interpreter, functions, and aggregation logic. It’s the DSL engine behind complex filtering and mappings.
– Other core modules in src (collections, config, graph, json, etc.) suggest support libraries and helper components for data manipulation and possibly visualization (e.g., ratatui, scales).
• Documentation (docs/) and tests (tests/) directories add context on usage and robust testing practices.
Key Components
a. CLI / Application Entry Point:
– Represent the main executable (src/main.rs) which parses user inputs and routes commands.
b. Command Dispatch Layer:
c. Core CSV Processing & Transformation Engine:
– Show the “Core Logic” modules that handle CSV ingestion, stream processing, and output formatting. This includes modules like src/util.rs, src/json.rs, src/dates.rs, src/graph.rs, etc.
d. DSL (Moonblade) Engine:
• Parsing (using grammar.pest)
• Expression Evaluation/Interpreter (src/moonblade/interpreter.rs & parser.rs)
• Aggregators and function libraries (src/moonblade/agg, src/moonblade/functions.rs, etc.)
• Documentation generation for DSL (src/moonblade/doc/)
– Indicate that this component is integrated into many command modules (especially those such as filter, map, transform, and agg).
e. Supporting Data Structures / Collections:
f. Testing & CI:
– Show that there is a separate Testing module (tests directory) that runs integration tests for almost every command.
– Mention CI/CD pipelines (files in .github/workflows) that ensure code quality and deployment reliability.
g. External Integrations and I/O:
Defining the Relationships and Data Flow
• Each command module then communicates with the “Core CSV Processing Engine” for reading, processing, and writing CSV data.
• Both the “Core CSV Processing Engine” and “Moonblade Engine” might share and update data in “Supporting Data Structures.”
• Also, add a section showing “Testing & CI” interacting with the code base to auto-run tests and builds.
Architectural Patterns and Design Principles
• Modularity: Every CSV command is a separate, independent module (Microkernel-like plugin approach) that leverages shared libraries.
• Layered Architecture: Separation between user interface (CLI), business logic (command and CSV processing), DSL evaluation (moonblade), and supporting libraries.
• Efficient Pipeline Processing: Emphasis on stream processing, low-memory footprint, and multithreading can be noted.
• Extensibility: The DSL engine (Moonblade) is designed to be composable and extensible, thus encouraging future enhancements.