# redis/riot architecture

> Architecture diagram of the GitHub repository redis/riot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/redis/riot
- Repository: https://github.com/redis/riot

## Overview

##### Main Components Identification
The system should be divided into these core components:

a) Core Framework Layer:
- Located in core/riot-core
- Contains base abstractions and utilities
- Handles job execution and processing

b) File Processing Layer:
- Located in core/riot-file
- Manages different file formats (CSV, JSON, XML)
- Handles I/O operations

c) Data Generation Layer:
- Located in core/riot-faker
- Handles synthetic data generation

d) Plugin System:
- Located in plugins/riot
- Contains specific implementations for different data sources/targets

##### Key Architectural Elements

a) Data Flow Architecture:
- Source Systems (Databases, Files, Faker)
- Processing Pipeline
- Target Systems (Redis instances)

b) Command Structure:
- Main Command Interface
- Specific Command Implementations (Import, Export, Replicate)

- RIOT CLI Interface                                   |

- ↓

- Core Command Processing                                   |

- Base Commands  |  | Job Management    |  | Progress      |  | Error Handling     |   |

- ↓

- Data Processing Pipeline                                   |

- Readers        |  | Processors        |  | Writers       |  | Transformers       |   |

- ↓

- Source/Target Adapters                                    |

- File Adapters  |  | Database Adapters |  | Redis Client  |  | Data Generators    |   |

##### Important Details

- Command flow paths
- Data transformation pipelines
- Error handling paths
- Configuration points
- Plugin extension points

##### Specific Features to Highlight

- Live replication capabilities
- Multiple format support
- Data generation features
- Database connectivity
- Redis cluster support

##### Integration Points

- Connections to external systems
- Protocol support (S3, Google Storage)
- Database connectivity options
- Redis deployment options (standalone/cluster)

Remember to show both the static architecture (components and their relationships) and the dynamic aspects (data flow and processing pipelines) of the system.

This will provide a comprehensive view of RIOT's architecture while maintaining clarity about its primary purpose as a data import/export tool for Redis.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph CLI["RIOT CLI Interface"]
        MainCmd["Main Command"]:::core
        click MainCmd "https://github.com/redis/riot/blob/main/core/riot-core/src/main/java/com/redis/riot/core/MainCommand.java"
    end

    subgraph Commands["Command Layer"]
        BaseCmd["Base Commands"]:::core
        JobCmd["Job Management"]:::core
        ProgHdl["Progress Handler"]:::core
        ErrHdl["Error Handler"]:::core
        click BaseCmd "https://github.com/redis/riot/blob/main/core/riot-core/src/main/java/com/redis/riot/core/AbstractJobCommand.java"
    end

    subgraph DataOps["Data Operations"]
        Import["Import Command"]:::cmd
        Export["Export Command"]:::cmd
        Replicate["Replicate Command"]:::cmd
        click Import "https://github.com/redis/riot/blob/main/plugins/riot/src/main/java/com/redis/riot/AbstractImportCommand.java"
        click Export "https://github.com/redis/riot/blob/main/plugins/riot/src/main/java/com/redis/riot/AbstractExportCommand.java"
        click Replicate "https://github.com/redis/riot/blob/main/plugins/riot/src/main/java/com/redis/riot/AbstractReplicateCommand.java"
    end

    subgraph Pipeline["Data Processing Pipeline"]
        Readers["Readers"]:::process
        Processors["Processors"]:::process
        Writers["Writers"]:::process
        Transform["Transformers"]:::process
        click Processors "https://github.com/redis/riot/tree/main/core/riot-core/src/main/java/com/redis/riot/core/processor/"
        click Writers "https://github.com/redis/riot/blob/main/plugins/riot/src/main/java/com/redis/riot/RedisWriterArgs.java"
        click Readers "https://github.com/redis/riot/blob/main/plugins/riot/src/main/java/com/redis/riot/RedisReaderArgs.java"
        click Transform "https://github.com/redis/riot/tree/main/plugins/riot/src/main/java/com/redis/riot/function/"
    end

    subgraph Adapters["Source/Target Adapters"]
        FileAdapt["File Adapters"]:::io
        DbAdapt["Database Adapters"]:::io
        RedisClient["Redis Client"]:::io
        DataGen["Data Generators"]:::io
        click FileAdapt "https://github.com/redis/riot/tree/main/core/riot-file/src/main/java/com/redis/riot/file/"
        click DataGen "https://github.com/redis/riot/blob/main/core/riot-faker/src/main/java/com/redis/riot/faker/FakerItemReader.java"
    end

    subgraph FileFormats["File Processing"]
        CSV["CSV Reader"]:::file
        JSON["JSON Reader"]:::file
        XML["XML Reader"]:::file
        click CSV "https://github.com/redis/riot/blob/main/core/riot-file/src/main/java/com/redis/riot/file/DelimitedReaderFactory.java"
        click JSON "https://github.com/redis/riot/blob/main/core/riot-file/src/main/java/com/redis/riot/file/JsonReaderFactory.java"
        click XML "https://github.com/redis/riot/blob/main/core/riot-file/src/main/java/com/redis/riot/file/XmlReaderFactory.java"
    end

    MainCmd --> BaseCmd
    BaseCmd --> JobCmd
    JobCmd --> ProgHdl
    JobCmd --> ErrHdl
    BaseCmd --> DataOps
    Import & Export & Replicate --> Pipeline
    Pipeline --> Adapters
    FileAdapt --> FileFormats
    
    %% Styling
    classDef core fill:#2374ab
    classDef cmd fill:#ff7f50
    classDef process fill:#4caf50
    classDef io fill:#ff9800
    classDef file fill:#9c27b0

    %% Legend
    subgraph Legend
        CoreComp["Core Components"]:::core
        CmdComp["Command Components"]:::cmd
        ProcComp["Processing Components"]:::process
        IOComp["I/O Components"]:::io
        FileComp["File Components"]:::file
    end
```
