# 4info/aerospike-loader-1 architecture

> Architecture diagram of the GitHub repository 4info/aerospike-loader-1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/4info/aerospike-loader-1
- Repository: https://github.com/4info/aerospike-loader-1

## Overview

##### Project type and purpose
- This is a **CLI-based, open-source data-loading tool** written in Java (Maven build).
- Purpose: Read user-provided DSV (delimiter-separated values) files and JSON schema/config files, parse them, and write data into an Aerospike cluster using the Aerospike Java client.

##### File structure
- Top-level script: `run_loader` – launches the Java application.
- `src/main/java/com/aerospike/load` – core modules:
- AerospikeLoad.java (entry point, orchestrator)
- Parser.java (DSV parser), AsWriterTask.java (writer threads), supporting classes (BinDefinition, ColumnDefinition, MappingDefinition, etc.)
- `config_files/` & `src/main/resources` – JSON config/schema and logging/property files.
- `doc/`, `example/` – documentation & sample inputs.
- Tests under `src/test` validating parsing, mappings, and end-to-end loading.

##### README for additional insights
- Command-line options (Apache Commons CLI) define hosts, ports, namespace, file locations, throughput limits, TLS settings, etc.
- Internal working: two thread pools (reader threads that parse DSV; writer threads that submit records to Aerospike).
- Dependencies: Aerospike Java client, commons-cli, log4j2, json-simple.

##### Main components & relationships
a. Components:
1. CLI Launcher (`run_loader` shell script)
2. Parameter Parser (Apache Commons CLI)
3. Configuration Loader (reads JSON schema into MappingDefinition, BinDefinition, etc.)
4. Reader Thread Pool (Parser tasks read DSV files)
5. Shared Queue or Task Handoff
6. Writer Thread Pool (AsWriterTask uses Aerospike Java client)
7. Aerospike Cluster (external service)
8. Logging subsystem (Log4j2)
b. Data & Control Flow:
- User runs `run_loader` → CLI Launcher invokes `AerospikeLoad.main()`
- Parameter Parser populates a `Parameters` object
- Config Loader builds in-memory metadata (`MappingDefinition`, `MetaDefinition`, `ColumnDefinition`)
- AerospikeLoad spins up reader threads: each thread uses `Parser` to read rows from DSV → emits record objects into a bounded queue
- Writer threads (`AsWriterTask`) consume from queue → call Aerospike Java client APIs to write records to the cluster
- Metrics (`PrintStat`, `Counter`) and logging report progress & errors
c. Architectural patterns & design principles:
- Producer-consumer pattern (reader vs. writer threads)
- Separation of concerns: CLI parsing, config mapping, data parsing, data writing
- Thread-safe queue for work handoff
- Externalized configuration (JSON, properties files)
d. Key technologies:
- Java 1.8+, Maven
- Aerospike Java client (v4.x)
- Apache Commons CLI
- log4j2 for logging
- json-simple for JSON parsing

##### Elements
- **Concurrency indicators** (e.g., “N reader threads”, “M writer threads”).
- **Fault-tolerance** or **error-handling** path (e.g., Error counter → abort when threshold reached).
- **Configuration sources**: JSON schema files, log4j2.properties, project.properties.

## Diagram (Mermaid)

```mermaid
flowchart TD
    CLI["CLI Launcher\n(run_loader)"]:::cli
    CLI -->|"invoke main()"| AP["Parameter Parser\n(Parameters)"]:::config

    subgraph "Configuration Loader"
        MappingDef["MappingDefinition"]:::config
        MetaDef["MetaDefinition"]:::config
        BinDef["BinDefinition"]:::config
        ColDef["ColumnDefinition"]:::config
    end

    AP -->|"parsed params"| MappingDef
    MappingDef -->|"metadata objects"| ReaderPool

    SampleConfig["Sample Config Files\n(config_files/)"]:::config
    ProjectProps["Project Properties\n(project.properties)"]:::config
    SampleConfig --> MappingDef
    ProjectProps --> MappingDef

    subgraph "Parsing Layer"
        ReaderPool["Reader Thread Pool\n(N reader threads)"]:::parsing
        ParserNode["Parser"]:::parsing
    end

    ReaderPool --> ParserNode
    ParserNode -->|"Parsed Record"| Queue["Shared Bounded Queue"]:::util

    Queue --> WriterPool["Writer Thread Pool\n(M writer threads)"]:::writing

    subgraph "Writing Layer"
        WriterPool
        AsWriter["AsWriterTask"]:::writing
        Constants["Aerospike Client Constants"]:::util
    end

    WriterPool --> AsWriter
    AsWriter -->|"Write API call"| Aerospike["Aerospike Cluster"]:::external

    Utils["Utility Functions\n(Utils)"]:::util
    ParserNode --> Utils
    AsWriter --> Utils

    subgraph "Metrics & Logging"
        PrintStat["PrintStat"]:::util
        Counter["Counter"]:::util
        Logging["Logging Subsystem\n(Log4j2)"]:::util
    end

    ReaderPool --> PrintStat
    WriterPool --> Counter
    LoggingConfig["Log4j2 Config\n(log4j2.properties)"]:::config
    LoggingConfig --> Logging

    click CLI "https://github.com/4info/aerospike-loader-1/tree/master/run_loader"
    click AP "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/Parameters.java"
    click MappingDef "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/MappingDefinition.java"
    click MetaDef "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/MetaDefinition.java"
    click BinDef "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/BinDefinition.java"
    click ColDef "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/ColumnDefinition.java"
    click SampleConfig "https://github.com/4info/aerospike-loader-1/tree/master/config_files/"
    click ProjectProps "https://github.com/4info/aerospike-loader-1/blob/master/src/main/resources/project.properties"
    click ParserNode "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/Parser.java"
    click Queue "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/AerospikeLoad.java"
    click AsWriter "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/AsWriterTask.java"
    click Constants "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/Constants.java"
    click Utils "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/Utils.java"
    click PrintStat "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/PrintStat.java"
    click Counter "https://github.com/4info/aerospike-loader-1/blob/master/src/main/java/com/aerospike/load/Counter.java"
    click LoggingConfig "https://github.com/4info/aerospike-loader-1/blob/master/src/main/resources/log4j2.properties"

    classDef config fill:#c8e6c9,stroke:#2e7d32,color:#2e7d32
    classDef parsing fill:#bbdefb,stroke:#1565c0,color:#1565c0
    classDef writing fill:#ffe0b2,stroke:#ef6c00,color:#e65100
    classDef util fill:#eceff1,stroke:#607d8b,color:#37474f
    classDef external fill:#e1bee7,stroke:#6a1b9a,color:#4a148c
    class CLI cli
    class AP,MappingDef,MetaDef,BinDef,ColDef,SampleConfig,ProjectProps,LoggingConfig config
    class ReaderPool,ParserNode parsing
    class WriterPool,AsWriter writing
    class Queue,Utils,PrintStat,Counter,Logging,Constants util
    class Aerospike external
```
