How 4info/aerospike-loader-1 works
Project type and purpose
• This is a CLI-based, open-source data-loading tool written in Java (Maven build).
• Purpose: Read user-provided DSV (delimiter-separated values) files and JSON schema/config files, parse them, and write data into an Aerospike cluster using the Aerospike Java client.
File structure
• Top-level script: run_loader – launches the Java application.
• src/main/java/com/aerospike/load – core modules:
• AerospikeLoad.java (entry point, orchestrator)
• Parser.java (DSV parser), AsWriterTask.java (writer threads), supporting classes (BinDefinition, ColumnDefinition, MappingDefinition, etc.)
• config_files/ & src/main/resources – JSON config/schema and logging/property files.
• doc/, example/ – documentation & sample inputs.
• Tests under src/test validating parsing, mappings, and end-to-end loading.
README for additional insights
• Command-line options (Apache Commons CLI) define hosts, ports, namespace, file locations, throughput limits, TLS settings, etc.
• Internal working: two thread pools (reader threads that parse DSV; writer threads that submit records to Aerospike).
• Dependencies: Aerospike Java client, commons-cli, log4j2, json-simple.
Main components & relationships
a. Components:
1. CLI Launcher (run_loader shell script)
2. Parameter Parser (Apache Commons CLI)
3. Configuration Loader (reads JSON schema into MappingDefinition, BinDefinition, etc.)
4. Reader Thread Pool (Parser tasks read DSV files)
5. Shared Queue or Task Handoff
6. Writer Thread Pool (AsWriterTask uses Aerospike Java client)
7. Aerospike Cluster (external service)
8. Logging subsystem (Log4j2)
b. Data & Control Flow:
• User runs run_loader → CLI Launcher invokes AerospikeLoad.main()
• Parameter Parser populates a Parameters object
• Config Loader builds in-memory metadata (MappingDefinition, MetaDefinition, ColumnDefinition)
• AerospikeLoad spins up reader threads: each thread uses Parser to read rows from DSV → emits record objects into a bounded queue
• Writer threads (AsWriterTask) consume from queue → call Aerospike Java client APIs to write records to the cluster
• Metrics (PrintStat, Counter) and logging report progress & errors
c. Architectural patterns & design principles:
• Producer-consumer pattern (reader vs. writer threads)
• Separation of concerns: CLI parsing, config mapping, data parsing, data writing
• Thread-safe queue for work handoff
• Externalized configuration (JSON, properties files)
d. Key technologies:
• Java 1.8+, Maven
• Aerospike Java client (v4.x)
• Apache Commons CLI
• log4j2 for logging
• json-simple for JSON parsing
Elements
• Concurrency indicators (e.g., “N reader threads”, “M writer threads”).
• Fault-tolerance or error-handling path (e.g., Error counter → abort when threshold reached).
• Configuration sources: JSON schema files, log4j2.properties, project.properties.