How gregsh/parallel-zip works
Project type and purpose
• This is a CLI tool (open-source utility) that runs on the JVM. It provides a “parallel zip” implementation in both Java and Kotlin, leveraging multiple cores to compress input files concurrently into a single ZIP archive far faster than the standard sequential approach. It also includes a merging capability (recombine existing ZIPs without recompression) and supports both a fat-jar and a GraalVM native image distribution.
File structure
• Top-level build files and config
• build.gradle.kts, gradlew, gradlew.bat, gradle/wrapper/…
• LICENSE.md, README.md, .gitignore
• Source folders
• src/main/java/parallelZip
– MainJava.java – Java entry point for parallel mode
– Sequential.java – Java entry point for the sequential comparison
• src/main/kotlin/parallelZip
– MainKotlin.kt – Kotlin entry point for parallel mode
– SingleEntryZipOutputStream.kt – helper to write one entry without a central directory
– ZipInputStreamEx.kt – extension to merge existing zips without recompression
• Resources
• META-INF/native-image/reflect-config.json – GraalVM reflection config
Key README insights
• Algorithm
1. Parallel phase: for each file, spin up a worker that
a. Wraps a ByteArrayOutputStream with SingleEntryZipOutputStream
b. Streams file bytes into that stream (skips central directory)
c. Stores (ZipEntry, compressed bytes) in a concurrent map
2. Sequential phase: single thread writes all pre-compressed entries’ raw bytes into a real ZipOutputStream, then closes to emit the central directory
• Uses Java Reflection to inject entries and offsets into java.util.zip.ZipOutputStream internals
• Performance numbers: ~8× speedup on multi-core
• Supports fat-jar and GraalVM native-image packaging
a. Main components
• CLI Launcher
– MainJava/MainKotlin
– Parses CLI args (output ZIP path + input files/dirs)
• File Enumerator
– Recursively lists all inputs
• Parallel Compressor Pool
– Thread-pool or parallel stream workers
– SingleEntryZipOutputStream + ByteArrayOutputStream per file
– Outputs (ZipEntry metadata + compressed byte array)
• In-Memory Buffer
– ConcurrentHashMap<ZipEntry, byte[]>
– Backpressure mechanism (e.g., blocking queue) if implemented
• Sequential Writer
– ZipOutputStream wrapping FileOutputStream
– Reflective injection of XEntry and offset
– Writes raw bytes from buffer, closes stream to flush central directory
• Merge Module (optional)
– ZipInputStreamEx reads existing zips, extracts entry headers + raw data pointers
• Packaging/Build
– Gradle → fat-jar or GraalVM native image
b. Relationships and interactions
1. CLI Launcher invokes File Enumerator
2. File list fed into Parallel Compressor Pool
3. Each compressor worker writes into In-Memory Buffer
4. Once compression phase completes (or incrementally), Sequential Writer drains the buffer in entry order
5. Final ZIP output on disk
6. (Optional) Merge Module can feed data directly into Sequential Writer without recompressing
c. Architectural patterns and design principles
• Producer-Consumer (parallel compression producers, sequential writer consumer)
• Single Responsibility (separate classes for entry streaming, CLI parsing, merge logic)
• Reflection as an Extension Point (to work around missing ZIP API)
• Immutable data handoff (ByteArrayOutputStream content + ZipEntry metadata)
d. Key technologies/frameworks
• JVM languages: Java 17+ and Kotlin
• java.util.zip APIs
• Kotlin Coroutines or Java parallel streams / ExecutorService
• Gradle with Kotlin DSL
• GraalVM native-image for AOT compilation