How alexpreynolds/sample works
Project Type Identification
This is a command-line utility tool focused on performing reservoir sampling on large text files. The architecture should reflect its primary purpose of efficient sampling with minimal memory usage.
Main Components
a. Input Processing Layer:
• Regular File Input (emphasize that stdin is not supported)
• mmap-based file access component
• Optional C stdio component (alternative implementation)
b. Core Processing Components:
• Reservoir Sampling Engine
• Mersenne Twister RNG (mt19937)
• Storage Manager (for byte offsets)
c. Output Processing Layer:
• Sample Output Generator
• Order Preservation Component (optional)
[Input File] → [File Access Layer (mmap/stdio)] →
• [Position Storage Manager] ↔ [Reservoir Sampling Engine] ↔ [MT19937 RNG] →
• [Sample Output Generator] → [Ordered Output (optional)] → [Final Output]
Component Details
a. File Access Layer:
• Two possible paths: mmap (primary) and stdio (secondary)
• Two-pass processing nature
• Byte offset storage instead of full line storage
b. Reservoir Sampling Engine:
• To Position Storage Manager
• Interaction with RNG
• Sampling modes (with/without replacement)
c. Storage Manager:
• 8-byte position storage
• Memory efficiency aspect
Important Flows to Highlight
• First Pass: File → mmap → Position Storage → RNG
• Second Pass: Position Storage → mmap → Output Generation
Technical Details to Note
• Version control information
• Build/test infrastructure
• File format support (BED, SAM, VCF, etc.)
• Configuration options
Implementation Details
• Relationship between:
• src/sample-library/mt19937.c
• src/bin/sample.c
• Files (mt19937.h, sample.h)
Performance Characteristics
A sidebar or notes showing:
• 2.25-2.75x faster than shuf for small k
• Memory efficiency metrics
• Trade-offs vs. traditional approaches