How vicent-dev/exporter works
Project type and purpose
• This is a single-binary Go CLI tool (“exporter”) that connects to a SQL database, runs an arbitrary query, and writes the full result set as a CSV file. It is neither full-stack nor microservices; it’s a focused, high-performance command-line application.
File structure
• Top-level files:
• main.go – entry point, flag parsing, orchestration
• env.go – loads environment variables/configuration
• storage.go – abstracts the database snapshot and query execution
• file.go – handles CSV file creation and writing
• Makefile, .env.dist – build and default config templates
• .github/workflows/go.yml – CI pipeline
• go.mod/go.sum capture dependencies; no sub-directories implies a monolithic structure.
• No external services beyond the SQL database; no web server or UI components.
README for additional insights
• CLI flags: -f (filename), -d (directory), -q (SQL query)
• Uses Go’s goroutines to parallelize chunks of the result set.
• Creates a consistent database snapshot, divides the full row count into ranges, and spawns X workers (where X = CPU cores) to fetch and write each chunk concurrently.
a. Main components:
1. CLI Interface (main.go)
2. Configuration Loader (env.go)
3. Database Client & Snapshot Manager (storage.go)
4. Chunk Manager & Worker Pool (storage.go + main.go)
5. CSV Writer (file.go)
6. Local Filesystem (output .csv)
b. Relationships & interactions:
• User invokes CLI → CLI parses flags → Config Loader reads env → initialize DB client
• DB client creates snapshot → Query “SELECT COUNT(*)” to get total rows
• Chunk Manager divides [0…total] into N ranges → Worker Pool of goroutines
• Each Worker pulls its range from DB, streams rows → sends to CSV Writer
• CSV Writer writes rows sequentially or via buffered channel to output file
• After all workers finish → Snapshot is torn down → Tool exits with CSV file on disk
c. Architectural patterns & principles:
• Single-responsibility modules (env loader, storage, file writer)
• Concurrency pattern: worker pool limiting parallel DB queries by CPU count
• Streaming I/O: avoids buffering entire result in memory
• Use of database snapshot to ensure consistent reads across workers
d. Key technologies:
• Go runtime (goroutines, channels)
• Go SQL driver (database/sql)
• CSV encoder (encoding/csv)
• OS file I/O (os package)
• Environment configuration (os.Getenv or a small helper)