How vicent-dev/turbo_ids works
Project Type and Purpose
• This is a Go-based, command-line exporter tool (an open-source utility) whose core function is to connect to a MongoDB instance, stream out matching rows according to user-supplied “row checkers,” and write the results to files in parallel.
File Structure Analysis
• Top-level Go files:
• main.go: CLI entry point and orchestrator
• env.go: loads environment variables or .env config
• storage.go: MongoDB connection, query construction, row dispatch, concurrency management
• rowCheckers.go: user-definable filters that decide if a row should be exported
• file.go: file writer(s), CSV or other output formats
• Additional files:
• .env.dist – sample environment var template
• Makefile – build/run targets
• docs/ – supporting images or supplementary docs
• .github/workflows/go.yml – CI pipeline to build/test
Key Architecture Insights from README
• Multi-threaded exporter: storage instantiates worker goroutines that pull from MongoDB and push to file writers.
• Extensibility points: new filters go into rowCheckers.go; query criteria adjusted in storage.go constructor.
• External dependency: MongoDB cluster (connection string via env).
a. Main Components
1. CLI Frontend (main.go + Makefile/flags)
2. Configuration Loader (env.go)
3. Storage Manager (storage.go):
– MongoDB Client
– Query Builder
– Row Dispatcher (goroutines + channel)
4. Row Checker Module (rowCheckers.go)
5. File Output Module (file.go)
6. External Service: MongoDB
b. Relationships & Data Flow
• User invokes CLI → config is loaded → storage.NewStorage establishes Mongo session → kicks off worker pool
• Each worker queries a batch from MongoDB → hands raw documents to rowChecker(s)
• Filtered records are sent over a channel to fileWriter goroutines → persisted to disk
c. Architectural Patterns
• Worker Pool / Pipeline: decouples I/O (DB reads) from CPU-bound filtering and file writes
• Dependency Injection for storage constructor (allows swapping criteria)
• Plugin-style extensibility: row checkers implement a common signature and can be registered dynamically
d. Key Technologies
• Go modules (go.mod)
• MongoDB Go Driver
• Goroutines & Channels
• CSV/JSON writers in file.go