# vicent-dev/turbo_ids architecture

> Architecture diagram of the GitHub repository vicent-dev/turbo_ids, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vicent-dev/turbo_ids
- Repository: https://github.com/vicent-dev/turbo_ids

## Overview

##### Project Type and Purpose
- This is a Go-based, command-line exporter tool (an open-source utility) whose core function is to connect to a MongoDB instance, stream out matching rows according to user-supplied “row checkers,” and write the results to files in parallel.

##### File Structure Analysis
- Top-level Go files:
- main.go: CLI entry point and orchestrator
- env.go: loads environment variables or .env config
- storage.go: MongoDB connection, query construction, row dispatch, concurrency management
- rowCheckers.go: user-definable filters that decide if a row should be exported
- file.go: file writer(s), CSV or other output formats
- Additional files:
- .env.dist – sample environment var template
- Makefile – build/run targets
- docs/ – supporting images or supplementary docs
- .github/workflows/go.yml – CI pipeline to build/test

##### Key Architecture Insights from README
- Multi-threaded exporter: storage instantiates worker goroutines that pull from MongoDB and push to file writers.
- Extensibility points: new filters go into rowCheckers.go; query criteria adjusted in storage.go constructor.
- External dependency: MongoDB cluster (connection string via env).

a. Main Components
1. CLI Frontend (main.go + Makefile/flags)
2. Configuration Loader (env.go)
3. Storage Manager (storage.go):
– MongoDB Client
– Query Builder
– Row Dispatcher (goroutines + channel)
4. Row Checker Module (rowCheckers.go)
5. File Output Module (file.go)
6. External Service: MongoDB
b. Relationships & Data Flow
- User invokes CLI → config is loaded → storage.NewStorage establishes Mongo session → kicks off worker pool
- Each worker queries a batch from MongoDB → hands raw documents to rowChecker(s)
- Filtered records are sent over a channel to fileWriter goroutines → persisted to disk
c. Architectural Patterns
- Worker Pool / Pipeline: decouples I/O (DB reads) from CPU-bound filtering and file writes
- Dependency Injection for storage constructor (allows swapping criteria)
- Plugin-style extensibility: row checkers implement a common signature and can be registered dynamically
d. Key Technologies
- Go modules (go.mod)
- MongoDB Go Driver
- Goroutines & Channels
- CSV/JSON writers in file.go

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Core Modules"
        CLI["CLI Frontend"]:::core
        CONFIG["Configuration Loader"]:::core
        STORAGE["Storage Manager"]:::core
        FILE["File Output Module"]:::core
    end

    CHECKER["Row Checker Module"]:::extension

    subgraph "Worker Pool"
        direction TB
        WORKER1["Worker Goroutine"]:::core
        WORKER2["Worker Goroutine"]:::core
    end

    CHANNEL[(Channel)]:::queue
    EXTERNAL["MongoDB"]:::external
    CI["Continuous Integration Pipeline"]:::ci

    CLI -->|"invoke CLI"| CONFIG
    CONFIG -->|"load env"| STORAGE
    STORAGE -->|"establish session & query"| EXTERNAL
    EXTERNAL -->|"Query Results"| STORAGE
    STORAGE -->|"dispatch jobs"| WORKER1
    STORAGE -->|"dispatch jobs"| WORKER2
    WORKER1 -->|"document"| CHECKER
    WORKER2 -->|"document"| CHECKER
    CHECKER -->|"filtered rows"| CHANNEL
    CHANNEL -->|"write request"| FILE
    CI -.-> STORAGE

    click CLI "https://github.com/vicent-dev/turbo_ids/blob/main/main.go"
    click CLI "https://github.com/vicent-dev/turbo_ids/tree/main/Makefile"
    click CONFIG "https://github.com/vicent-dev/turbo_ids/blob/main/env.go"
    click CONFIG "https://github.com/vicent-dev/turbo_ids/blob/main/.env.dist"
    click STORAGE "https://github.com/vicent-dev/turbo_ids/blob/main/storage.go"
    click CHECKER "https://github.com/vicent-dev/turbo_ids/blob/main/rowCheckers.go"
    click FILE "https://github.com/vicent-dev/turbo_ids/blob/main/file.go"
    click CI "https://github.com/vicent-dev/turbo_ids/blob/main/.github/workflows/go.yml"

    classDef core fill:#d5f5e3,stroke:#27ae60,stroke-width:1px
    classDef external fill:#aed6f1,stroke:#2980b9,stroke-width:1px
    classDef extension fill:#fcf3cf,stroke:#f1c40f,stroke-width:1px
    classDef queue fill:#eeeeee,stroke:#7f8c8d,stroke-width:1px,stroke-dasharray: 2 2
    classDef ci fill:#f5b7b1,stroke:#c0392b,stroke-width:1px
```
