How bishopfox/raink works
Project type and purpose
• This is an open-source command-line tool (written in Go) named “raink” that uses large language models (LLMs) via the OpenAI API to perform document ranking. It inputs a list of items (sentences, files, etc.), splits them into batches, issues batched API calls in parallel, and then aggregates the scored results into a ranked output.
File structure
• Top-level files:
– main.go: the single entrypoint implementing CLI flags, batching logic, concurrency, API calls, and output formatting
– go.mod / go.sum: Go module definitions
– README.md / LICENSE / .gitignore: project metadata
• Directories:
– img/: logo assets for the README
– testdata/: sample input files (sentences.txt, tlds.txt) used in examples or tests
• Configuration: Reads OPENAI_API_KEY from the environment.
• No separate “pkg” or “internal” directories – everything lives in main.go, so treat it as a monolithic CLI process with discrete functional modules.
Key architectural insights from the README
– The tool handles non-determinism, context window limits, and output constraints by:
a. Splitting input into fixed-size batches (flag: -s)
b. Running multiple randomized “runs” (flag: -r) to smooth out nondeterminism
c. Parallelizing OpenAI calls for each run
d. Aggregating and scoring results numerically
– Outputs JSON arrays of { value, score, id } for downstream processing (e.g., piping to jq).
Components & Interactions
a. Main Components
1. CLI Frontend (User)
2. Input Loader (reads file or directory)
3. Batch Manager (splits items into batches)
4. Concurrency Controller (goroutine pool or worker pool)
5. OpenAI API Adapter (HTTP client, handles API key, retries)
6. Response Processor (parses LLM output, extracts scores)
7. Aggregator & Scorer (collects all runs, computes final ranking)
8. Output Formatter (JSON emitter to stdout)
9. Configuration Module (reads flags, env vars)
10. (Optional) Logger / Metrics collector
b. Relationships & Data Flow
– User invokes “raink” → Configuration Module parses flags & ENV
– Input Loader reads the specified file(s) → passes list of items to Batch Manager
– Batch Manager divides items into N batches per run
– Concurrency Controller spins up workers → each worker:
→ calls OpenAI API Adapter with a batch + system prompt
→ receives raw JSON/stream from LLM
→ Response Processor extracts “value” + numeric “score”
– Aggregator collects results from all workers & runs → merges per‐item scores
– Output Formatter emits the final ranked list to stdout (JSON)
– (Optionally) Logger records API latencies, errors
c. Architectural Patterns & Design Principles
– Modular layering: separate file I/O, batching, API communication, processing, and output
– Concurrency via worker pools (goroutines + channels)
– Resilience: retries on API errors, backpressure via batch size tuning
– Configurability: flags for prompt, run count, batch size
– Single-process monolith (no microservices), but with clear internal module boundaries
d. Key Technologies & Libraries
– Go standard library: flag, io, os, net/http, encoding/json, sync
– OpenAI HTTP REST API (v1/chat/completions)
– jq (downstream consumer example)