How bitbomdev/minefield works
Project Overview
• Type & Purpose: Minefield is a command line and server-based tool designed for rapid, offline processing and analysis of SBOMs (Software Bill Of Materials). It makes heavy use of roaring Bitmaps for graphing dependencies. The project is optimized for speed, security (air-gapped operation), scalability, and customizability.
• Key Use Cases: Ingesting SBOM files, caching and indexing package dependency data, running complex queries (including leaderboard, custom queries, and dependency intersections), and visualizing query results.
Analyzing the File Structure
• Top-Level Directories/Files:
• "cmd": Contains all main command entrypoints such as “server”, “ingest”, “cache”, “query”, “leaderboard”, and “llm”.
• "api": Contains API definitions (proto files are here), signaling that the system exposes a programmatic (RPC/HTTP) interface.
• "pkg": Contains the core business logic modules:
· "graph": Implements the graph engine including parsing, caching, storage—for managing dependencies using roaring Bitmaps.
· "storages": Provides different storage implementations (e.g., Redis, SQL) that the system can use.
· "tools": Contains helper components for various ingestion methods (e.g., for sbom, scorecard, vulnerability analysis).
· "weightedNACD": Contains algorithms and risk calculation tools.
· "utils": Provides miscellaneous utility functions.
• "testdata" & "e2e": Test folders to validate ingestion and query functionality.
• Supporting files: Dockerfile, docker-compose.yml, Makefile, and buf configuration files indicate containerization and proto generation for API.
Identifying Main Components and Their Relationships
• Client/CLI Interface:
• Entry Point (main.go) that passes commands to various subcommands (server, ingest, cache, query, leaderboard).
• Commands are organized under the "cmd" directory.
• API Server:
• cmd/server: Hosts the API server which could be invoked to expose functionalities programmatically.
• Uses generated API code (under "api", "gen/api") and proto definitions.
• Ingestion Module:
• cmd/ingest: Handles the ingestion functionality. This is subdivided further for different types of data (OSV, SBOM, and Scorecard).
• This module accepts SBOM files or directories and processes them before feeding into the graph and storage subsystems.
• Caching Mechanism:
• cmd/cache: Provides commands for caching processed data.
• Works closely with the graph module (pkg/graph) and storage implementations (pkg/storages which support SQL and Redis).
• Query Module:
• cmd/query: Contains various query functions including custom queries, glob searches, metadata retrieval, etc.
• Interacts with the graph engine to return dependency relationships, intersect queries, and more (utilizing pkg/graph logic).
• Leaderboard Module:
• cmd/leaderboard: Generates ranked lists of packages based on dependency counts.
• Offers different flavors of leaderboard results (custom queries, key-based).
• Internal Libraries and Tools:
• cmd/helpers: Provides common utility functions used by different commands.
• pkg/tools: Contains ingestion helpers, SBOM parsing, vulnerability analysis, and risk calculation algorithms.
• Data Storage & External Dependencies:
• pkg/storages: Abstracts different storage mechanisms (Redis, SQL).
• The README and architecture indicate the system operates in air-gapped mode, emphasizing local processing without internet dependency.
Architectural Patterns & Design Principles
• Modularity: Code is highly modular with clearly separated commands (each implemented as a submodule under “cmd”) and internal libraries (under “pkg”), making the system extensible.
• Layered Architecture: There’s a clear separation between user commands (CLI/API), business logic (graphing, ingestion, caching), and persistence (storage modules).
• Dependency Injection: Likely practiced in components such as “cmd/server/wire.go” (using wire_gen.go) to hook different dependencies.
• Air-Gapped Philosophy: All components are designed to function without external network calls, emphasizing local and secure data processing.
• Use of Protocol Buffers: Protobuf definitions under “api” and generated API code in “gen/api” indicate service communication might be based on RPC or gRPC.
• Additional Annotations: Optionally include notes such as “Air-Gapped Operation” near all modules to reinforce the secure, offline design philosophy.