# yelp/detect-secrets architecture

> Architecture diagram of the GitHub repository yelp/detect-secrets, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yelp/detect-secrets
- Repository: https://github.com/yelp/detect-secrets

## Overview

Based on our analysis of the file tree and README, the project is an open-source Python tool designed for scanning code for secrets. It provides multiple command‐line utilities (such as scanning for new secrets, auditing an existing baseline, and functioning as a pre-commit hook) and includes a variety of built-in plugins and filters. The overall goal is to prevent secrets from entering the codebase by providing an automated means of detecting and later auditing these secrets (for example, via a “baseline” file).

##### Main Components
- Command-Line Interface (CLI):
– This is the entry point that handles user commands (e.g., “scan”, “audit”, “detect-secrets-hook”) and arguments.
– Located in files like detect_secrets/__main__.py and main.py.
- Core Scanning Engine:
– Contains modules for scanning files, maintaining the secrets baseline (see detect_secrets/core/scan.py, baseline.py), and running heuristics.
– Manages the orchestration of scanning processes.
- Plugin System:
– A major component where different secret detection plugins are located (see detect_secrets/plugins/).
– Plugins include a variety of secret detectors (e.g., AWSKeyDetector, GitHubTokenDetector) that follow a common interface.
– Includes special plugins for regex, entropy detection, and keyword detection.
- Audit Module:
– Responsible for labeling and verifying potential secrets, updating the baseline (found in detect_secrets/audit/).
– Supports an interactive audit process to separate false positives from true secrets.
- Filter System:
– Contains modules and functionalities to filter out false positives (located in detect_secrets/filters/).
– Able to process inline comments (like “pragma: allowlist secret”) as well as command-line options to exclude specific patterns.
- Utility and Support Modules:
– Various utility modules (detect_secrets/util/) handle file I/O, git integration, semantic versioning, and other supporting functionalities.
- Configuration and Settings:
– Settings and configuration utilities (e.g., detect_secrets/settings.py) allow customization, including specifying plugins and filters at runtime.
- Testing and Documentation:
– Tests are separated into directories (tests/ and testing/) to ensure the components work together.
– Documentation (docs/) provides design rationale, usage examples, plugin directions, and filtering mechanisms.

##### Relationships and Interactions
- The CLI layer passes user commands to the core scanning engine.
- The core engine coordinates with the plugin system to scan input files and identify potential secrets.
- For each file processed, the engine applies various plugins (which are dynamically loaded) as well as security filters to reduce false positives.
- The Audit module consumes the output from scanning and lets users interactively mark secrets (true/false positives) enabling iterative improvements.
- Utility modules (for git operations, file path handling, etc.) are called by both the scanning engine and plugins.
- External integrations such as pre-commit hooks (configured via .pre-commit-config.yaml) interact with the scanning functionality by running baseline comparisons.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Notice how scanning, auditing, plugin management, and utility functions are separated into distinct modules.
- Modular and Extensible Design: The plugin system is designed so that new detectors can be added without modifying the core logic.
- Configurability: Settings, including plugin configuration and filter options, are adjusted via a consistent configuration object (via the “transient_settings” or “default_settings” context managers).
- Test-Driven Organization: A comprehensive test suite (across tests/ directories) reinforces adherence to design contracts.

##### Relevant Technologies, Frameworks, and Libraries
- Python as the central programming language.
- CLI frameworks by using Python’s built-in libraries (argparse) or similar modules.
- Git integration (for scanning only tracked files) and pre-commit hooks.
- External libraries such as pyahocorasick (for wordlist filtering) and a gibberish detection model via the “gibberish-detector” package.

## Diagram (Mermaid)

```mermaid
flowchart TD
    CLI["Command-Line Interface (CLI)"]:::CLI
    CORE["Core Scanning Engine"]:::CORE
    PLUGIN["Plugin System"]:::PLUGIN
    FILTER["Filter System"]:::FILTER
    AUDIT["Audit Module"]:::AUDIT
    UTIL["Utility & Support Modules"]:::UTIL
    CONFIG["Configuration & Settings"]:::CONFIG
    TESTDOC["Testing & Documentation"]:::TESTDOC

    CLI -->|"triggers"| CORE
    CORE -->|"invokes"| PLUGIN
    CORE -->|"applies"| FILTER
    CORE -->|"reports"| AUDIT
    CONFIG -->|"configures"| CORE
    CONFIG -->|"configures"| PLUGIN
    CONFIG -->|"configures"| FILTER
    UTIL -->|"supports"| CORE
    UTIL -->|"supports"| PLUGIN

    %% Click Events for Command-Line Interface (CLI)
    click CLI "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/__main__.py"
    click CLI "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/main.py"
    click CLI "https://github.com/yelp/detect-secrets/blob/master/.pre-commit-config.yaml"
    click CLI "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/pre_commit_hook.py"

    %% Click Events for Core Scanning Engine
    click CORE "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/core/scan.py"
    click CORE "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/core/baseline.py"
    click CORE "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/core/potential_secret.py"
    click CORE "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/core/secrets_collection.py"

    %% Click Events for Plugin System
    click PLUGIN "https://github.com/yelp/detect-secrets/tree/master/detect_secrets/plugins/"
    click PLUGIN "https://github.com/yelp/detect-secrets/tree/master/detect_secrets/core/plugins/"

    %% Click Event for Audit Module
    click AUDIT "https://github.com/yelp/detect-secrets/tree/master/detect_secrets/audit/"

    %% Click Event for Filter System
    click FILTER "https://github.com/yelp/detect-secrets/tree/master/detect_secrets/filters/"

    %% Click Events for Utility and Support Modules
    click UTIL "https://github.com/yelp/detect-secrets/tree/master/detect_secrets/util/"
    click UTIL "https://github.com/yelp/detect-secrets/tree/master/detect_secrets/transformers/"

    %% Click Event for Configuration and Settings
    click CONFIG "https://github.com/yelp/detect-secrets/blob/master/detect_secrets/settings.py"

    %% Click Events for Testing and Documentation
    click TESTDOC "https://github.com/yelp/detect-secrets/tree/master/tests/"
    click TESTDOC "https://github.com/yelp/detect-secrets/tree/master/docs/"

    %% Styles
    classDef CLI fill:#f2a900,stroke:#333,stroke-width:2px;
    classDef CORE fill:#b0e0e6,stroke:#333,stroke-width:2px;
    classDef PLUGIN fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef AUDIT fill:#ffe4b5,stroke:#333,stroke-width:2px;
    classDef FILTER fill:#ffb6c1,stroke:#333,stroke-width:2px;
    classDef UTIL fill:#d3d3d3,stroke:#333,stroke-width:2px;
    classDef CONFIG fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef TESTDOC fill:#e6e6fa,stroke:#333,stroke-width:2px;
```
