How komalsh100/secrets-scanner works
This repository is a small Python command-line application organized around a single scanning pipeline with clear separation between detection logic, scanning orchestration, and output/reporting.
At a high level, the architecture is:
• A CLI boundary exposed as a console command (secrets-scanner)
• A core scanning engine that walks files or git history and produces findings
• A detection layer made of regex-based secret patterns plus entropy analysis
• A reporting layer that formats results for terminal, JSON, and HTML
• A CI/CD integration layer in GitHub Actions that runs the scanner and publishes results
Main subsystems
1. CLI entrypoint
Located in secrets_scanner/cli.py.
This is the user-facing boundary. It parses command-line flags such as:
• target path
• --git-history
• severity filtering
• --no-entropy
• output file paths for JSON/HTML
• CI failure flags like --fail-on-critical and --fail-on-high
Its job is to translate user intent into scanner configuration, invoke the engine, then present results and set the process exit code. In architecture terms, this is the presentation/orchestration layer, not where detection logic lives.
2. Core scanning engine
Located in secrets_scanner/scanner.py.
This is the heart of the repository. It is responsible for:
• traversing the target directory, or reading from git history when enabled
• applying ignore rules from .secretsignore
• reading file contents and evaluating them against detection rules
• aggregating findings and severity counts
• deciding whether the run should fail based on CLI policy
This module is the main boundary between input sources and security findings. It likely coordinates both types of detectors:
• regex/pattern matches
• entropy-based heuristic matches
The scanner is the main data-flow hub: it consumes files/commits and emits findings.
3. Pattern library
Located in secrets_scanner/patterns.py.
This is the rule catalog for deterministic detection. The README indicates 20+ secret types, including:
• AWS keys
• GitHub tokens
• Stripe keys
• Slack tokens
• JWTs and bearer tokens
• database connection strings
• generic API keys and hardcoded credentials
Architecturally, this module is a pure rules/data layer. It should be thought of as a catalog of regex definitions plus metadata such as severity and human-readable descriptions. The scanner depends on it, but it should not depend on the scanner.
4. Entropy analysis
Located in secrets_scanner/entropy.py.
This subsystem provides heuristic detection for strings that do not match a known secret pattern but still look suspicious, especially:
• high-entropy base64-like strings
• high-entropy hex-like strings
This is a second detector type alongside regex matching. It helps catch unknown or custom secrets and is usually optional because it costs more time than straight regex checks. The CLI exposes that boundary with --no-entropy.
5. Reporting layer
Located in secrets_scanner/reporter.py.
This module transforms internal findings into consumable outputs:
• human-readable terminal output
• JSON report for automation
• HTML report for artifact review
This is a downstream boundary from the scanner. It does not discover secrets; it formats scanner output for humans and pipelines. The README also implies that severity breakdowns and summary counts are generated here or with its help.
6. Package metadata and distribution
Files:
• setup.py
• secrets_scanner.egg-info/
• secrets_scanner/__init__.py
This repository is packaged as an installable Python CLI, likely using setuptools and a console-script entry point. The presence of egg metadata and the install instructions in the README suggest the command is installed into the environment and used from CI/CD or local shells.
7. Tests
Located in tests/test_scanner.py.
The tests appear focused on the scanning engine, which is appropriate because the core risk is false positives/false negatives in detection. The test boundary is likely centered on scanner behavior, pattern matching, ignore rules, and maybe entropy classification.
8. GitHub Actions workflow
Located in .github/workflows/secrets-scan.yml.
This is the external integration boundary for CI/CD. The workflow runs on pushes and pull requests and does the following:
• invokes the scanner on the repository
• likely enables git-history scanning
• generates JSON and HTML reports
• uploads artifacts
• posts a PR comment summary
• fails the build when configured thresholds are exceeded
This makes the scanner part of a security gate in the delivery pipeline rather than just a local developer tool.
Data flow
A typical scan follows this flow:
1. User or CI invokes the CLI
- Target path and options are passed into secrets_scanner.cli.
2. CLI constructs scan configuration
- Includes severity threshold, ignore behavior, git-history mode, entropy toggle, and output options.
3. Scanner collects input
- Reads filesystem contents or git commit history.
- Applies .secretsignore exclusions and any CLI excludes.
4. Detection runs
- Regex patterns are applied first or in parallel via patterns.py.
- If enabled, entropy.py flags suspicious high-entropy strings.
- Each match becomes a finding with file/path, line number, severity, type, and explanation.
5. Findings are aggregated
- The scanner computes totals by severity and overall scan summary.
6. Reporting occurs
- Console output is shown.
- Optional JSON and HTML reports are written by reporter.py.
7. Exit code is set
- The process exits non-zero if the configured severity threshold is violated, enabling CI enforcement.
Important architectural boundaries
Detection vs orchestration
The biggest separation is between:
• what counts as a secret: patterns.py and entropy.py
• how the repository is scanned: scanner.py
• how results are presented: cli.py and reporter.py
This keeps the engine reusable and the output layer replaceable.
Heuristic vs deterministic detection
There are two detection strategies:
• Regex-based: explicit known secret formats
• Entropy-based: probabilistic suspicion for unknown secrets
That distinction matters for architecture because it affects runtime cost, false positives, and explainability.
Local analysis vs git-history analysis
The README explicitly supports scanning:
• the current working tree
• all commits in git history
That means the scanner has two input modes and likely two internal traversal paths. In a diagram, this is a key fork before the detection layer.
Ignore policy boundary
.secretsignore acts as a policy filter between inputs and detection. It prevents the engine from wasting effort on excluded paths and reduces noise from tests, examples, and fixtures.
Human output vs machine output
The reporting layer serves two audiences:
• developers reading terminal output
• automation consuming JSON/HTML artifacts
That matters because CI behavior depends on machine-readable findings, while local usage depends on readable summaries.
Technologies and operational context
• Language/runtime: Python
• Packaging: setuptools-style installable CLI
• CI/CD: GitHub Actions
• Execution context: local developer machines and CI runners
• External system dependency: Git repository history when --git-history is enabled
No database, web server, or message broker is present. This is a self-contained analysis tool with filesystem and git as its primary inputs.
How to draw the architecture
If you are diagramming it, the cleanest layout is:
• Left side: inputs
- filesystem target
- git history
- .secretsignore
- CLI flags
• Center: scanner pipeline
- CLI parser
- scan orchestrator
- regex detector
- entropy detector
- ignore filter
- result aggregator
• Right side: outputs
- terminal summary
- JSON report
- HTML report
- exit code
- GitHub Actions PR comment/artifacts
The key design characteristic is that this repository is a single-purpose security scanning tool with a thin CLI shell around a reusable detection engine.
Main entry points: scanner.py Scanner · reporter.py Reporter · cli.py CLI · secrets-scan.yml GitHub Actions
Components 16
Inputs
Scan targets and policy inputs
- Target path · Repository or directory scanned by the tool.
- Git history · Optional commit history source when history scanning is enabled.
- .secretsignore · Path exclusion rules applied before detection.
.secretsignore
Core pipeline
CLI orchestration and detection engine
- CLI · Console command boundary that parses flags and starts scans.
secrets_scanner/cli.py - Scanner · Coordinates filesystem or git-history scanning and aggregates findings.
secrets_scanner/scanner.py - Patterns · Regex-based secret definitions and metadata.
secrets_scanner/patterns.py - Entropy · High-entropy string analysis for suspicious secrets.
secrets_scanner/entropy.py
Outputs
Human and machine reporting
- Reporter · Formats findings for terminal, JSON, and HTML outputs.
secrets_scanner/reporter.py - Console · Terminal summary and scan status for local use.
- JSON report · Structured results written for automation and CI artifacts.
- HTML report · Renderable report for review as a build artifact.
- Exit code · Process failure signal based on severity thresholds.
Delivery
Packaging, tests, and CI integration
- GitHub Actions · Workflow that runs scans on pushes and pull requests.
.github/workflows/secrets-scan.yml - Package · Installable Python CLI package metadata and entry point.
setup.py - Tests · Scanner-focused tests for detection and scan behavior.
tests/test_scanner.py - Project init · Package root used by the installed CLI.
secrets_scanner/__init__.py
Connections 20
- GitHub Actions → CLI: runs · CI invokes the scanner through the CLI.
- Package → CLI: exposes · Packaging installs the console command entry point.
- CLI → Scanner: starts · CLI translates flags into scanner configuration.
- CLI → Reporter: formats · CLI sends findings to the reporting layer.
- Scanner → Target path: scans · Primary filesystem target is traversed and read.
- Scanner → Git history: optionally scans · Git history is used when history mode is enabled.
- Scanner → .secretsignore: filters · Ignore policy excludes paths before analysis.
- Scanner → Patterns: matches · Scanner applies deterministic regex patterns.
- Scanner → Entropy: checks · Scanner applies entropy heuristics when enabled.
- Patterns → Scanner: defines · Pattern catalog supplies secret signatures.
- Entropy → Scanner: flags · Entropy detector contributes suspicious findings.
- Scanner → Reporter: emits · Findings and summaries are passed to reporting.
- Reporter → Console: prints · Human-readable terminal output is produced.
- Reporter → JSON report: writes · Structured JSON report is generated.
- Reporter → HTML report: writes · HTML artifact report is generated.
- Scanner → Exit code: decides · Severity policy determines the process exit status.
- GitHub Actions → JSON report: publishes · Workflow collects machine-readable results as artifacts.
- GitHub Actions → HTML report: publishes · Workflow collects HTML output as artifacts.
- Tests → Scanner: verifies · Tests exercise scanner behavior and detection outcomes.
- Project init → Package: supports · Package root participates in the installable module layout.