How talkaboutcybersecurity/gitmonitor works
Project Type and Purpose
• GitMonitor is a Python‐based CLI tool for continuously discovering GitHub repositories of interest and scanning them for sensitive information, then reporting findings (via Slack).
• It is not a web app but a background service scheduled (e.g., by cron) that uses two rule sets:
• Searching rules (YAML) to find repositories on GitHub
• Filtering rules (regex) to detect secrets in cloned repos
High-Level Components
a. Configuration Loader
• Reads src/config.ini (and/or environment variables for GitHub credentials and Slack webhook).
b. Rule Manager
• Loads “searching” YAML files from src/rules
• Loads “filtering” regex patterns from src/libs/regexs.py
c. GitHub API Client (src/libs/github.py)
• Queries GitHub search endpoints by keywords, languages, file patterns, commits
d. Repository Manager (src/libs/directory.py)
• Clones matched repos to local src/source directory
• Deletes local copy after scanning
e. Scanner Engine (src/libs/rules.py & src/libs/utils.py)
• Applies regex patterns on cloned code files
• Records matches, categorizes by rule
f. Reporter (src/libs/slack.py)
• Constructs Slack messages using Slack Incoming Webhook
• Posts start‐scan, per‐repo results, and summary banners
g. State Persister
• Reads/writes JSON logs in src/log directory to track which repos were already scanned
Relationships & Data Flows
1. Cronjob → Invoke gitmonitor.py
2. Config Loader → Rule Manager → loads YAML + regex sets
3. GitHub Client → “Searching” rules → GitHub REST API → list of new repos
4. For each repo:
a. Repository Manager → git clone → local path
b. Scanner Engine → apply filtering regex → produce findings
c. State Persister → update JSON log
d. Repository Manager → cleanup local clone
5. Reporter → aggregates per‐repo findings → sends Slack messages
Architectural Patterns & Principles
• Modular design: each responsibility (config, API client, scanning, reporting) is isolated in src/libs
• Rule‐based engine: plug‐and‐play YAML for search, regex for filtering
• Stateless core: temporary clones are cleaned up, persistent state limited to JSON logs
• Cron‐driven scheduling: externalized orchestration without long‐lived processes
Technology & Frameworks
• Python 3.x
• PyYAML (for parsing search rules)
• requests (for GitHub & Slack REST calls)
• Git CLI (for cloning)
• Slack Incoming Webhooks
Essential Elements Checklist
• Call out scheduling (cron) as the trigger