# talkaboutcybersecurity/gitmonitor architecture

> Architecture diagram of the GitHub repository talkaboutcybersecurity/gitmonitor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/talkaboutcybersecurity/gitmonitor
- Repository: https://github.com/talkaboutcybersecurity/gitmonitor

## Overview

##### Project Type and Purpose
- GitMonitor is a Python‐based CLI tool for continuously discovering GitHub repositories of interest and scanning them for sensitive information, then reporting findings (via Slack).
- It is not a web app but a background service scheduled (e.g., by cron) that uses two rule sets:
- Searching rules (YAML) to find repositories on GitHub
- Filtering rules (regex) to detect secrets in cloned repos

##### High-Level Components
a. Configuration Loader
- Reads src/config.ini (and/or environment variables for GitHub credentials and Slack webhook).
b. Rule Manager
- Loads “searching” YAML files from src/rules
- Loads “filtering” regex patterns from src/libs/regexs.py
c. GitHub API Client (src/libs/github.py)
- Queries GitHub search endpoints by keywords, languages, file patterns, commits
d. Repository Manager (src/libs/directory.py)
- Clones matched repos to local src/source directory
- Deletes local copy after scanning
e. Scanner Engine (src/libs/rules.py & src/libs/utils.py)
- Applies regex patterns on cloned code files
- Records matches, categorizes by rule
f. Reporter (src/libs/slack.py)
- Constructs Slack messages using Slack Incoming Webhook
- Posts start‐scan, per‐repo results, and summary banners
g. State Persister
- Reads/writes JSON logs in src/log directory to track which repos were already scanned

##### Relationships & Data Flows
1. Cronjob → Invoke gitmonitor.py
2. Config Loader → Rule Manager → loads YAML + regex sets
3. GitHub Client → “Searching” rules → GitHub REST API → list of new repos
4. For each repo:
a. Repository Manager → `git clone` → local path
b. Scanner Engine → apply filtering regex → produce findings
c. State Persister → update JSON log
d. Repository Manager → cleanup local clone
5. Reporter → aggregates per‐repo findings → sends Slack messages

##### Architectural Patterns & Principles
- Modular design: each responsibility (config, API client, scanning, reporting) is isolated in src/libs
- Rule‐based engine: plug‐and‐play YAML for search, regex for filtering
- Stateless core: temporary clones are cleaned up, persistent state limited to JSON logs
- Cron‐driven scheduling: externalized orchestration without long‐lived processes

##### Technology & Frameworks
- Python 3.x
- PyYAML (for parsing search rules)
- requests (for GitHub & Slack REST calls)
- Git CLI (for cloning)
- Slack Incoming Webhooks

##### Essential Elements Checklist
- Call out scheduling (cron) as the trigger

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Trigger
    Cron["Cronjob Trigger"]:::external
    Cron -->|invokes| GitMonitor["gitmonitor.py"]:::internal

    %% Configuration Loader
    subgraph "Configuration Loader"
        ConfigINI["config.ini"]:::storage
        ConfigParser["Config Parser"]:::internal
    end
    GitMonitor --> ConfigINI
    GitMonitor --> ConfigParser

    %% Rule Manager
    subgraph "Rule Manager"
        SearchRules["Searching Rules Loader"]:::internal
        FilteringRules["Filtering Rules Loader"]:::internal
    end
    ConfigINI --> SearchRules
    ConfigParser --> FilteringRules

    %% Rule Engines
    subgraph "Rule Engines"
        SearchRules --> GitHubClient["GitHub API Client"]:::internal
        FilteringRules --> ScannerEngine["Scanner Engine"]:::internal
    end

    %% External Services
    GitHubClient -->|queries| GitHubAPI["GitHub API"]:::external
    SlackWebhook["Slack Webhook"]:::external

    %% Repository Loop
    subgraph "Per-Repository Loop"
        direction TB
        Clone["Clone Repository"]:::internal
        LocalStorage["Local Clone Storage"]:::storage
        Clone --> LocalStorage
        ScannerEngine --> ScannerUtils["Scanner Utilities"]:::internal
        ScannerUtils -->|scans| LocalStorage
        ScannerUtils --> Findings["Findings"]
        Findings --> StateStore["JSON Log Store"]:::storage
        Findings --> Reporter["Reporter (Slack)"]:::internal
        StateStore --> Cleanup["Cleanup Clone"]:::internal
        Cleanup -->|removes| LocalStorage
        Reporter -->|posts| SlackWebhook
    end

    GitHubAPI --> Clone
    GitHubClient --> Clone

    %% Legend
    subgraph Legend
        direction LR
        A["Internal Module"]:::internal
        B["External Service"]:::external
        C["Storage"]:::storage
    end

    %% Click Events
    click GitMonitor "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/gitmonitor.py"
    click ConfigINI "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/config.ini"
    click ConfigParser "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/parse/config.py"
    click SearchRules "https://github.com/talkaboutcybersecurity/gitmonitor/tree/master/src/rules/"
    click FilteringRules "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/libs/regexs.py"
    click GitHubClient "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/libs/github.py"
    click Clone "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/libs/directory.py"
    click ScannerEngine "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/libs/rules.py"
    click ScannerUtils "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/libs/utils.py"
    click Reporter "https://github.com/talkaboutcybersecurity/gitmonitor/blob/master/src/libs/slack.py"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef external fill:#90EE90,stroke:#333,stroke-width:1px
    classDef storage fill:#D3D3D3,stroke:#333,stroke-width:1px
```
