# securityriskadvisors/repoharvester architecture

> Architecture diagram of the GitHub repository securityriskadvisors/repoharvester, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/securityriskadvisors/repoharvester
- Repository: https://github.com/securityriskadvisors/repoharvester

## Overview

##### Project Type Identification
This is a command-line utility tool written in Go that harvests email addresses from GitHub repositories' commit histories. The project is relatively simple in structure but involves concurrent operations and external system interactions.

##### Main Components

a. Input Layer:
- Command-line interface (flags and arguments parsing)
- Input validation for target types (user/org/url)
- Configuration parameters (working directory, git path, etc.)

b. Core Processing Components:
- Repository Queue Manager
- Queue size configuration
- Worker pool management
- Repository Harvester
- Concurrent workers (configurable)
- Git command executor
- Email Extractor
- Commit parsing logic
- Email collection and deduplication

c. External Interactions:
- GitHub API Interface
- Repository listing
- API rate limiting handling
- Git Binary Interface
- Local git command execution
- Repository cloning

d. Output Handlers:
- JSON output formatter
- Flat file output formatter
- Console logger (verbose/quiet modes)

##### Specific Elements

**[CLI Interface] → [Input Validator]**
- ↓
**[Config Manager] → [Worker Pool Manager]**
- ↓
- [Repository Queue] ←→ [GitHub API]
- ↓
- [Worker Processes] ←→ [Git Binary]
- ↓
- [Email Extractor]
- ↙     ↘
- [JSON Writer]    [Flat File Writer]

##### Important Details to Highlight

- Concurrent processing: Show the worker pool and how it manages multiple repository processing tasks
- External dependencies: Clearly mark the GitHub API and Git binary as external systems
- Configuration points: Indicate where size filters and fork filters are applied
- Output formats: Show the dual output paths (JSON and flat file)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input
        CLI[CLI Interface]
        IV[Input Validator]
        CM[Config Manager]
    end

    subgraph Core
        WPM[Worker Pool Manager]
        RQ[Repository Queue]
        WP[Worker Processes]
        EE[Email Extractor]
    end

    subgraph External
        GH[GitHub API]
        GB[Git Binary]
    end

    subgraph Output
        JW[JSON Writer]
        FF[Flat File Writer]
    end

    CLI --> IV
    IV --> CM
    CM --> WPM
    WPM --> RQ
    RQ <--> GH
    RQ --> WP
    WP <--> GB
    WP --> EE
    EE --> JW
    EE --> FF

    click CLI "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click IV "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click CM "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click WPM "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click RQ "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click WP "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click EE "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click JW "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"
    click FF "https://github.com/SecurityRiskAdvisors/repoharvester/blob/master/repoharvester.go"

    classDef input fill:#a2d2ff,stroke:#333,stroke-width:2px
    classDef core fill:#90be6d,stroke:#333,stroke-width:2px
    classDef external fill:#f9c74f,stroke:#333,stroke-width:2px
    classDef output fill:#c77dff,stroke:#333,stroke-width:2px

    class CLI,IV,CM input
    class WPM,RQ,WP,EE core
    class GH,GB external
    class JW,FF output

    subgraph Legend
        L1[Input Components]:::input
        L2[Core Processing]:::core
        L3[External Systems]:::external
        L4[Output Handlers]:::output
    end
```
