# lc/gau architecture

> Architecture diagram of the GitHub repository lc/gau, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lc/gau
- Repository: https://github.com/lc/gau

## Overview

##### Project Type and Purpose
- This is a command-line application written in Go that fetches known URLs from multiple external providers (AlienVault OTX, the Wayback Machine, Common Crawl, and URLScan). As a security and reconnaissance tool, it integrates with various external APIs to retrieve URL data for any given domain.
- The README outlines its usage, configuration options, and even provides Docker build instructions, pointing to a simple utility with a clear input (domain or domains) and output (fetched URLs).

##### File Structure
- At the top level, you have key configuration files (.gau.toml, Dockerfile, GitHub workflows, go.mod, etc.).
- The “cmd” directory contains the entry point (main.go) where execution starts.
- The “runner” directory (especially runner.go and associated flags/flags.go) is likely responsible for parsing command-line options and setting up the execution flow.
- The “pkg” directory holds the core functionality divided into sub-packages:
- “httpclient” – encapsulates HTTP logic used across the application.
- “output” – handles the formatting and presentation of the results.
- “providers” – contains subdirectories for each external URL source (commoncrawl, otx, urlscan, wayback) and a common provider aggregator. This separation suggests that each external API is encapsulated as its own component.

##### Identifying Main Components
a. CLI / Entry Point:
- Component: "Command-Line Interface (CLI)"
- Located in: cmd/gau/main.go
- Role: Accepts user input and flags.

b. Runner / Application Controller:
- Component: "Runner/Flag Parser"
- Located in: runner/runner.go, runner/flags/flags.go
- Role: Handles parsing of command line flags, setting up configuration, and initiating the workflow.

c. Providers:
- Component: "Providers Module"
- Located in: pkg/providers – with submodules:
- commoncrawl
- otx
- urlscan
- wayback
- Role: For a given domain, each provider is queried to fetch URL data. There’s also a “providers.go” file that likely aggregates results or handles provider selection.

d. HTTP Client:
- Component: "HTTP Client Module"
- Located in: pkg/httpclient/client.go
- Role: Centralized HTTP request logic used by provider modules to communicate with external services.

e. Output Processing:
- Component: "Output Module"
- Located in: pkg/output/output.go
- Role: Formatting and output (plain text vs. JSON) of fetched URLs, handling file writing if specified.

##### Relationships and Data Flow
- The user starts the application via the CLI. Data flows from the user input into the Runner component, which parses and validates flags and options.
- The Runner then coordinates with the Providers module. For each provider (e.g., otx, commoncrawl, etc.), the runner invokes methods that use the HTTP Client to perform API calls.
- Fetched data is then processed by the Output module, which formats the final list of URLs based on parameters like blacklist filters or output format (JSON/text).
- All interactions (from HTTP calls to provider aggregations) are decoupled into separate, modular packages, reflecting a clean separation of concerns.

##### Architectural Patterns and Principles
- Modularity: Each external service is encapsulated in its own sub-package within “providers”. This makes it easier to add or modify integrations.
- Configuration Over Convention: Uses configuration files with overrides via command-line flags.
- Separation of Concerns: Distinct responsibilities for CLI parsing, HTTP communication, provider integration, and output formatting.
- Dependency Injection: Although not explicit, the modular design suggests that components like the HTTP client can be injected across providers for uniform behavior (e.g., handling retries, timeouts).

##### Technologies and Frameworks
- Built in Go, relies on the standard Go toolchain (go.mod, go.sum indicate dependency management via Go Modules).
- Docker support is provided (Dockerfile and instructions in the README), indicating containerization is a deployment option.
- GitHub Actions workflows indicate CI/CD pipelines that probably validate and build the project.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Node
    CLI["Command-Line Interface (CLI)"]:::cli

    %% Runner subgraph
    subgraph "Runner Module"
        RunnerCtrl["Runner Controller"]:::runner
        RunnerFlags["Flag Parser"]:::runner
    end

    %% HTTP Client Node
    HTTP["HTTP Client Module"]:::http

    %% Providers subgraph
    subgraph "Providers Module"
        Providers_Agg["Provider Aggregator"]:::providers
        Provider_Common["CommonCrawl Provider"]:::providers
        Provider_OTX["AlienVault OTX Provider"]:::providers
        Provider_URLScan["URLScan Provider"]:::providers
        Provider_Wayback["Wayback Machine Provider"]:::providers
    end

    %% Output Module Node
    Output["Output Module"]:::output

    %% External Services subgraph
    subgraph "External Services"
        External_Common["CommonCrawl"]:::external
        External_OTX["AlienVault OTX"]:::external
        External_URLScan["URLScan"]:::external
        External_Wayback["Wayback Machine"]:::external
    end

    %% Connections
    CLI -->|"initiates"| RunnerCtrl
    CLI -->|"initiates"| RunnerFlags

    RunnerCtrl -->|"configures"| HTTP
    RunnerFlags -->|"parses-flags"| HTTP

    HTTP -->|"sends-HTTP-request"| Providers_Agg

    Providers_Agg -->|"delegates-query"| Provider_Common
    Providers_Agg -->|"delegates-query"| Provider_OTX
    Providers_Agg -->|"delegates-query"| Provider_URLScan
    Providers_Agg -->|"delegates-query"| Provider_Wayback

    Provider_Common -->|"fetches-URLs"| External_Common
    Provider_OTX -->|"fetches-URLs"| External_OTX
    Provider_URLScan -->|"fetches-URLs"| External_URLScan
    Provider_Wayback -->|"fetches-URLs"| External_Wayback

    Providers_Agg -->|"aggregates-results"| Output

    %% Click Events
    click CLI "https://github.com/lc/gau/blob/master/cmd/gau/main.go"
    click RunnerCtrl "https://github.com/lc/gau/blob/master/runner/runner.go"
    click RunnerFlags "https://github.com/lc/gau/blob/master/runner/flags/flags.go"
    click HTTP "https://github.com/lc/gau/blob/master/pkg/httpclient/client.go"
    click Providers_Agg "https://github.com/lc/gau/blob/master/pkg/providers/providers.go"
    click Provider_Common "https://github.com/lc/gau/tree/master/pkg/providers/commoncrawl"
    click Provider_OTX "https://github.com/lc/gau/blob/master/pkg/providers/otx/otx.go"
    click Provider_URLScan "https://github.com/lc/gau/tree/master/pkg/providers/urlscan"
    click Provider_Wayback "https://github.com/lc/gau/blob/master/pkg/providers/wayback/wayback.go"
    click Output "https://github.com/lc/gau/blob/master/pkg/output/output.go"

    %% Styles
    classDef cli fill:#f9c,stroke:#333,stroke-width:2px;
    classDef runner fill:#ccf,stroke:#333,stroke-width:2px;
    classDef http fill:#cfc,stroke:#333,stroke-width:2px;
    classDef providers fill:#fc9,stroke:#333,stroke-width:2px;
    classDef output fill:#9cf,stroke:#333,stroke-width:2px;
    classDef external fill:#ccc,stroke:#333,stroke-width:2px;
```
