# kacakb/jsfinder architecture

> Architecture diagram of the GitHub repository kacakb/jsfinder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kacakb/jsfinder
- Repository: https://github.com/kacakb/jsfinder

## Overview

##### Project Type and Purpose
- Project Type: CLI utility / open‐source tool
- Purpose: Read a list of web‐page URLs (from a file or stdin), issue concurrent HTTP GET requests, parse each response to extract JavaScript file references, and write those URLs to an output file.

##### Main Architectural Components
a. CLI & Configuration Parser
– Parses flags: input source (–l or –read), concurrency (–c), output file (–o), verbose/silent (–s).
b. Input Reader
– Reads URLs from stdin or from the specified input file.
c. URL Dispatcher / Worker Pool Controller
– A buffered channel that holds incoming URLs.
– A pool of worker goroutines limited by the concurrency flag.
d. HTTP Fetcher (Workers)
– Each worker takes a URL, issues an HTTP GET via Go’s net/http client, returns the response body or an error.
e. JS Extractor (Parser)
– Applies a compiled regular expression against the HTML body to find `<script src=…>` and other JS‐link attributes.
f. Result Aggregator / Output Writer
– Deduplicates (optional) and writes found JS URLs to the output file (`output.txt` by default) or to a custom file.
g. Logger / Console Reporter
– Prints status, progress messages, or remains silent based on the verbose/silent flag.

##### Relationships & Data Flow
1. User runs `jsfinder` CLI →
2. CLI Parser reads flags →
3. Input Reader streams URLs →
4. URLs pushed into Dispatcher channel →
5. Worker Pool Controller spawns up to C concurrent workers →
6. Each HTTP Fetcher worker pulls a URL →
7. Worker calls Parser on the HTTP response →
8. Parsed JS URLs emitted to Result Aggregator →
9. Output Writer writes to file →
10. Logger emits status to console

##### Architectural Patterns & Design Principles
- Concurrency via Go routines + buffered channels (worker‐pool pattern)
- Single responsibility for each pipeline stage (input, fetch, parse, write)
- Flag‐driven configuration for flexibility
- Decoupled I/O and processing logic for testability and extensibility

##### Relevant Technologies & Libraries
- Go 1.20, `net/http` for requests
- `regexp` for JS‐URL extraction
- Go’s standard `flag` package for CLI options
- Built‐in file I/O (`os`, `bufio`) and logging (`fmt` / `log`)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    WebServers("(External Web Servers)"):::external

    %% Project Metadata
    subgraph "Project Metadata"
        direction TB
        GoMod["go.mod"]:::io
        Readme["README.md"]:::io
        License["LICENSE"]:::io
        Gitignore[".gitignore"]:::io
        CI[".github/workflows/shiftleft.yml"]:::io
        Funding[".github/FUNDING.yml"]:::io
    end

    %% Main CLI Tool
    subgraph "jsFinder CLI Tool" 
        direction TB
        CLI["CLI & Configuration Parser\n(flags: -l, -read, -c, -o, -s)"]:::config
        Input["Input Reader\n(file or stdin)"]:::io
        Dispatcher["URL Dispatcher /\nWorker Pool Controller"]:::process
        Fetcher["HTTP Fetcher (Worker)"]:::process
        Parser["JS Extractor (Parser)"]:::process
        Aggregator["Result Aggregator"]:::process
        Writer["Output Writer\n(output.txt)"]:::io
        Logger["Logger / Console Reporter"]:::process
    end

    %% Data Flows
    CLI -->|"parses flags"| Input
    Input -->|"streams URLs"| Dispatcher
    Dispatcher -->|"dispatch URL"| Fetcher
    Fetcher -->|"HTTP GET"| WebServers
    WebServers -->|"response body"| Fetcher
    Fetcher -->|"HTML body"| Parser
    Parser -->|"extracted JS URLs"| Aggregator
    Aggregator -->|"deduplicates"| Writer
    Writer -->|"writes URLs"| Writer
    Writer -->|"writes to file"| FileOut[/"output.txt"/]:::io
    CLI -->|"controls verbosity"| Logger
    Dispatcher -->|"spawn workers"| Logger
    Fetcher -->|"errors/status"| Logger
    Parser -->|"parsing status"| Logger
    Aggregator -->|"aggregation status"| Logger
    Writer -->|"write status"| Logger

    %% Click Events
    click CLI "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click Input "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click Dispatcher "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click Fetcher "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click Parser "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click Aggregator "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click Writer "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click Logger "https://github.com/kacakb/jsfinder/blob/main/jsfinder.go"
    click GoMod "https://github.com/kacakb/jsfinder/blob/main/go.mod"
    click Readme "https://github.com/kacakb/jsfinder/blob/main/README.md"
    click License "https://github.com/kacakb/jsfinder/tree/main/LICENSE"
    click Gitignore "https://github.com/kacakb/jsfinder/blob/main/.gitignore"
    click CI "https://github.com/kacakb/jsfinder/blob/main/.github/workflows/shiftleft.yml"
    click Funding "https://github.com/kacakb/jsfinder/blob/main/.github/FUNDING.yml"

    %% Styles
    classDef io fill:#D0E8FF,stroke:#005FA1,color:#005FA1;
    classDef process fill:#DFFFD0,stroke:#3A7F0D,color:#3A7F0D;
    classDef config fill:#FFE8B0,stroke:#A15F00,color:#A15F00;
    classDef external fill:#E0E0E0,stroke:#7F7F7F,color:#7F7F7F;
    class FileOut io
```
