# francisconeves97/jxscout architecture

> Architecture diagram of the GitHub repository francisconeves97/jxscout, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/francisconeves97/jxscout
- Repository: https://github.com/francisconeves97/jxscout

## Overview

##### Project Type & Purpose
- jxscout is an **open-source CLI tool** (written in Go) for capturing, organizing, and analyzing JavaScript assets (HTML, JS, source maps, bundles) flowing through a proxy (Burp or Caido).
- It exposes an HTTP server to receive requests, persists assets to disk/SQLite, and runs analysis modules (chunk discovery, AST analysis, beautifier, sourcemap reverse, ingestion, overrides).

##### High-Level Components
a. External Components
- Proxy Plugin (Burp / Caido) – forwards HTTP requests/responses to jxscout
- Bun & Prettier – JavaScript runtime & formatter for certain modules
- VSCode Extension – WebSocket client for AST visualization
b. jxscout Core
- HTTP Server (internal/core/websocket + cmd/jxscout/http.go)
- CLI/TUI Prompt (internal/core/tui + cmd/jxscout/main.go)
- Configuration & Logging (pkg/jxscout/*.go + SQLite under `~/.jxscout/`)
- Persistence
– Disk storage: `~/jxscout/<project>/…`
– Database: SQLite (internal/core/database)
- Event Bus
– DB eventbus (internal/core/dbeventbus)
– Core eventbus (internal/core/eventbus)
c. Processing Modules (internal/modules & pkg/*)
- Asset Fetcher & Service
- Chunk Discoverer
- Sourcemap Reverser
- Beautifier
- AST Analyzer
- HTML/JS Ingestion
- Git Committer, Overrides, Custom Ingestion, etc.

##### Relationships & Data Flow
1. Proxy Plugin → HTTP Server
– HTTP POST of request/response metadata & bodies
2. HTTP Server → Asset Service
– Saves raw asset to disk, records metadata in SQLite
– Emits “new asset” event on Core EventBus
3. Core EventBus → Module Orchestrator
– Listens for new‐asset events
– Dispatches jobs to various Modules (chunk‐discoverer, sourcemaps, AST analyzer, beautifier) via concurrent queues
4. Modules → Disk & Database
– Write enriched/processed artifacts back to disk (`~/jxscout/...`)
– Insert analysis results into SQLite
– Some modules invoke Bun (JS) or WASM code (sourcemaps) under the hood
5. CLI/TUI & VSCode Extension → Core
– CLI/TUI: Direct function calls into core services for commands (list assets, override, config)
– VSCode Extension: WebSocket client connects to internal/core/websocket to fetch AST result tree in real time

##### Architectural Patterns & Principles
- **Event-Driven Pipeline:** Core EventBus and DB EventBus decouple asset ingestion from processing modules.
- **Modular Plugin Architecture:** Each analysis/discovery task lives in its own `internal/modules/*` or `pkg/*` package.
- **Concurrency Controls:** Concurrency limits (flags) govern parallelism when fetching, beautifying, analyzing.
- **Separation of Concerns:** Clear layers for HTTP/API, persistence, business logic (asset service), and presentation (CLI/TUI, WebSocket).

##### Technologies & Frameworks
- Go standard library (HTTP server, CLI prompt)
- SQLite (via internal/core/database)
- Bun (JavaScript runtime) + Prettier for JS formatting
- WASM (sourcemaps parser)
- WebSocket (internal/core/websocket) for VSCode integration
- Concurrent queues & event buses (internal/core/eventbus, dbeventbus)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Swimlanes
    subgraph "External"
        direction TB
        ProxyPlugin["Proxy Plugin (Burp/Caido)"]:::external
        BunPrettier["Bun & Prettier"]:::external
    end

    subgraph "Clients"
        direction TB
        VSCodeExt["VSCode Extension"]:::client
        CLITUIClient["CLI/TUI"]:::client
    end

    subgraph "Core"
        direction TB
        HTTPServer["HTTP Server"]:::service
        WebSocketServer["WebSocket Server"]:::service
        CLIEntrypoint["CLI Entrypoint"]:::service
        ConfigLogging["Configuration & Logging"]:::service
        ApplicationCore["Application Core"]:::service
        FileService["File Storage Service"]:::service
        AssetRepo["Asset Repository"]:::service
        AssetService["Asset Ingestion Service"]:::service
        AssetFetcher["Asset Fetcher Service"]:::service
        SQLiteDB["SQLite Database"]:::storage
        CoreEventBus["Core EventBus"]:::service
        DBEventBus["DB EventBus"]:::service
        ConcurrentQueue["Shared Concurrency Queue"]:::service
    end

    subgraph "Modules"
        direction TB
        ASTAnalyzer["AST Analyzer"]:::module
        Beautifier["Beautifier"]:::module
        ChunkDiscoverer["Chunk Discoverer"]:::module
        HTMLIngestion["HTML Ingestion"]:::module
        JSIngestion["JS Ingestion"]:::module
        GenericIngestion["Generic Ingestion"]:::module
        Overrides["Overrides"]:::module
        SourcemapReverser["Sourcemap Reverser"]:::module
        GitCommitter["Git Committer"]:::module
    end

    %% Flows
    ProxyPlugin -->|"POST /capture"| HTTPServer
    HTTPServer -->|"save raw asset"| FileService
    HTTPServer -->|"record metadata"| SQLiteDB
    HTTPServer -->|"emit NewAsset"| CoreEventBus
    CoreEventBus -->|"dispatch jobs"| ASTAnalyzer
    CoreEventBus -->|"dispatch jobs"| Beautifier
    CoreEventBus -->|"dispatch jobs"| ChunkDiscoverer
    CoreEventBus -->|"dispatch jobs"| HTMLIngestion
    CoreEventBus -->|"dispatch jobs"| JSIngestion
    CoreEventBus -->|"dispatch jobs"| GenericIngestion
    CoreEventBus -->|"dispatch jobs"| Overrides
    CoreEventBus -->|"dispatch jobs"| SourcemapReverser
    CoreEventBus -->|"dispatch jobs"| GitCommitter

    ASTAnalyzer -->|"write results"| FileService
    ASTAnalyzer -->|"insert analysis"| SQLiteDB
    Beautifier -->|"write results"| FileService
    ChunkDiscoverer -->|"write results"| FileService
    HTMLIngestion -->|"write results"| FileService
    JSIngestion -->|"write results"| FileService
    GenericIngestion -->|"write results"| FileService
    Overrides -->|"write results"| FileService
    SourcemapReverser -->|"write results"| FileService
    GitCommitter -->|"write results"| FileService

    CLITUIClient -->|"function calls"| ApplicationCore
    VSCodeExt -->|"WebSocket"| WebSocketServer
    WebSocketServer -->|"serve AST tree"| ASTAnalyzer

    %% Click Events
    click HTTPServer "https://github.com/francisconeves97/jxscout/blob/main/pkg/jxscout/http.go"
    click WebSocketServer "https://github.com/francisconeves97/jxscout/blob/main/internal/core/websocket/websocket.go"
    click CLITUIClient "https://github.com/francisconeves97/jxscout/blob/main/internal/core/tui/tui.go"
    click CLITUIClient "https://github.com/francisconeves97/jxscout/blob/main/internal/core/tui/prompt.go"
    click CLITUIClient "https://github.com/francisconeves97/jxscout/blob/main/internal/core/tui/commands.go"
    click CLIEntrypoint "https://github.com/francisconeves97/jxscout/blob/main/cmd/jxscout/main.go"
    click ConfigLogging "https://github.com/francisconeves97/jxscout/blob/main/pkg/jxscout/options.go"
    click ConfigLogging "https://github.com/francisconeves97/jxscout/blob/main/pkg/jxscout/logger.go"
    click ConfigLogging "https://github.com/francisconeves97/jxscout/blob/main/pkg/jxscout/util.go"
    click ApplicationCore "https://github.com/francisconeves97/jxscout/blob/main/pkg/jxscout/jxscout.go"
    click ApplicationCore "https://github.com/francisconeves97/jxscout/blob/main/pkg/jxscout/prompt.go"
    click ApplicationCore "https://github.com/francisconeves97/jxscout/blob/main/pkg/jxscout/scope-checker.go"
    click FileService "https://github.com/francisconeves97/jxscout/blob/main/internal/core/asset-service/file-service.go"
    click AssetRepo "https://github.com/francisconeves97/jxscout/blob/main/internal/core/asset-service/repository.go"
    click AssetService "https://github.com/francisconeves97/jxscout/blob/main/internal/core/asset-service/service.go"
    click AssetFetcher "https://github.com/francisconeves97/jxscout/blob/main/internal/core/asset-fetcher/service.go"
    click SQLiteDB "https://github.com/francisconeves97/jxscout/blob/main/internal/core/database/database.go"
    click CoreEventBus "https://github.com/francisconeves97/jxscout/blob/main/internal/core/eventbus/eventbus.go"
    click DBEventBus "https://github.com/francisconeves97/jxscout/blob/main/internal/core/dbeventbus/eventbus.go"
    click ASTAnalyzer "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/ast-analyzer/module.go"
    click Beautifier "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/beautifier/beautifier.go"
    click ChunkDiscoverer "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/chunk-discoverer/module.go"
    click HTMLIngestion "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/html-ingestion/html-ingestion.go"
    click JSIngestion "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/js-ingestion/js-ingestion.go"
    click GenericIngestion "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/ingestion/ingestion.go"
    click Overrides "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/overrides/module.go"
    click SourcemapReverser "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/sourcemaps/sourcemaps.go"
    click GitCommitter "https://github.com/francisconeves97/jxscout/blob/main/internal/modules/git-committer/git-commiter.go"
    click ConcurrentQueue "https://github.com/francisconeves97/jxscout/blob/main/pkg/concurrent-queue/concurrentqueue.go"

    %% Styles
    classDef external fill:#FDEBD0,stroke:#E67E22,color:#A04000;
    classDef client fill:#D6EAF8,stroke:#3498DB,color:#1B4F72;
    classDef service fill:#D5F5E3,stroke:#27AE60,color:#145A32;
    classDef module fill:#F9EBEA,stroke:#C0392B,color:#78281F;
    classDef storage fill:#EBF5FB,stroke:#5DADE2,color:#154360;
```
