How gerbenjavado/linkfinder works
Project Type and Purpose
• LinkFinder is a Python-based command-line tool for penetration testers and bug hunters. Its main purpose is to scan JavaScript files (or collections, including domains and Burp files) to discover endpoints by using a combination of jsbeautifier and regular expressions.
• Since it focuses on processing inputs (JS files/URLs) and producing an output (HTML or text results), it’s a utility rather than a full web application. However, note that there’s also a Docker integration for containerized usage.
File Structure
• Key Files:
• linkfinder.py: The main executable script that coordinates input handling, processing (beautification and regex scanning), and output generation.
• template.html: Likely the HTML template used when outputting results in a styled format.
• test_parser.py: Contains unit tests to validate the endpoint parsing logic, which helps us see that the core functionality lies in parsing and transforming data.
• setup.py and requirements.txt: Used for installation and dependency management; they hint at a standard packaging setup.
• Dockerfile and .dockerignore: Support running the tool within a container for portability.
• There’s no traditional layered “backend/frontend” architecture here. Instead, the design revolves around:
• Input handling (reading from diverse sources: URL, file, folder, Burp file).
• Processing (jsbeautifier pre-processing and applying multiple regexes to extract endpoints).
• Output generation (either to CLI or rendered HTML using template.html).
Diagramming the System Architecture
a. Main Components:
• Input Module: Accepts and validates user input (whether it’s a URL, file, or directory). It also handles command-line arguments (via argparse).
• Pre-Processing Module: If needed, it "beautifies" the raw JavaScript input using jsbeautifier.
• Parsing Module: Applies the series of regular expressions to extract endpoints from the JavaScript content.
• Output Module: Formats and outputs the discovered endpoints, either by writing them to an HTML file (using the provided template.html) or directly to STDOUT.
• Unit Testing Module: Comprised of test_parser.py to verify parsing correctness.
• Deployment & Containerization: Dockerfile and associated scripts which wrap the application for containerized use.
b. Relationships and Interactions:
• The Input Module feeds the raw JavaScript content (from file, URL, or domain crawl) into the Pre-Processing Module.
• The Pre-Processing Module standardizes/beautifies the JS content.
• The standardized content is then passed to the Parsing Module, where regular expressions are applied to detect endpoints.
• The Parsing Module produces a structured set of endpoints that is passed to the Output Module.
• The Output Module formats the results either by substituting into the HTML template (template.html) or printing them in plain text.
• Optional: Docker acts as an isolated environment container that encapsulates the above components.
c. Architectural Patterns / Design Principles:
• Separation of Concerns: Even though the project is relatively small, different responsibilities are clearly separated (input reading, processing, output generation).
• Extensibility: Being a command-line tool that relies on flags and parameters, it can easily be extended to add new processing or output options.
• Testability: The presence of a dedicated test file for its parser implies a design that supports unit testing.
• Containerization: Docker support to ensure a consistent runtime environment.
d. Relevant Technologies & Frameworks:
• Python 3 – Core development language.
• jsbeautifier – For reformatting/minifying the JavaScript input.
• Standard Python libraries: argparse (for command-line interface) and regex (for text processing).
• Docker – For containerizing the application.
Elements
• Labels for Each Component:
• “Input Module” (handles CLI arguments, file/URL reading)
• “Pre-Processing Module (jsbeautifier)”
• “Parsing Module (Regex Engine)”
• “Output Module (HTML/CLI)”
• “Docker Container (optional encapsulation)”
• “Unit Testing (test_parser.py)”
• From CLI -> Input Module -> Pre-Processing -> Parsing -> Output.
• Processing components versus configuration/deployment components.
• Dependency flows (e.g., how jsbeautifier integrates with the Pre-Processing Module).
Final Recommendations
• To reflect the modular design and potential for future extensions, as noted in the README usage examples and Docker container instructions.