How noahfraiture/coma works
Project Identification
• Project Type & Purpose: Coma is a CLI tool written in Rust for scraping web content. It extracts text, comments, links, images, etc., from websites. The README explains various command-line options and extraction commands like print, save, and graph – emphasizing data extraction and presentation.
• Main Use Cases: Given a URL and specific commands/options, the tool scrapes web pages and either outputs the data in the terminal, saves it to files, or generates a graph (HTML topology).
Overall Architectural Components
• CLI Interface: Located in src/cli.rs and main.rs, this component handles command-line parsing, option flags, and dispatching commands.
• Configuration Management: Handled by src/config.rs for interpreting user inputs and setting defaults.
• Web Scraping / Extraction Logic: Found in modules like src/extract.rs and src/browser.rs, which are likely responsible for fetching the webpage and processing its content.
• Data Presentation & Formatting: Modules such as src/display.rs and src/format.rs control how results are presented (in terminal or saved files) and how they’re formatted.
• Graph Generation: The src/graph.rs (along with templates/index.html) indicates a specialized component that creates an HTML representation (a topology) of the scraped site structure.
• State Management & Data Structures: The src/state.rs and src/node.rs files suggest that there’s an internal model storing scraped nodes or states, possibly to support multi-page scraping and graph topology.
• Static Assets & Templates: The static/ folder (with files like the shrimp.jpg logo) and templates/ folder (with HTML for topology) provide support for rendering and presentation.
Relationships and Data Flow
• User Input & CLI Parser: The process begins at main.rs, where the CLI input is parsed. The configuration module picks up the input options (e.g., URL, content type, depth).
• Decision & Dispatching: Depending on the command (print, save, graph), main.rs (with help from cli.rs) dispatches to the respective processing modules.
• Extraction Component: The browser and extract modules work together to fetch the webpage content and extract the requested data type (text, links, images, etc.).
• Intermediate State & Data Structures: The extraction process populates state or node models (from state.rs, node.rs) that might be used for maintaining a graph or hierarchy of pages.
• Output Modules: Based on the chosen command, either display.rs presents data in the terminal; format.rs prepares file-formatted outputs; or graph.rs creates an HTML file utilizing templates for graphing.
• External Libraries & Tools: The project uses Cargo (Rust’s package manager) as seen with Cargo.toml. Also, flake.nix and flake.lock suggest a Nix-based build/deployment environment. Tailwind.config.js and HTML templates signal integration with frontend styling for generated graphs.
Architectural Patterns and Design Principles
• Modular Separation: Different concerns are separated into distinct modules (CLI, config, extraction, formatting, graph generation), making it clear from the file tree that a modular architecture is in use.
• Single Responsibility: Each module (e.g., extract.rs focuses on scraping; cli.rs on interface) encapsulates a single concern, which aids maintainability and ease of extension.
• Extensibility: With planned future features (e.g., additional output formats, proxy settings, improved topology generation), the design is intended to be extended by adding or enhancing modules.