How ray-d-song/web-archive works
• Browser Plugin – for saving web pages as single HTML files and uploading them.
• Server – built on Cloudflare Workers that accepts the uploads, processes them, stores them in a Cloudflare D1 database and R2 storage bucket, and exposes APIs.
• Web Client – for users to search, view, and share the archived pages.
Main Components
a. Browser Plugin:
– Located under “packages/plugin”. It includes directories for background scripts, content scripts, popup UI (with React/TSX components), internationalization, and various helper libraries.
b. Server:
– Found in “packages/server”. It contains API endpoints (auth, folders, pages, data, etc.), middleware for token validation, database models, SQL migrations, and utility functions. This server runs on Cloudflare Workers, leveraging Cloudflare D1 (SQL database) and R2 (object storage bucket).
c. Web Client:
– Located in “packages/web”. It has the web entry point, pages, components, routing, state management (store), asset management, and configuration files (postcss, tailwind, vite).
Out Relationships and Data Flow
a. The Browser Plugin is responsible for capturing a webpage, converting it into a single HTML file, and then uploading it via API calls.
b. The Server receives this HTML file. It then:
– Processes authentication (via a token middleware).
– Stores meta-information and metadata into the D1 database.
– Saves the full HTML file in the R2 storage bucket.
c. The Web Client communicates with the Server API to query archive records, display content, and search or filter through folders, tags, and other metadata.
d. There are shared components and utilities under “packages/shared” indicating that some UI components or helper functions are reused between the Web Client and Plugin (“DRY” principle).
Architectural Patterns and Design Principles
a. Separation of Concerns: Each part – Plugin, Server, and Client – has its own dedicated directory and clear responsibilities.
b. Modular Design: The project is split into packages (monorepo structure using pnpm-workspace).
c. Reusability: The “packages/shared” folder contains common UI components and utilities that are shared across different parts.
d. Cloud-based Serverless Architecture: The Server part leverages Cloudflare’s serverless functions, along with managed services (D1 and R2).
– Browser Plugin (Client-Side Extension)
– Server (Cloudflare Worker with API endpoints, authentication, processing logic, and interaction with storage)
– Web Client (User Interface Web Application)
– Underlying Systems: Cloudflare D1 database and Cloudflare R2 storage bucket.
– Plugin: Highlight Background, Content Scripts, Popup UI, and the internationalization support.
– Web Client: Outline the routing, state management (store modules), and UI components.
– From Browser Plugin → Server: Upload API calls (HTML file and metadata).
– From Web Client → Server: Queries and API calls to retrieve archived pages.
Final Elements
• The Plugin sending “save webpage” requests to the Server.
• The Server writing metadata to D1 and storing HTML files in R2.
• The Web Client fetching and displaying data from the Server.
– Include annotations (or legends) explaining architectural patterns, technology choices (e.g., Cloudflare Workers, pnpm workspace, React, Tailwind, etc.), and any interdependencies.