How zisra/substack works
Project’s Main Components
a. Frontend (Client-Side):
• Located in the "src" directory and built with React (TypeScript), using Vite as the build tool.
• Contains various UI components (found under "src/components" and "src/components/ui") for displaying articles, notes, comments, modals (e.g., About, Preferences), command palettes, and overall theming.
• Has separate routing for different views (e.g., article, note, archived, wiki) under "src/routes", along with utility hooks and libraries.
b. Backend (Server-Side):
• Found in the "server" directory. Implements Node/TypeScript-based APIs.
• Uses Express-like route modules (e.g., "download-article.ts", "download-comments.ts", "download-note.ts", and "image-proxy.ts") to handle scraping and proxy functions.
• Contains specialized scraping modules under "server/scrapers" (such as scrapers for Substack articles, notes, comments, and even Wikipedia) for fetching and processing external data.
• Other utility files (e.g., "turndown.ts", "utils.ts", "metaTags.ts") support content processing and metadata management.
c. Configuration & Build Files:
• "package.json", "pnpm-lock.yaml", "tsconfig.json", "vite.config.ts", "tailwind.config.ts" – defining dependencies, toolchain configurations, and build processes.
• "index.html" and public assets like "public/robots.txt" aid in deployment.
Relationships and Interactions
• The Frontend communicates with the Backend via HTTP API calls to the defined routes. For example, when a user requests to download an article or view comments, the client makes a request to a specific endpoint (like /download-article or /download-comments).
• The Backend routes internally use scraper modules to fetch, process, and transform the content from Substack, and then deliver it back to the client.
• Additionally, components from the frontend (such as the ArticleViewer or CommandPalette) render UI based on the data received from these API endpoints.
• There is likely a data/state management layer inside the frontend (using hooks and contexts found in "src/lib/DatabaseContext.tsx" and "src/lib/database.ts") to manage interactions and caching.
Architectural Patterns and Design Principles
• Separation of concerns is a clear design principle here, with the frontend responsible for user interaction and the backend focusing on data scraping and processing.
• An API-driven approach is used where the backend exposes REST endpoints that the frontend consumes.
• Modular design with a clear directory structure: different modules handle UI components, routing, state management, and backend scrapers.
• Use of modern frontend technologies (React, TypeScript, Vite, Tailwind) and backend modularization (using isolated routes and scraper modules).
Technologies, Frameworks, and Libraries
• Frontend: React (with TypeScript), Vite, Tailwind CSS.
• Backend: Node.js with TypeScript.
• HTTP routing and middleware possibly via Express (or similar), though not explicitly named.
• Scraping modules and utilities for content extraction and conversion (e.g., turndown for converting HTML to Markdown).