# zisra/substack architecture

> Architecture diagram of the GitHub repository zisra/substack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zisra/substack
- Repository: https://github.com/zisra/substack

## Overview

##### Project’s Main Components
a. Frontend (Client-Side):
- Located in the "src" directory and built with React (TypeScript), using Vite as the build tool.
- Contains various UI components (found under "src/components" and "src/components/ui") for displaying articles, notes, comments, modals (e.g., About, Preferences), command palettes, and overall theming.
- Has separate routing for different views (e.g., article, note, archived, wiki) under "src/routes", along with utility hooks and libraries.

b. Backend (Server-Side):
- Found in the "server" directory. Implements Node/TypeScript-based APIs.
- Uses Express-like route modules (e.g., "download-article.ts", "download-comments.ts", "download-note.ts", and "image-proxy.ts") to handle scraping and proxy functions.
- Contains specialized scraping modules under "server/scrapers" (such as scrapers for Substack articles, notes, comments, and even Wikipedia) for fetching and processing external data.
- Other utility files (e.g., "turndown.ts", "utils.ts", "metaTags.ts") support content processing and metadata management.

c. Configuration & Build Files:
- "package.json", "pnpm-lock.yaml", "tsconfig.json", "vite.config.ts", "tailwind.config.ts" – defining dependencies, toolchain configurations, and build processes.
- "index.html" and public assets like "public/robots.txt" aid in deployment.

##### Relationships and Interactions
- The Frontend communicates with the Backend via HTTP API calls to the defined routes. For example, when a user requests to download an article or view comments, the client makes a request to a specific endpoint (like /download-article or /download-comments).
- The Backend routes internally use scraper modules to fetch, process, and transform the content from Substack, and then deliver it back to the client.
- Additionally, components from the frontend (such as the ArticleViewer or CommandPalette) render UI based on the data received from these API endpoints.
- There is likely a data/state management layer inside the frontend (using hooks and contexts found in "src/lib/DatabaseContext.tsx" and "src/lib/database.ts") to manage interactions and caching.

##### Architectural Patterns and Design Principles
- Separation of concerns is a clear design principle here, with the frontend responsible for user interaction and the backend focusing on data scraping and processing.
- An API-driven approach is used where the backend exposes REST endpoints that the frontend consumes.
- Modular design with a clear directory structure: different modules handle UI components, routing, state management, and backend scrapers.
- Use of modern frontend technologies (React, TypeScript, Vite, Tailwind) and backend modularization (using isolated routes and scraper modules).

##### Technologies, Frameworks, and Libraries
- Frontend: React (with TypeScript), Vite, Tailwind CSS.
- Backend: Node.js with TypeScript.
- HTTP routing and middleware possibly via Express (or similar), though not explicitly named.
- Scraping modules and utilities for content extraction and conversion (e.g., turndown for converting HTML to Markdown).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Frontend (Client-Side)"
        FE_Entry["Frontend Entry Point"]:::frontend
        FE_UI["UI Components"]:::frontend
        FE_Routes["Routing & Views"]:::frontend
        FE_State["State Management & Utility Hooks"]:::frontend
    end

    subgraph "Backend (Server-Side)"
        BE_API["Backend API Endpoints"]:::backend
        BE_Scrapers["Backend Scraping Modules"]:::backend
        BE_Processing["Content Processing Utilities"]:::backend
    end

    subgraph "Configuration & Build"
        CFG["Configuration & Build Files"]:::config
    end

    %% Frontend internal connections
    FE_Entry --> FE_UI
    FE_Entry --> FE_Routes
    FE_Entry --> FE_State

    %% Frontend to Backend API
    FE_Routes -->|"HTTP_API"| BE_API
    FE_UI -->|"HTTP_API"| BE_API
    FE_State -->|"HTTP_API"| BE_API

    %% Backend internal flow
    BE_API -->|"calls_scraper"| BE_Scrapers
    BE_API -->|"calls_processing"| BE_Processing
    BE_Scrapers -->|"returns_scraped_data"| BE_API
    BE_Processing -->|"returns_processed_content"| BE_API

    %% Backend to Frontend response
    BE_API -->|"HTTP_response"| FE_UI

    %% Configuration influences
    CFG --- FE_Entry
    CFG --- BE_API

    %% Click Events
    click FE_Entry "https://github.com/zisra/substack/blob/main/src/main.tsx"
    click FE_UI "https://github.com/zisra/substack/tree/main/src/components"
    click FE_Routes "https://github.com/zisra/substack/tree/main/src/routes"
    click FE_State "https://github.com/zisra/substack/tree/main/src/lib"
    click BE_API "https://github.com/zisra/substack/tree/main/server/routes"
    click BE_Scrapers "https://github.com/zisra/substack/tree/main/server/scrapers"
    click BE_Processing "https://github.com/zisra/substack/blob/main/server/turndown.ts"
    click CFG "https://github.com/zisra/substack/blob/main/package.json"

    %% Styles
    classDef frontend fill:#AED581,stroke:#558B2F,stroke-width:2px;
    classDef backend fill:#81D4FA,stroke:#0288D1,stroke-width:2px;
    classDef config fill:#F8BBD0,stroke:#C2185B,stroke-width:2px;
```
