# ourongxing/newsnow architecture

> Architecture diagram of the GitHub repository ourongxing/newsnow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ourongxing/newsnow
- Repository: https://github.com/ourongxing/newsnow

## Overview

##### Project type and purpose
- By examining the file tree and README, we see that NewsNow is a full-stack, SSR/PWA news‐aggregator web application.
- The “src” folder holds a React/Vite frontend (TSX, atoms/hooks/components), the “server” folder holds serverless API endpoints (Nitro config, endpoints under server/api, scrapers under server/sources), and “shared” holds common types/constants used by both.
- Key features: real-time scraping of multiple news sources, GitHub OAuth login, user data sync, caching (default 30m), PWA support, adaptive scraping intervals.

##### File structure
- Top-level:
- Dockerfile, docker-compose.yml – for containerized deployment.
- nitro.config.ts, vite.config.ts, pwa.config.ts – builder configs (Nitro for SSR, Vite for bundling, PWA setup).
- server/api – entry points for HTTP APIs (login, sync, news queries).
- server/sources – individual scraper modules for each external source.
- server/database – caching and user DB logic.
- shared – shared types, constants, metadata used by both front and back.
- src – React app: components, hooks, routes, styling.
- Patterns:
- Monorepo with code-sharing
- Serverless function pattern (each file in server/api becomes a route)
- PWA offline capability (public/sw.js)
- Middleware for auth (server/middleware/auth.ts)

##### README for additional insights
- Describes SSR + static build for Cloudflare Pages or Vercel, with output dist/output/public
- Guides on GitHub OAuth flow (API entrypoints, callback URL, JWT)
- Advises Cloudflare D1 for persistent storage

a. Main Components
1) Client (Browser/PWA)
3) SSR/Serverless Layer (Nitro engine)
4) API Routes (server/api/*)
5) Scraper Modules (server/sources/*)
6) Database Layer (Cloudflare D1 or other connectors via server/database/*.ts)
7) External Services:
- GitHub OAuth
- External news websites (36kr, HackerNews, Zhihu, etc.)
b. Relationships & Data Flow
- Client requests page → CDN/static server → index.html + bundled JS
- Client (or service worker) → API endpoints (/api/latest, /api/s, /api/me/sync)
- API endpoint → Middleware auth (JWT verification) → Database (user cache)
- API endpoint → Scraper module → External news site → returns raw data → server/utils (normalize → RSS/JSON)
- Normalized data cached in Database → returned to client
- On login: Client → /api/oauth/github → GitHub OAuth → callback → issue JWT → store in cookie/localStorage
c. Architectural Patterns & Design Principles
- Serverless functions (one route per file)
- Monorepo with shared code (avoid duplication)
- Atomic React state via Jotai atoms (src/atoms)
- PWA strategy: service worker caching + overlay-scrollbar + offline reading
- Adaptive scraping: central “source” util orchestrates intervals to avoid IP bans
d. Key Technologies & Frameworks
- Frontend: React, Vite, Typescript, Jotai, Tailwind/UnoCSS (uno.config.ts)
- Server: Nitro, Node.js, Typescript, Cloudflare Workers/D1 adapter
- Deployment: Docker Compose, Cloudflare Pages/Workers, Vercel
- CI/CD: GitHub Actions (auto-refresh, docker build, release)

##### Elements
- Data flows: “Cache lookup,” “Fetch & normalize,” “Issue JWT.”

##### Final Advice
- PWA aspects explicitly (service worker, offline cache).

## Diagram (Mermaid)

```mermaid
graph TD
    %% Client Side
    subgraph "Client Side"
        direction TB
        Browser["Browser"]:::frontend
        ReactApp["React App"]:::frontend
        ServiceWorker["Service Worker"]:::frontend
        IndexedDB["IndexedDB Cache"]:::frontend
    end

    %% CDN / Static Hosting
    CDN["CDN / Static Hosting\n(index.html, JS/CSS, Icons)"]:::infra

    %% Server Side
    subgraph "Server Side"
        direction TB
        Nitro["SSR/Serverless Engine\n(Nitro)"]:::backend
        subgraph "API Routes"
            direction TB
            Login["/api/login & enable-login"]:::api
            OAuthGithub["/api/oauth/github"]:::api
            Sync["/api/me & /api/me/sync"]:::api
            Latest["/api/latest"]:::api
            Search["/api/s & /api/s/entire.post"]:::api
            Proxy["/api/proxy"]:::api
        end
        AuthMiddleware["Auth Middleware"]:::backend
        Scrapers["Scraper Modules"]:::backend
        Database["Database Layer"]:::backend
    end

    %% External Services
    subgraph "External Services"
        direction TB
        GitHub["GitHub OAuth"]:::external
        NewsSites["External News Sites"]:::external
    end

    %% Deployment & CI/CD
    Docker["Docker & Compose"]:::infra
    CI["GitHub Actions"]:::infra

    %% Data Flows
    Browser -->|Loads| CDN
    ReactApp -->|Requests| CDN
    CDN -->|Serves| ReactApp
    ReactApp -->|Calls| Nitro
    ServiceWorker -->|Intercepts & Caches| CDN
    ServiceWorker -->|Serves Offline| ReactApp

    Nitro -->|Routes Requests| AuthMiddleware
    AuthMiddleware -->|Verify JWT| Database
    AuthMiddleware --> API_Route_Fanout[ ]
    API_Route_Fanout -->|login| Login
    API_Route_Fanout -->|oauth| OAuthGithub
    API_Route_Fanout -->|sync| Sync
    API_Route_Fanout -->|latest| Latest
    API_Route_Fanout -->|search| Search
    API_Route_Fanout -->|proxy| Proxy

    Login -->|Issue JWT| Database
    OAuthGithub -->|OAuth Flow| GitHub
    OAuthGithub -->|Callback| Login

    Latest -->|Cache Lookup| Database
    Latest -->|Fetch if Miss| Scrapers
    Search -->|Fetch & Normalize| Scrapers
    Scrapers -->|Fetch Data| NewsSites
    Scrapers -->|Normalize| Database
    Sync -->|User Data| Database

    Docker -->|Deploy| Nitro
    Nitro -->|Containerized| Docker
    CI -->|Build & Deploy| Docker

    %% Click Events
    click ReactApp "https://github.com/ourongxing/newsnow/tree/main/src/"
    click ServiceWorker "https://github.com/ourongxing/newsnow/blob/main/public/sw.js"
    click CDN "https://github.com/ourongxing/newsnow/tree/main/public/"
    click Nitro "https://github.com/ourongxing/newsnow/blob/main/nitro.config.ts"
    click Login "https://github.com/ourongxing/newsnow/blob/main/server/api/login.ts"
    click OAuthGithub "https://github.com/ourongxing/newsnow/blob/main/server/api/oauth/github.ts"
    click Sync "https://github.com/ourongxing/newsnow/blob/main/server/api/me/index.ts"
    click Sync "https://github.com/ourongxing/newsnow/blob/main/server/api/me/sync.ts"
    click Latest "https://github.com/ourongxing/newsnow/blob/main/server/api/latest.ts"
    click Search "https://github.com/ourongxing/newsnow/blob/main/server/api/s/index.ts"
    click Search "https://github.com/ourongxing/newsnow/blob/main/server/api/s/entire.post.ts"
    click Proxy "https://github.com/ourongxing/newsnow/tree/main/server/api/proxy/"
    click AuthMiddleware "https://github.com/ourongxing/newsnow/blob/main/server/middleware/auth.ts"
    click Scrapers "https://github.com/ourongxing/newsnow/tree/main/server/sources/"
    click Database "https://github.com/ourongxing/newsnow/tree/main/server/database/"
    click ReactApp "https://github.com/ourongxing/newsnow/tree/main/shared/"
    click Docker "https://github.com/ourongxing/newsnow/tree/main/Dockerfile"
    click Docker "https://github.com/ourongxing/newsnow/blob/main/docker-compose.yml"
    click CI "https://github.com/ourongxing/newsnow/tree/main/.github/workflows/"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085;
    classDef backend fill:#d4edda,stroke:#155724,color:#155724;
    classDef api fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
    classDef external fill:#fff3cd,stroke:#856404,color:#856404;
    classDef infra fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
```
