# iliane5/meridian architecture

> Architecture diagram of the GitHub repository iliane5/meridian, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iliane5/meridian
- Repository: https://github.com/iliane5/meridian

## Overview

##### Main Components
- Scrapers and Workflows: Located under apps/scrapers. This module is responsible for fetching RSS feeds, parsing source articles (with tools like Puppeteer and custom rate limiting logic), and executing workflows for different feed types (e.g., processArticles.workflow.ts and rssFeed.workflow.ts).
- Brief Generation & Analysis: Found in apps/briefs, which includes Jupyter notebooks (reportV5.ipynb), scripts for handling events and LLM interactions (llm.py, events.py), and markdown output generation.
- Frontend: Under apps/frontend, which is built with Nuxt 3/Vue 3. It includes pages for briefs (with dynamic routes like [slug].vue, latest.vue), plugins (for markdown rendering), and server API endpoints (reports.get.ts and subscribe.post.ts) for delivering content.
- Database & Migrations: Provided in packages/database, which uses PostgreSQL along with Drizzle for ORM. Database schema definitions, migrations (SQL scripts), and related configuration indicate how persistent data (articles, processed briefs, logs, etc.) is stored.
- Shared Configurations & Utilities: The workspace has configurations via Turborepo, pnpm workspace files, and TypeScript config packages that provide cross-project settings.

##### Interactions and Data Flows
- The Scrapers process the feeds and store metadata in an “Article Metadata DB” (likely a table within your PostgreSQL instance managed by the packages/database module).
- Next, another workflow (Article Processor Workflow) extracts content (via standard fetch or browser API call for challenging pages) and performs an LLM-driven analysis (Gemini, etc.) which results in processed article data stored in a separate “Processed Articles DB.”
- The Brief Generation module uses these processed articles, performing clustering (UMAP/HDBSCAN) and then further deep analysis via additional LLM steps. This eventually produces a Markdown brief that is stored in the Reports DB.
- Finally, the Frontend API (under apps/frontend/server) fetches the final brief data from the Reports DB and serves it through the Nuxt UI.

##### Architectural Patterns and Relationships
- Component Separation: Clearly separate the backend data ingestion, AI processing, and frontend presentation layers.
- Microservices/Function-based Architecture: The workflows (scraping and processing) are isolated and can be seen as serverless functions (Cloudflare Workers) that then interact with centralized databases.
- Cloud Deployment: Notice the use of Cloudflare (Workers, Pages, Workflows) and Turborepo managing multiple packages, indicating that automated deployments and microservice orchestration are essential parts of the architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Inputs
    subgraph "External Inputs"
        A["RSS Feed URLs"]:::external
    end

    %% Scraping & Ingestion
    subgraph "Scraping & Ingestion"
        B["Scraper Workflow (Cloudflare Workers)"]:::backend
        C["Article Metadata DB"]:::database
    end

    %% Processing & Analysis
    subgraph "Processing & Analysis"
        D["Article Processor Workflow"]:::backend
        E["LLM Article Analysis (Gemini)"]:::backend
        F["Processed Articles DB"]:::database
        G["Clustering (UMAP/HDBSCAN)"]:::backend
        H["LLM Cluster Review"]:::backend
    end

    %% Brief Generation
    subgraph "Brief Generation"
        I["Brief Generation (Python Notebook)"]:::backend
        J["Reports DB"]:::database
    end

    %% Frontend Delivery
    subgraph "Frontend Delivery"
        K["Frontend API (Nuxt Server)"]:::frontend
        L["Frontend UI (Nuxt Pages)"]:::frontend
    end

    %% Shared Configurations
    M["Shared Configurations & Utilities"]:::config

    %% Data Flow Connections
    A -->|"scrapes"| B
    B -->|"stores metadata"| C
    C -->|"triggers"| D
    D -->|"analyzes"| E
    E -->|"saves processed"| F
    F -->|"clusters"| G
    G -->|"reviews clusters"| H
    H -->|"generates brief"| I
    I -->|"saves brief"| J
    J -->|"feeds"| K
    K -->|"renders"| L

    %% Config influence (non-data flow)
    M ---|"configures"| B
    M ---|"configures"| D
    M ---|"configures"| I
    M ---|"configures"| K

    %% Click Events
    click B "https://github.com/iliane5/meridian/tree/main/apps/scrapers"
    click I "https://github.com/iliane5/meridian/blob/main/apps/briefs/reportV5.ipynb"
    click E "https://github.com/iliane5/meridian/blob/main/apps/briefs/src/llm.py"
    click K "https://github.com/iliane5/meridian/blob/main/apps/frontend/src/server/api/reports.get.ts"
    click L "https://github.com/iliane5/meridian/tree/main/apps/frontend/src/pages"
    click C "https://github.com/iliane5/meridian/blob/main/packages/database/src/schema.ts"
    click M "https://github.com/iliane5/meridian/blob/main/turbo.json"

    %% Style Definitions
    classDef external fill:#FFDD57,stroke:#333,stroke-width:2px;
    classDef backend fill:#B3E5FC,stroke:#333,stroke-width:2px;
    classDef database fill:#FFCCBC,stroke:#333,stroke-width:2px;
    classDef frontend fill:#D1C4E9,stroke:#333,stroke-width:2px;
    classDef config fill:#E0E0E0,stroke:#333,stroke-width:2px;
```
