How iliane5/meridian works
Main Components
• Scrapers and Workflows: Located under apps/scrapers. This module is responsible for fetching RSS feeds, parsing source articles (with tools like Puppeteer and custom rate limiting logic), and executing workflows for different feed types (e.g., processArticles.workflow.ts and rssFeed.workflow.ts).
• Brief Generation & Analysis: Found in apps/briefs, which includes Jupyter notebooks (reportV5.ipynb), scripts for handling events and LLM interactions (llm.py, events.py), and markdown output generation.
• Frontend: Under apps/frontend, which is built with Nuxt 3/Vue 3. It includes pages for briefs (with dynamic routes like [slug].vue, latest.vue), plugins (for markdown rendering), and server API endpoints (reports.get.ts and subscribe.post.ts) for delivering content.
• Database & Migrations: Provided in packages/database, which uses PostgreSQL along with Drizzle for ORM. Database schema definitions, migrations (SQL scripts), and related configuration indicate how persistent data (articles, processed briefs, logs, etc.) is stored.
• Shared Configurations & Utilities: The workspace has configurations via Turborepo, pnpm workspace files, and TypeScript config packages that provide cross-project settings.
Interactions and Data Flows
• The Scrapers process the feeds and store metadata in an “Article Metadata DB” (likely a table within your PostgreSQL instance managed by the packages/database module).
• Next, another workflow (Article Processor Workflow) extracts content (via standard fetch or browser API call for challenging pages) and performs an LLM-driven analysis (Gemini, etc.) which results in processed article data stored in a separate “Processed Articles DB.”
• The Brief Generation module uses these processed articles, performing clustering (UMAP/HDBSCAN) and then further deep analysis via additional LLM steps. This eventually produces a Markdown brief that is stored in the Reports DB.
• Finally, the Frontend API (under apps/frontend/server) fetches the final brief data from the Reports DB and serves it through the Nuxt UI.
Architectural Patterns and Relationships
• Component Separation: Clearly separate the backend data ingestion, AI processing, and frontend presentation layers.
• Microservices/Function-based Architecture: The workflows (scraping and processing) are isolated and can be seen as serverless functions (Cloudflare Workers) that then interact with centralized databases.
• Cloud Deployment: Notice the use of Cloudflare (Workers, Pages, Workflows) and Turborepo managing multiple packages, indicating that automated deployments and microservice orchestration are essential parts of the architecture.