How ericciarla/trendfinder works
Main Components
a) External Services Layer:
• Twitter/X API
• Firecrawl Service
• Together AI Service
• Slack Webhooks
b) Application Core:
• Express.js Server
• Cron Job Controller
• Service Modules
• Data Processing Pipeline
┌─ Data Sources ─┐ ┌─ Processing ─┐ ┌─ Output ─┐
• Twitter/X API │ │ Express.js │ │ Slack │
• Firecrawl │ -> │ Application │ -> │ Webhook │
• ┘ └─────────────┘ └─────────┘
Detailed Component Breakdown
a) Create three main vertical sections:
• Input (Data Collection)
• Processing
• Output (Notification)
• Internal services: Rectangles
• Processing units: Hexagons
Data Flow Representation
1. Data Collection Flow:
• Twitter/X API ─────┐
• Firecrawl ────┼─→ getCronSources
2. Processing Flow: v
• scrapeSources ────→ generateDraft ────→ AI Analysis
• (Together AI)
3. Output Flow:
• AI Analysis ────→ sendDraft ────→ Slack Notification
Additional Elements
Required environment variables:
• TOGETHER_API_KEY
• FIRECRAWL_API_KEY
• X_API_BEARER_TOKEN
• SLACK_WEBHOOK_URL
b) Cron Job Controller:
Scheduling component that triggers the data collection process
Specific Implementation Details
src/services/
• getCronSources.ts → Data collection initialization
• scrapeSources.ts → Raw data gathering
• generateDraft.ts → AI processing
• sendDraft.ts → Notification dispatch
Notes for Implementation
• Use consistent spacing
• Version/date
• Brief component descriptions
Delivers notifications
Remember to maintain a balance between detail and clarity, focusing on the core functionality while still showing the complete system architecture.