# ericciarla/trendfinder architecture

> Architecture diagram of the GitHub repository ericciarla/trendfinder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ericciarla/trendfinder
- Repository: https://github.com/ericciarla/trendfinder

## Overview

##### Main Components

a) External Services Layer:
- Twitter/X API
- Firecrawl Service
- Together AI Service
- Slack Webhooks

b) Application Core:
- Express.js Server
- Cron Job Controller
- Service Modules
- Data Processing Pipeline

**┌─ Data Sources ─┐    ┌─ Processing ─┐    ┌─ Output ─┐**
- Twitter/X API  │    │ Express.js   │    │  Slack   │
- Firecrawl     │ -> │ Application  │ -> │ Webhook  │
- ┘    └─────────────┘    └─────────┘

##### Detailed Component Breakdown

a) Create three main vertical sections:
- Input (Data Collection)
- Processing
- Output (Notification)

- Internal services: Rectangles
- Processing units: Hexagons

##### Data Flow Representation

**1. Data Collection Flow:**
- Twitter/X API ─────┐
- Firecrawl     ────┼─→ getCronSources

**2. Processing Flow:   v**
- scrapeSources ────→ generateDraft ────→ AI Analysis
- (Together AI)
**3. Output Flow:**
- AI Analysis ────→ sendDraft ────→ Slack Notification

##### Additional Elements

Required environment variables:
- TOGETHER_API_KEY
- FIRECRAWL_API_KEY
- X_API_BEARER_TOKEN
- SLACK_WEBHOOK_URL

b) Cron Job Controller:
Scheduling component that triggers the data collection process

##### Specific Implementation Details

**src/services/**
- getCronSources.ts   → Data collection initialization
- scrapeSources.ts    → Raw data gathering
- generateDraft.ts    → AI processing
- sendDraft.ts        → Notification dispatch

##### Notes for Implementation

- Use consistent spacing
- Version/date
- Brief component descriptions

##### Delivers notifications

Remember to maintain a balance between detail and clarity, focusing on the core functionality while still showing the complete system architecture.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef external fill:#a8d1f0,stroke:#3a84c3
    classDef core fill:#90EE90,stroke:#006400
    classDef process fill:#FFB347,stroke:#FF8C00
    classDef notification fill:#DDA0DD,stroke:#800080
    classDef config fill:#F0E68C,stroke:#BDB76B

    %% External Services
    subgraph ExternalServices["External Services"]
        TwitterAPI[Twitter/X API]:::external
        Firecrawl[Firecrawl Service]:::external
        TogetherAI[Together AI]:::external
        SlackWebhook[Slack Webhooks]:::external
    end

    %% Core Application
    subgraph CoreApp["Core Application"]
        ExpressServer["Express.js Server"]:::core
        CronController["Cron Job Controller"]:::core
    end

    %% Service Layer
    subgraph Services["Service Layer"]
        GetSources["Source Retrieval"]:::process
        ScrapeSources["Data Scraping"]:::process
        GenerateDraft["AI Draft Generation"]:::process
        SendDraft["Notification Dispatch"]:::notification
    end

    %% Configuration
    subgraph Config["Configuration"]
        Docker["Docker Container"]:::config
        EnvConfig["Environment Config"]:::config
        TypeScript["TypeScript Config"]:::config
    end

    %% Relationships
    TwitterAPI & Firecrawl --> GetSources
    GetSources --> ScrapeSources
    ScrapeSources --> GenerateDraft
    TogetherAI --> GenerateDraft
    GenerateDraft --> SendDraft
    SendDraft --> SlackWebhook
    
    CronController --> GetSources
    ExpressServer --> CronController

    %% Click Events
    click ExpressServer "https://github.com/ericciarla/trendFinder/blob/main/src/index.ts"
    click CronController "https://github.com/ericciarla/trendFinder/blob/main/src/controllers/cron.ts"
    click GetSources "https://github.com/ericciarla/trendFinder/blob/main/src/services/getCronSources.ts"
    click ScrapeSources "https://github.com/ericciarla/trendFinder/blob/main/src/services/scrapeSources.ts"
    click GenerateDraft "https://github.com/ericciarla/trendFinder/blob/main/src/services/generateDraft.ts"
    click SendDraft "https://github.com/ericciarla/trendFinder/blob/main/src/services/sendDraft.ts"
    click Docker "https://github.com/ericciarla/trendFinder/tree/main/Dockerfile"
    click EnvConfig "https://github.com/ericciarla/trendFinder/blob/main/.env.example"
    click TypeScript "https://github.com/ericciarla/trendFinder/blob/main/tsconfig.json"

    %% Legend
    subgraph Legend["Legend"]
        L1["External Services"]:::external
        L2["Core Components"]:::core
        L3["Processing Services"]:::process
        L4["Notification Services"]:::notification
        L5["Configuration"]:::config
    end
```
