# theendofline/webhook-firestore-sheets-pipeline architecture

> Architecture diagram of the GitHub repository theendofline/webhook-firestore-sheets-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theendofline/webhook-firestore-sheets-pipeline
- Repository: https://github.com/theendofline/webhook-firestore-sheets-pipeline

## Overview

##### Main Components

a) External Systems:
- Service X (Data Source)
- Google Sheets (Final Destination)

b) Google Cloud Components:
- Cloud Functions (2 separate functions)
- Cloud Firestore
- Cloud Scheduler
- IAM/Service Accounts

##### Component Relationships

- Service X → webhook-to-firestore-pipeline (HTTP POST/Webhook)
- webhook-to-firestore-pipeline → Firestore (Write)
- Cloud Scheduler → firestore-to-google-sheets-pipeline (HTTP Trigger)
- firestore-to-google-sheets-pipeline → Firestore (Read)
- firestore-to-google-sheets-pipeline → Google Sheets (Write)

##### Additional Details

Annotations for:
- Data types being transferred (date, name, hyperlink)
- Authentication points (service account)
- Validation steps
- Transformation processes (timestamp localization)
- Scheduling frequency

##### Process Highlights

- Data validation
- Deduplication
- Timestamp conversion
- Data reconciliation

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef external fill:#e6f3ff,stroke:#666
    classDef gcloud fill:#e6ffe6,stroke:#666
    classDef database fill:#f9f9f9,stroke:#666
    classDef trigger fill:#ffe6cc,stroke:#666
    classDef process fill:#fff,stroke:#666

    %% Components
    ServiceX[("Service X")]:::external
    Sheets[("Google Sheets")]:::external
    
    subgraph Ingestion
        WebhookFn["Webhook to Firestore Function"]:::gcloud
        Validation["Data Validation & Deduplication"]:::process
    end
    
    subgraph Storage
        Firestore[(Cloud Firestore)]:::database
    end
    
    subgraph Export
        Scheduler["Cloud Scheduler"]:::trigger
        SheetsFn["Firestore to Sheets Function"]:::gcloud
        Transform["Timestamp Conversion"]:::process
        ServiceAccount["Service Account Auth"]:::gcloud
    end

    %% Relationships
    ServiceX -->|HTTP POST| WebhookFn
    WebhookFn --> Validation
    Validation -->|Write| Firestore
    Scheduler -->|HTTP Trigger| SheetsFn
    SheetsFn -->|Read| Firestore
    SheetsFn --> Transform
    Transform --> ServiceAccount
    ServiceAccount -->|Write| Sheets

    %% Click Events
    click WebhookFn "https://github.com/theendofline/webhook-firestore-sheets-pipeline/blob/main/webhook-to-firestore-pipeline/main.py"
    click SheetsFn "https://github.com/theendofline/webhook-firestore-sheets-pipeline/blob/main/firestore-to-google-sheets-pipeline/main.py"
    click ServiceAccount "https://github.com/theendofline/webhook-firestore-sheets-pipeline/blob/main/firestore-to-google-sheets-pipeline/service_account_credentials_file.json"

    %% Legend
    subgraph Legend
        L1[("External Service")]:::external
        L2["Google Cloud Service"]:::gcloud
        L3[(Database)]:::database
        L4["Trigger"]:::trigger
        L5["Process"]:::process
    end
```
