# anisayari/my-public-data architecture

> Architecture diagram of the GitHub repository anisayari/my-public-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anisayari/my-public-data
- Repository: https://github.com/anisayari/my-public-data

## Overview

##### Main Components
- External Data Sources: Twitter API and YouTube API (the project uses these to fetch data).
- GitHub Actions Workflows: These are scheduled (every 6 hours for Twitter and every 48 hours for YouTube) and can also be manually triggered. They manage the automation of data fetching, token refresh, and update routines.
- Scripts / Data Fetching Modules: Under the “scripts” folder you have TypeScript programs for Twitter (fetch-bookmarks.ts, oauth-server.ts, refresh-token.ts, etc.) and JavaScript for YouTube (update.js). These scripts perform the actual fetching and processing of data.
- Configuration & Secrets: The “.env.example” file (to be copied to .env) and GitHub Secrets represent the secure credentials needed to access the external APIs.
- Data Output Files: The “twitter” and “youtube” directories holding JSON files (e.g., twitter/bookmarks.json, twitter/metadata.json, youtube/data.json, youtube/metadata.json) represent the outputs that are ultimately consumed by the personal website.

- That the scripts are triggered by GitHub Actions workflows. This can be depicted by illustrating “GitHub Actions” as a component that schedules and executes these scripts.
- That the credentials (via the .env file and GitHub Secrets) are prerequisites for these interactions.

##### Architectural Patterns and Technologies
- Decoupled architecture: GitHub Actions decouples the scheduling and triggering of data updates from the implementation details within the scripts.
- Use of containerization/testing elements such as Dockerfile.test and docker-compose.test.yml, which may be used for local testing but are not part of the production data flow.
- Point out that this design follows a pipeline pattern where data is fetched from external sources and stored in a static, public repository.
- Language details – TypeScript for Twitter scripts and JavaScript for YouTube scripts – alongside any specific workflow jobs like “refresh-token” or “update-data” relevant to the GitHub Actions.

##### Instructing the Principal Software Engineer
- GitHub Actions manage automation and scheduling.
- The scripts handle fetching and processing the data.
- The JSON files serve as the public data sources for the personal website.
- Do not hesitate to include supplementary information such as testing containers (e.g., Docker components) if local development testing is critical.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External Services"
        TW("Twitter API"):::external
        YT("YouTube API"):::external
    end

    CS("Configuration & Secrets"):::config

    subgraph "Automation"
        GA("GitHub Actions Workflows"):::automation
    end

    subgraph "Data Fetching Scripts"
        TS("Twitter Scripts"):::processing
        YS("YouTube Script"):::processing
    end

    subgraph "Data Output Files"
        TOUT("Twitter Data Files"):::storage
        YOUT("YouTube Data Files"):::storage
    end

    subgraph "Testing & Containerization Setup"
        DFT("Dockerfile.test"):::testing
        DCT("docker-compose.test.yml"):::testing
    end

    GA -->|"triggers"| TS
    GA -->|"triggers"| YS

    TW -->|"fetches"| TS
    YT -->|"fetches"| YS

    CS -->|"provides_creds"| TS
    CS -->|"provides_creds"| YS

    TS -->|"updates"| TOUT
    YS -->|"updates"| YOUT

    click GA "https://github.com/anisayari/my-public-data/tree/main/.github/workflows"
    click TS "https://github.com/anisayari/my-public-data/tree/main/scripts/twitter"
    click YS "https://github.com/anisayari/my-public-data/blob/main/scripts/youtube/update.js"
    click CS "https://github.com/anisayari/my-public-data/blob/main/.env.example"
    click TOUT "https://github.com/anisayari/my-public-data/tree/main/twitter"
    click YOUT "https://github.com/anisayari/my-public-data/tree/main/youtube"
    click DFT "https://github.com/anisayari/my-public-data/blob/main/Dockerfile.test"
    click DCT "https://github.com/anisayari/my-public-data/blob/main/docker-compose.test.yml"

    classDef external fill:#a2d2ff,stroke:#000,stroke-width:1px;
    classDef automation fill:#caffbf,stroke:#000,stroke-width:1px;
    classDef processing fill:#ffbc42,stroke:#000,stroke-width:1px;
    classDef config fill:#f1ffc4,stroke:#000,stroke-width:1px;
    classDef storage fill:#ffdfba,stroke:#000,stroke-width:1px;
    classDef testing fill:#e0bbff,stroke:#000,stroke-width:1px;
```
