How anisayari/my-public-data works
Main Components
• External Data Sources: Twitter API and YouTube API (the project uses these to fetch data).
• GitHub Actions Workflows: These are scheduled (every 6 hours for Twitter and every 48 hours for YouTube) and can also be manually triggered. They manage the automation of data fetching, token refresh, and update routines.
• Scripts / Data Fetching Modules: Under the “scripts” folder you have TypeScript programs for Twitter (fetch-bookmarks.ts, oauth-server.ts, refresh-token.ts, etc.) and JavaScript for YouTube (update.js). These scripts perform the actual fetching and processing of data.
• Configuration & Secrets: The “.env.example” file (to be copied to .env) and GitHub Secrets represent the secure credentials needed to access the external APIs.
• Data Output Files: The “twitter” and “youtube” directories holding JSON files (e.g., twitter/bookmarks.json, twitter/metadata.json, youtube/data.json, youtube/metadata.json) represent the outputs that are ultimately consumed by the personal website.
• That the scripts are triggered by GitHub Actions workflows. This can be depicted by illustrating “GitHub Actions” as a component that schedules and executes these scripts.
• That the credentials (via the .env file and GitHub Secrets) are prerequisites for these interactions.
Architectural Patterns and Technologies
• Decoupled architecture: GitHub Actions decouples the scheduling and triggering of data updates from the implementation details within the scripts.
• Use of containerization/testing elements such as Dockerfile.test and docker-compose.test.yml, which may be used for local testing but are not part of the production data flow.
• Point out that this design follows a pipeline pattern where data is fetched from external sources and stored in a static, public repository.
• Language details – TypeScript for Twitter scripts and JavaScript for YouTube scripts – alongside any specific workflow jobs like “refresh-token” or “update-data” relevant to the GitHub Actions.
Instructing the Principal Software Engineer
• GitHub Actions manage automation and scheduling.
• The scripts handle fetching and processing the data.
• The JSON files serve as the public data sources for the personal website.
• Do not hesitate to include supplementary information such as testing containers (e.g., Docker components) if local development testing is critical.