How discus0434/nook works
Main Components
• AWS Lambda Functions:
– Several Lambda functions exist (found under nook/lambda/) for different sources, such as:
▪ reddit_explorer – collects and summarizes Reddit posts (with subreddits defined in a TOML file)
▪ hacker_news – collects Hacker News articles
▪ github_trending – collects trending GitHub repositories (languages defined in a TOML file)
▪ tech_feed – fetches tech blog posts via RSS (includes a Dockerfile for the tech feed Lambda)
▪ paper_summarizer – collects and summarizes arXiv papers
▪ viewer – provides the Web interface and chat functionality
– There is also a common utilities module (e.g., gemini_client in nook/lambda/common) that interfaces with the Google Gemini LLM API for Japanese summarization.
• AWS EventBridge:
– Acts as the scheduling mechanism that triggers the various Lambda functions on a daily schedule.
• Data Storage (Amazon S3):
– Collected and processed data (summaries, articles, feeds) are saved in Amazon S3, to be later retrieved by the viewer or other applications.
• External APIs and Data Sources:
– Reddit API
– Hacker News API
– GitHub API
– RSS feeds for tech blogs
– arXiv API
– Google Gemini API for LLM-based summarization
Relationships and Interactions
• EventBridge triggers most of the Lambda functions (reddit_explorer, hacker_news, github_trending, tech_feed, paper_summarizer) on a fixed schedule.
• Each Lambda function fetches data from its respective source(s); for example:
– reddit_explorer queries the Reddit API.
– github_trending queries the GitHub API.
– tech_feed processes multiple RSS feeds.
– paper_summarizer contacts the arXiv API.
• The Lambda functions then use the Google Gemini API (via gemini_client) to summarize the data into Japanese.
• All collected and processed content is then stored in S3.
• The viewer Lambda function retrieves stored data from S3 and renders the dynamic Web interface, which also includes the chat functionality for follow-up questions.
• Users interact with the viewer component (the front-end) to access the daily summaries.
Architectural Patterns & Important Technologies
• Serverless design utilizing AWS Lambda and EventBridge for scalable, event-driven data processing.
• Modular design ensures that adding or modifying new data sources is streamlined by simply editing their respective configuration files (TOML files in this case) and deploying updated Lambda functions.
• AWS CDK is used to manage infrastructure as code (the cdk.json file and nook_stack.py indicate CDK usage).
• The system clearly separates data collection/processing (back-end Lambda functions) and presentation (viewer Lambda, Web interface).
– Emphasize the modular design and the separation of data ingestion from presentation.
– Document, using notes or callouts, the configuration files (like subreddits.toml, languages.toml, feed.toml) that control data collection for each Lambda.