# discus0434/nook architecture

> Architecture diagram of the GitHub repository discus0434/nook, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/discus0434/nook
- Repository: https://github.com/discus0434/nook

## Overview

##### Main Components
- AWS Lambda Functions:
– Several Lambda functions exist (found under nook/lambda/) for different sources, such as:
▪ reddit_explorer – collects and summarizes Reddit posts (with subreddits defined in a TOML file)
▪ hacker_news – collects Hacker News articles
▪ github_trending – collects trending GitHub repositories (languages defined in a TOML file)
▪ tech_feed – fetches tech blog posts via RSS (includes a Dockerfile for the tech feed Lambda)
▪ paper_summarizer – collects and summarizes arXiv papers
▪ viewer – provides the Web interface and chat functionality
– There is also a common utilities module (e.g., gemini_client in nook/lambda/common) that interfaces with the Google Gemini LLM API for Japanese summarization.

- AWS EventBridge:
– Acts as the scheduling mechanism that triggers the various Lambda functions on a daily schedule.

- Data Storage (Amazon S3):
– Collected and processed data (summaries, articles, feeds) are saved in Amazon S3, to be later retrieved by the viewer or other applications.

- External APIs and Data Sources:
– Reddit API
– Hacker News API
– GitHub API
– RSS feeds for tech blogs
– arXiv API
– Google Gemini API for LLM-based summarization

##### Relationships and Interactions
- EventBridge triggers most of the Lambda functions (reddit_explorer, hacker_news, github_trending, tech_feed, paper_summarizer) on a fixed schedule.
- Each Lambda function fetches data from its respective source(s); for example:
– reddit_explorer queries the Reddit API.
– github_trending queries the GitHub API.
– tech_feed processes multiple RSS feeds.
– paper_summarizer contacts the arXiv API.
- The Lambda functions then use the Google Gemini API (via gemini_client) to summarize the data into Japanese.
- All collected and processed content is then stored in S3.
- The viewer Lambda function retrieves stored data from S3 and renders the dynamic Web interface, which also includes the chat functionality for follow-up questions.
- Users interact with the viewer component (the front-end) to access the daily summaries.

##### Architectural Patterns & Important Technologies
- Serverless design utilizing AWS Lambda and EventBridge for scalable, event-driven data processing.
- Modular design ensures that adding or modifying new data sources is streamlined by simply editing their respective configuration files (TOML files in this case) and deploying updated Lambda functions.
- AWS CDK is used to manage infrastructure as code (the cdk.json file and nook_stack.py indicate CDK usage).
- The system clearly separates data collection/processing (back-end Lambda functions) and presentation (viewer Lambda, Web interface).

– Emphasize the modular design and the separation of data ingestion from presentation.
– Document, using notes or callouts, the configuration files (like subreddits.toml, languages.toml, feed.toml) that control data collection for each Lambda.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Deployment and Scheduling
    subgraph "Deployment"
        CDK("AWS CDK"):::deployment
    end
    CDK --> event_bridge["AWS EventBridge"]:::scheduler

    %% Data Ingestion Lambdas
    subgraph "Data Ingestion Lambdas"
        lambda_reddit("Lambda: reddit_explorer"):::lambda
        lambda_hacker("Lambda: hacker_news"):::lambda
        lambda_github("Lambda: github_trending"):::lambda
        lambda_tech("Lambda: tech_feed"):::lambda
        lambda_paper("Lambda: paper_summarizer"):::lambda
    end

    event_bridge --> lambda_reddit
    event_bridge --> lambda_hacker
    event_bridge --> lambda_github
    event_bridge --> lambda_tech
    event_bridge --> lambda_paper

    %% External Data Sources
    reddit_api("Reddit API"):::externalAPI
    hacker_api("Hacker News API"):::externalAPI
    github_api("GitHub API"):::externalAPI
    rss_feeds("RSS Feeds"):::externalAPI
    arxiv_api("arXiv API"):::externalAPI

    lambda_reddit -->|"fetches"| reddit_api
    lambda_hacker  -->|"fetches"| hacker_api
    lambda_github  -->|"fetches"| github_api
    lambda_tech    -->|"fetches"| rss_feeds
    lambda_paper   -->|"fetches"| arxiv_api

    %% Summarization Service
    subgraph "Summarization Service"
        gemini_client("Gemini Client"):::lambda
        google_gemini("Google Gemini API"):::externalAPI
    end

    lambda_reddit -->|"summarizes"| gemini_client
    lambda_hacker -->|"summarizes"| gemini_client
    lambda_github -->|"summarizes"| gemini_client
    lambda_tech   -->|"summarizes"| gemini_client
    lambda_paper  -->|"summarizes"| gemini_client
    gemini_client -->|"calls"| google_gemini

    %% Data Storage
    s3("Amazon S3"):::storage
    lambda_reddit -->|"stores"| s3
    lambda_hacker -->|"stores"| s3
    lambda_github -->|"stores"| s3
    lambda_tech   -->|"stores"| s3
    lambda_paper  -->|"stores"| s3

    %% Presentation
    viewer("Lambda: viewer (Web Interface & Chat)"):::lambda
    s3 -->|"retrieves"| viewer
    user("User"):::client
    viewer -->|"interacts"| user

    %% Click Events for Component Mappings
    click lambda_reddit "https://github.com/discus0434/nook/blob/main/nook/lambda/reddit_explorer/reddit_explorer.py"
    click lambda_reddit "https://github.com/discus0434/nook/blob/main/nook/lambda/reddit_explorer/subreddits.toml"
    click lambda_hacker "https://github.com/discus0434/nook/blob/main/nook/lambda/hacker_news/hacker_news.py"
    click lambda_github "https://github.com/discus0434/nook/blob/main/nook/lambda/github_trending/github_trending.py"
    click lambda_github "https://github.com/discus0434/nook/blob/main/nook/lambda/github_trending/languages.toml"
    click lambda_tech "https://github.com/discus0434/nook/blob/main/nook/lambda/tech_feed/tech_feed.py"
    click lambda_tech "https://github.com/discus0434/nook/blob/main/nook/lambda/tech_feed/feed.toml"
    click lambda_tech "https://github.com/discus0434/nook/tree/main/nook/lambda/tech_feed/Dockerfile"
    click lambda_paper "https://github.com/discus0434/nook/blob/main/nook/lambda/paper_summarizer/paper_summarizer.py"
    click viewer "https://github.com/discus0434/nook/blob/main/nook/lambda/viewer/viewer.py"
    click viewer "https://github.com/discus0434/nook/blob/main/nook/lambda/viewer/templates/index.html"
    click gemini_client "https://github.com/discus0434/nook/blob/main/nook/lambda/common/python/gemini_client.py"
    click CDK "https://github.com/discus0434/nook/blob/main/nook/nook_stack.py"
    click CDK "https://github.com/discus0434/nook/blob/main/cdk.json"

    %% Styles
    classDef lambda fill:#C1E1C1,stroke:#2E8B57,stroke-width:2px;
    classDef scheduler fill:#E0E0E0,stroke:#808080,stroke-width:2px;
    classDef storage fill:#FFFFCC,stroke:#B8860B,stroke-width:2px;
    classDef externalAPI fill:#FFC0CB,stroke:#FF69B4,stroke-width:2px;
    classDef client fill:#ADD8E6,stroke:#1E90FF,stroke-width:2px;
    classDef deployment fill:#87CEFA,stroke:#4682B4,stroke-width:2px;
```
