How fuzzy-memory/caffeine-print works
Main Components
• News Retrieval Module (src/get_news.py): This component pulls news from worldnewsapi.com.
• Data Processing Modules:
• Clustering (src/similarity_clustering.py): Uses S-BERT sentence embeddings and DBSCAN to cluster similar news stories.
• Ranking (src/ranking.py): Produces weighted scores based on text, source, sentiment, and ChatGPT-generated feedback.
• HTML Generation Module (src/htmls.py): Arranges the picked articles into an HTML email body.
• Email Delivery (likely integrated into src/main.py, together with recipient-related logic in src/db/recipients.py): Handles the actual sending process.
• Data Models and Database Handling (src/data_models.py, src/db/data_models.py, and src/db/__init__.py): Abstract the data schema and possibly manage persistent or temporary storage.
• Configuration and Other Utilities:
• Properties (src/properties.py): Likely holds configuration parameters (e.g., API keys, thresholds).
• Utilities (src/utils.py): Contains helper functions used across components.
• External Data (assets/news_sources.json): Provides information on news sources and their ranking bases.
Relationships and Interactions
• The News Retrieval Module fetches data from an external API.
• The raw news data is sent to the Clustering Module where similar stories are grouped.
• The Ranking Module then takes clustered stories (and possibly additional data from the configuration or database modules) to rank articles.
• The HTML Generator receives the selected and ranked articles, formatting them into an email-ready HTML body.
• The Email Delivery system (within the main execution path in src/main.py) draws on the recipients (from src/db/recipients.py) and sends the final email.
• Cross-cutting concerns like data models, utility functions, and configuration support multiple components, ensuring data consistency and proper processing flow.
Architectural Patterns and Technologies
• Design follows a modular separation of concerns ensuring each module (retrieval, clustering, ranking, formatting, and mailing) is encapsulated.
• Although not explicitly an MVC, the separation helps distinguish data (models), processing logic (controllers, clustering/ranking), and presentation (HTML generation).
• Use of a configuration (pyproject.toml and src/properties.py) allows flexibility in deployment and possibly dependency injection.
• GitHub Actions (in .github/workflows/daily_run.yml) indicates an automation pipeline (likely for scheduled runs).