How isaacbacker/aus-sent works
Project’s Nature and Purpose
• This project is a full-stack application that builds an Australian Sentiment Index using Google Trends data.
• The model’s core is written in R (with several scripts to download data, process it, run validations, and construct predictions), while the web interface is a full-stack framework combining a Python FASTAPI backend (with an SQLite database) and two different frontends (one using Chart.js and the other using Plotly).
Main Components
• Data Collection & Preprocessing:
– The “model” directory contains submodules such as “Aus_sent3”, “aus_sent”, “aus_sent2”, etc., which include R and Python scripts that fetch Google Trends data, perform stationarity tests, generate datasets, and execute model evaluations.
– Indicate scripts that process raw CSV files (in “raw” subdirectories) and produce outputs like predicted sentiment indexes (e.g., daily_predicted_sentiment.csv).
• Model & Analytics Engine:
– The R-based sentiment modeling, regression validations (with “LASSO”, “ChoiModel.R”, “ConstructIndex.R”, etc.), and testing routines (as seen in the “ValidationTests2” and “prediction_tests” directories).
– Represent the transformation from raw economic & sentiment data into a computed sentiment index.
• Data Storage:
– The SQLite database (sentiment-dash/backend/sentiment.db) that is being populated by the output CSV files.
– Show how the CSV outputs from the model layer are read and written into the database.
• API Backend:
– The Python FASTAPI application (sentiment-dash/backend/app.py) that connects to the SQLite database.
– This component provides RESTful endpoints serving the sentiment data to the frontends.
• Web Frontend:
– Two separate web interfaces: one built using Chart.js and the other using Plotly (located in sentiment-dash/frontend, with different HTML files and accompanying static assets).
– These frontends interact with the backend to display the sentiment index in a user-friendly manner.
• Deployment & Configuration:
– Configuration files like render.yaml and requirements.txt for both the model and sentiment-dash.
– Note the instructions in the README regarding manual deployment on render.com.
Establishing Hierarchical and Data Flow Relationships
• Processing pipelines in the Model/Analytics Engine, where R and Python scripts transform these inputs into “prediction” CSV files.
• FASTAPI backend querying the SQLite database and exposing API endpoints.
• Optionally, include a feedback loop node if there are any monitoring/validation tests that feed into model improvement.
Architectural Patterns & Design Principles
• The system follows a separation-of-concerns design:
– The model and data processing (in R and accompanying Python scripts) are distinct from the full-stack web application (sentiment-dash).
– Data ingestion, processing, storage, and presentation are logically separated.
• Modularity:
– Multiple subdirectories (e.g., various “aus_sent” versions) indicate parallel experimentation or versioning in model development.
– The frontends are decoupled from the backend service via standardized API calls.
• Use of different technologies:
– R (for modeling and analytics), Python (FASTAPI, deployment scripts), SQLite (storage), HTML/CSS/JS (frontend visualization frameworks).
Essential Elements
• Clear labels on each component:
– “Data Collection & Preprocessing (Model – R/Python scripts)”
– “Sentiment Modeling & Validation”
– “Data Storage (SQLite Database)”
– “Backend API (FASTAPI)”
– “Frontend Interfaces (Chart.js and Plotly)”
– Data flows from raw files and Google Trends inputs → processing modules → CSV outputs → database insertion → API requests → data rendered in the web dashboards.