# isaacbacker/aus-sent architecture

> Architecture diagram of the GitHub repository isaacbacker/aus-sent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/isaacbacker/aus-sent
- Repository: https://github.com/isaacbacker/aus-sent

## Overview

##### Project’s Nature and Purpose
- This project is a full-stack application that builds an Australian Sentiment Index using Google Trends data.
- The model’s core is written in R (with several scripts to download data, process it, run validations, and construct predictions), while the web interface is a full-stack framework combining a Python FASTAPI backend (with an SQLite database) and two different frontends (one using Chart.js and the other using Plotly).

##### Main Components
- Data Collection & Preprocessing:
– The “model” directory contains submodules such as “Aus_sent3”, “aus_sent”, “aus_sent2”, etc., which include R and Python scripts that fetch Google Trends data, perform stationarity tests, generate datasets, and execute model evaluations.
– Indicate scripts that process raw CSV files (in “raw” subdirectories) and produce outputs like predicted sentiment indexes (e.g., daily_predicted_sentiment.csv).
- Model & Analytics Engine:
– The R-based sentiment modeling, regression validations (with “LASSO”, “ChoiModel.R”, “ConstructIndex.R”, etc.), and testing routines (as seen in the “ValidationTests2” and “prediction_tests” directories).
– Represent the transformation from raw economic & sentiment data into a computed sentiment index.
- Data Storage:
– The SQLite database (sentiment-dash/backend/sentiment.db) that is being populated by the output CSV files.
– Show how the CSV outputs from the model layer are read and written into the database.
- API Backend:
– The Python FASTAPI application (sentiment-dash/backend/app.py) that connects to the SQLite database.
– This component provides RESTful endpoints serving the sentiment data to the frontends.
- Web Frontend:
– Two separate web interfaces: one built using Chart.js and the other using Plotly (located in sentiment-dash/frontend, with different HTML files and accompanying static assets).
– These frontends interact with the backend to display the sentiment index in a user-friendly manner.
- Deployment & Configuration:
– Configuration files like render.yaml and requirements.txt for both the model and sentiment-dash.
– Note the instructions in the README regarding manual deployment on render.com.

##### Establishing Hierarchical and Data Flow Relationships
- Processing pipelines in the Model/Analytics Engine, where R and Python scripts transform these inputs into “prediction” CSV files.
- FASTAPI backend querying the SQLite database and exposing API endpoints.
- Optionally, include a feedback loop node if there are any monitoring/validation tests that feed into model improvement.

##### Architectural Patterns & Design Principles
- The system follows a separation-of-concerns design:
– The model and data processing (in R and accompanying Python scripts) are distinct from the full-stack web application (sentiment-dash).
– Data ingestion, processing, storage, and presentation are logically separated.
- Modularity:
– Multiple subdirectories (e.g., various “aus_sent” versions) indicate parallel experimentation or versioning in model development.
– The frontends are decoupled from the backend service via standardized API calls.
- Use of different technologies:
– R (for modeling and analytics), Python (FASTAPI, deployment scripts), SQLite (storage), HTML/CSS/JS (frontend visualization frameworks).

##### Essential Elements
- Clear labels on each component:
– “Data Collection & Preprocessing (Model – R/Python scripts)”
– “Sentiment Modeling & Validation”
– “Data Storage (SQLite Database)”
– “Backend API (FASTAPI)”
– “Frontend Interfaces (Chart.js and Plotly)”
– Data flows from raw files and Google Trends inputs → processing modules → CSV outputs → database insertion → API requests → data rendered in the web dashboards.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        DS1["Google Trends & CSV Files"]:::datasource
    end

    %% Data Collection & Preprocessing (Model Scripts)
    subgraph "Data Collection & Preprocessing (Model Scripts)"
        DP1["Aus_sent3 Scripts"]:::preprocess
        DP2["aus_sent Scripts"]:::preprocess
        DP3["aus_sent2 Scripts"]:::preprocess
    end

    %% Model & Analytics Engine
    subgraph "Model & Analytics Engine"
        MA1["Aus_sent3 Modeling"]:::analytics
        MA2["aus_sent2 Modeling"]:::analytics
        MA3["aus_sent_0.1 Modeling"]:::analytics
        MA4["Prediction Tests"]:::analytics
    end

    %% Data Storage
    subgraph "Data Storage"
        DB["SQLite Database"]:::storage
    end

    %% Backend API
    subgraph "Backend API"
        API["FASTAPI Application"]:::api
    end

    %% Web Frontend
    subgraph "Web Frontend"
        FE1["Chart.js Interface"]:::frontend
        FE2["Plotly Interface"]:::frontend
    end

    %% Deployment & Configuration
    subgraph "Deployment & Configuration"
        DC1["render.yaml"]:::deployment
        DC2["requirements.txt"]:::deployment
    end

    %% Data Flow Connections
    DS1 -->|"feeds"| DP1
    DS1 -->|"feeds"| DP2
    DS1 -->|"feeds"| DP3

    DP1 -->|"processes"| MA1
    DP2 -->|"processes"| MA2
    DP3 -->|"processes"| MA3
    MA1 -->|"triggers"| MA4

    MA4 -->|"outputs CSV to"| DB
    DB -->|"queried by"| API
    API -->|"serves data to"| FE1
    API -->|"serves data to"| FE2

    %% Deployment linkage (configuration influences API)
    DC1 -.-> API
    DC2 -.-> API

    %% Click Events for Data Collection & Preprocessing
    click DP1 "https://github.com/isaacbacker/aus-sent/tree/main/model/Aus_sent3"
    click DP2 "https://github.com/isaacbacker/aus-sent/tree/main/model/aus_sent"
    click DP3 "https://github.com/isaacbacker/aus-sent/tree/main/model/aus_sent2"

    %% Click Events for Model & Analytics Engine
    click MA1 "https://github.com/isaacbacker/aus-sent/tree/main/model/Aus_sent3"
    click MA2 "https://github.com/isaacbacker/aus-sent/tree/main/model/aus_sent2"
    click MA3 "https://github.com/isaacbacker/aus-sent/blob/main/model/Aus_sent_0.1"
    click MA4 "https://github.com/isaacbacker/aus-sent/tree/main/model/prediction_tests"

    %% Click Event for Data Storage
    click DB "https://github.com/isaacbacker/aus-sent/blob/main/sentiment-dash/backend/sentiment.db"

    %% Click Event for API Backend
    click API "https://github.com/isaacbacker/aus-sent/blob/main/sentiment-dash/backend/app.py"

    %% Click Events for Web Frontend
    click FE1 "https://github.com/isaacbacker/aus-sent/blob/main/sentiment-dash/frontend/index_chartjs.html"
    click FE2 "https://github.com/isaacbacker/aus-sent/blob/main/sentiment-dash/frontend/index_plotly.html"

    %% Click Events for Deployment & Configuration
    click DC1 "https://github.com/isaacbacker/aus-sent/blob/main/render.yaml"
    click DC2 "https://github.com/isaacbacker/aus-sent/blob/main/requirements.txt"

    %% Class Definitions for Colors
    classDef datasource fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef preprocess fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef analytics fill:#F9E79F,stroke:#333,stroke-width:2px;
    classDef storage fill:#D5F5E3,stroke:#333,stroke-width:2px;
    classDef api fill:#F5CBA7,stroke:#333,stroke-width:2px;
    classDef frontend fill:#D2B4DE,stroke:#333,stroke-width:2px;
    classDef deployment fill:#A9DFBF,stroke:#333,stroke-width:2px;
```
