# ameliawong1996/from_news_to_forecast architecture

> Architecture diagram of the GitHub repository ameliawong1996/from_news_to_forecast, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ameliawong1996/from_news_to_forecast
- Repository: https://github.com/ameliawong1996/from_news_to_forecast

## Overview

##### Project type and purpose
- This is a machine-learning research system that integrates unstructured news data with numerical time series data to improve forecasting accuracy using Large Language Models (LLMs) and generative agents.
- Key use cases: electricity load forecasting, exchange‐rate prediction, Bitcoin price prediction, traffic volume forecasting.

##### File‐structure analysis
- Top‐level directories:
- Agent/ – contains the generative‐agent code, data, and notebooks for iterative event reasoning.
- data/ – raw and paired time series & news data.
- code files at root: llm-finetune, model_download, validation, evaluation notebooks.
- utils/, templates/ – supporting code and template prompts for LLM interactions.
- Patterns:
- Data ingestion & preprocessing (data/raw_*, Agent/Data_all).
- Agent‐based news selection & reasoning (Agent/code, AgentBuilding.ipynb).
- Fine‐tuning pipeline (llm-finetune.py, finetune-lls4ts.sh).
- Validation & evaluation (validation.py, Evaluation.ipynb, accuracy-ts.py).

##### Architecture components
a. Data Sources (on left):
– Numerical time series CSVs (energy, exchange, bitcoin, traffic)
– Raw news JSONs (AU-news, finance, etc.)
– Supplementary data (weather, calendar, economic indicators)
b. Data Processing Layer:
– Raw data loader & cleaner (scripts in Agent/Data_all and data/raw_*)
– News preprocessing (tokenization, filtering)
– Time-series pairing logic (paired_time_series_news_training_data)
c. Generative Agent Component:
– LLM‐based news selector (Agent/code/gpt_news_evaluation.py, select_news_logic)
– Reflection loop for iterative reasoning (AgentBuilding.ipynb)
d. Fine-tuning Pipeline:
– Model download (model_download.py)
– Deepspeed‐driven training (llm-finetune.py, finetune-lls4ts.sh)
– LoRA adapters and hyperparameter configuration
e. Inference & Validation:
– Validation script (validation.py)
– Prompt templates (templates/ts_test.json)
– Metrics calculation (Evaluation.ipynb, accuracy-ts.py)
f. Outputs & Visualization:
– Forecast JSON results
– Plots and comparison charts (plot_comparision.py)
– Notebooks summarizing results

##### Relationships & data flow
- Process blocks feeding cleaned & paired datasets into both the Agent Component and directly into Fine-tuning.
- Agent outputs (selected news events) feed back into the paired dataset.
- Paired dataset flows into the Fine-tuning block.
- Trained model flows into Inference & Validation.
- Validation outputs feed into Evaluation & Visualization.

##### Key architectural patterns & design principles
- Modular pipeline: clear separation of data ingestion, agent reasoning, model training, and evaluation.
- Iterative feedback loop: Agent’s reflection loop refines news selection before re-training.
- Reusable components: prompt templates, utility functions, LoRA adapters.
- Scalability: deepspeed for distributed training, configurable data loaders.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Sources"
        TS1["Exchange CSV"]:::datastore
        TS2["Bitcoin CSV"]:::datastore
        TS3["Traffic CSV"]:::datastore
        TS4["Weather CSV"]:::datastore
        News1["AU-news JSON"]:::datastore
        News2["Bitcoin News JSON"]:::datastore
        News3["Traffic News JSON"]:::datastore
    end

    subgraph "Data Processing Layer"
        Loader["Raw Data Loader & Cleaner"]:::processing
        NewsPrep["News Preprocessing"]:::processing
        Pairing["Time-Series Pairing Logic"]:::processing
    end

    subgraph "Generative Agent Component"
        Selector["LLM-based News Selector"]:::agent
        Reflect["Iterative Reflection Notebook"]:::agent
        Reselect["News Reselection Logic"]:::agent
    end

    subgraph "Fine-tuning Pipeline"
        ModelDownload["Model Download"]:::training
        Train["Deepspeed-driven Training<br/>LoRA adapters"]:::training
        Shell["Shell Training Wrapper"]:::training
    end

    subgraph "Inference & Validation"
        Validation["Validation Script"]:::validation
        Templates["Prompt Templates"]:::validation
        Metrics["Metrics Calculation"]:::validation
    end

    subgraph "Supporting Utilities"
        Prompter["Prompt Construction"]:::util
        Alpaca["LLM Prompt Templates"]:::util
    end

    subgraph "Outputs & Visualization"
        Forecast["Forecast JSON Results"]:::output
        Plots["Comparison Plots"]:::output
        Notebooks["Result Summary Notebooks"]:::output
    end

    TS1 --> Loader
    TS2 --> Loader
    TS3 --> Loader
    TS4 --> Loader
    News1 --> NewsPrep
    News2 --> NewsPrep
    News3 --> NewsPrep

    Loader --> Pairing
    NewsPrep --> Pairing

    Pairing --> Selector
    Selector --> Reselect
    Reselect --> Pairing
    Reflect --> Selector

    Pairing --> ModelDownload
    ModelDownload --> Train
    Shell --> Train
    Prompter --> Train
    Alpaca --> Train

    Train --> Validation
    Templates --> Validation
    Validation --> Metrics

    Metrics --> Forecast
    Forecast --> Plots
    Forecast --> Notebooks
    Plots --> Notebooks

    click TS1 "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/data/raw_time_series_data/Exchang_all_data_2018-2022_D_final.csv"
    click TS2 "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/data/raw_time_series_data/bitcoin_daily.csv"
    click TS3 "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/data/raw_time_series_data/traffic_hourly.csv"
    click TS4 "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/data/raw_time_series_data/weather_load_2019-2022.csv"
    click News1 "https://github.com/ameliawong1996/from_news_to_forecast/tree/main/data/raw_news_data/AU-news/"
    click News2 "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/data/raw_news_data/bitcoin_news.json"
    click News3 "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/data/raw_news_data/traffic_news.json"
    click Loader "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/Agent/code/AU_LF_dataload_final.py"
    click Pairing "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/data/paired_time_series_news_training_data/AULF_train_data_2019-2020.json"
    click Selector "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/Agent/code/gpt_news_evaluation.py"
    click Reflect "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/Agent/code/AgentBuilding.ipynb"
    click Reselect "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/Agent/code/reselect_news.py"
    click ModelDownload "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/model_download.py"
    click Train "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/llm-finetune.py"
    click Shell "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/finetune-lls4ts.sh"
    click Validation "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/validation.py"
    click Templates "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/templates/ts_test.json"
    click Metrics "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/accuracy-ts.py"
    click Plots "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/Agent/code/plot_comparision.py"
    click Prompter "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/utils/prompter.py"
    click Alpaca "https://github.com/ameliawong1996/from_news_to_forecast/blob/main/templates/alpaca.json"

    classDef datastore fill:#cce5ff,stroke:#004085
    classDef processing fill:#d4edda,stroke:#155724
    classDef agent fill:#d1ecf1,stroke:#0c5460
    classDef training fill:#fff3cd,stroke:#856404
    classDef validation fill:#d6d8db,stroke:#1b1e21
    classDef output fill:#f8d7da,stroke:#721c24
    classDef util fill:#e2e3e5,stroke:#6c757d
```
