# pipiku915/finmem-llm-stocktrading architecture

> Architecture diagram of the GitHub repository pipiku915/finmem-llm-stocktrading, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pipiku915/finmem-llm-stocktrading
- Repository: https://github.com/pipiku915/finmem-llm-stocktrading

## Overview

##### Project Type and Purpose
- FinMem is an LLM-based autonomous trading agent that leverages layered memory to process multi-source financial data. It integrates real-time news and market data with a sophisticated memory module inspired by human cognitive processes to make informed investment decisions.
- The project includes two main modes: “train” mode (where the agent builds or populates its memory and learns from historical data) and “test” mode (where it uses this memory alongside live input to execute trading strategies).

##### File Structure to Identify Main Components
- The top-level directories include:
- .devcontainer (Docker configuration for reproducible environments)
- config (containing various TOML configuration files for different LLM backends, e.g., Gemini, GPT, TGI)
- data-pipeline (scripts for downloading data via SQL and API, cleaning, summarizing, sentiment analysis, and statistical tests)
- data (organized into folders for raw, intermediate, model input/output, checkpoints, and results)
- puppy (core source code for implementing the trading agent, including its modules for chat, embedding, memory functions, portfolio, and reflection)
- The entry point is run.py with additional shell scripts (run_gemini.sh, run_opeai.sh, run_tgi.sh) for different execution scenarios.

a. Main Components to Include:
- Data Pipeline Module:
- Scripts under data-pipeline (ranging from data download via SQL/APIs, data cleaning, summarization, sentiment analysis, visualization, and performance metrics such as Wilcoxon-Test)
- Interaction with external data sources (e.g., Refinitiv, Alpaca, SEC APIs)
- Data Storage:
- Various data folders (e.g., data/01_raw, data/02_intermediate, etc.) that store data at different stages: ingestion, cleaning, model input, and output logs
- Core Trading Agent (within the "puppy" package):
- Memory Module: Includes layered memory functions such as recency, decay, importance and access counting, which help the agent in emulating human-like memory.
- Decision-making and Character Design: Agent, chat, reflection, embeddings, and portfolio management components that collectively transform processed data into trading decisions.
- Configuration and Environment:
- Config directory files: They determine which LLM backend (e.g., TGI, GPT-4) and endpoint settings to use.
- Docker/.devcontainer: Ensure consistent, containerized execution
- Environment Variables (.env): For API keys and tokens (OPENAI_API_KEY, HF_TOKEN)
- Execution Flow:
- run.py (the central entry point), which handles simulation (both train and test modes) and checkpoint resumption.
- Shell scripts to build and run the Docker container, ensuring reproducibility and environment standardization.

b. Relationships and Interactions:
- Data is first ingested and prepared via the Data Pipeline. Extracted data flows from external data sources into the “data” directories.
- Prepared data is then used to train the agent’s memory module (the “train” mode in run.py), where the memory functions from the puppy/memory_functions directory process and layer the data.
- The trading decision-making component (using modules like agent, chat, portfolio, etc.) consumes both historical memories and new data in “test” mode, producing trading outputs.
- The configuration files and environment settings (from config, .env, and Docker setup) ensure that the data pipeline and agent operate under the correct parameters and endpoints.

##### Architectural Patterns and Design Principles
- Modularity: Clear separation between data ingestion/processing (data-pipeline), persistent data storage (data folders), and agent functioning (puppy).
- Layered Memory Architecture: Emulates human cognitive processes with layered memory (access, decay, recency), which is a unique design feature.
- Configurability and Extensibility: Use of TOML configuration files allows the system to be adapted for different LLM backends and trading environments.
- Reproducibility: Containerization (.devcontainer) ensures execution uniformity.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Nodes
    Config["Configuration & Environment"]:::config
    Exec["Execution Flow (run.py & Shell Scripts)"]:::exec
    External["External Data Sources"]:::external
    Pipeline["Data Pipeline Module"]:::data
    Storage["Data Storage"]:::storage
    Memory["Memory Module (Layered Memory Functions)"]:::memory
    Trading["Core Trading Agent (Agent Logic)"]:::agent

    %% Flow Connections
    Config -->|"sets up"| Exec
    Config -->|"configures"| Pipeline
    Config -->|"configures"| Trading
    Exec -->|"orchestrates"| Pipeline
    Exec -->|"orchestrates"| Trading
    External -->|"feeds data to"| Pipeline
    Pipeline -->|"stores processed data in"| Storage
    Storage -->|"supplies historical data to"| Memory
    Storage -->|"supplies fresh data to"| Trading
    Memory -->|"enhances decision-making of"| Trading

    %% Click Events
    click Pipeline "https://github.com/pipiku915/finmem-llm-stocktrading/tree/main/data-pipeline/"
    click Storage "https://github.com/pipiku915/finmem-llm-stocktrading/tree/main/data/"
    click Trading "https://github.com/pipiku915/finmem-llm-stocktrading/tree/main/puppy/"
    click Memory "https://github.com/pipiku915/finmem-llm-stocktrading/tree/main/puppy/memory_functions/"
    click Config "https://github.com/pipiku915/finmem-llm-stocktrading/tree/main/config/"
    click Exec "https://github.com/pipiku915/finmem-llm-stocktrading/blob/main/run.py"

    %% Styles
    classDef config fill:#F39C12,stroke:#000,stroke-width:2px,color:#fff;
    classDef exec fill:#8E44AD,stroke:#000,stroke-width:2px,color:#fff;
    classDef external fill:#5DADE2,stroke:#000,stroke-width:2px,color:#fff,stroke-dasharray: 5 5;
    classDef data fill:#3498DB,stroke:#000,stroke-width:2px,color:#fff;
    classDef storage fill:#1ABC9C,stroke:#000,stroke-width:2px,color:#fff;
    classDef memory fill:#27AE60,stroke:#000,stroke-width:2px,color:#fff;
    classDef agent fill:#2ECC71,stroke:#000,stroke-width:2px,color:#fff;
```
