How pipiku915/finmem-llm-stocktrading works
Project Type and Purpose
• FinMem is an LLM-based autonomous trading agent that leverages layered memory to process multi-source financial data. It integrates real-time news and market data with a sophisticated memory module inspired by human cognitive processes to make informed investment decisions.
• The project includes two main modes: “train” mode (where the agent builds or populates its memory and learns from historical data) and “test” mode (where it uses this memory alongside live input to execute trading strategies).
File Structure to Identify Main Components
• The top-level directories include:
• .devcontainer (Docker configuration for reproducible environments)
• config (containing various TOML configuration files for different LLM backends, e.g., Gemini, GPT, TGI)
• data-pipeline (scripts for downloading data via SQL and API, cleaning, summarizing, sentiment analysis, and statistical tests)
• data (organized into folders for raw, intermediate, model input/output, checkpoints, and results)
• puppy (core source code for implementing the trading agent, including its modules for chat, embedding, memory functions, portfolio, and reflection)
• The entry point is run.py with additional shell scripts (run_gemini.sh, run_opeai.sh, run_tgi.sh) for different execution scenarios.
a. Main Components to Include:
• Data Pipeline Module:
• Scripts under data-pipeline (ranging from data download via SQL/APIs, data cleaning, summarization, sentiment analysis, visualization, and performance metrics such as Wilcoxon-Test)
• Interaction with external data sources (e.g., Refinitiv, Alpaca, SEC APIs)
• Data Storage:
• Various data folders (e.g., data/01_raw, data/02_intermediate, etc.) that store data at different stages: ingestion, cleaning, model input, and output logs
• Core Trading Agent (within the "puppy" package):
• Memory Module: Includes layered memory functions such as recency, decay, importance and access counting, which help the agent in emulating human-like memory.
• Decision-making and Character Design: Agent, chat, reflection, embeddings, and portfolio management components that collectively transform processed data into trading decisions.
• Configuration and Environment:
• Config directory files: They determine which LLM backend (e.g., TGI, GPT-4) and endpoint settings to use.
• Docker/.devcontainer: Ensure consistent, containerized execution
• Environment Variables (.env): For API keys and tokens (OPENAI_API_KEY, HF_TOKEN)
• Execution Flow:
• run.py (the central entry point), which handles simulation (both train and test modes) and checkpoint resumption.
• Shell scripts to build and run the Docker container, ensuring reproducibility and environment standardization.
b. Relationships and Interactions:
• Data is first ingested and prepared via the Data Pipeline. Extracted data flows from external data sources into the “data” directories.
• Prepared data is then used to train the agent’s memory module (the “train” mode in run.py), where the memory functions from the puppy/memory_functions directory process and layer the data.
• The trading decision-making component (using modules like agent, chat, portfolio, etc.) consumes both historical memories and new data in “test” mode, producing trading outputs.
• The configuration files and environment settings (from config, .env, and Docker setup) ensure that the data pipeline and agent operate under the correct parameters and endpoints.
Architectural Patterns and Design Principles
• Modularity: Clear separation between data ingestion/processing (data-pipeline), persistent data storage (data folders), and agent functioning (puppy).
• Layered Memory Architecture: Emulates human cognitive processes with layered memory (access, decay, recency), which is a unique design feature.
• Configurability and Extensibility: Use of TOML configuration files allows the system to be adapted for different LLM backends and trading environments.
• Reproducibility: Containerization (.devcontainer) ensures execution uniformity.