# hananedupouy/llms-in-finance architecture

> Architecture diagram of the GitHub repository hananedupouy/llms-in-finance, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hananedupouy/llms-in-finance
- Repository: https://github.com/hananedupouy/llms-in-finance

## Overview

##### Project type and purpose
- This is an educational monorepo of Jupyter‐notebook demos applying Generative AI (LLM Agents, RAG, Multimodal models) to financial workflows.
- Rather than a single deployed product, it’s a collection of modules showcasing patterns: multi‐agent orchestration, retrieval pipelines, chart interpretation, synthetic data generation, and evaluation.

##### File structure
- Top‐level modules:
· Agents/… (several subfolders: AutoGen, OpenAI Agents SDK, Anthropic, LlamaIndex, CrewAI, etc.)
· RAG/… (workflows using different LLM providers + evaluation submodule)
· Multimodal_llms/financial_analysis (chart/PDF loaders + multimodal LLM experiments)
· Datasets/Synthetic_data (data generation notebooks + images)
· Papers/ (reference summaries, images)
- No monolithic backend/frontend—it’s a set of pipelines orchestrated in notebooks.
- Key “infrastructure” files: pyproject.toml under CRAG (packaging), data folders under AutoGen, evaluation tool configs.

##### README for additional insights
- Describes the four pillars: AI Agents, RAG, Multimodals, Papers.
- Mentions frameworks (OpenAI Agents SDK, Anthropic, AutoGen, LlamaIndex, CrewAI, LangChain).
- Emphasizes retrieval, parsing, chart interpretation, evaluation with DeepEval/GiskardAI.

##### Components
a. Main components (group into layers):
1. User interface layer: Jupyter Notebooks (entry points for each demo)
2. Orchestration layer: Agent frameworks (AutoGen, OpenAI Agents SDK, Anthropic agents, LlamaIndex agents, CrewAI, LangChain)
3. Retrieval/Index layer: LlamaIndex packs (Corrective_RAG), prompt caching, PDF/data loaders
4. Multimodal Analysis layer: chart/PDF loaders + multimodal LLMs (GPT-4o, Claude Sonnet, o1 Reasoning)
5. Data layer: financial data (CSV, PDFs), synthetic data generator (LLamaIndex notebooks)
6. Evaluation layer: DeepEval, GiskardAI modules under RAG/evaluation
7. External services layer: LLM APIs (OpenAI, Anthropic, Sonnet), storage (S3, local file system)
b. Relationships / interactions:
- Notebooks invoke Orchestration layer → orchestrator spins up multiple agents → agents call LLM APIs.
- Agents or RAG workflows consult the Retrieval/Index layer (LlamaIndex packs, cached prompts) → fetch documents or cached embeddings → send to LLM for response.
- Multimodal notebooks load charts/PDFs → pass to multimodal LLMs → receive analysis.
- Synthetic data notebooks generate datasets → feed into Agents or RAG demos for testing.
- Evaluation notebooks retrieve model outputs → use DeepEval/GiskardAI → produce evaluation reports.
c. Architectural patterns / design principles:
- Modular decomposition: each framework/demo lives in its own folder.
- Pipeline style: data ingestion → indexing → LLM call → post‐processing → evaluation.
- Multi‐agent choreography: agents collaborate via an orchestrator.
- Extensibility points: add new Agent framework simply by adding a folder with notebooks & config.
d. Key technologies & libraries:
- Jupyter Notebooks (UI)
- Agent frameworks: openai-agents-sdk, AutoGen, LlamaIndex, CrewAI, LangChain, Anthropic client
- Retrieval/Index: LlamaIndex packs, prompt caching modules
- Multimodal: GPT-4o, Claude Sonnet, o1 Reasoning, LlamaParse
- Evaluation: DeepEval, GiskardAI
- Data storage: local CSV/PDF, possible cloud buckets (if extended)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Notebook UI Layer
    subgraph "Notebook UI Layer" 
        direction TB
        README["README.md"]:::ui
        AG_NOTES["Agents Notebooks"]:::ui
        RAG_NOTES["RAG Notebooks"]:::ui
        MM_NOTES["Multimodal Notebooks"]:::ui
        SD_NOTES["Synthetic Data Notebooks"]:::ui
        EV_NOTES["Evaluation Notebooks"]:::ui
    end

    %% Orchestration Layer – Agent Frameworks
    subgraph "Orchestration Layer" 
        direction TB
        AG_AUTOGEN["AutoGen Agents"]:::orchestration
        AG_OPENAI["OpenAI Agents SDK"]:::orchestration
        AG_ANTHROPIC["Anthropic Agents"]:::orchestration
        AG_CREWAI["CrewAI Agents"]:::orchestration
        AG_LANGCHAIN["LangChain Agents"]:::orchestration
        AG_SMOL["SmolAgents"]:::orchestration
        AG_LLAMAIDX["LlamaIndex Agents"]:::orchestration
        AG_MCP["MCP (OpenAI Agents SDK)"]:::orchestration
    end

    %% Retrieval / Index Layer
    subgraph "Retrieval / Index Layer"
        direction TB
        CORR_RAG["Corrective RAG Pack"]:::retrieval
        PROMPT_CACHE["Prompt Caching Workflows"]:::retrieval
    end

    %% Multimodal Analysis Layer
    subgraph "Multimodal Analysis Layer"
        direction TB
        MM_CHARTS["Chart/PDF Analysis Notebooks"]:::multimodal
        MM_LOADERS["PDF/Image Loaders"]:::multimodal
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        FIN_DATA["Financial Data (CSV/PY)"]:::data
        SYN_DATA["Synthetic Data Generation"]:::data
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        direction TB
        DEEPEVAL["DeepEval Module"]:::evaluation
        GISKARD["GiskardAI Module"]:::evaluation
    end

    %% External Services Layer
    subgraph "External Services"
        direction TB
        LLM_APIS["External LLM APIs"]:::external
        STORAGE["Storage (S3/Local FS)"]:::external
    end

    %% Reference Papers
    PAPERS["Papers/ Reference Summaries"]:::reference

    %% Connections
    README -->|uses| AG_NOTES
    README -->|uses| RAG_NOTES
    README -->|uses| MM_NOTES
    README -->|uses| SD_NOTES
    README -->|uses| EV_NOTES

    AG_NOTES -->|invokes| AG_AUTOGEN
    AG_NOTES -->|invokes| AG_OPENAI
    AG_NOTES -->|invokes| AG_ANTHROPIC
    AG_NOTES -->|invokes| AG_CREWAI
    AG_NOTES -->|invokes| AG_LANGCHAIN
    AG_NOTES -->|invokes| AG_SMOL
    AG_NOTES -->|invokes| AG_LLAMAIDX
    AG_NOTES -->|invokes| AG_MCP

    AG_AUTOGEN -->|fetches| CORR_RAG
    AG_OPENAI -->|fetches| PROMPT_CACHE
    AG_ANTHROPIC -->|fetches| PROMPT_CACHE

    AG_AUTOGEN -->|calls| LLM_APIS
    AG_OPENAI -->|calls| LLM_APIS
    AG_ANTHROPIC -->|calls| LLM_APIS
    AG_CREWAI -->|calls| LLM_APIS
    AG_LANGCHAIN -->|calls| LLM_APIS
    AG_SMOL -->|calls| LLM_APIS
    AG_LLAMAIDX -->|calls| LLM_APIS
    AG_MCP -->|calls| LLM_APIS

    RAG_NOTES -->|uses| CORR_RAG
    RAG_NOTES -->|uses| PROMPT_CACHE
    RAG_NOTES -->|calls| LLM_APIS

    MM_NOTES -->|loads| MM_CHARTS
    MM_NOTES -->|loads| MM_LOADERS
    MM_CHARTS -->|sends image| LLM_APIS
    MM_LOADERS -->|parses PDF| MM_CHARTS

    SD_NOTES -->|produces| SYN_DATA
    SD_NOTES -->|reads| FIN_DATA
    AG_NOTES -->|tests with| SYN_DATA
    RAG_NOTES -->|tests with| SYN_DATA

    EV_NOTES -->|reads outputs| AG_AUTOGEN
    EV_NOTES -->|reads outputs| RAG_NOTES
    EV_NOTES -->|uses| DEEPEVAL
    EV_NOTES -->|uses| GISKARD
    DEEPEVAL -->|evaluates| AG_NOTES
    GISKARD -->|evaluates| MM_NOTES

    AG_NOTES -->|reads| FIN_DATA
    RAG_NOTES -->|reads| FIN_DATA

    LLM_APIS -->|stores logs| STORAGE

    README -->|references| PAPERS

    %% Click Events Notebook UI
    click README "https://github.com/hananedupouy/llms-in-finance/blob/main/README.md"
    click AG_NOTES "https://github.com/hananedupouy/llms-in-finance/tree/main/Agents"
    click RAG_NOTES "https://github.com/hananedupouy/llms-in-finance/tree/main/RAG"
    click MM_NOTES "https://github.com/hananedupouy/llms-in-finance/tree/main/Multimodal_llms"
    click SD_NOTES "https://github.com/hananedupouy/llms-in-finance/tree/main/Datasets/Synthetic_data"
    click EV_NOTES "https://github.com/hananedupouy/llms-in-finance/tree/main/RAG/evaluation"

    %% Click Events Orchestration Layer
    click AG_AUTOGEN "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/AutoGen/AutoGen_Financial_AI_Agents.ipynb"
    click AG_AUTOGEN "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/AutoGen/Multi_agents_framework_AutoGen_Financial_AI_Agent.ipynb"
    click AG_OPENAI "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/OpenAI/OpenAI_AI_Assistant_Agentic_System.ipynb"
    click AG_OPENAI "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/OpenAI/OpenAI_Agents_SDK_Financial_News_Bot.ipynb"
    click AG_ANTHROPIC "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/Anthropic/Anthropic_Agent_With_LlamaIndex_Financial_data.ipynb"
    click AG_CREWAI "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/CrewAI/CrewAI_Financial_Analysis.ipynb"
    click AG_LANGCHAIN "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/LangChain/Agent_Trading_Strategy_Implementation_with_Mistral_and_LangGraph.ipynb"
    click AG_SMOL "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/SmolAgents/Smolagents_Financial_Analyst.ipynb"
    click AG_LLAMAIDX "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/llamaIndex/Financial_AI_Agent_package_from_LlamaIndex.ipynb"
    click AG_MCP "https://github.com/hananedupouy/llms-in-finance/blob/main/Agents/MCP/mcp_github_openai_agents_sdk.py"

    %% Click Events Retrieval / Index Layer
    click CORR_RAG "https://github.com/hananedupouy/llms-in-finance/tree/main/Agents/CRAG-Corrective-RAG/corrective_rag_pack/llama_index/packs/corrective_rag/"
    click PROMPT_CACHE "https://github.com/hananedupouy/llms-in-finance/blob/main/RAG/Anthropic/Anthropic_Prompt_Caching.ipynb"
    click PROMPT_CACHE "https://github.com/hananedupouy/llms-in-finance/blob/main/RAG/llamaindex/PromptCaching_RAG_Workflow.ipynb"

    %% Click Events Multimodal
    click MM_CHARTS "https://github.com/hananedupouy/llms-in-finance/blob/main/Multimodal_llms/financial_analysis/multimodal_claude3_5_sonnet_gpt4o_claude3_opus_charts_analysis.ipynb"
    click MM_CHARTS "https://github.com/hananedupouy/llms-in-finance/blob/main/Multimodal_llms/financial_analysis/o1_Reasoning_Over_Complex_Financial_Charts.ipynb"
    click MM_LOADERS "https://github.com/hananedupouy/llms-in-finance/blob/main/Multimodal_llms/financial_analysis/Anthropic_PDF_loader_claude_3-5_sonnet_api.ipynb"

    %% Click Events Data Layer
    click FIN_DATA "https://github.com/hananedupouy/llms-in-finance/tree/main/Agents/AutoGen/data/"
    click SYN_DATA "https://github.com/hananedupouy/llms-in-finance/blob/main/Datasets/Synthetic_data/llamaIndex/Synthetic_data_generation_LLamaIndex.ipynb"
    click SYN_DATA "https://github.com/hananedupouy/llms-in-finance/tree/main/Datasets/Synthetic_data/images/"

    %% Click Events Evaluation
    click DEEPEVAL "https://github.com/hananedupouy/llms-in-finance/blob/main/RAG/evaluation/deepeval/README.md"
    click GISKARD "https://github.com/hananedupouy/llms-in-finance/blob/main/RAG/evaluation/giskard_ai/Giskard_AI_LlamaParse_GPT_4o_mini_10K_report.ipynb"

    %% Click Events Reference Papers
    click PAPERS "https://github.com/hananedupouy/llms-in-finance/tree/main/Papers/"

    %% Styles
    classDef ui fill:#f9f,stroke:#333,stroke-width:1px;
    classDef orchestration fill:#bbf,stroke:#333,stroke-width:1px;
    classDef retrieval fill:#bfb,stroke:#333,stroke-width:1px;
    classDef multimodal fill:#fbf,stroke:#333,stroke-width:1px;
    classDef data fill:#ffb,stroke:#333,stroke-width:1px;
    classDef evaluation fill:#fbb,stroke:#333,stroke-width:1px;
    classDef external fill:#cff,stroke:#333,stroke-width:1px;
    classDef reference fill:#ddd,stroke:#333,stroke-width:1px;
```
