# foundryservices/snporacle architecture

> Architecture diagram of the GitHub repository foundryservices/snporacle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/foundryservices/snporacle
- Repository: https://github.com/foundryservices/snporacle

## Overview

##### Project Overview and Purpose
- This project is a specialized financial oracle that leverages neural network models to forecast short-term S&P 500 prices.
- It integrates miners (which run prediction models) and validators (which compare these predictions to real market outcomes) within the broader Bittensor ecosystem.
- Technologies such as TensorFlow, Bittensor, yfinance, and HuggingFace are key, as are machine learning models (stored under “mining_models”) and modular Python components.

##### Analyzing the File Structure
- Top-Level Components:
- Directories like “snp_oracle”, “mining_models”, “docs”, and “tests” indicate that the core functionality is implemented in “snp_oracle”.
- “snp_oracle” is subdivided into:
▪ base_miner: Likely responsible for fetching data (using get_data.py), defining a basic model and prediction logic.
▪ neurons: Contains specialized scripts for miners and validators – here, “miner.py” and “validator.py” indicate components managing prediction and validation logic.
▪ predictionnet: A more advanced set of modules including an API (in the “api” directory), base logic, protocol design, utilities (like configuration, dataset manager, and HuggingFace integration), and separate validator files.
- The “mining_models” directory holds pre-trained model assets (e.g., joblib and h5 files) that the neural network components use.
- Configuration and CI/CD:
- Files like .flake8, Makefile, GitHub workflows in .github/workflows, and environment templates (.env.miner.template, .env.validator.template) imply a well-structured coding, testing, and deployment process.
- The pyproject.toml underscores dependency management (including TensorFlow, Bittensor, and yfinance).

##### Key Components and Their Roles
- Data Input and Preprocessing:
- External data is likely sourced using yfinance (as evident from README badges), feeding market data (S&P 500 prices) into the system.
- The “snp_oracle/base_miner/get_data.py” script suggests a dedicated data fetching and preprocessing module.
- Prediction (Miner) Components:
- “snp_oracle/base_miner/model.py” and “predict.py” are central to generating forecasts.
- “snp_oracle/neurons/miner.py” likely refines these predictions using neural network techniques, relying on pre-trained models in “mining_models”.
- Integration with HuggingFace (in utilities) provides an additional layer for open-sourcing or reloading models.
- Validation Components:
- “snp_oracle/neurons/validator.py” and “snp_oracle/predictionnet/validator” handle the scoring of miner predictions against actual market moves.
- Metrics such as Directional Accuracy and Mean Absolute Error (from README) are computed here.
- API and External Interaction:
- The “snp_oracle/predictionnet/api” folder contains endpoints (example.py, prediction.py, metagraph.py) and query utilities, providing a way for external systems or users to interact with the oracle.
- Utility and Configuration:
- The utilities (config.py, dataset_manager.py, uids.py, misc.py) layer supports configuration, data management, and ensures smooth operation across modules.

##### Architectural Patterns
- Layered/Modular Architecture:
- Separate layers for data ingestion (base_miner), processing and prediction (neurons and mining_models), interfacing (predictionnet API), and validation.
- Separation of Concerns:
- Clear distinctions exist between miners (predicting prices), validators (ensuring prediction accuracy), and the API (external access).
- Integration with External Services:
- CI/CD and Testing:
- Though not part of runtime architecture, including a note for deployment processes with GitHub workflows and testing (in “tests” folder) can help illustrate overall development and deployment quality practices.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Sources
    MD["Market Data (yfinance)"]:::external

    %% Data Ingestion
    DI["Data Ingestion (get_data.py)"]:::internal
    MD -->|"fetches"| DI

    %% Subgraph: Base Miner (snp_oracle/base_miner)
    subgraph "snp_oracle/base_miner"
        BM1["Model (model.py)"]:::internal
        BM2["Predict (predict.py)"]:::internal
        BM1 -->|"generates"| BM2
    end
    DI -->|"prepares data"| BM1

    %% Miner Neural Network Component
    MNN["Miner Neural Network (miner.py)"]:::internal
    BM2 -->|"feeds prediction"| MNN

    %% Pre-Trained Models Dependency
    PT["Pre-Trained Models (mining_models)"]:::internal
    PT -->|"supplies models"| MNN

    %% Validator Modules
    V1["Validator (neurons/validator.py)"]:::internal
    AV["Advanced Validator (predictionnet/validator)"]:::internal
    MNN -->|"sends output"| V1
    MNN -->|"sends output"| AV

    %% Prediction API Layer
    API["Prediction API (predictionnet/api)"]:::api
    MNN -->|"offers results"| API
    V1 -->|"reports metrics"| API
    AV -->|"reports metrics"| API

    %% Utility and Configuration
    UTIL["Utilities & Config (predictionnet/utils)"]:::utility
    UTIL -->|"supports"| MNN
    UTIL -->|"supports"| API

    %% External Dependencies
    TF["TensorFlow"]:::external
    BT["Bittensor"]:::external
    HF["HuggingFace"]:::external
    TF -->|"computes"| MNN
    BT -->|"integrates"| MNN
    HF -->|"integrates"| UTIL

    %% CI/CD and Testing (Deployment Pipelines)
    CI["CI/CD & Testing (GitHub workflows, tests)"]:::internal
    CI -.->|"deploys"| API

    %% Click Events
    click DI "https://github.com/foundryservices/snporacle/blob/main/snp_oracle/base_miner/get_data.py"
    click BM1 "https://github.com/foundryservices/snporacle/blob/main/snp_oracle/base_miner/model.py"
    click BM2 "https://github.com/foundryservices/snporacle/blob/main/snp_oracle/base_miner/predict.py"
    click MNN "https://github.com/foundryservices/snporacle/blob/main/snp_oracle/neurons/miner.py"
    click PT "https://github.com/foundryservices/snporacle/tree/main/mining_models"
    click V1 "https://github.com/foundryservices/snporacle/blob/main/snp_oracle/neurons/validator.py"
    click AV "https://github.com/foundryservices/snporacle/tree/main/snp_oracle/predictionnet/validator"
    click API "https://github.com/foundryservices/snporacle/tree/main/snp_oracle/predictionnet/api"
    click UTIL "https://github.com/foundryservices/snporacle/tree/main/snp_oracle/predictionnet/utils"

    %% Style Definitions
    classDef internal fill:#ffebcd,stroke:#8b4513,stroke-width:2px;
    classDef external fill:#add8e6,stroke:#00008b,stroke-width:2px;
    classDef api fill:#90ee90,stroke:#006400,stroke-width:2px;
    classDef utility fill:#d3d3d3,stroke:#696969,stroke-width:2px;
```
