# fastmachinelearning/ml4fg architecture

> Architecture diagram of the GitHub repository fastmachinelearning/ml4fg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fastmachinelearning/ml4fg
- Repository: https://github.com/fastmachinelearning/ml4fg

## Overview

##### Project type and purpose
- This is an open-source tutorial/reference design for “Machine Learning for Frame Grabbers.”
- It combines Python-based model definition (Jupyter notebooks + hls4ml) with FPGA hardware build flows (Vivado HLS, VHDL, Tcl scripts) to deploy neural networks onto Euresys frame-grabber FPGAs in high-throughput imaging.

##### File structure
- Top‐level notebooks:
- hls4ml-frame-grabber_tutorial.ipynb (introductory flow)
- part9_FOLO_frame_grabbers_advanced_features.ipynb (YOLO-style advanced example)
- environment.yml defines the Conda env (Python, hls4ml, Euresys SDK, etc.)
- weights/ holds pretrained model files (h5, quantized, pruned)
- images/MNIST as sample image data
- lambda_cpp/ contains helper headers for layer reductions
- board_files/ is the heart of the FPGA flow, subdivided into:
- 05_model_design_hls/ (HLS C++ sources, headers, testbench data, run_hls.tcl)
- CustomLogic.vhd and related Tcl scripts (Vivado project creation, synthesis, implementation)
- platform‐specific directories (octo/, quad/) each with CustomLogic.h and Vivado‐flow Tcl scripts
- .gitignore, README.md, customlogic_functions.tcl, result_writeout.v, signal_inference.v

##### Extract architecture insights from README
- Uses hls4ml to convert Keras/TensorFlow models to HLS C++
- Leverages CustomLogic toolkit to glue neural‐network IP with frame‐grabber interfaces
- Workflow: define/train model → export weights → hls4ml generates HLS code → Vivado HLS → Vivado implementation → FPGA bitstream → frame-grabber for in situ inference

##### Main components & their relationships
a. Host environment (PC)
- Python + Jupyter notebooks
- Conda environment (hls4ml, Euresys SDK)
- Weights store & test images
b. ML conversion layer
- hls4ml library converts Keras model + weights → HLS C++ sources (myproject_axi.cpp/.h)
- lambda_cpp utilities for layer reductions
c. HLS testbench
- tb_data/ holds input/output golden data
- run_hls.tcl drives Vivado HLS for synthesis & C/RTL cosimulation
d. CustomLogic integration
- CustomLogic.vhd (wrapper around the hls4ml core)
- platform-specific headers (CustomLogic.h)
- Tcl scripts for creating Vivado projects and running implementation (create_vivado_project.tcl, run_impl.tcl)
e. FPGA hardware
- Frame-grabber FPGA on Euresys device (connected to imaging sensor)
- myproject_axi IP (neural network accelerator) + result_writeout.v + signal_inference.v
f. I/O flow
- Sensor → FPGA front end → neural-network IP → post-processing → host via PCIe/Ethernet

##### Architectural patterns & design principles
- Clear separation of concerns: Python layer vs. HLS layer vs. FPGA integration
- Modular hardware IP: hls4ml‐generated cores + CustomLogic wrapper + testbench + post-processing modules
- Reusable platform flows: identical Tcl scripts for multiple board configurations (octo, quad)
- Dataflow pipeline: streaming image data → inference → streaming results

a. Components to include
1. Host PC block (Python/Jupyter, hls4ml, weights repo, notebooks)
2. hls4ml conversion block (model import, code generation)
3. Vivado HLS block (run_hls.tcl, myproject_axi.cpp/.h, tb_data)
4. Vivado Implementation block (CustomLogic.vhd, create_vivado_project.tcl, run_impl.tcl)
5. FPGA block (frame-grabber, sensor I/O, neural accelerator IP, result_writeout, signal_inference)
6. Imaging sensor and external storage (optional)
b. Interactions & data flows
c. Technologies & labels
– Blue for software components (Python, hls4ml, notebooks)
– Green for HLS/C++ IP
– Orange for FPGA/hardware modules (VHDL, CustomLogic)
– Gray for platform scripts (Tcl)
– Rectangles for processes/modules
– Clouds for external dependencies (Euresys SDK)
d. Annotations
- Input formats (raw image formats from images/MNIST) and output formats (prediction stream)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Host Environment
    subgraph "Host Environment" 
        direction TB
        Notebook1["hls4ml-frame-grabber_tutorial.ipynb"]:::software
        Notebook2["part9_FOLO_frame_grabbers_advanced_features.ipynb"]:::software
        Env["environment.yml"]:::scripts
        Readme["README.md"]:::software
        WeightsCyl[(Weights Store)]:::storage
        ImagesCyl[(Sample Image Data)]:::storage
    end

    %% ML Conversion Layer
    subgraph "ML Conversion" 
        direction TB
        HLS4ML["hls4ml Library"]:::cloud
        Lambda["nnet_reduce.h"]:::hls
    end

    %% HLS Design & Testbench
    subgraph "Vivado HLS Flow" 
        direction TB
        A1["myproject_axi.cpp"]:::hls
        A2["myproject_axi.h"]:::hls
        A3["myproject_test.cpp"]:::hls
        TBin[(tb_input_features.dat)]:::storage
        TBout[(tb_output_predictions.dat)]:::storage
        RunHLS["run_hls.tcl"]:::scripts
    end

    %% CustomLogic Integration
    subgraph "CustomLogic Integration" 
        direction TB
        CLVHDL["CustomLogic.vhd"]:::hardware
        CLFuncs["customlogic_functions.tcl"]:::scripts
        Wrp["myproject_axi_wrp.vhd"]:::hardware
    end

    %% Platform-Specific Implementation
    subgraph "Platform Flows" 
        direction TB
        subgraph "Octo Platform"
            O1["CustomLogic.h"]:::hardware
            O2["create_vivado_project.tcl"]:::scripts
            O3["run_impl.tcl"]:::scripts
        end
        subgraph "Quad Platform"
            Q1["CustomLogic.h"]:::hardware
            Q2["create_vivado_project.tcl"]:::scripts
            Q3["run_impl.tcl"]:::scripts
        end
    end

    %% FPGA & I/O
    subgraph "FPGA & I/O" 
        direction TB
        Sensor["Imaging Sensor"]:::hardware
        FPGA["Frame-Grabber FPGA"]:::hardware
        IP["myproject_axi IP"]:::hardware
        Post1["result_writeout.v"]:::hardware
        Post2["signal_inference.v"]:::hardware
        HostResults["Host Results Stream"]:::cloud
    end

    %% Flows
    Notebook1 -->|uses| Env
    Notebook2 -->|uses| Env
    Notebook1 -->|reads/writes| WeightsCyl
    Notebook2 -->|reads/writes| WeightsCyl
    Notebook1 -->|reads| ImagesCyl
    HLS4ML -->|generates| A1
    HLS4ML -->|generates| A2
    Lambda -->|supports| A1
    Env -->|provides| HLS4ML
    A1 & A2 & A3 -->|input data| TBin
    RunHLS -->|drives| A1
    RunHLS -->|drives| A2
    RunHLS -->|drives| A3
    TBin -->|validate| A3
    TBout -->|validate| A3
    A1 & A2 -->|wrapped by| CLVHDL
    CLFuncs -->|configures| CLVHDL
    CLVHDL -->|wrapped into| Wrp
    Wrp -->|packaged into| FPGA
    O1 & O2 & O3 -->|implements| FPGA
    Q1 & Q2 & Q3 -->|implements| FPGA
    Sensor -->|stream| FPGA
    FPGA -->|accelerates via| IP
    IP -->|post-process| Post1
    IP -->|post-process| Post2
    Post1 & Post2 -->|send results| HostResults

    %% Click Events
    click Notebook1 "https://github.com/fastmachinelearning/ml4fg/blob/master/hls4ml-frame-grabber_tutorial.ipynb"
    click Notebook2 "https://github.com/fastmachinelearning/ml4fg/blob/master/part9_FOLO_frame_grabbers_advanced_features.ipynb"
    click Env "https://github.com/fastmachinelearning/ml4fg/blob/master/environment.yml"
    click Readme "https://github.com/fastmachinelearning/ml4fg/blob/master/README.md"
    click WeightsCyl "https://github.com/fastmachinelearning/ml4fg/blob/master/weights/folo_weights.h5"
    click WeightsCyl "https://github.com/fastmachinelearning/ml4fg/blob/master/weights/folo_weights_qat.h5"
    click WeightsCyl "https://github.com/fastmachinelearning/ml4fg/blob/master/weights/folo_weights_qat_pruned.h5"
    click ImagesCyl "https://github.com/fastmachinelearning/ml4fg/tree/master/images/MNIST/"
    click Lambda "https://github.com/fastmachinelearning/ml4fg/blob/master/lambda_cpp/nnet_reduce.h"
    click A1 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/05_model_design_hls/srcs/myproject_axi.cpp"
    click A2 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/05_model_design_hls/srcs/myproject_axi.h"
    click A3 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/05_model_design_hls/srcs/myproject_test.cpp"
    click TBin "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/05_model_design_hls/tb_data/tb_input_features.dat"
    click TBout "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/05_model_design_hls/tb_data/tb_output_predictions.dat"
    click RunHLS "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/05_model_design_hls/scripts/run_hls.tcl"
    click CLVHDL "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/CustomLogic.vhd"
    click CLFuncs "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/customlogic_functions.tcl"
    click Wrp "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/myproject_axi_wrp.vhd"
    click O1 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/octo/CustomLogic.h"
    click O2 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/octo/create_vivado_project.tcl"
    click O3 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/octo/run_impl.tcl"
    click Q1 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/quad/CustomLogic.h"
    click Q2 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/quad/create_vivado_project.tcl"
    click Q3 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/quad/run_impl.tcl"
    click Post1 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/result_writeout.v"
    click Post2 "https://github.com/fastmachinelearning/ml4fg/blob/master/board_files/signal_inference.v"

    %% Styles
    classDef software fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef hls fill:#90EE90,stroke:#333,stroke-width:1px
    classDef hardware fill:#FFA500,stroke:#333,stroke-width:1px
    classDef scripts fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef storage fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef cloud fill:#F0F8FF,stroke:#333,stroke-width:1px
```
