How fastmachinelearning/ml4fg works
Project type and purpose
• This is an open-source tutorial/reference design for “Machine Learning for Frame Grabbers.”
• It combines Python-based model definition (Jupyter notebooks + hls4ml) with FPGA hardware build flows (Vivado HLS, VHDL, Tcl scripts) to deploy neural networks onto Euresys frame-grabber FPGAs in high-throughput imaging.
File structure
• Top‐level notebooks:
• hls4ml-frame-grabber_tutorial.ipynb (introductory flow)
• part9_FOLO_frame_grabbers_advanced_features.ipynb (YOLO-style advanced example)
• environment.yml defines the Conda env (Python, hls4ml, Euresys SDK, etc.)
• weights/ holds pretrained model files (h5, quantized, pruned)
• images/MNIST as sample image data
• lambda_cpp/ contains helper headers for layer reductions
• board_files/ is the heart of the FPGA flow, subdivided into:
• 05_model_design_hls/ (HLS C++ sources, headers, testbench data, run_hls.tcl)
• CustomLogic.vhd and related Tcl scripts (Vivado project creation, synthesis, implementation)
• platform‐specific directories (octo/, quad/) each with CustomLogic.h and Vivado‐flow Tcl scripts
• .gitignore, README.md, customlogic_functions.tcl, result_writeout.v, signal_inference.v
Extract architecture insights from README
• Uses hls4ml to convert Keras/TensorFlow models to HLS C++
• Leverages CustomLogic toolkit to glue neural‐network IP with frame‐grabber interfaces
• Workflow: define/train model → export weights → hls4ml generates HLS code → Vivado HLS → Vivado implementation → FPGA bitstream → frame-grabber for in situ inference
Main components & their relationships
a. Host environment (PC)
• Python + Jupyter notebooks
• Conda environment (hls4ml, Euresys SDK)
• Weights store & test images
b. ML conversion layer
• hls4ml library converts Keras model + weights → HLS C++ sources (myproject_axi.cpp/.h)
• lambda_cpp utilities for layer reductions
c. HLS testbench
• tb_data/ holds input/output golden data
• run_hls.tcl drives Vivado HLS for synthesis & C/RTL cosimulation
d. CustomLogic integration
• CustomLogic.vhd (wrapper around the hls4ml core)
• platform-specific headers (CustomLogic.h)
• Tcl scripts for creating Vivado projects and running implementation (create_vivado_project.tcl, run_impl.tcl)
e. FPGA hardware
• Frame-grabber FPGA on Euresys device (connected to imaging sensor)
• myproject_axi IP (neural network accelerator) + result_writeout.v + signal_inference.v
f. I/O flow
• Sensor → FPGA front end → neural-network IP → post-processing → host via PCIe/Ethernet
Architectural patterns & design principles
• Clear separation of concerns: Python layer vs. HLS layer vs. FPGA integration
• Modular hardware IP: hls4ml‐generated cores + CustomLogic wrapper + testbench + post-processing modules
• Reusable platform flows: identical Tcl scripts for multiple board configurations (octo, quad)
• Dataflow pipeline: streaming image data → inference → streaming results
a. Components to include
1. Host PC block (Python/Jupyter, hls4ml, weights repo, notebooks)
2. hls4ml conversion block (model import, code generation)
3. Vivado HLS block (run_hls.tcl, myproject_axi.cpp/.h, tb_data)
4. Vivado Implementation block (CustomLogic.vhd, create_vivado_project.tcl, run_impl.tcl)
5. FPGA block (frame-grabber, sensor I/O, neural accelerator IP, result_writeout, signal_inference)
6. Imaging sensor and external storage (optional)
b. Interactions & data flows
c. Technologies & labels
– Blue for software components (Python, hls4ml, notebooks)
– Green for HLS/C++ IP
– Orange for FPGA/hardware modules (VHDL, CustomLogic)
– Gray for platform scripts (Tcl)
– Rectangles for processes/modules
– Clouds for external dependencies (Euresys SDK)
d. Annotations
• Input formats (raw image formats from images/MNIST) and output formats (prediction stream)