How texasinstruments/edgeai-tidl-tools works
Project Type and Purpose
• This is an open-source toolchain (“Edge AI TIDL Tools”) for compiling and running DNNs on TI embedded devices. It provides:
• A model‐compilation pipeline (Python & C++ APIs)
• Integration with three “Open Source Runtimes” (TFLite, ONNX Runtime, TVM/DLR)
• Examples, scripts, test data, and Docker support for both X86_PC host and TI SOC target
File‐Structure Analysis
• Top‐level directories
• examples/ – CMake–based C++ examples (examples/osrt_cpp), Python APIs (examples/osrt_python), Jupyter notebooks, model prototxts
• osrt-model-tools/ – Python package for ONNX and TFLite model optimization (src/ops, utilities, setup.py)
• scripts/ – Docker build/run, orchestration scripts to compile and run examples, debug tools
• test_data/ – golden outputs for validation
• setup.sh, setup_env.sh – environment-variable setup scripts
• Architectural patterns
• Layered: API layer → compilation layer → runtime layer → hardware abstraction
• Plugin architecture: OSRT runtimes load TIDL delegate to offload to DSP/MMA
• Build orchestration via CMake for C++ and pip/setup.py for Python
Insights from README
• Two main streams: Data‐scientist workflow (model compile on X86 → model artifacts) and Embedded‐engineer workflow (inference on X86_PC or TI SOC)
• Three supported runtimes with identical high‐level flow (compile → generate artifacts → load + infer)
• Setup paths: native or Docker for host, CMake build for C++ examples, Python APIs for scripting
Components
a. Main Components
• Host Environment (X86_PC)
– Setup Scripts (setup.sh, Dockerfiles)
– Python API & osrt_model_tools (model optimizer)
– C++ Examples / CMake build system
– Runtimes (TFLite, ONNX Runtime, TVM/DLR)
– TIDL Delegate / DSP Compiler
– Model Artifacts, Output Binaries, Output Images (file stores)
– Orchestration Scripts (run_python_examples.sh, run_compile_model_zoo.sh)
• Target Environment (TI SOC)
– Deployed Model Artifacts
– Python & C++ Inference Apps
– TIDL Runtime Libraries (lib files)
– On-device hardware: Cortex-A cores, C7x DSP, MMA accelerator
• External Services
– Docker Engine
– pip / system package manager
– Git (versioning)
b. Relationships & Data Flow
1. User invokes “compile” via Python CLI or C++ example →
2. osrt_model_tools / C++ compiler loads ONNX/TFLite model →
4. Inference layer loads artifacts into chosen OSRT runtime →
5. Runtime schedules layers: unsupported layers run on ARM (via CPU kernels), supported layers offloaded to C7x/MMA through TIDL delegate →
6. Outputs written back to file store → comparison/validation (scripts, test_data)
c. Architectural Patterns & Principles
• Layer separation: clear API vs compile vs runtime vs hardware abstraction
• Plugin/Delegate pattern for runtime extensibility
• Scripted orchestration for reproducible builds & benchmarks
• Modular Python package for model optimizations
d. Key Technologies & Frameworks
• CMake, GCC/Clang (C++ build)
• Python 3.10, pip, setuptools (osrt_model_tools)
• Docker (optional host setup)
• ONNX Runtime, TensorFlow Lite, TVM/DLR
• TI’s TIDL libraries (DSP compiler & runtime)