How marks0kolov/touch-grass works
Project Type and Purpose
• This is a lightweight, full-stack AI application that combines:
a) A static web GUI (HTML/CSS/JavaScript) running in the browser via a simple HTTP server.
b) A Python-based inference layer that loads two ONNX models (grassNet and poseNet) to decide if an uploaded image shows “touching grass.”
c) A training pipeline (separate from runtime) used to fine-tune the ResNet-18 grass classifier and to adopt an existing hand-pose model.
File Structure Analysis
• Top-level files:
– README.md, LICENSE, .gitignore, presentation.pdf
• gui/
– index.html, script.js, styles.css → Static front-end assets.
• models/
– grassNet/ (labels.txt, model.onnx)
– poseNet/ (hand_pose.json, model.onnx) → Persisted ML artifacts.
• scripts/
– check_grass.py, check_hand.py, check_touching_grass.py → Python CLI entrypoints for inference.
– requirements.txt → Python dependencies (e.g., onnxruntime, numpy, Pillow).
README Insights
• Describes two modes of use: command-line inference and GUI with a recurring timer.
• CLI mode loads both ONNX models via onnxruntime and prints a boolean result.
• GUI mode: user sets a timer, upon timeout the page prompts for an image upload; the alarm persists until a valid “touch grass” photo is provided.
• Training is done offline with a PyTorch classification script (ResNet-18) and external datasets; not part of the deployed runtime.
Main Components
a. Front-End (Browser)
– HTML/CSS (gui/index.html, styles.css)
– JavaScript (gui/script.js): timer logic, file-upload UI, AJAX/fetch calls to inference service (or wrapper).
b. Static HTTP Server
– Python’s built-in HTTP server (python3 -m http.server) serving the gui/ folder.
c. Inference Service
– Python scripts (check_touching_grass.py)
– ONNX Runtime (runtime engine)
– ML Models (grassNet.onnx, poseNet.onnx)
d. Training Pipeline (offline)
– PyTorch training script (external train.py)
– Dataset sources (roboflow, kaggle, Places365)
– Versioned model artifacts committed to models/
e. External Services & Data
– ONNX Model Zoo (for Pose-ResNet)
– NVIDIA Jetson-Orin Nano (target hardware)
Relationships and Interactions
1. User → Browser GUI: set timer → on timeout, select image.
2. Browser → Inference Service: upload image via HTTP POST or local CLI invocation.
3. Inference Service loads ONNX models via ONNX Runtime, runs two inferences (grassNet, poseNet), applies business logic (AND condition) → returns result.
4. Browser updates UI (stop alarm if result is true).
5. Training Pipeline (offline) produces updated model.onnx → developer commits to models/
Architectural Patterns & Design Principles
• Separation of Concerns: static UI vs. inference logic vs. training pipeline.
• Layered architecture: Presentation (GUI) → Application (Inference) → Data/Model (ONNX Artifacts).
• Reuse of standard components (ONNX Runtime, built-in HTTP server).
• Configurability: CLI scripts accept model paths as arguments.
Technologies and Libraries
• Front-end: HTML5, CSS3, Vanilla JS (fetch API)
• Back-end/Inference: Python 3.x, onnxruntime, numpy, Pillow
• Model Training: PyTorch (via dusty-nv train.py), ResNet-18
• Hosting: python3 -m http.server (port 1008)
Critical Details to Capture
• Two distinct inference sub-models and how they combine.
• The lack of a heavyweight web-framework — using the built-in HTTP server.
• Offline vs. online components.
• The flow from user action (timer → upload) to Python inference and back to UI.