How flowdriveai/flowpilot works
Project type and purpose
• Project type: Full-stack, multi-platform driver assistance system
• Purpose: Runs on Windows/Linux PCs or Android devices to provide ACC, ALC, FCW, LDW, and DM by orchestrating hardware, sensors, ML inference, control algorithms, and user interfaces.
File structure
• Top-level “android”, “desktop”, “headless” → three front-end targets (Android app, desktop OpenGL UI, headless CLI).
• “selfdrive” → core daemons (boardd, sensord, modeld, controlsd, calibrationd, locationd, loggerd, thermald, manager, ui).
• “common” → shared libraries (C/C++/Python/Java) for data structures, FFI, Kalman filters, transformations, logging, timing, file I/O.
• “system/hardware” → hardware abstraction across platforms.
• “libs” → third-party dependencies (acados MPC, ONNX/TNN, OpenCV, ZeroMQ).
• Build scripts: SConstruct/SConscript (Python build), Gradle (Java/Android), Pipfile (Python deps).
README for additional insights
• Describes Flowpilot as an extension/fork of openpilot.
• Emphasizes support for real cars (via Panda CAN device) and virtual cars (CARLA, FlowStreamer).
• Mentions logging of sensor and user data for offline analysis.
Main components and their interactions
a. Hardware layer
– Panda device (black/red/white) connects via USB to host → interfaces with car CAN bus.
– Cameras (road-facing, driver-facing), IMU, GPS, thermal sensors.
b. OS & Abstraction
– system/hardware module abstracts platform specifics (PC, Android, TICI).
– Panda driver in boardd talks to hardware via libusb/CAN APIs.
c. Core daemons (selfdrive)
– sensord: captures sensors & camera frames → publishes messages on ZMQ/CAPNP bus.
– locationd: GNSS processing, Kalman filters for localization.
– modeld: runs neural nets (ONNX/TNN/SNPE) → produces perception outputs.
– controlsd: vehicle control (longitudinal & lateral planners, MPC via acados, PID).
– boardd: sends control commands to CAN, receives CAN data, time sync.
– calibrationd: camera intrinsics calibration routines.
– loggerd: records all message traffic and camera streams to disk.
– thermald: monitors hardware temperature.
– manager: lifecycle manager for daemons (start/stop/restart) based on services.yaml.
– ui: overlays, maps, driver-monitoring, settings; publishes user inputs.
d. Front-ends
– Android app (“android/…”): Java/NDK code, SNPEModelRunner, CameraManager → talks to core daemons over local socket or over IPC bus.
– Desktop app (“desktop/…”): Java LWJGL UI.
– Headless CLI (“headless/…”): minimal Java entry point for command-line testing.
e. Tools & Simulation
– CARLA integration via tools/sim.
– FlowStreamer video injection.
– Plotting and joystick tools.
Architectural patterns & design principles
• Microservices/Daemon pattern: each selfdrive component is a separate process communicating over a message bus.
• Pub-sub via ZeroMQ/CAPNP for loose coupling and replay capability.
• Hardware abstraction layer hides platform specifics.
• Shared common library for data types, FFI, transforms, logging.
• Build-time code generation (kaitai, acados templates).
• Cross-platform mobile & desktop UIs sharing core daemons.