# flowdriveai/flowpilot architecture

> Architecture diagram of the GitHub repository flowdriveai/flowpilot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flowdriveai/flowpilot
- Repository: https://github.com/flowdriveai/flowpilot

## Overview

##### Project type and purpose
- Project type: Full-stack, multi-platform driver assistance system
- Purpose: Runs on Windows/Linux PCs or Android devices to provide ACC, ALC, FCW, LDW, and DM by orchestrating hardware, sensors, ML inference, control algorithms, and user interfaces.

##### File structure
- Top-level “android”, “desktop”, “headless” → three front-end targets (Android app, desktop OpenGL UI, headless CLI).
- “selfdrive” → core daemons (boardd, sensord, modeld, controlsd, calibrationd, locationd, loggerd, thermald, manager, ui).
- “common” → shared libraries (C/C++/Python/Java) for data structures, FFI, Kalman filters, transformations, logging, timing, file I/O.
- “system/hardware” → hardware abstraction across platforms.
- “libs” → third-party dependencies (acados MPC, ONNX/TNN, OpenCV, ZeroMQ).
- Build scripts: SConstruct/SConscript (Python build), Gradle (Java/Android), Pipfile (Python deps).

##### README for additional insights
- Describes Flowpilot as an extension/fork of openpilot.
- Emphasizes support for real cars (via Panda CAN device) and virtual cars (CARLA, FlowStreamer).
- Mentions logging of sensor and user data for offline analysis.

##### Main components and their interactions
a. Hardware layer
– Panda device (black/red/white) connects via USB to host → interfaces with car CAN bus.
– Cameras (road-facing, driver-facing), IMU, GPS, thermal sensors.
b. OS & Abstraction
– system/hardware module abstracts platform specifics (PC, Android, TICI).
– Panda driver in boardd talks to hardware via libusb/CAN APIs.
c. Core daemons (selfdrive)
– sensord: captures sensors & camera frames → publishes messages on ZMQ/CAPNP bus.
– locationd: GNSS processing, Kalman filters for localization.
– modeld: runs neural nets (ONNX/TNN/SNPE) → produces perception outputs.
– controlsd: vehicle control (longitudinal & lateral planners, MPC via acados, PID).
– boardd: sends control commands to CAN, receives CAN data, time sync.
– calibrationd: camera intrinsics calibration routines.
– loggerd: records all message traffic and camera streams to disk.
– thermald: monitors hardware temperature.
– manager: lifecycle manager for daemons (start/stop/restart) based on services.yaml.
– ui: overlays, maps, driver-monitoring, settings; publishes user inputs.
d. Front-ends
– Android app (“android/…”): Java/NDK code, SNPEModelRunner, CameraManager → talks to core daemons over local socket or over IPC bus.
– Desktop app (“desktop/…”): Java LWJGL UI.
– Headless CLI (“headless/…”): minimal Java entry point for command-line testing.
e. Tools & Simulation
– CARLA integration via tools/sim.
– FlowStreamer video injection.
– Plotting and joystick tools.

##### Architectural patterns & design principles
- Microservices/Daemon pattern: each selfdrive component is a separate process communicating over a message bus.
- Pub-sub via ZeroMQ/CAPNP for loose coupling and replay capability.
- Hardware abstraction layer hides platform specifics.
- Shared common library for data types, FFI, transforms, logging.
- Build-time code generation (kaitai, acados templates).
- Cross-platform mobile & desktop UIs sharing core daemons.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Hardware" 
        Panda["Panda CAN Device"]:::hardware
        Sensors["Sensors (Camera, IMU, GPS, Thermal)"]:::hardware
    end

    subgraph "Hardware Abstraction Layer"
        HAL["system/hardware"]:::hardware
    end

    MessageBus["ZMQ/CAPNP Bus"]:::bus

    subgraph "Core Daemons"
        SD["sensord"]:::daemons
        LD["locationd"]:::daemons
        MD["modeld"]:::daemons
        CD["controlsd"]:::daemons
        BD["boardd"]:::daemons
        CalD["calibrationd"]:::daemons
        LogD["loggerd"]:::daemons
        ThermD["thermald"]:::daemons
        Mgr["manager"]:::manager
        UI["ui"]:::ui
    end

    subgraph "Front-ends"
        Android["Android App"]:::frontend
        Desktop["Desktop UI"]:::frontend
        Headless["Headless CLI"]:::frontend
    end

    subgraph "Tools & Simulation"
        Sim["Simulation (CARLA)"]:::tools
        VInject["FlowStreamer"]:::tools
        Joystick["Joystick Tool"]:::tools
        Plot["Plotting Tool"]:::tools
    end

    %% Flows
    Panda -->|CAN/USB| HAL
    Sensors -->|Data| HAL
    HAL -->|Abstracted I/O| MessageBus
    MessageBus -->|pub-sub| SD
    SD -->|sensor data| MessageBus
    MessageBus -->|GNSS data| LD
    LD -->|localization| MessageBus
    MessageBus -->|images| MD
    MD -->|perception| MessageBus
    MessageBus -->|planning| CD
    CD -->|control commands| MessageBus
    MessageBus -->|CAN cmds| BD
    BD -->|CAN bus| Panda
    MessageBus -->|logs| LogD
    allDaemons(( )):::invisible
    subgraph "Lifecycle & UI"
        Mgr -->|manage| SD
        Mgr -->|manage| LD
        Mgr -->|manage| MD
        Mgr -->|manage| CD
        Mgr -->|manage| BD
        Mgr -->|manage| CalD
        Mgr -->|manage| LogD
        Mgr -->|manage| ThermD
        Mgr -->|manage| UI
        UI -->|user events| MessageBus
    end

    Android -->|IPC| MessageBus
    Desktop -->|IPC| MessageBus
    Headless -->|IPC| MessageBus

    Sim -->|inject| MessageBus
    VInject -->|video stream| MessageBus
    Joystick -->|control| MessageBus
    Plot -->|visualize| MessageBus

    %% Click Events
    click HAL "https://github.com/flowdriveai/flowpilot/tree/master/system/hardware"
    click BD "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/boardd"
    click SD "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/sensord"
    click LD "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/locationd"
    click MD "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/modeld"
    click CD "https://github.com/flowdriveai/flowpilot/blob/master/selfdrive/controls/controlsd.py"
    click CD "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/controls/lib"
    click CalD "https://github.com/flowdriveai/flowpilot/blob/master/selfdrive/calibration/calibrationd.py"
    click LogD "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/loggerd"
    click ThermD "https://github.com/flowdriveai/flowpilot/blob/master/selfdrive/thermald/thermald.py"
    click Mgr "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/manager"
    click UI "https://github.com/flowdriveai/flowpilot/tree/master/selfdrive/ui"
    click Android "https://github.com/flowdriveai/flowpilot/tree/master/android/"
    click Desktop "https://github.com/flowdriveai/flowpilot/tree/master/desktop/"
    click Headless "https://github.com/flowdriveai/flowpilot/tree/master/headless/"
    click Sim "https://github.com/flowdriveai/flowpilot/tree/master/tools/sim"
    click VInject "https://github.com/flowdriveai/flowpilot/tree/master/scripts/flowpilot-setup-env-android"
    click VInject "https://github.com/flowdriveai/flowpilot/tree/master/tools/flow-player"
    click Joystick "https://github.com/flowdriveai/flowpilot/blob/master/tools/joystick/joystickd.py"
    click Plot "https://github.com/flowdriveai/flowpilot/tree/master/tools/plotjuggler"
    click ThirdParty "https://github.com/flowdriveai/flowpilot/tree/master/libs/acados"
    click Common "https://github.com/flowdriveai/flowpilot/tree/master/common/"
    click ODBC "https://github.com/flowdriveai/flowpilot/tree/master/opendbc/"
    click PandaUtil "https://github.com/flowdriveai/flowpilot/tree/master/panda/"
    click Build "https://github.com/flowdriveai/flowpilot/tree/master/SConstruct"
    click Build "https://github.com/flowdriveai/flowpilot/tree/master/SConscript"
    click Build "https://github.com/flowdriveai/flowpilot/blob/master/build.gradle"
    click Build "https://github.com/flowdriveai/flowpilot/tree/master/Pipfile"

    %% Styles
    classDef hardware fill:#e0e0e0,stroke:#666,stroke-width:1px
    classDef daemons fill:#cce5ff,stroke:#005cbf,stroke-width:1px
    classDef manager fill:#d4edda,stroke:#155724,stroke-width:1px
    classDef ui fill:#d4edda,stroke:#155724,stroke-width:1px
    classDef frontend fill:#d1ecf1,stroke:#0c5460,stroke-width:1px
    classDef tools fill:#fff3cd,stroke:#856404,stroke-width:1px
    classDef bus fill:#f8d7da,stroke:#721c24,stroke-width:1px
    classDef invisible fill:none,stroke:none
```
