Low-power Computer Vision

Embedded Computer Vision is a highly multi-disciplinary field that requires expertise in optics, image sensors, hardware, firmware and so on. As a result, the bar to playing with technology in this field is quite high.

The Vision FPGA SoM is a Lattice FPGA based System on Module that integrates an ultra-low-power vision sensor, a 3-axis accelerometer/gyroscope, and an I²S MEMS microphone in a small form-factor (3 cm x 2 cm).

This device is built for developers who want to not only play with this technology, but also have a path to integrating it into real world products. We designed the device with the following thoughts in mind:

• Low Power Image Sensor + FPGA: Overall power consumption is minimized and in the 10-20 mW range to enable battery powered applications.
• Well-designed Host Interface: Easy to integrate into a larger system using an interrupt driven SPI host interface, programmable IO voltages to support a glueless HW interface.
• Modular Design: Complexity will be localized to the SoM so that the developer can quickly integrate this device into their system using a breadboard as well as a simple API. All required components for vision/audio/motion (eg. image sensor and illumination) will be integrated so the developer doesn't have to cobble together parts. The solution translates into a final product with minimal changes.
• Flexible and Easy to Use: Easy to integrate mechanically and electrically. The SoM will appear like an SPI device with a SW library to support it. The user has the ability to configure the FPGA CRAM instead of having to program the flash followed by a reset to configure the FPGA. The developer has the flexibility to choosing an image sensor while providing a reasonable, low-cost default option.
• Strong Set of Features: Vision FPGA SoM is not just an FPGA board. Require a reasonably sized SRAM to allow for temporary data storage at low power (not DRAM!) which is especially important for vision applications where multiple image frames may need to be captured/processed. Audio and IMU are commonly used together with vision and will make sense to integrate on this platform.
• Documented and Open FPGA Code: The SoM supports an open source toolchain. Well-documented code and plenty of simple examples for each subsystem exists, as well as a large design that ties together various parts of the SoM.
• Production Capable: The SoM form-factor and connectorized interface enable a smooth transition from prototype to production.

The modular concept is summarized in a talk given at a tinyML meetup in 2019.

Flexible and Tightly Integrated

By including most commonly used sensors in a configurable platform utilizing an FPGA with an open source toolchain, this device enables developers to experiment and build quickly. Sample FPGA, as well as host code will be provided as a jumping off point for backers.

Features & Specifications

• The main processing element is the Lattice iCE40UP5k FPGA
  • 5K LUT's
  • 1 Mb RAM
  • 8 MAC units
• Main Onboard Sensor:
  • Color, rolling shutter imager (Himax HM01B0)
• Other Onboard Sensors:
  • Knowles MEMS I²S microphone, expandable to a stereo configuration with an off-board I²S microphone
  • InvenSense IMU 60289 6-axis Gyro/accelerometer
• Memory:
  • 4 Mb qSPI Flash for FPGA bitstream/code storage
  • 64 Mb qSPI SRAM for temporary data
• LED's:
  • Tri-colour LED for a user interface driven by the FPGA
  • IR LED for low-light illumination with frame exposure synchronization
• GPIO:
  • Four GPIO, each with programmable IO voltage
  • Four-wire SPI host interface with programmable IO voltage
• Flexible Power Options:
  • Single 3.3 V operation, can supply 1.8 V and 1.2 V at 100 mA (max) to external devices using onboard LDO's
  • External 3.3 V, 1.8 V, 1.2 V for lower power operation
• Neural Network Support: Supports the Lattice SensAI toolchain using Tensorflow/Caffe/Keras for model development, quantization, and mapping to the SensAI Neural Network engines. Inference at >30 FPS at ~25 mW (average). Sample FPGA code provided for:
  • Image capture
  • Vision-based people detection
  • Audio keyword detection
• Small size: 2 cm x 3 cm

Accessories for Development and Prototyping

SoM Developer Board

The Developer Board provides the following functionality that enables application development using the module:

• Brings out all IO to headers
• Micro USB based power and FPGA programming
• LED's on all GPIO for signal monitoring
• Module power monitoring to allow for application development
• Standard headers (dual Pmod and Qwiic) for attaching expansion boards such as Wi-Fi/BLE and other sensors for quick prototyping
• Small prototyping area
• Support for up to 4 Servo's using PWM control (power must be provided externally!) under FPGA control

The Developer board also has a Raspberry Pi Hat connector. This leverages the extensive RPi ecosystem to develop applications using the Vision FPGA SoM. This enables new use cases for a Raspberry Pi Zero W (including the ones as shown in Adafruit Pi Hat:

• Image Sensing: The RPi can capture images (qVGA or 320x240) without adding the high resolution camera (Normally $29.95 or so).
• Microphone Capability: The Pi can use the MEMS microphone on the SoM over the I2S port, enabling you RPi to listen!
• Accelerometer/Gyroscope: The Pi can talk directly to the IMU on the SoM enabling motion sensing on the Pi.
• AI acceleration: The RPi, while being quite capable in terms of running Linux, isn't as good at running Neural Networks as the FPGA on the SoM. The RPi hat option enables the RPi and Vision SoM to take advantage of the strengths of each device.

SoM Breakout Board

The breakout board is in the Adafruit Feather form factor and has the following features:

• All pins of the Vision SoM are brought out to 2.54mm breadboard compatible headers
• Micro USB charging, Li poly charger circuit with automatic switchover
• QWIIC connector to allow interfacing with the QWIIC ecosystem of sensors, displays etc.

Note: the FPGA programming must be done using an external SPI master. The USB port is purely to supply power.

Comparisons

Support & Documentation

All data about the SoM is captured in the Vision FPGA SoM GitHub repo and will be updated over time with sample code, Colab notebooks, etc.