Low-power Computer Vision
Embedded Computer Vision is a highly multi-disciplinary field that requires expertise in optics, image sensors, hardware, firmware and so on. As a result, the bar to playing with technology in this field is quite high.
The Vision FPGA SoM is a Lattice FPGA based System on Module that integrates an ultra-low-power vision sensor, a 3-axis accelerometer/gyroscope, and an I²S MEMS microphone in a small form-factor (3 cm x 2 cm).
This device is built for developers who want to not only play with this technology, but also have a path to integrating it into real world products. We designed the device with the following thoughts in mind:
- Low Power Image Sensor + FPGA: Overall power consumption is minimized and in the 10-20 mW range to enable battery powered applications.
- Well-designed Host Interface: Easy to integrate into a larger system using an interrupt driven SPI host interface, programmable IO voltages to support a glueless HW interface.
- Modular Design: Complexity will be localized to the SoM so that the developer can quickly integrate this device into their system using a breadboard as well as a simple API. All required components for vision/audio/motion (eg. image sensor and illumination) will be integrated so the developer doesn't have to cobble together parts. The solution translates into a final product with minimal changes.
- Flexible and Easy to Use: Easy to integrate mechanically and electrically. The SoM will appear like an SPI device with a SW library to support it. The user has the ability to configure the FPGA CRAM instead of having to program the flash followed by a reset to configure the FPGA. The developer has the flexibility to choosing an image sensor while providing a reasonable, low-cost default option.
- Strong Set of Features: Vision FPGA SoM is not just an FPGA board. Require a reasonably sized SRAM to allow for temporary data storage at low power (not DRAM!) which is especially important for vision applications where multiple image frames may need to be captured/processed. Audio and IMU are commonly used together with vision and will make sense to integrate on this platform.
- Documented and Open FPGA Code: The SoM supports an open source toolchain. Well-documented code and plenty of simple examples for each subsystem exists, as well as a large design that ties together various parts of the SoM.
- Production Capable: The SoM form-factor and connectorized interface enable a smooth transition from prototype to production.
The modular concept is summarized in a talk given at a tinyML meetup in 2019.
Flexible and Tightly Integrated
By including most commonly used sensors in a configurable platform utilizing an FPGA with an open source toolchain, this device enables developers to experiment and build quickly. Sample FPGA, as well as host code will be provided as a jumping off point for backers.
Vision FPGA SoM top.
Features & Specifications
- The main processing element is the Lattice iCE40UP5k FPGA
- 5K LUT's
- 1 Mb RAM
- 8 MAC units
- Main Onboard Sensor: Color, rolling shutter imager (Himax HM01B0)
- Other Onboard Sensors:
- Knowles MEMS I²S microphone, expandable to a stereo configuration with an off-board I²S microphone
- InvenSense IMU 60289 6-axis Gyro/accelerometer
- Memory:
- 4 Mb qSPI Flash for FPGA bitstream/code storage
- 64 Mb qSPI SRAM for temporary data
- LED's:
- Tri-colour LED for a user interface driven by the FPGA
- IR LED for low-light illumination with frame exposure synchronization
- GPIO:
- Four GPIO, each with programmable IO voltage
- Four-wire SPI host interface with programmable IO voltage
- Flexible Power Options:
- Single 3.3 V operation, can supply 1.8 V and 1.2 V at 100 mA (max) to external devices using onboard LDO's
- External 3.3 V, 1.8 V, 1.2 V for lower power operation
- Neural Network Support: Supports the Lattice SensAI toolchain using Tensorflow/Caffe/Keras for model development, quantization, and mapping to the SensAI Neural Network engines. Inference at >30 FPS at ~25 mW (average). Sample FPGA code provided for:
- Vision-based people detection
- Audio keyword detection
- Small size: 2 cm x 3 cm
Accessories for Development and Prototyping
SoM Developer Board
The Developer Board provides the following functionality that enables application development using the module:
- Brings out all IO to headers
- Micro USB based power and FPGA programming
- LED's on all GPIO for signal monitoring
- Module power monitoring to allow for application development
- Standard headers (dual Pmod and Qwiic) for attaching expansion boards such as Wi-Fi/BLE and other sensors for quick prototyping
- Small prototyping area
Vision FPGA SoM on the SoM developer board.
SoM Breakout Board
The Breakout Board brings out all the pins on the SoM to 2.54 mm pitch breadboard friendly format to allow you to integrate the SoM into your project. Note that the FPGA must be programmed using an external SPI flash programmer (programming supported using a FT232 USB adapter, Raspberry Pi, or microcontroller among other options).
SoM breakout board.
Comparisons
| Item | FPGA Vision SoM | iCEBreaker | TinyFPGA BX | Tomu FPGA | |
|---|---|---|---|---|---|
| OSHW? | OSHW | OSHW | OSHW | OSHW | Closed |
| Price | $70 | $69 | $38 | $45 | $49.49 |
| Schematics Published? | Yes | Yes | Yes | Not yet | Yes |
| Design Files Published? | Not Yet | Yes | Yes | Not yet | No |
| Volume Production Friendly? | Yes | No | No | No | No |
| FPGA | |||||
| FPGA Model | iCE40UP5K | iCE40UP5K | iCE40LP8K | iCE40UP5K | iCE40HX1K |
| Logic Capacity (LUTs) | 5280 | 5280 | 7680 | 5280 | 1280 |
| Internal RAM (bits) | 120k + 1024k | 120k + 1024k | 128k | 120k + 1024k | 64k |
| Multipliers | 8 | 8 | 0 | 8 | 0 |
| Peripherals | |||||
| USB Interface | FTDI 2232HQ (dev kit) | FTDI 2232HQ | On FPGA bootloader | On FPGA bootloader | FTDI 2232HL |
| User IOs | 8 | 27 + 7 | 41 + 2 | 4 + 2 | 18 |
| Pmod Connectors | 1 (dev board) | 3 | 0 | 0 | 1 |
| User Buttons | 0 | 1 tactile + 3 tactile on breakoff Pmod | 1 reset | 2 capacitive | 0 |
| User LED | 1 tricolor, high-power IR LED | 2 on-board | 4 on dev board | 2 + 5 on breakoff Pmod | 5 |
| Onboard Clock | 12 MHz, shared with FTDI | 12 MHz, shared with FTDI | 16 MHz | 12 MHz | 12 MHz, shared with FTDI |
| Flash | 8 Mb QSPI | 128 Mbit QSPI DDR | 8 Mbit SPI | 16 Mbit SPI | 32 Mbit SPI |
| SRAM | 64 Mbit QSPI PSRAM | No | No | No | No |
| IMU | InvenSense 6 DOF (accellerometer + gyroscope) | No | No | No | No |
| Mic | MEMS I²S microphone | No | No | No | No |
| Power Measurement | On dev kit | No | No | No | No |
| Open Source Toolchain | Yes | Yes | Yes | Yes | Yes |
| APIO | Yes | Yes | Yes | Yes | Yes |
| Icestudio | Yes | Yes | Yes | Not yet | Yes |
| Migen | No | Yes | Yes | No | Yes |
Support & Documentation
All data about the SoM is captured in the Vision FPGA SoM GitHub repo and will be updated over time with sample code, Colab notebooks, etc.
Manufacturing Plan
Once the campaign concludes successfully, we will initiate completion of the verification run for the SoM, developer kit and breakout boards, followed by thorough testing to verify functionality. We have a couple of assembly vendors already lined up and will be negotiating the final assembly pricing in parallel to the design effort. The final delivery dates for the devices will be dialed in based on the lead-times for parts and agreed upon build schedules with the assembler and communicated to backers. PWB’s will be ordered from ALLPCB as in previous runs followed by assembly and test and fulfilled by Crowd Supply.
Fulfillment & Logistics
The devices will be manufactured/tested by tinyVision.ai Inc. and bulk shipped to Crowd Supply from where they will be shipped to the backers. For more information, you can refer to this useful guide to ordering, paying, and shipping.
Risks & Challenges
Some sources of risk are:
Design: Prototypes have been built and tested for functionality.
- SoM: The SoM functionality has been verified for the most part. The prototype was designed for both a global shutter imager and a rolling shutter imager for flexibility. Feedback from a survey indicates that most potential backers would prefer a color imager and don't care as much for rolling vs. global shutter. This change needs to be made and a final pre-production run will be required to verify the design.
- Developer Board: The current prototype is overly complex in that it adds a large number of level translators when not needed. This has been significantly simplified and a final design has been completed. This needs to be built and tested.
- Breakout Board: The prototype board is functional electrically. However, once the 2.54 mm pitch pins were soldered on, we realized that these would short very easily to the SoM. Design changes are being made to avoid this issue.
Manufacturing:
- The builtin image sensors are chosen for their low power profile. However, these devices are not commonly available. We have built up good relationships with the suppliers and also obtained quotes from the vendors in anticipation along with lead times.
- As with any hardware product, there are challenges of maintaining quality and testing. We are not new to manufacturing products for customers: we have been building and selling the UPduino and Himax adapter boards for a couple of years.
That said, if there are any delays or complications with manufacturing or delivery, we will communicate this clearly and promptly with backers.
You can learn more about how Crowd Supply protects it’s backers in their Guide.
