Project update 10 of 14
The NeTV2 production pipeline is really starting to take shape. In about one week I will be heading to South Korea to bring up a new line for NeTV2 production. Although I had originally intended to do production in Dongguan, China, those plans were a quick casualty of the trade war between the US and China. Ironically, US trade war policy makes moving production into America the least favorable option, because the heaviest tariffs are levied upon the components I would need to import to the US to build NeTV2 using American hands. In fact, the main reasons for moving production out of China are a precaution against the trade war deepening and a heavier tax being levied on the American side on the import of audiovisiual appliances, combined with a general stiffening of China’s trade and tax policies over the past year due to President Xi’s policy reforms and general response to the trade war.
Fortunately, the CM I’ve engaged for production, AQS, has very strong ties to South Korea, so getting a quotation for moving production to South Korea wasn’t difficult. While the cost of production is higher in South Korea, it should be more than offset by the savings in duties and tariffs once all things are considered, so NeTV2 backers are safe from becoming casualties of the trade war (for now). Of course, moving to a new line always has risks — I haven’t had a chance to personally audit the line, which is why I’m flying to South Korea next week to oversee the bring-up of the SMT line.
As alluded to in the previous update, I’ve spent most of the last month building the NeTV2 test infrastructure. Testers are typically as complicated as the product itself. In this case, I had to design a custom Raspberry Pi HAT which adds a pair of 12-bit ADCs with current monitoring facilities as well as loopback facilities to assist with testing the high speed GTP transceivers on the NeTV2 FPGAs. The curious can see the schematic and board layout for the tester HAT at the Alphamax GitHub repo. The goal of the tester infrastructure is 100% coverage of all solder joints and features — no pin shall be left unwiggled, no connector left empty during testing. This meant having to build an FPGA design that exercised every possible option built into the PCB — and there are a lot of options built into this board. The tester master branch of the NeTV2 FPGA contains all the code for this configuration.
I also spent way too much time debugging a subtle problem in the BCM2835 I/O block on the Raspberry Pi. This block is important because it’s what I use to drive the JTAG interface on the FPGA, necessary for configuring the FPGA. Because of the bug, I had been using the sysfsgpio interface which is orders of magnitude slower than the BCM2835 I/O block, effectively doubling the amount of time it takes to test and provision an NeTV2. The bug, in a nutshell, is that the GPIO block runs at a fraction of the CPU core speed, and so commands to wiggle the GPIOs always arrive in order but not necessarily at the time you think they would — GPIO commands would get bunched up and committed in a burst, which would take your nice even square wave clock and turn it into a series of glitches. This has bad implications for driving a JTAG interface. The fix is to insert a series of dummy GPIO operations. I tried submitting a patch upstream to OpenOCD, but it’s currently in purgatory so to keep things moving I’ve forked OpenOCD to build a version for inclusion in the production tester as well as in the user firmware release. The good news is that while submitting the patch another developer (Salvador Arroyo) had also encountered the problem and had a higher-performance workaround than mine, so the JTAG interface is running faster than ever.
In addition to a core FPGA design and Raspberry Pi HAT, a large amount of firmware had to be written to string together the tests and report the results. I used Xobs’ Exclave tester infrastructure to pull this off. Exclave is a framework written in Rust that gives a systemd-like environment for scripting dependencies between tests, along with support for testing triggers and status feedback. In the case of NeTV2, the test infrastructure reports test results in JSON records that are relayed to a web page which renders the pass/fail status of every hardware block in real time. The actual test is divided into two phases, one "quick test" which takes about 20 seconds to run and covers the most critical 80% of the board, and one "comprehensive test" which tests the other 20% that was missed by the quick test, but takes several minutes to run and set up because of the sheer number of connectors and cables that have to be plugged in to prepare for the test.
The idea here is to yield out the most critical, numerous flaws quickly from the production line before investing the effort to do a comprehensive test. Fortunately, preparing two test scenarios like this is something that Exclave was written explicitly to support — being able to rebase tests and optimize test flow by just editing a single dependency file based upon the actual types of production problems encountered is extremely important in a production environment. While it’s tempting to write the whole framework using a scripting language like Python or Bash, in my experience this can lead to subtle problems, such as forgetting a test condition or a test dependency. This is especially easy to do in the high-stress circumstance of a "line down" situation due to a tester problem and you’re furiously hacking code to develop a new scenario to capture an edge case that was previously missed. Exclave’s formalization of dependencies and simplified methodology for ordering tests help prevent stupid mistakes that can lead to numerous irate customers and expensive returns and exchanges.
I also decided to solve the problem of imaging Raspberry Pi SD cards in a more elegant fashion than doing a full-card "dd". The base image for NeTV2 consumes less than 4 GB, but the smallest cards you can get are around 16 GB in size. So a full-card "dd" would be spending 75% of its time writing empty disk space, making it the slowest operation of the entire production process. I made a suite of tools, which you can check out at the usb-pyromaniac repository, for extracting partitions from a Rasbpian or Fedberry card and optimizing the burn process by first copying a minimally-sized partition to the disk and then resizing it to fill the card. The usb-pyromaniac tool furthermore includes a facility that allows you to plug several blank SD cards at once into a PC and burn them in parallel, increasing the effective throughput of the burning process to about one card every ninety seconds or so. The framework could theoretically scale up to a very large number of USB ports, but you also need a commensurately larger number of CPU cores to keep the ports busy, so in practice we burn only four cards at once, one per core on the NUC-style PC installed at the factory for this purpose.
Finally, I did manage to add a couple finishing touches to the user firmware, such as a click-to-update script for the FPGA and some helper scripts to stop and start the MagicMirror infrastructure to ease debugging for the uninitiated. The FPGA core itself got a minor upgrade to the sampling point state machine for the video inputs, making the design a little more robust against bad cables and crappy video sources. However, I would like to emphasize to all users that NeTV2 will perform best with good-quality HDMI cables. At 1080p, some of the oldest HDMI cables I have laying around (pre-HDMI 1.3) were very marginal; it’s very difficult if not impossible to establish a reliable video link using those cables. I have, however, had good success with modern cables, especially those rated for 4k resolution (HDMI 1.4+) or better, even at several meters in length. So, before you plug in your NeTV2, please make sure you’re using a late-generation, high-quality HDMI cable or else you may have trouble esablishing a stable video link!
Well, that’s it for now. After checking out the production facility in Korea and setting up the test and quality control pipeline there, the next challenge will be pushing out the plastics. If all goes well, you’ll be hearing more about the outcome of the plastic injection molds in a bit over a month.