Xous: Bug Scrub

This is another one of those OS updates where we have no fancy screenshots to show you. After the Xous 0.9 push, we spent a couple of weeks cleaning up bugs and testing things so we’d have a clean base to move forward from.

It turns out that std is pretty awesome to have in Rust, but also, std isn’t some magical thing that springs into immaculate existence on its own. It’s another layer of software that hides a lot of complexity and has ample room for bugs. It’s also one of the few parts of Rust that changes with every release, which means that every few weeks Xobs has to forward-port all the Xous-specific patches to the latest Rust, so we can have a std that works with it. It’s a thankless maintenance task and, if you ever get a chance, buy Xobs a beer and ask him about the travails of maintaining your own Rust libstd.

I actually think it’s kind of neat that we get front-row seats looking directly into a part of the machine that rarely gets attention, even if it is for the purpose of fixing subtle bugs. From the perspective of building a computer you can trust through-and-through, the runtime and standard libraries are fundamental, and simply delegating that to an unknown/untrusted third party introduces risk in the software supply chain. It’s a lot harder to do it yourself, but I also have more trust in the final product as a result.

Xous: Stupid Network Tricks

I said we don’t have any fancy screenshots to show in this update, but we do have some remarkably pedestrian screenshots to share:

Thanks in large part to a big effort by Samblenny, we can now do some rudimentary networking: we can acquire DHCP on an IPv4 wireless network, answer ARP queries, and we have a framework for filtering/dropping packets on the embedded controller. All of which means this all operates with the main CPU powered down!

Xous itself sprouted a network stack based on Whitequark’s smoltcp. Since we’re cobbling everything together from the ground-up, we decided to start with UDP, which is an essential prerequisite for traditional DNS resolution. At the moment, we have the core bindings for UDP, along with a DNS resolver and a cache. The screenshots above show these two features in action. We also have an ICMP Echo (e.g., ping) facility, but as of this post it’s going through a major refactor to fix some significant implementation problems.

Significantly, we’re stopping just short of wiring up TCP. Why not just go all the way and implement the whole suite of standard networking protocols? I feel like that would paper over how immature the network stack is – I’m not happy with the amount of testing it has received to date and I feel like it still has a couple of iterations to go before it’s really ready for applications to run on top of it. Let’s be honest about it: I (bunnie) ended up writing a non-trivial portion of the network code, but I have never implemented a network stack. I had to keep a copy of the TCP/IP Guide on my desk for the past month to learn the most basic things: what even is ARP? What is the structure of a UDP packet? How does DNS work? I might be good at soldering, and I may have implemented PHY-level demodulators, but in all truth, I have zero qualifications to write code for a network stack. You’ve been warned!

That being said, the suite of protocols we have now should be adequate for doing a good measure of stress-testing: for example, we need to figure out why the stack crashes in the face of a ping flood and why every hour or so we’re getting mysterious “host interface” errors from the WF200. Thus, while wiring smoltcp’s TCP implementation to the WF200’s packet buffer is ostensibly “easy”, the lack of TCP is meant to be a very clear signal to the community that “thar be dragons” in the network stack and we could use help making things more stable, cleaner, and generally better before icing the cake and putting a cherry on top.

Personally, I’m taking a break from network implementation and testing, to try and push out a very rough draft of how storage is going to work on Precursor: the “Plausibly Deniable DataBase” (PDDB). This is an ambitious project to keep users from becoming the weakest link. The situation was elegantly summarized by Randall Munroe in this xkcd strip:

Right. Enhanced security must come hand-in-hand with enhanced plausible deniability (PD) of any secrets that may or may not be contained within the system. The PDDB will map a key/value store onto the Rust std::io read and write abstractions, while blending patterns of secure and insecure data access together all the way down to the hardware. We’re assuming the adversary is knowledgeable and has forensic analysis capabilities on both static images of the filesystem and even live snooping of the read/write data to the FLASH itself, but is otherwise unable to access the root keys within the SoC (admittedly, a big assumption and perhaps not a good one, but we have to draw a line somewhere). Given that production is about to ramp up, we’ll see how far we get, but I’d like to have at least the basic APIs fleshed out before the units arrive in user’s hands, so there is less of a temptation to port FAT32 to Precursor and call it a day.

Production Update: Getting Busy!

After a long lull waiting for parts to come in, we’re now just one part shy of starting production. In theory, that part arrives the second week of November, at which point we’re full steam ahead. Thanks to changes in travel policy, at least as of this writing, I’m planning on traveling to South Korea to be on-site to inspect the first articles running off the line, hopefully at the end of November.

Here are the most recent highlights from the production front:

• We now have an approved PVD steel process. The CNC shop is churning them out!
• We have full custom screws for your Precursor! That’s right, each stainless-steel SUS410 grade screw has a precision turned surface that gives it a handsome radial shine, a custom black PVD finish, and a patch of blue nylon on the threads to keep it from backing out. This also means each of your screws cost about $0.65 (USD) a piece in volume – keep that in mind as you take your units apart.
• We’ve got a gift box and manual designed, and we’re currently running production samples for approval. This is running a little bit behind schedule because it was more fun figuring out UDP sockets than designing fiddly cardboard trays to hold accessories in place. But, I’m hopeful that a few smart decisions on material choices and design will allow us to sidestep the worst supply chain issues and help make up for some lost time.
• If you want to hear a good “supply chain war story”, buy me a beer and ask me about the SiT8021. It’s got everything from cancellations, to price hikes, to gougers, to total substitution inflexibility due to EMC certification rules, to incompetent distributors losing orders, to bugs/misconfigurations in websites preventing new orders. The bottom line is we have enough parts to deliver units to our initial backers, but currently not enough to support any significant new business. But: “We’re working on it.”
• The battery is now certified for shipping from our factory to Crowd Supply, but Crowd Supply requires a UN38.3 certification for us to ship spare batteries to you. Because the certification lapses at the end of every calendar year and costs about $200 to renew, we're going to kick off the certification process in January. It means we may not be able to ship you replacement batteries on day one, but we’ll eventually get there.
• The production test now has just about 100% automated feature coverage. I say “just about” because it’s a bit tough to automatically test the “insert a paperclip” reset button, and some limitations on tester calibration means the sleep current thresholds are a bit wide. However, you can get a flavor of our coverage by checking out the tests that run on the firmware burner and the tests that run on the fully assembled unit. Every key gets pressed, audio is checked with an FFT, network addresses are acquired, power levels measured, backlights lit, vibe motors vibed... you get the idea.

Hopefully the next update will include some photos of circuit boards rolling down the SMT line in our factory!

Compliance Update: We Will Boycott the UKCA (and You Should Too!)

The EMC compliance saga seems to be never-ending. The certification house is still picking over the verbiage in our user manuals (apparently, we need to explicitly inform EU users that it’s safe to hold a Precursor to your head), and we’re revising our labeling to fall in line with what they say are the minimum requirements. I guess if you’re an Apple or Samsung you can get away with simply “Apple / Cupertino, CA” or “Samsung / Seoul, Korea” as your “address”; but for a small player like us, apparently we need to include the entire postal address of our factory from the street number down to the postal code on the outside of the product. I’m not sure exactly what that’s supposed to accomplish, but when you flip over your Precursor, you’ll be greeted by a wall of text silkcreened on the radome, just in case you need to physically mail a letter to the contract manufacturer to inquire about compliance issues.

Also, to our current and potential users in the UK: apparently, as part of Brexit, your illustrious lawmakers have decided it’d be a great idea to refuse the EU’s CE certification, so in order to import products to the UK they must also get the “UKCA” mark. This was originally slated to go into effect on Jan 1, 2022, and I was preparing to inform all of you that I was simply going to cancel your shipments and refund your money. Fortunately for the early backers, that deadline was just extended to Jan 1, 2023, so you’ll be able to import a Precursor with the CE certifications that, with great effort and cost, we’ve managed to procure.

That being said, we plan to disallow new sales of Precursor into the UK starting in the fourth quarter of 2022, because it simply is not worth it for us to seek a separate regulatory marking for the UK. Although the UKCA has a 1:1 mapping to the CE today, there’s no guarantee it will stay that way. Furthermore, it is still a separate marking, which costs real money to process; and our manuals, boxes, and case parts would have to be scrapped or reworked to bear the mark. This is both wasteful and unsustainable, as the endpoint of this road is our products bearing the unique regulatory marks of hundreds of territories and regions with no substantive benefit to the consumer or the producer; it’s already silly that we even have to have to bear both FCC and CE markings.

Thus, I also want to send an unambiguous message that this return to pre-industrial mercantilism and rent-seeking behavior by the certification bodies will not be tolerated. I ask that other makers of hardware join me in boycotting the UKCA. The global certification regime is already fractured and frankly a bit silly, and we need to make it clear that there are real costs that we the manufacturers shall not bear when trade barriers are raised for the sake of political theater. If manufacturers keep on eating escalating tariffs and compliance costs, then there is virtually no pressure on politicians to reign in these kinds of trade policies: they are a cheap & visible political win, with no local economic cost.

Instead of creating fresh regulatory barriers for the sake of having barriers, we should be moving toward harmonizing the world around a single common compliance and safety data reporting standard. At the end of the day, the laws of physics are the same whether you’re in the EU, USA, China, or even the UK. We should be promulgating traceable, accessible calibration standards while lowering barriers to the submission and access of raw compliance data: let the data speak for itself, leave the geopolitics out of it!

Happy hacking,

- bunnie & Xobs