Project update 33 of 36
I was thinking we’d be light on new features due to the holidays, but I was wrong! Our development community has rallied, and we’ve got a nice list of compelling improvements in this release.
vaultUpgrade to FIDO2.1
vault application has been upgraded from FIDO2.0 to FIDO2.1. The most significant user-facing improvement is support for "residential keys" on Precursor. In other words, if your SSH client is up-to-date, you can store SSH private keys on Precursor, instead of leaving it laying around in your
.ssh/ folder. This allows you to "carry your keys" with you, allowing one to login using a less-trusted computer, without having to install a copy of your private keys on that machine.
If you’re running some flavor of Linux, you can generate such an SSH key by plugging in your Precursor and running
ssh-keygen -t ed25519-sk -O resident
If you haven’t already set a PIN on your device, you need to do that first, using the following command:
fido2-token -S /dev/hidraw0 (or whatever the path is to your device, consult
If you’re using openSSH, you’ll need a version later than 8.3 or so for this to work. Note that many distros are still pushing out versions older than that, so if your SSH complains about
-O resident being an unrecognized option, that’s probably why.
In addition to supporting residential keys, the
vault application migrates your existing authenticator database to a new schema. The purpose of the migration is to make future upgrades of our OpenSK port much easier.
The migration should be seamless; if you encounter issues, you can always downgrade to an older version of Xous and your original database will be unscathed. Please open an issue if you do encounter a migration problem.
Xobs has been very hard at work improving the performance and stability of the Xous kernel. After a heroic effort porting the Rust
std tests, a huge number of bugs were fixed, and some very subtle performance regressions were also fixed. Here’s a partial list of the changes:
DoubleFreeis now returned instead of
scalarreturn calls are now unified
asm!rather than external object files
condvarimplementation has been completely overhauled
RecalculateSleepwhen sent internally
Chasing down bugs is a never-ending task, and Xobs is still hard at work moving the furniture to find the bugs that scattered when the lights were shone on them. As Xobs quipped in the dev-chat channel: "There are two hard problems in computer science: Cache invaAnd concurrency.lidation".
We also added a chapter to the Xous Book about performance, aided by cycle-accurate, system-level simulations of the hardware.
I was pleasantly surprised to discover that Verilog simulators are performant enough to boot Xous in a cycle-accurate simulation. This means I can dig through saved waveforms from a single run, and learn things such as cache and TLB miss rates, or the overhead of kernel to userspace transitions. These sorts of stats are usually very hard to come by because any on-line performance logger will incur some penalty for creating trace messages. It’s also hard to dig into things like the state of the instruction decoder, branch predictor, or L1/L2 cache interactions using code-based profiling techniques. However, with a system-level Verilog simulation (it even models the overhead of fetching data out of our SPINOR flash chip!), we can drill down into every flip flop and gate inside the system, and pinpoint code problems, bus contention, and/or logic bugs in the RTL.
Above: an example analysis of the simulated RTL in a waveform browser. The diagram shows just a few signals out of thousands: here we have some external SRAM signals, a couple ranges of the program counter (rendered as an "analog" waveform), and the state of the CPU’s ASID register. Steps in the program counter "waveform" height correlate to jumps between various functions, some of which are annotated.
I feel like this is one of the stronger arguments for open RTL CPUs — being able to straddle the boundary between hardware and software with a simulator like this is extremely powerful for identifying regressions and bugs. Of course, a lot of this can be done with an emulator; but simulating with the RTL captures not only the intended behavior, but also the unintended bugs baked into the chip.
I think it’s an interesting enough topic that I’ll probably write a blog post dedicated to this in the coming months. For now, I’m polishing the technique a bit so that it’s more accessible to users and less of a pile of Rube Goldberg scripts that break when you look at them funny.
@gsora has spearheaded an initiative to build a USB mass storage driver into Precursor, and it is starting to bear fruit. At the moment, an app he wrote called
transientdisk is available in the source tree. It’s in early days, so for now, you’ll have to make a custom build to play with it. When activated, the app causes Precursor to enumerate a 1.44MiB USB mass storage device to a connected host. All the data is stored in RAM. As its name indicates, the data is transient — once you leave the app for any reason, the disk is de-allocated, and if you reboot all memory is zeroized.
The app is currently more of a proof-of-concept that the USB mass storage layer is sound and performant, but even in this state it’s useful for sneakernetting sensitive encryption keys between machines. You could do that right now with a USB stick, but then you’d have to grind up the stick to ensure the data is not recoverable; with
transientdisk, you can easily inspect the code and see that erased means erased, and not simply shuffled to a list of blocks to eventually be erased as part of some opaque wear-levelling algorithm.
Now that the USB mass storage driver has been proven, we’re exploring using the mass storage interface to present virtual filesystems to allow for easier backup/restore of key data, and possibly even OS updates.
I’ve had at least one documented success of an unbiased third party going through the key burning flow without incident, so the feature is now available in the stable release train.
It is still marked as Beta and has a stern warning about the risks, but at least you don’t have to jump through special hoops to try it out. If you are successful at executing the flow, I’d be interested in knowing. And of course, if it fails — please file a detailed bug report.
Thanks to significant efforts by @tmarble,
mtxcli has some significant bug fixes and stability improvements! It now auto-syncs messages from servers more reliably, and it has filters so if you have a lot of active chats, the device is not overwhelmed by a deluge of chat logs.
Many other fixes and features are in this release, including:
.Systembasis is assumed).
fatand pushing FFT test code onto the tester. Note that any users who wish to write code that relies on built-in floating point transcendental functions will have to restore lto=
thin, at least until https://github.com/rust-lang/rust/issues/105734 is resolved.
bip-utilsdependency removed from Python packages. This allows
precursorupdaterto run on older platforms that don't have the latest-greatest Python. A hand-rolled BIP-39 word-to-bits converter is used instead.
vaultpasswords path. Records are re-used instead of re-allocated if they don't change. This should speedup switching to
vaultpasswords by about 2x after the very first time the records are loaded (the first time will take longer because the records have to be built up).
By the time you read this update, I’ll hopefully have shepherded a new batch of Precursors through our factory in South Korea. Either that, or I’ll be cursing vendors for having sent me substandard parts. For the past couple years, the pain in the supply chain has been real. While there has been some recovery in the supply chain, it has been uneven; some parts are readily available, but some are still extremely hard to get.
Of course, you can’t build a system until every single part is on the line, so a partial recovery is still a headache for small-scale producers like us. I have managed to acquire enough parts to do another build only by twisting some arms fairly hard. The risk of arm-twisting — especially when you’re a small player — is that vendors extract revenge by sending substandard or incorrect parts. They can afford to piss you off much more than you can afford to piss them off (cue the "You’re not Apple" refrain). This is why I’m traveling to the factory to oversee the build and personally check the parts for irregularities.
Unfortunately, simply waiting for parts to arrive on their "natural course" was not an option — without constant pushing we would have sold out months ago, with no hope for a resupply in over a year. Such a protracted lack of availability risks shaking confidence in the project, sapping precious momentum from our nascent developer community. I’d personally much rather take a measurable supply chain risk, than an immeasurable risk of eroding community goodwill.
I’ll share the outcome of the build in our next backer update. Wish me luck!