Project update 32 of 75
The Free Software Foundation (FSF) posted an article on August 10, 2016 entitled "Support the Libre Tea Computer Card, a candidate for Respects Your Freedom certification". We are extremely grateful for their support, which has significantly boosted contributions for the EOMA68 campaign.
In this update, which comes just after Paul Boddie wrote his very insightful summary, we’d like to give everyone a bit of history regarding both our relationship with the FSF, and the reasoning behind our selection of the Allwinner A20 processor for the first ever EOMA68 Computer Card. There has been much debate going on in online forums, and we would like to both clarify for and educate our community about the practical realities we face in making libre hardware available to the world.
I have been talking on and off with Dr. Stallman for almost a decade. Dr. Stallman is one of the smartest people I know and a stickler for clear communication. If I said anything that had the slightest possibility for ambiguity, he would pick up on it immediately and ask for clarification. Through my interactions with him, I have developed an even stronger propensity for avoiding ambiguous language. So when well over two years ago I brought up the possibility of creating a modern Respects Your Freedom (RYF)-certified laptop that is also easily upgraded, I felt our conversations progressed smoothly.
What is RYF Certification? From FSF’s website: "The ‘Respects Your Freedom’ computer hardware product certification program encourages the creation and sale of hardware that will do as much as possible to respect your freedom and your privacy, and will ensure that you have control over your device."
Independent from my FSF conversations, I was contacted by Chris Waid from ThinkPenguin, which is a business that is focused on making the adoption of free software simple and completely hassle-free, so that anyone including your granny can be using a GNU/Linux computer. Chris was very keen to sponsor the creation of a laptop that would ultimately go even beyond the FSF’s strict requirements (the byline at ThinkPenguin? "A Company Where Freedom Matters"). The fact that Intel and AMD have backdoor co-processors is a serious threat that he felt compelled to tackle head-on.
I explained to Chris that the design I had been developing was: (1) based around a modular, easily upgradeable standard; (2) based on my desire to seriously advance Software Freedom; (3) and cost-effective to reconfigure, which would enable us to bring in newer and faster (Libre) Computer Cards on an ongoing basis at a fraction of the costs incurred in traditional computer design manufacturing. Together, we spent some time thrashing out specs and then I got on with it.
A hilarious aside: unknown to me at the time, Chris had been applying for RYF Certification for ThinkPenguin products for years, and so was in regular communication with his FSF contacts. So while I was talking with Dr. Stallman about the progress being made on the EOMA68 designs, Chris was simultaneously talking to the FSF about the laptop design. It was only very recently we realized we had two independent conversations going on with the FSF!
Initially, two EOMA68 Computer Cards were developed: one using an Ingenic jz4775, the other with an Allwinner A20. We planned to apply for RYF Certification on the jz4775 card because not only is there no GPU (there’s a Vector FPU similar to AltiVec), but the video processor’s source code is entirely GPL compliant, making it a seemingly perfect candidate for RYF Certification. However, upon close investigation, the only FSF-endorsed operating systems available for MIPS32 were so horribly-out-of-date that the resulting device would be pretty much unsaleable even to FSF supporters. Reluctantly we had to drop the jz4775 from immediate consideration.
As the laptop housing neared completion around six months ago, Chris introduced me into the conversations with his contacts at the FSF. We went over the problems with Operating Systems for the jz4775 and despite my initial reservations, discussed the A20 instead, despite my believing its VPU required additional reverse-engineering (not realizing it was nearly 100% completed!) and that its GPU (MALI-400) is nowhere near Libre, with limadriver requiring funding to complete the work.
As first outlined early in the campaign on the Libreplanet list, I learned that the FSF makes a clear and unequivocal distinction between two types of people: (1) those people who are capable of assessing the risks associated with proprietary software; (2) those that are not. They take the view that anyone capable of running command-line tools and understanding the output is by definition capable of making an informed judgement about the difference between Libre and proprietary software.
The FSF’s primary concern is therefore that users who are NOT able to assess the dangers of proprietary software might accidentally install it without ever knowing of the potential risks to which they have exposed themselves, whilst, interestingly at the same time, not preventing or prohibiting anyone in any way from making an informed decision.
For example, in a "worst-case scenario", consider Debian’s "synaptics package manager". It’s a simple GUI that is accessible to non-technically-aware individuals. They can easily click on the "non-free" section (with no warning given as to the consequences), followed by clicking on the "update packages" command (again with no warning given that a list of proprietary software package descriptions is about to be downloaded). No warning is given either when, through this same simple mouse-driven process, proprietary Wi-Fi firmware driver is installed. The same GUI package manager can even be configured to tick the "I agree to the terms of the proprietary license" associated with that proprietary firmware without the end-user ever being presented with a copy of that license for them to review.
However, they also take the view that if we advocate that software writers should respect their users and give them freedom from subjugation and subversion, by the same logic the FSF also respects user’s right to choose to install proprietary software. Thus we have the seemingly paradoxical rules that web pages which promote Software Freedom - particularly and especially those associated with RYF Certification - must be clearly not seen to endorse the installation of proprietary software.
So, although the FSF has sometimes been harshly criticized for its firm stance on what qualifies for certification, those critics seldom, if ever, consider the possibility that the number of "non-technically inclined" computer users far outnumbers the geeks who are "command-line fluent" and that those who have a superior knowledge of the issues involving computer security might feel they have an ethical responsibility to protect those who are not.
So while individuals on forums or even some "Open-Source Hardware Vendors" have actively complained about the FSF’s criteria for certification, the FSF is simply taking on the role of a guardian in an area which requires a combined sense of ethical responsibility and deep and comprehsensive technical knowledge.
So now that I understood the FSF perspective more clearly, the A20’s GPU could be assessed as to whether there would be risk to an average non-technical person of "easily" installing the proprietary Mali-400 drivers (remember: by the FSF definition, if installation requires typing at a command-line, the answer is "No they certainly could not"). At this point in the conversations, I said, "Hang on a minute… the Mali-400 GPU is memory-mapped!", and I was asked if it could be visible by any kind of commands such as "lspci" or "lsusb" and, after some thought, I was able to emphatically answer "No".
That’s when we realized that not only could the A20 be part of a product that could potentially receive RYF Certification, but that products based around the Freescale iMX6 or other SoCs with memory-mapped proprietary peripherals could also be part of "carefully considered" RYF-Certified products.
Remember that RYF Certification requires a 100% commitment from us, the people who are bringing you these devices. RYF Certification is not automatic. The FSF is trusting us - and every vendor of RYF-Certified devices - implicitly with its full reputation. It’s a huge responsibility.
The irony is we are even getting flak from people who believe we are being irresponsible (or hypocritical) for not being "purist" enough. They feel that we should not be compromising in this way by even applying for RYF Certification on a device that has an un-reverse-engineered proprietary GPU!
Actually, we’re not very happy about it either folks, but after spending five years evaluating almost 100 different SoCs (read more about this crazy world in our "Picking a Processor" update), the A20 is at the top of a very, very short list.. a list of ONE remaining candidate.
I once made an attempt to raise money for an FSF-Endorseable Processor which, it has to be admitted, is fortunate that it did not go ahead at the time, as there are now much better candidate processor cores around which to create such an effort. But we have to raise funds to get to that point.
Despite the existence of naysayers who would have us shelve such an ambitious project because it is NOT PERFECT… well, we are heartened that as of this writing, 1300+ of you share our vision and have supported this campaign with your wallets, your forum posting, your blogging, your social media posts and podcasts, and so much more.
We want to bring you truly libre hardware. We have been working on this for several years, and we will continue to do so for as long as it takes.
I am talking to Allwinner about the possibility of them creating a new SoC that uses Vivante, so that now that CEDAR has been reverse-engineered, the proprietary Mali-400 could be dropped and etnaviv used instead. I have spoken often that it would take somewhere between US\$5m and US\$10m to create an entirely new SoC (using the OpenSPARC T2, or Loongson’s MIPS64 core, or other suitable cores). MIAOU and GPLGPU are available and could be used in an open core as part of the GPU. All of these options and many more can be explored, without requiring yet another two years of R&D invested just in the design of the "Housings" (e.g. the laptop) alone.
As brought insightfully to us by Paul Boddie in different words, there is one remaining issue that, whilst we are on the topic, is relevant and has come up a lot throughout the campaign. Despite showing in this update that the A20 is quite literally the last remaining available choice, the campaign has sparked a number of people on several public forums to vent their frustration at Allwinner’s apparently blatant disregard for copyright law. They feel that it would be better to boycott both Allwinner and this entire campaign than it would be to help us achieve the goal of being in a position to influence Allwinner and other companies.
Now, bear in mind that the Far Eastern market literally dwarfs the Western one by something like a factor of three or four to one. Even if every single person in the West were to "boycott" Allwinner in the way that I hear people advocating, our ability to influence the direction that Allwinner are headed, through our miniscule - no negligible - impact on their revenue stream - would be zero. In short: Boycotting Allwinner only hurts ourselves.
In order to bring libre hardware to the world, we consciously seek to improve the situation with Chinese fabless semiconductor companies in a respectful way. To do this, we need to understand Allwinner and how to relate to them in an influential way. This involves examining the key differences between how products are developed in the East and the West.
In the West, access to the source code is critically important as part of the development and maintenance of products. In the Far East, thanks to the "let’s copy it" mentality, source code access is not only unnecessary, but is actually a hindrance to sales.
To explain this further, we need to share some additional insights. Rockchip, for example, is so overwhelmed with requests for access to their SoC from factories with no software engineering expertise whatsoever that they set a $100,000 flat-rate no-exceptions fee for access to information (including providing Linux kernel source under NDA, which is a known violation of the GPL license). The simple logic goes: the factories are incompetent at software development, and their questions tie up at least one of our engineers full-time, costing us money. Therefore, we will charge them up-front.
Tom Cubie, the developer behind the Radxa Rock, was one of the very first to break Rockchip of this habit. By citing his prior success with the Cubieboard series, and promising never to contact them for any form of support requests, he was able to convince them to reduce the $100,000 up-front fees to only $5,000 for access to the hardware documentation and a one-way code-drop. He did not even have to ask them for samples of their own processor because it’s available in the open market in Shenzhen. Rockchip were extremely surprised by the effectiveness of this strategy, and now actually approach members of the sunxi community asking for their help!
However, this is an exception to the golden rule in Asia. The fabless semiconductor companies assume they have to shoulder the burden of getting the product out to the factories, hand-holding them the entire way until production-quality units begin shipping. In certain product divisions, MediaTek even go so far as to handle the production on behalf of their customers, such that it becomes impossible for even their clients to check the legality (or otherwise) of the product’s source code!
In complete contrast, the Western approach is that the developers take full responsibility on themselves for the development of a product. An SoC is viewed as merely a component. It is extremely frustrating to be presented with a bare-bones "technical" document from a Chinese fabless semiconductor company of some 50 scant pages, when the equivalent documentation from Intel, Samsung, Freescale, or Texas Instruments is often a whopping 1,000- to 2,000-page densely packed PDF.
Even more frustrating is the lack of software support. Allwinner has lost at least three opportunities that even I know of to sell between 10,000 and 50,000 units due to the GPL violations and "black box" limitations of the libcedarx video encode / decode engine. The attractiveness of the low cost of Allwinner’s SoCs turns into total frustration (and wasted funds for the Western Companies) on discovery that the hurriedly-developed libcedarx was only envisioned to be used with Android, so is not properly re-entrant and cannot be used for simultaneous video encode and video decode, wiping out the possibility for its use in Internet-connected security camera / webcam markets. It’s taken nearly three years for CEDAR to be reverse-engineered so a proper and effective library is now available but it’s always going to be for older SoCs.
Therefore, we can see there is:
of all hardware and software development being on the fabless semiconductor company itself, thus driving them to customise SoCs to specific products and markets.
which makes Eastern-developed SoCs initially appear more attractive…
license compliance causes major product development issues for any company (mostly Western) to put the processor to a different purpose than the one envisioned by the fabless semiconductor company.
Next, we will go over Allwinner’s history so that we can understand their specific situation.
Allwinner was a spin-off from another fabless semiconductor company at a time when, back in 2010, the price of competitor SoCs was around $11 to $13. The average retail cost of an IPTV box or tablet based around those processors was around \$50 to \$80. Along came Allwinner with a superbly-engineered and highly-integrated $7.50 processor (including the PMIC), a BOM of $15 for their Reference Design, such that the world’s first $35 tablets became a reality, and suddenly everyone wanted one. Now, the problem was that the market grew so quickly that anyone holding stock of any of the components for the older designs quickly went out of business. Even factories that had placed deposits on components reneged on their contracts and walked away from deals, knowing full well that they simply couldn’t even sell $50 tablets when there were $35 ones that were significantly better. As a result, huge numbers of OEMs and ODMs were left holding stock of components unique to the incumbent Reference Designs, and went out of business, and a major recession occurred due to the disruption brought about by the introduction of one processor!
Several heavyweight China investors jumped in on Allwinner’s success. Over the next few years, these investors created their own niches inside Allwinner, creating competitive products not only to Rockchip, Actions Semi, and AMLogic’s offerings but to other Allwinner processors. Note however that the list of competitors does not include companies that are well-known in the West, notably Freescale, Texas Instruments, and Samsung! Freescale and Texas Instruments are simply unable to compete on price, nor are they taking the same cost-saving short-cuts (incomplete documentation, GPL violations) that are the hallmark of being first to market in China. Samsung is a special case as not only do they make processors, but the processors are driven by a highly successful mass-volume smartphone and tablet brand.
In the Far East we know that China has a completely different perspective on Copyright thanks to Bunnie Huang’s "Gongkai" article. Add to that mixture a "get it done and out the door" general attitude to business the world over, the concept of "face", the language barrier (many Chinese developers have never read the GPL and the fact that "Free" Software actually translates literally to "Anarchy" Software as in "socially and culturally you will be respected during periods of Governmental Repression for abandoning laws and responsibilities and running away to save your life" Software instead of the more accurate "Mutual Benefit Collaboration" Software really is not helping), as well as the internal power struggles and "niche carving" caused by the many independent investors who would normally never work together, and we start to see that we’re lucky that this process has successfully brought out such ultra-low-cost processors at all.
I have already stated my view that the only real way we can influence Chinese fabless semiconductor companies is financially, and that boycotting is ineffective. Very recent historical examples that we can draw from are two other crowdfunding campaigns using Allwinner processors: the pine64 project and the CHIP.
Examining the forums of each, in the case of the pine64 team we can see the sunxi community (as well as pine64’s own backers) were extremely annoyed with the pine64 team’s ineffectiveness in obtaining a fully libre codebase. Despite receiving comprehensive advice, the pine64 team’s meeting with Allwinner regarding source code did not go as expected. Chinese companies will often use the argument that they are "protecting their intellectual property" by keeping certain parts of the source code secret (despite the fact that in doing so they have broken copyright law, making it impossible for products to be sold in countries where copyright law is enforceable). Assuming that these same arguments were what was repeated to pine64, the pine64 team could have responded by informing Allwinner that without the full source, there would simply be no orders. That the pine64 was supplied to its backers with a proprietary bootloader, for which the source code is not available, tells us everything that we need to know about what actually happened during that meeting.
Learning from this lesson, when it came to NextThingCo’s turn (with the $9 CHIP), the oxyacetalene blow-torch of complaints on their forum was turned up a rather loud notch and it got results. NextThingCo’s prime marketing attraction was the price. However, like many prior examples, they forgot the strategic value of providing full source code. The sunxi community therefore reminded them, publicly, right during the middle of their crowd-funding campaign, of the need for full source. NextThingCo heeded this advice, and made their order for processors conditional on receipt of the full source code. Bear in mind that they’ve got something like 50,000 orders to fulfil, now all of a sudden that’s a big deal.
This is the way to influence Chinese fabless semiconductor companies: through their wallets. They’re not going to be swayed by arguments about how "bad" it is that they’re not respecting copyright law: to do so would cost them money… so they don’t do it. And the law in China is (mostly) on their side: to bring a copyright case against someone in China you have to be a Chinese citizen! Thanks to "gongkai" they see the source code as being "there for the taking." It doesn’t even matter that more of the Chinese engineers people are communicating with on various Linux-related forums now fully understand what copyright law is about, because Chinese society operates on strict one-way hierarchical rules: you can’t even question your superior, even if you haven’t actually understood what it is he’s asked you to do!
The EOMA68 modular strategy allows us to get to market with a newer processor with significantly less money in a lot less time. Dropping in a completely new design of Computer Card is, with a $20,000 budget, a matter of about 3-4 weeks of very intense work.
Compare this to the average monolithc "designed for obsolescence" laptop, which requires around $250,000 and at least a year’s work, and the average PC motherboard around $50,000. That’s quite a large committment that has a lot of inertia. Therefore, it is no surprise to find that corporations cave in and compromise on ethics and continue to use proprietary or license-violating software (even at their own long-term expense and immediate risk). As we saw even with the recent example of pine64, they would have already spent too large a sum on the development of the pine64 board to back down.
EOMA68 designs have no such inertia. With even a small amount of financial backing we can very quickly show Chinese fabless semiconductor companies that their committment to software libre gets results - and by results I mean financial results.
Long-term we in the West know, as the example of the Valve/Steam-Intel publicly documented collaboration shows, that software libre source gets results, but Chinese fabless semiconductor companies simply haven’t yet seen any real financial benefits from complying with software licenses because of the culturally radically different product development lifecycle. We’ve also shown that even when companies in the West develop "open hardware" without linking the product sales to full software disclosure and full copyright compliance, they are actually harming, hindering, and undermining everybody’s collective efforts to work with Chinese fabless semiconductor companies.
Remember that our ultimate goal is to reach mass-volume production with a variety of libre hardware products. There are already enough companies out there in the world that sell you products that violate your privacy and/or cause you to have to throw them into landfill because there’s no proper source code availability. I think you would agree that simply copying the business practices of those companies (including distributing proprietary and license-violating software in mass-volume quantities) is not going to help.
We therefore choose to take responsibility for doing things in a different way, so as to achieve a different outcome. Your amazing support of the EOMA68 campaign will help us to do that. We will remain publicly accountable to you through the mailing lists, and invite you to join us on this ambitious journey.