Linux ARM support: A hot mess, an ugly clean-up

I’ll tell you what, after all of that talk about forks last week, it’ll be nice to turn my attention to something less controversial today. Let me check my Topic O’ the Day board and see what’s next for discussion… Monday… ah, here we go:

ARM in the Linux kernel.

…Really? Really, blog $DEITY!? That’s what I’ve got to go with today? Because the LibreOffice and OpenOffice.org argument looks like a schoolyard spat compared to the United Nations-level complexity of the the forks in the ARM section of the Linux kernel.

[Also see: Intel gets defensive about Windows 8 on ARM]

And yet, there they are. Relating back to last week’s discussion about energetic departures and their historical success rates, it would be better to describe the forks in the ARM branch has “low-energy” events. There weren’t philosophical calls for freedom or technological salvation that created these forks and for a long time only a specialized group of developers and manufacturers even knew what was going on.

But despite the quiet origins of these forks, the problem has grown to the level where Linux kernel maintainer Linus Torvalds has publicly threatened to stop pulling ARM-related changes into the mainline Linux kernel; a threat that could effect dozens of companies’ livelihoods and the course of embedded Linux development.

To get an idea of what’s going on, I reached out to Grant Likely, owner of Secret Lab Technologies, who also maintains the SPI, GPIO, and Device Tree core device driver infrastructure for the kernel, which is used by all architectures, including x86. Likely is going to speak about the problems of ARM Linux support at the August LinuxCon North America and I figured he would be a good person to help make sense of all of this.

Update: The preceding paragraph was corrected to reflect Likely’s actual relationship with the Linux kernel, erroneously described in the initial release of this article.

To get a sense of how this started, Likely explained, one has to go back quite a few years, to when ARM manufacturers first started adding Linux support on their devices. Unlike the desktop and server models for Linux development, where the motto for kernel features is “consolidate, consolidate, consolidate,” embedded device manufactures had a different set of pressures. Namely, the marketplace, and the accompanying pressure to build a device quickly and cheaply, get that device out on the market as fast as possible, then turn around and do it again.

This code, rinse, repeat cycle Likely described was undertaken not just by one ARM manufacturer, but pretty much all of them. Device-specific changes would be introduced to Linux kernels running on the hardware, with little thought about consolidation, even across individual device models. To make matters worse, even though the GPL v2 license on the Linux kernel requires these changes to be released back upstream to the main Linux kernel, often they were not.

Likely cited Harald Welte’s well-known GPL Violations blog as a record of known ARM manufacturer violations, though Likely emphasized that “the big-tier manufacturers did tend to get what to do.”

But even compliant behavior brought trouble. As many note about Torvalds in general and maintainers specifically, Linux ARM maintainer Russell King does not scale. Faced with increasingly frequent and often-redundant proposed changes to the kernel, King and his team of sub-architecture maintainers (of which Likely is a member) could often fall behind.

Now ARM support is a labyrinthine maze of wheels within reinvented wheels, to mix the metaphors. But in recent years there has been a concerted and growing movement in the ARM community to finally bring about consolidation across features.

Likely stressed that there were few big, dramatic moments that affected such a change in attitude, but he did point out a couple of them along the way. First, sub-architecture maintainers for ARM stopped pushing patches to King because he simply couldn’t keep up. Then, separate Git trees were created for the ARM branch of the kernel, which enabled Torvalds to start directly pulling patches into the mainline Linux kernel himself.

Once Torvalds was more directly involved in handling pull requests into the mainline kernel, his displeasure about the ARM situation was increasingly vocal, as illustrated by this March 31 message to the Linux kernel mailing list:

“What I’m saying is that we should not be adding any mindless board drivers for ARM.

“Because they don’t work. Most of them are totally unmaintainable crap in the long run. They are extra code that actually have negative worth. It shouldn’t be in the kernel at all…”

The debate continued into April, as solutions were offered and objections raised. Torvalds made his expectations entirely clear in an April 18 message:

Hint for anybody on the arm list: look at the dirstat that [King] posted, and if your “arch/arm/{mach,plat}-xyzzy” shows up a lot, it’s quite possible that I won’t be pulling your tree unless the reason it shows up a lot is because it has a lot of code removed.

“People need to realize that the endless amounts of new pointless platform code is a problem, and since my only recourse is to say ‘if you don’t seem to try to make an effort to fix it, I won’t pull from you’, that is what I’ll eventually be doing.

“Exactly when I reach that point, I don’t know.”

That Torvalds is taking a stance like this is entirely expected, since he is pushing back against a growing commercial effort to finally get ARM features consolidated and into the mainline as vendors have finally gotten the clue that they do not have to reinvent the wheel and can save even more money and resources by creating open standards within the ARM ecosystem.

Indeed, as Likely pointed out, this is entirely the reason why the non-profit Linaro consortium (formed by big-name ARM vendors like Freescale, IBM, Samsung, ST-Ericsson, Texas Instruments, and ARM itself) was put together in 2010. (Likely, by the way, is currently under contract with Linaro, but emphasized he was speaking about these topics on behalf of himself.)

Right now, the solution to the ARM mess is very actively being sought by Linaro and other players in the ARM arena. Linaro, Likely explained, is championing the formation of an ARM maintainer team for the Linux kernel, like the one that already exists for x86. Some of the x86 team members are already reaching out to the ARM community to help them set up such a team, which would help alleviate the scaling problems of the current ARM maintainer organization.

Something definitely needs to be done. According to Linaro Chief Technical Officer David Rusling, “…[a]s an indication of the scale of this problem, each new kernel release sees about 70,000 new lines of ARM code, whereas there’s roughly 5,000 lines of new x86 code added.”

No wonder Torvalds is ticked off.

The reasons for ARM consolidation are crystal clear from this kind of technical perspective. But there’s potential money-on-the-table reasons, too. While the respective development of the Android and Linux kernels has been separate, there is clear evidence that the relationship is getting better and the two kernels are becoming more in sync. With a clearer channel from Linux to Android in sight, silicon manufacturers are more motivated than ever to get their products as fast as possible into the exploding Android market, Likely added.

These two sources of pressure are about to collapse what has been a multitude of ARM-device-centric kernel forks. This collapse may not be pretty, and at times it could get downright ugly. But it’s a necessary step in the progress of ARM support in Linux.