Adventures of porting PostmarketOS to the Librem 5

I’ve been longing to drop the shackles of Android ever since I made the decision to stop using my Nokia N900. Nokia had given up on Linux phones, and it was clear that there would be no further security patches for my favorite smartphone of all time. Shaking Google out of Android had been my mission for years, and I had resorted to running my own builds of “de-Googled” LineageOS. I was longing for something better. I was out of the country when I first read about postmarketOS (“pmOS”) in May 2017. postmarketOS is a Linux distribution based on Alpine Linux, that strives to provide a Linux distribution running the mainline Linux kernel, as a means to revive old smartphones long forgotten by their manufacturers. My beloved N900 was one device with (rough) support! I quickly jumped on eBay to order a second N900 to meet me at home when I arrived back, because obviously two are needed. Obviously… Thus began my relationship with postmarketOS, one that continues to this day.

Things were not all rosy though… After some time it became clear that the older N900 CPU wasn’t going to get any faster for running “modern” applications and that there would never be a free userspace graphics driver for its GPU, so I was quite excited when I first learned about the Librem 5. Sure, it didn’t have a physical slide-out keyboard, but the promise of a device from a company that would treat Linux support as a first-class citizen was too good to pass up. I promptly pre-ordered a developer kit (“devkit”) and phone, with the full intention of porting postmarketOS to the device and eventually using it full time to replace the heaping pile of Android in my pocket.

The devkit couldn’t have arrived at a better time (sarcasm); right before the holiday season! After spending the first few days still in the box due to my “other obligations,” it was rapidly de-boxed and I started planning my next move. Some cursory research told me that the devkit used an SoC from NXP, used the bootloader u-boot, and was flashed with a utility called uuu. Since I wasn’t completely sure how uuu worked, and expected trouble once I started to experiment with using it to flash pmOS, the first task was to solder on headers for using the UART.

This, as I suspected, proved to be a very worthwhile step. Purism should consider just adding the header next time, though it was very simple to DIY at any rate. The only other modification I made to the devkit (completed a few days later) was adding an external 40mm fan right over the SoC’s heatsink. It wasn’t necessary, but I did cause the device to reboot at least once compiling Mesa, so I figured it couldn’t hurt. A photo of this fan (shown later on) ended up causing some angst on the Internet. Oops. After adding a battery, per Purism’s recommendation, I was now ready to get some work done!

“Don’t reinvent the wheel, just realign it” (Anthony J. D’Angelo)

Being new to using NXP (and Emcraft) devices, a large part of the first day was spent trying to gather as much information as possible about how this device boots, how to push images to it for booting, where in the image it expects a bootloader (u-boot) to be “installed,” and the required format and locations for the kernel and device tree file. Emcraft had a surprising amount of information on their website, along with some examples in the form of out-of-tree kernel patches and pre-compiled binaries. By far the greatest breakthrough was discovering Purism’s Jenkins CI instance. Being familiar with picking apart Jenkins output because of $daily_job, I was able to figure out how Purism was building all the major components. By searching Purism’s Gitlab instance for the script names run in CI, I could locate the appropriate repositories and source code for those scripts to use as a model for the Alpine Linux packages I’d have to create for supporting the devkit and phone in pmOS.

Armed with a basic idea of what I’d need to make an attempt at booting a custom u-boot and kernel, I started looking in-depth at uuu. This was a completely new tool to me, but it was very easy to package for pmOS/Alpine Linux. The uuu utility can be passed a script with arbitrary commands to execute, and once again, Purism’s prior work on this provided me with a helpful template to use later on when I had created a pmOS image for flashing to the device. I was able to use the u-boot image from Purism’s devkit artifacts to verify that my packaged uuu worked

The next step was to look at building/packaging the u-boot, the Arm Trusted Firmware (ATF), and the Cortex M4 firmware for DDR training. These were generally fairly straightforward to create an Alpine Linux package for, though, since none of these components used upstream/mainline source, the packages had to diverge from any that may already exist in Alpine Linux (e.g., can’t use their u-boot) in order to use the sources with any patches from Purism/Emcraft/NXP/etc. The biggest hurdle here was that the M4 firmware needed to be a 32-bit ARM binary, while everything else was compiled for AArch64. The package repository specific to pmOS (pmaports) includes cross compilers, so I was able to compile/assemble the single image bootloader into one Alpine Linux package. Not exactly ideal, and would almost certainly be rejected by upstream Alpine, but it was quick/easy, and simple to debug. Purism’s CI and supporting scripts were heavily utilized in order to figure out the right sequence of steps to build things, where to “install” resulting binaries, and how to generate a correct image with them all included.

Getting to the point of having a working u-boot/firmware image required some trial and error, some of which was no fault but my own, and having that UART header on the devkit was absolutely required for me to resolve all of those early, pre-Linux kernel boot issues.

Anatomy of a mobile device in postmarketOS

Devices in postmarketOS are supported by device-specific Alpine Linux packages that are responsible for installing any additional configuration unique to the device, and providing parameters to the tool (pmbootstrap) responsible for building pmOS images for flashing to the device. Common components of a device package in pmOS include things like configuration files for setting up audio, and parameters for specifying where to “install” u-boot in the image that is generated. pmbootstrap makes it very easy to create a skeleton device package, by just specifying the new device when running pmbootstrap init. The hard part is collecting all of the various configuration files that help enable and set up hardware for the device. Once again, Purism’s public repositories to the rescue! By grepping my way around their public repos (e.g.,librem5-base), most functionality on the Librem5 present in PureOS could be enabled in pmOS early on. Most devices in pmOS, unfortunately, require downstream kernels. The devkit and phone (as of today) are no exception, though it is close enough to mainline that I didn’t have to apply any of the various hacks that pmOS includes for dealing with ancient, abandoned downstream kernels. As a result, it was relatively straightforward to create an Alpine Linux package for building/installing Purism’s kernel fork. The last major piece was graphics. I could have ignored this if I only wanted to boot to a console using the framebuffer, but that’s no way to use a smartphone in 2020! (/troll)

I wanted to run Plasma Mobile and Phosh, and the only way I could do that was with a version of Mesa that supported the GPU on the devkit. At the time, support for this GPU was upstream in Mesa, but it had lots of issues. If I wanted to have a decent chance of running a hardware accelerated graphical environment, I would have to package Purism’s Mesa fork. This would end up creating all sorts of issues later with package dependency resolution in Alpine Linux (conflicts with Alpine’s Mesa package and subpackages were frequent), so thankfully this was only temporary until Purism landed all of their patches in upstream Mesa.

I now had everything needed to generate my first bootable postmarketOS image for the devkit, and flash it to the device!

Except I didn’t. Flashing the device manually with uuu was fine for me, but I expected anyone with a Librem 5 to be able to easily use pmbootstrap to build and flash pmOS to the device. The first change was teaching pmbootstrap to automate uuu. pmbootstrap has had support for flashing devices directly with external tools for quite some time. Most devices originally shipped with Android using fastboot for flashing, which isn’t an option for the Librem 5. Having image flashing support isn’t a requirement, but it’s much nicer than trying to use scary tools like dd to destroy flash disks, or requiring folks to run some other command/script separate from pmbootstrap to get pmOS on their device. Shoehorning uuu automation into pmbootstrap was basically just adding some new pmbootstrap configuration parameters to the device package that can be used to tell pmbootstrap to use uuu, and then patching pmbootstrap to call uuu with the right parameters when instructed to flash the image to the device.

The Librem 5 devkit (and phone) require two firmware files for booting, as mentioned previously: u-boot and M4 firmware. pmbootstrap knows how to deal with embedding u-boot firmware (used by some devices supported in pmOS), but did not know how to embed two separate firmware into the flashable pmOS image. Another patch adding this functionality, and some more device package config parameters were all it took. Now I was ready for…

First boot!

At the time, the devkit’s onboard display was not functional, so I had to make do with HDMI to an external display which, fortunately, worked perfectly. XFCE is a fairly lightweight desktop environment with (most) Batteries Included by default, so it’s a favorite of mine for the first boot of pmOS on a device.

A lot of folks in the Purism Matrix channels are excited about “device convergence,” in particular using your phone with an external keyboard/mouse/monitor as you would a desktop PC. Not to be the one left out, I thought it might be fun to play Elder Scrolls 3: Morrowind (via OpenMW), which basically requires a keyboard/mouse to use. After packaging OpenMW in Alpine Linux, I was now able to play one of the best RPGs ever made on the devkit. Albeit, not at a super smooth frame rate, but that was to be expected given how early this hardware was and the current state of its support in Mesa.

Fast forward a few weeks, and Purism is starting to ship their first phones to the general public. I deferred receiving the phone I ordered until the “Evergreen” batch, because I wanted a higher quality device to use for years. I was bummed that I was stuck using a devkit with pmOS, and was starting to regret my decision when Purism was kind enough to lend me a Birch phone to play with. No time was wasted deleting the PureOS image on it (sorry Purism!) to make room for pmOS. Luckily the same SoC was chosen for the phone, so only minor changes were needed to the device package in postmarketOS since the phone shared the same kernel and u-boot source code as the devkit. The u-boot configuration was different, but that was just a matter of selecting the appropriate defconfig at compile time.

UARTs UARTs everywhere, but not enough to use

One thing that was worrisome from the start was the lack of readily accessible UART on the phone for debugging issues pre-Linux boot. Now, this wasn’t something most users of the phone would care about, but I had relied on this extensively on the devkit to find the right combination of configuration parameters for booting pmOS. The schematics for the phone show there are 4 UARTs, but none of them have headers pre-installed, and 2 of them are available in the super tiny test point pads under the back cover. A few devices that we’ve used with postmarketOS support serial console out via a 3.5mm headphone jack. That would have been a nice feature for the Librem 5, but wishing for it wouldn’t help my current situation.

Eric Kuzmenko from Purism pointed me to an M.2 debug breakout board that apparently Purism was using in-house at one point, and provided me with a version of the layout that exposed only UART. I had this board fabbed, but still (as of today) have not succeeded in getting u-boot to output a serial console over it. So, if there are pre-Linux u-boot-related issues, I’ve had to fall back to using the devkit to debug them. Luckily I haven’t yet had any pre-Linux boot issues specific to the phone.

It’s worth noting that this is largely a problem for me and anyone else who wants to port an OS to run on this device. The vast, vast majority of users won’t miss having easy UART serial console access.

Fortunately for my first attempt at booting on the phone, the only issue I ran into was accidentally using the u-boot image built for the devkit. Oops. And, fortunately, that didn’t result in any magic smoke being released!

If it looks like a phone and acts like a phone, it must be a phone

It wasn’t long after this that Purism was demonstrating the modem work they had enabled in their PureOS distribution. Enabling the phone’s modem with ModemManager was easy, but audio was not being routed correctly during a phone call. Purism released a small application called wys that handles audio routing during phone calls, so after packaging this application for Alpine Linux (so it was then available in pmOS), I was able to make and receive phone calls to some degree. Audio quality and proximity sensor support still needed a little work, but the makings of this being an actual phone were already there. SMS just worked on pmOS with no changes at all, which was a very pleasant surprise after having spent a lot of time in the past figuring out how to send and receive an SMS using ofono test scripts on the N900. Mobile data connections also just worked in pmOS with Phosh and ModemManager; another very pleasant surprise!

from Hacker News https://ift.tt/2RB1zph