Android Q(&A): Android Engineers take us on a deep dive of Android Q

As has become tradition for Ars at Google I/O, we recently sat down with some of the people who make Android to learn more about Google's latest OS. For 2019, the talk was all about Android Q and this year's big engineering effort, Project Mainline. Mainline's goal is to enable Google (and sometimes OEMs!) to directly update core parts of the OS without pushing out a whole system update. If that sounds technical and challenging, well, it is.

This year running the Ars Android Interview Gauntlet we have veteran interviewee Dave Burke, VP of engineering for Android. As the head of Team Android, Burke is an encyclopedia of Android knowledge and always manages to come up with insightful answers to my grab bag of esoteric questions. And returning for the second year in a row is Iliyan Malchev, principal engineer at Android, the lead of Project Treble, and all-around Linux integration guru.

But to help up the ante for this latest deep dive, Ars was also joined by Anwar Ghuloum, Android's senior director of engineering and the lead of Project Mainline. Ghuloum's insight was especially welcomed given this year's I/O headliner: as "The Next Great Android Update Project," Mainline was easily the biggest news to come out of the conference.

So, buckle up for a long Android Q(&A) if you will—but first, some background on Mainline.

Project Mainline: A "fundamental shift" in Android OS development

For years, we've seen Google continually work to chop Android up into more easily updatable pieces. Early on in Android's life, the Google apps and core system apps were offloaded to the Android app store, allowing Google to pump out new user-facing features whenever it wanted. Google Play Services then took many developer APIs and offloaded those to the Android app store, allowing Google to pump out developer-facing API updates whenever it wanted. More recently Android 8.0 brought us Project Treble, which separated the OS from the hardware support, allowing for easier update development.

With Android Q, the big new modularization effort is "Project Mainline." Along the same lines as Google's early-days move to put apps in the Play Store, Mainline modularizes several core system components and moves those to the Play Store. Mainline goes deeper into the system than the surface-level apps, though—these are big chunks of system functionality like the media framework and ART, the Android RunTime.

Traditionally, the Play Store has distributed apps only in the form of APK files, but for many of the components being modularized in Project Mainline, they wouldn't work if packaged up as an APK. Since the APK system was built for system and user-level apps, there are limitations for things like permissions and when they can turn on in the boot up process. For modularizing these core components, Google came up with something more powerful than an APK: the "APEX" file type. APEX files can have essentially root-level permissions, and they get to start up very early in the boot process, allowing Google (or your OEM) to update many more components. APK files are packages for system- and user-level apps, and APEX files are packages for core system components. This table shows the first batch of them in Android Q:

Google's Project Mainline components for Android Q.

Credit: Google

In the future, we'll probably see Project Mainline modules grow to encompass more and more of the Android system. For this first Android Q release, though, Google chose to focus on three themes: "Consistency," "Security," and "Privacy." Before our I/O interview, Google provided us with the above table of the Project Mainline components in Android Q, detailing which components are being modularized and what the recommendations are for OEMs. And that brought us to the first question.

What follows is a transcript, with some of the interview lightly edited for clarity. For a fuller perspective, we've also included some topical background comments in italics.

Ars: So I have this Project Mainline table, which details which component are recommended or not. How did you go about picking what is and is not mandatory?

Anwar Ghuloum, the head of Project Mainline: Ideally, we'd want everything to be mandatory. The way we worked on these modules was to talk to all our device manufacturers and say, "Hey, we're doing this, work with us on it." They upstreamed a bunch of code. They had a bunch of future requests for things that they were beginning the process of working on, and, for those modules where we could actually meet all those requirements, we made those mandatory. For the modules where there are still gaps, we made them optional for this release, and for the next release they'll be mandatory. So that gives us time to get to parity, because we don't want to regress their device experience, but pushing these modules, we want to make sure their stuff gets in.

Dave Burke, VP of engineering: I think part of this work is upstreaming with our partners. When I say partners, I'm talking about device makers. They add changes into the device they build, and we want to get them all upstreamed to mainline code branch, so we have consistency. It just takes time.

Ghuloum: Yeah, I mean, we've done a ton of upstreaming. It's amazing. For some of these packages, we upstreamed more in the last year than we've upstreamed in the previous 10 years.

Burke: Yeah, it's important.

I think part of the background here is that Project Mainline represents Google clawing back final ownership of some core system code from OEMs (aka device manufacturers). If device manufacturers are going to give up ownership of that code, Google wants to make sure all the customizations OEMs used to add are now supported in the normal AOSP (Android Open Source Project) code base that everyone uses.

You can imagine Google going around the ecosystem for each module and asking things like "Do you really need to customize the way the DNS resolver works?" When the answer was "no," Google's version was made mandatory. For modules where the answer was "Well actually...," the plan is to upstream all that into the Google version in AOSP and eventually adopt the Google version. Ghuloum's declaration that some packages were able to "upstream more in the last year than we've upstreamed in the previous 10 years" sounds like they're making a ton of progress. More code upstreamed into AOSP means less code to keep track of for OEMs, which leads to easier, less complicated system updates. 

The initial focus of Project Mainline.

Credit: Google

Ghuloum: What we explained to our teams is that the premise of using a Mainline module is that you will get to release once a month. That you are actively working with the partners, co-developing, planning your roadmap, and stuff like that. People in the team have been pretty compelled by that, and excited about it.

Ars: Oh, is that the plan—a once-a-month release for Mainline modules?

Ghuloum: Well, that's our trained cadence and that's driven by our security update schedule, because some of the components are security sensitive. The media component in particular comprises primarily codecs and extractors. One of the reasons that's a module is that we looked at vulnerabilities over the last year, and nearly 40 percent of patch vulnerabilities in our security updates came from those modules. So, we're like, "Hey, what if we could just push these out to the entire ecosystem, instead of putting the burden on the OEM to take these, test them, and push them out themselves?"

Android's media playback engine has to load all sorts of scary file types from across the Web, and doing so in a safe manner has always been a security challenge. Android's media playback engine is called "Stagefright," a name you might recognize from the news cycle in 2015, when series of remote code execution vulnerabilities were discovered in the Stagefright engine. The formal monthly security update schedule that Ghuloum is talking about was started up in response to those vulnerabilities, and the media framework hardening continues to this day.

Today, Google produces the AOSP security updates every month and hands them over to OEMs, but that still isn't a perfect solution. Not every phone supports monthly security updates, not every phone ships these updates every month in a timely manner, and most phones only support them for two years. As Ghuloum says, in addition to Google taking over the testing, integration, and roll out of these fixes, Project Mainline would also (eventually, once everything is on Android Q) let Google roll out these fixes every month to the entire ecosystem instead of just a few flagship phones. 

Burke: The other thing is, we often hear from developers on what we could do to make their lives easier on Android. One of the things that comes up often is fragmentation of slightly different behaviors in different parts of the OS, even within the same manufacturer—the media framework is one they bring up. And so more consistency there is good for the developers, too. It reduces errors and the work they have to do, and it increases the quality of apps, which is good for users.

Ghuloum: I was calling this "bug consistency" yesterday.

Burke: (laughing) Bug consistently! Yeah, that's true.

Ghuloum: There's this module called "ANGLE;" it's basically OpenGL implemented on Vulcan. Right now it's mandatory for OEMs, but developers can opt in to whether they use it or not. The idea is to lean into the kind of the support for Vulcan that's coming on all these devices. Having a consistent GL implementation—not necessarily a bug-free one, because we never ship bug-free software, nobody ever does—but the thing for game devs that they struggle with is, they're used to bugs in drivers, but all these different bugs and different drivers are super painful. We can make that much more consistent.

Burke: The other way to think about this is: it's generally good hygiene. You look at the GPS rollover that happened on April 6, for example. It grounded some airplanes because they couldn't cope with the clock rolling over. There's always something that's going to happen in software, and you want to have the ability to get this updated, especially, like, really low-level stuff—like with Conscript, which is our secure library, SSL library, and TLS. That's updatable, as well. And that's another area that when certificates expire, or you're cert provider suddenly goes out of business, you can fix that.

Ghuloum: Or, the BoringSSL bugs.

Burke: Or, the BoringSSL bugs, yeah exactly. They're kind of unsexy but fundamental components in the system.

Iliyan Malchev, Project Treble lead: Years ago, there was a bug in Bionic (Android's standard C programming library) that was introduced by one of our partners, who had the sign tables wrong. So, trade functions were randomly failing, and a range of the curve in a way could break games. So, stuff like this is incredibly hard to catch before you ship.

Ghuloum: And the developers had to live with it for the next few years—unless it ever gets patched. I've seen that with some of our own first party apps, [they] have to work around bugs throughout the ecosystem. It's just spaghetti code.

Ars: So, was there a test update that went out to Beta Q users?

Ghuloum: Yes—actually, as of Beta 2, we started pushing updates. There are threads on Reddit about this.

Ars: Right, OK. 

Ghuloum: Their devices were rebooting and, yes, we were pushing updates, testing updates, and we were rebooting people's devices. We're only doing this in the Beta, actually. During production, when Q ships, all reboots will just be organic reboots from the user. We looked at the numbers, and it looks like over a couple of weeks, that gets us to a reasonable saturation level of people taking the update. Plus, we have monthly security updates we're going to be rebooting at least once a month, anyway. So, you'll just take it. We don't want to put UX in the user's face. If there's an update waiting, we just wait for them to reboot.

Ars: OK. Do you think that's what the final version is going to look like—kind of a quiet background thing that won't be very visible?

Ghuloum: Yeah.

In the betas, there has been some UI for a Mainline update. The about screen has a new "Mainline Module Version" field, and tapping on it will check for a Mainline update. On some versions Mainline has been called a "Google Play System Update." I still don't think there's a final, user-facing name yet.

Credit: Ron Amadeo

Burke: The thing about Mainline that may not come through in the blog posts or in other discussions is, it's probably the biggest fundamental shift in how we do software development for the operating systems since we started. The thing about how Android works is, we build a reference, or now a full product. You know, the operating system and the product, which is Pixel, and then that code is open source, available to partners. Partners like OnePlus take it and they build devices, but, really, we hand out that open source code, and they take it from there. So, the scaling happens in all the leaf nodes, if you want to call it that.

When you have a model like Mainline, where we have a common set of code for modules, and we, the Android team, are pushing it out, now we're responsible for that code on every device. So, if a bug happened, it occurs on a OnePlus phone, that bug comes to us, and we've got to fix it and push it out. To facilitate a model like that, we've had to seriously scale up our, what we call, "engineering productivity," or our test automation. We've had to significantly scale up our telemetry. We've had to have rollback protection. We've had checkpointing. We've had to have canary models. It's really changed how we've done things. That's also why we're starting this set of modules, we prove them all out, and once we have that right we can do more.

Ars: It sounds like you imagine the scope of Project Mainline expanding in the future.

Burke: Yeah, I think it will increase. I would say, with caveats, which are like, "This is an open source project. We are a partner of that project, so we have to do it in conjunction with our partners in a way that makes sense for everybody." But, yeah, the infrastructure allows us to do more.

Ars: OK, because it caught my eye that the APEX documentation mentions ART (the Android RunTime) and hardware abstraction layers (HALs), and those are not Mainline modules yet.

Ghuloum: Actually, if you look in the system image for Pixel 3—it's no secret since it's out there now—you'll see that ART is packaged with Libcore as a mainline module. I think it was in Beta 3. We have Libcore and Bionic in an APEX, but we're not updating it for Q.

Engineer to English translation: Libcore houses ART and a lot of Java components. Bionic is Android's C standard library. 

Ars: Oh, OK—I did see an "APEX" folder in the last release.

Burke: We still believe very deeply in this being open source. We believe in a collaborative ecosystem, and we also believe deeply in device makers being able to differentiate. So, we need to balance all that. But there's a lot of things that aren't differentiating, like Conscript library. Bringing things like that into a module will make devices more secure, it will reduce bugs, and it will make developers' lives easier. That's a clear win.

Ghuloum: The interesting thing about customizability is we picked these modules in areas where they were security or privacy sensitive or where OEMs weren't really differentiating. Media's one interesting area where people have extensions, and so that's the one place we actually allow—and we're going to look more into this in the future—but we allow partners to ship their own APEXes. Which is, to ship APEXes alongside the media component APEX so that they can extend functionality.

Burke: Yeah, we're using an extension model. There's a common core and extra functionality modules.

Ars: So there will be, like, a "Samsung media" module with extra bits?

Burke: Yeah, maybe you want to ship Dolby Vision, or Dolby Atmos.

Ghuloum: Or, some other random codec we don't support.

Ars: OK, so that lets them adopt the AOSP implementations and add their little extras at the end of it.

Ghuloum: It's funny, we've been kind of moving more toward that in general. If you look at what we did with the adaptive battery stuff last year, the way we built that was there was a bunch of system APIs that an AI model that we use can interact with. It can sort of predict what things you are going to run next. Partners can replace that wholesale, right? So, now, instead of patching over all this code and having to re-base, they can just use these stable interfaces to extend. We're going to go more in that direction.

Burke: I feel like there should be a name for this kind of software development approach. Twenty years ago when you wrote software, you put it on a CD-ROM, you sent it out, and then you might have like a service pack or something. In this model, where you're pushing software out to scale, it's all about telemetry, doing canaries, launching it, and then you have to rollback, adapt to it quickly, and do frequent updates. [You need to] have very rigorous testing, so you know you don't regress. It needs a name. Maybe it's "modern software development."

Ghuloum: Progressive?

Burke: Progressive! Progressive software development. But it's very different. You look at other operating systems, and maybe they have a bug in their video telephony and they have to push a whole operating system, right? So, that's kind of not as progressive. But if you look at Android, in general, a lot of pieces are already updating. Google apps update very frequently, and Google Play Services updates very frequently. To us, it's a sort of natural progression.

APEX—and I swear this is in the official documentation—is actually an acronym for "Android Pony EXpress." How can you not want to know the story behind that?

Ars: So what are the requirements for supporting Mainline? Is it mandatory for phones that launch with Q?

Ghuloum: Yeah, any phone that launches with Q has to support it. Then there's a small number of modules that are required on devices that are upgrading to Q. The reason—you probably want reasons for this....

Ars: Sure!

Ghuloum: The two things that are required for devices upgrading to Q are ExtServices and Permissions Controller because those are things that were already signed by Google and delivered to the partner to ship as part of their system. So, now we're just updating that.

Signing an Android app is the final step before the distribution process, so when Ghuloum says the two packages are "signed by Google," he means they are finished components that OEMs aren't allowed to change. Before manufacturers would just slap these finished components in their existing OTA updates unmodified, and in that case it makes more sense to have Google deliver them directly via the Play Store. 

Ars: So what is in ExtServices? Because I wrote about it years ago...

Burke: Yeah I remember that!

Ars: And it doesn't seem like anything happened with it. 

Ghuloum: The glib answer is it's a grab bag of things that we want to update in persistently running system processes. The more concrete answer is they are bits of some privacy stuff in there that we need to be able to update, and fix bugs in, or rev, or change the policy, and so on. There's some bits of auto-fill field detection in there. There's a bit of Mainline in there that's updatable, too, like our Mainline on-device watchdog. Stuff like that is actually in ExtServices.

Ars: Is there a reason you ended up with this weird grab bag?

Ghuloum: We wanted some place to put very small things that are going to run all the time—no UX, no large memory footprint items. That's why we created ExtServices. By design, if you're going in there, you're small—you know, like a little algorithmic thing that we can rev.

Ars: There's a Mainline support exception for devices with low RAM. Is there a reason for that?

Ghuloum: The primary concern is not so much the RAM, it's the storage. A lot of low RAM devices, especially, like, 512 and 1 gigabyte devices, are pretty tight on storage.

Burke: The APEX module has to live with the data partition, and so, if you're replacing a system partition, you're effectively duplicating files.

Ars: Right, just like APK updates.

Ghuloum: We are going to work to eliminate that loophole, but it was just a practical consideration.

Burke: I have a question. Who came up with Android Pony Express?

Dave Burke has taken over this interview and is now asking his own questions! APEX—and I swear this is in the official documentation—is actually an acronym for "Android Pony EXpress." How can you not want to know the story behind that?

Malchev: I can tell you exactly. I can send you the link to the commit to prove my authorship of this wonderful, wonderful name. We were going to call it "NPK," originally, for "Native PacKage."

Burke: But it does Jars as well.

Malchev: But it does Jars. So, Dianne Hackborn—who was one of the architects of Android, in general—she's like, "This is too close to APK, I don't want it." And then we started bikeshedding, and I'm like, "Just call it APEX." It's Android Pony Express.

Ghuloum: By the way, when you came up with APEX, people were like, "Yeah, that's it, we're done." We stopped.

Burke: Yeah, that was pretty good.

Malchev: So, is the word a "backronym?" The less fun backronym was "Android Portable Exchange," but "Android Pony Express" is better.

Burke: Much better.

Ghuloum: It's funner.

Burke: We need more names like that.

Dynamic System Updates—Like a Linux Live CD, for Android

After exhausting the topic of Project Mainline, we turned to other interesting Android developments that were shown off at Google I/O. In the Android sandbox, there was a demo of a previously undisclosed Android Q feature called "Dynamic System Updates." This was a dual boot solution for Android, allowing you to be in one version of the Android OS and switch to a different version of the OS with a simple reboot. For developers, manufacturers, tinkerers, and other people who might want to quickly boot into an old or new-and-beta build of Android, this sounds like a very interesting feature.

It also seemed very Project Treble-adjacent. The I/O demo unit was switching between a retail build of Android and a "Generic System Image (GSI)." Android was previously packaged as an embedded operating system, with the OS and hardware support baked into a single image. Project Treble split the OS from the hardware support, and the ultimate expression of that is a GSI—a "generic" version of Android that will run on any phone with Project Treble support. With the GSI, Android hardware support works a lot less like an embedded operating system and more like Windows or desktop Linux. You have one build of the OS that works on many devices.

Treble support—and successfully booting the GSI—are actually a requirement for Android compatibility now (as of 8.0 Oreo). Google even released a GSI version of the Android Q Beta. Combine that with this dual boot feature, and next year when the Android R beta comes out, it's possible you could just dual boot into it instead of having to wipe out a device.

Since my three-person panel of interviewees included Iliyan Malchev—the head of Project Treble—I figured I could get some solid answers on this dual boot project.

Dynamic System Updates. In this demo, it was a notification.

Ron Amadeo

Dynamic System Updates. In this demo, it was a notification. Ron Amadeo

Here's the main system, a Google version of the Android Q Beta.

Ron Amadeo

Here's the main system, a Google version of the Android Q Beta. Ron Amadeo

This is the same phone, but a quick reboot later and we're on a totally different version of Android! This is an open source-only version of Android Q.

Ron Amadeo

This is the same phone, but a quick reboot later and we're on a totally different version of Android! This is an open source-only version of Android Q. Ron Amadeo

Here's the main system, a Google version of the Android Q Beta. Ron Amadeo

This is the same phone, but a quick reboot later and we're on a totally different version of Android! This is an open source-only version of Android Q. Ron Amadeo

Here's the first build on this phone.

Ron Amadeo

And here's the second build. Again, this is all on a single device. One phone, two builds of Android.

Ron Amadeo

Ars: So what is this live boot feature? First is it "Dynamic Android," or "Dynamic System Updates?" I've heard several different names for this while asking around.

Malchev: It started out as "live images," or "live boot," because the first implementation of it actually had a Generic System Image in m memory. So it mimicked what you would do when you boot from a Linux Live CD. But then the security team made us completely rewrite it in October-November. And when we rewrote it, it actually ended up being better.

Burke: That was like a little insight into internal Android teams—the security team "made them rewrite it." (Laughing.) We have a very strong security team. They are a great team. They actually help make Android more secure.

Malchev: I'm sorry, so "pending security review feedback," we rewrote everything and the second iteration was much better! We made it persist on the user data image, so it was no longer "live." We called it "Android on tap."

Ars: I've heard that name, too.

(What is it with Googlers and naming things?)

Malchev: Then, when we started discussing the name with people whose job it is to decide names, they figured that it wasn't as descriptive. We settled with "Dynamic System Updates."

Ars: OK, good. I think that's what the sandbox demo UI says. So, how does this work? Is there an extra partition that this gets loaded onto?

Malchev: No. Actually, one of the other Q launch requirements that are meant to make updates easier is dynamic partitions. It's unrelated to this, but we reuse the same mechanism. Dynamic partitions allow you to resize partitions with an OTA without risking sudden death and messing up the partition table.

I regret that I did not jump on this little tidbit during the interview, but changing the partition size to make updates easier sounds like a boon for cheaper devices that are tight on storage. An Android phone ships with two main partitions, the "System" partition, which holds the factory OS and can only be changed via an OTA (over the air update, AKA a system update), and a "data" partition, which houses the user's files. Today, the size of these partitions is set in stone at the factory, and while that's not a big deal on devices with tons of storage, when you have these cheap "Android Go" devices with only 8GB of storage, it's a challenge to pick a partition split that gives the user enough storage to work with while still having a large enough system partition for the OS and any future OS updates. 

Resizable partitions would mean manufacturers wouldn't have to worry about reserving system space for future updates. They could give users as large of a data partition as possible at launch, and then later down the line could resize the system partition for future updates if needed. 

Malchev: So, the underlying mechanism is we collect blocks from storage and assemble logical partitions from that. We're using the same mechanism for Dynamic System Updates, and we create two files on /data.

Burke: One of the files is the system image and one for user data, so it's like a virtual partition.

Malchev: Yeah, one for system, one for data, and it would span their extents. We would find out what blocks were underpinning them, and we would use that mechanism to create a virtual partition out of those. And on top of that partition, we would dump our GSI, and we would create an empty F2FS or EXT4 user data partition on the data-in-data partition. We need a better name for this.

Malchev: And then, we have a setting where init (part of the Android boot process) would reroute the boot flow to boot from the stashed partition. So you can boot from it, delete it—the idea is you can try new versions of Android, without hurting your device.

Ars: It sounds a lot like the virtual machine, which is cool.

Burke: Yes, it is somewhat, except it boots into it.

Ars: So, then is this extra version of Android a permanent thing? Like, you'll be able to keep running it for however long you want?

Malchev: You know, Dave (Burke) gave this feedback: "We don't want people to get stuck in this." So if you reboot your device, it will go back to your factory OS. We don't want people to get stuck in this. But, you can toggle a flag—it would be in ADB—that will allow you to reboot into the GSI. The flag is ephemeral, so you have to keep toggling the flag to keep rebooting to the GSI. You could set up your account and everything, and your data is just in its own little island. There's no cross talk between the two.

Ars: And this doesn't need an unlocked bootloader, right?

Malchev: Dynamic System Update is not a launch requirement, but, if enabled, it does not need the device to be unlocked. That's kind of the point of it.

Ars: Right, OK. So the goal is you can test an Android Beta on a Samsung phone, or something that's locked down.

Malchev: Yep—and we have it enabled on our Pixel devices. Which is what you saw at the table.

Burke: This is another thing: as you know, it was facilitated by Treble. The very clean hardware interface allows you to do this. This would be crazy talk, like three or four years ago.

Malchev: If we did it a few years ago, you wouldn't have anything uniform to flash to. So, we wouldn't know what to do with that mechanism. But it's very easy to support this and we would like for our partners to enable it. We're talking to a number of them to see.

Other Android Q-uestions

Now for the "why is this here?" portion of the interview. The first oddity that we addressed is the status of the "Snooze notification" feature in Android Q. This feature arrived with Android 8.0 Oreo, and it survived all the way up to Android Q Beta 3, where it disappeared. It returned in Android Q Beta 4. So, why?

Ars: Android Q stuff: Why was snooze removed from the notifications panel—bug or feature?

Burke: It may or may not be gone in the future. So, we added this new "gentle" and "priority" setting for notifications. We're trying to simplify all the concepts. The problem with notifications, I think, is that they have gotten a little too complicated. You have this concept of channels, you've got snoozing. We wanted to simplify and just have this gentle and priority setting. So one part of our team is arguing to remove snooze—I'm on that part of the team—to simplify things for general users. Obviously, we always worry that there's going to be some users who really liked snoozing, but what's the right thing for the majority of users? Is simple better? And, so, that's why it's sort of coming and going. I think it was in one beta, it's not in another. It may end up back there. It has very low usage.

Ars: It's hard to find, yeah.

Burke: This is kind of one of the advantages or disadvantages of doing betas. You can see us debating.

Adaptive sleep, which appeared briefly in an Android Q beta for some people.

Next oddity: This strange "Adaptive Sleep" setting was showing up for some people in one of the Android Q betas. It described itself as a feature that kept the screen away when it detected you were looking at it, but it didn't work and it was unclear how it would work. Some devices, like Samsung Galaxy phones, already had a feature like this that was powered by the front camera. It would turn on when the screen was on, and if it detected a face, it would stay on. 

Ars: What is this adaptive sleep thing in the settings? Did everybody see the picture?

Burke: That's kind of an experimental thing that isn't ready. I think we had a bug where it showed up in certain builds.

Ars: Yep.

Burke: So a couple of people have asked us for this, and basically the idea is to keep the phone awake when you're looking at it. This is just a skeleton framework right now, so it could be anything. I don't think it's actually implemented in AOSP. It's just like, "Here is where you would put it in the OS, and then, a device maker would provide a back end to it." I don't think it's going to make it out to Q. It wasn't ready, so I was surprised to see it, too.

Google does this a lot. OEMs come to it with a feature they want implemented that isn't in Android, and often they either implement it themselves or add API hooks for OEMs to plug into. In the case of something like split screen support in Android 7.0 Nougat, OEMs build a feature first and Google arrives with its own implementation and standard rules for the entire ecosystem later. In the case of "Adaptive Sleep," Samsung will have a clean, tidy place to plug its front camera implementation into. Either option is a good thing as it reduces the amount of custom code OEMs have to keep track of, even if it doesn't show up in the next Pixel device. 

A picture of Android's (not Chrome's!) desktop mode.

Credit: XDA Developers

The next weird item on my list was Android Q's desktop mode. With a carefully crafted ADB command—you can read all about it on XDA Developers—you can enable a desktop OS interface for Android. The desktop interface looks, well, like Chrome OS, with a desktop, floating windows, and an app drawer button that could double as a "start" button. Why would Google build such a thing? Is Chrome OS being replaced? Is Android going to take on Windows? Will there be Pixel computers sold with a keyboard and mouse?!

Ars: Speaking of weird features, why is there a desktop mode?

Burke: Where's the "desktop mode?"

Ars: With some ADB commands you can enable a desktop mode with floating windows and stuff. 

Burke: Oh, right.

Ghuloum: I can give the history on that. This is a part of the work we do with the Foldables team and teams at various OEMs. We build better support for simultaneous multi-window and multiple displays. One of the interesting use cases here, that we don't officially support on Pixel but our partners support, is Samsung's DEX mode and other OEM's desktop mode. Everything we did there works great for the Android Runtime on Chrome and works great for desktop mode. So, the team was just using that as a way to sort of demo and experiment with stuff before we actually had physical form factor devices.

Let's talk Linux!

Credit: Iliyan Malchev

Malchev is also the Google representative that shows up at some of the Linux conferences. Malchev was a speaker at Linaro Connect 2017, where he made the announcement that LTS (Long Term Support) Linux kernels would be supported for triple the previous lifespan, or six years. This was a big deal for Android device development, which took so long that it was typical for a device support window to outlast the Linux kernel support window.

Kernel.org lists two kernel versions on the six-year LTS plan: Linux kernel 4.4 and 4.9. Newer releases of the Linux kernel have gone back to a two-year support window, though, so it seems not every release will be a six-year LTS. Since Malchev kicked off the announcement of this plan, I figured he would know what the selection process is like.

Ars: You announced six-year LTS kernel and now it seems not every release is getting a six year LTS. How does that work? 

Malchev: Six-year LTS kernels are a thing, and we will keep announcing six-year LTS kernels. Not every two-year LTS kernel becomes a six-year LTS one because not every LTS kernel is what ends up on Android devices.

Ars: So basically this is driven by Qualcomm?

Malchev: Yeah basically, whatever kernel Qualcomm, MediaTech, and the other SOC manufacturers choose to commercialize their SOCs on, this is what we announce as six-year LTS. For example: for Pie, you are required to ship on one of three kernel versions. We provide the minimum version for each one. Of these three, I think 4.4 and 4.9 are six-year LTS ones. The third one, which, I think, is 4.14, is not announced to be six-year LTS yet because we are still waiting to see if there's an uptick in the ecosystem. There's a literal bootstrapping problem, where we need to support a version on the Android common kernel for silicon vendors to consider it, but for us to announce six-year LTS it needs to actually ship. So, we're in this waiting period for 4.14. Going into Q, there's going to be two more, I think, that we'll add to this range of kernels. It's like a sliding window, right?

Ars: OK, so everybody kind of huddles up and decides what the blessed six-year kernel will be.

Malchev: Yeah. The kernel moves fast, so it's a lot of effort to do even two-year LTSs, let alone six. The gap increases, and so cherry picking is no longer trivial after the third year or so. We need to be judicious with what we decide support for six-year LTS.

As Malchev said, Pie's requirement of Linux kernel 4.4-4.14 is for newly shipping devices only. Android devices don't update to new major kernel versions, though, and since Google wants older devices to be able to update to Android 9 Pie, this means Pie must support kernel versions much older than 4.4. Pie support actually goes back to 3.18, which was released in 2014. It's typical for the Android platform to have to support Linux kernels that are around six years old.

It would be nice to not actually have to support a rolling six-year history of the Linux kernel in every single Android release, but getting there would mean getting devices and device vendors to actually upgrade to major new versions of Android. When we talk about Project Treble modularizing Android away from the hardware, the Linux kernel lives on the "hardware" side of that split. So when it comes to major versions, the current implementation of Android is to drop Linux on a phone and forget about it forever. It does not have to be that way, though. 

Sandeep Patil, a senior engineer on the Android team, gave a talk last year at the Linux Plumbers Conference and detailed a possible future where the Linux kernel moved forward in time along with the Android Platform. Just as Project Treble has the "Generic System Image"—a raw AOSP build that must boot on all Project Treble devices—Patil detailed the beginnings of the concept of a "GKI"—the Generic Kernel Image—a kernel that could run on any Treble device. Patil said this kernel would be something Google releases, and someday that might even be an upstream, mainline Linux kernel with drivers and other specific hardware support pushed into SoC-specific modules.

Patil's GKI slide. This shows a potential future with Android and Linux on one side of the Project Treble split, (the colored background) and the Vendor HAL Interface and kernel modules on the other side. Today, Android lives on the updatable left side of this slide and the Penguin lives on the etched-in-stone right side.

Credit: Sandeep Patil

Patil mentioned that Treble laid the groundwork for a GKI project logistically, giving Google a blueprint for the interfaces that would need to be built, cross-ecosystem cooperation that would be needed, and the testing that would need to be instituted. It all sounded like a "Project Treble 2: Linux Boogaloo," but I'm not sure if that's overselling it. I was also not very sure of the progress of this. So...

Ars: What is this I hear about a Generic Kernel Image? Will you really start distributing a kernel with GSIs? Does anyone want to give a status update on this?

Malchev: Everything else being equal, we want to unify things in a way that makes sense, right? Subject to our requirements that we want to be open source, we want to let people customize. Be together, not the same, all this awesome stuff. We have been unifying kernel sources in source form for a while. The Generic Kernel Image is an idea that, "Hey, we have the sources there, why don't we build a kernel out of them that we can use to test devices with?" It's in the spirit of the Generic System Images. They don't exist yet, though. They're interesting conceptually, we just need to do a lot of work to figure out if it makes sense in the ecosystem.

Ars: Is the plan really to have a mainline Linux kernel work on a phone?

Malchev: I'll go out on a limb and say that we want that. The work is tremendously wide in space and difficult. It's an ecosystem wide problem. So, we take it one step at a time. We unify sources. But also like, forget a single kernel, let's get people to upgrade their kernels to the latest LTS. That's what we're focusing on now, right? We're making strides there.

Burke: By the way, this is obvious, but doing that just fixes so many security issues.

Ars: But, when you say "upgrade," you're still not upgrading a kernel on a phone, right?

Malchev: Right, implicit here is that, once a device ships, you won't jump from 4.14 to 5.1, right? You won't jump to next kernel versions, but since these are LTS kernels, we would like for them to take the LTS updates that we so painstakingly enable. So, these LTS trains, we want them to land on actual devices. Even that's a struggle, right? [That's] because of everything that makes updatability difficult, as in user space, for the kernel. So, we're focusing on that, actually, with the security team and our partners. Actually, we're making a lot of progress.

It turns out the minor LTS releases do get updated on some phones. The Pixel 3 XL started on 4.9.96 and today (on the Android Q Beta, at least) it runs 4.9.165. And with that, our time wrapped—thanks to Dave, Iliyan, and Anwar for taking the time to answer my questions. Let's do it again next year!