Offline software updates in Endless OS

Motivation

A lot of the software most of us use day-to-day is built with the assumption that Internet access is fast, cheap, unlimited*, and ubiquitous.

For a lot of Endless’ current and target users, most of those assumptions are not true. Internet connectivity is often capped to low throughput and monthly quotas and may be relatively expensive or unreliable. Or it may be inaccessible entirely due to cost or lack of infrastructure.

When these assumptions fail, a lot of modern software fails.

This includes software updates.

A solution

Endless OS is built on top of OSTree for OS management and Flatpak for app management. OS updates and apps are presented to the user in Gnome Software.

Offline app and OS updates have been some of our longest-running development projects and it’s been quite a challenge. There are a handful of complex components involved and there’s a lot of plumbing and API work that needed to be designed, discussed, and tested in real-world scenarios. And because there was some but not total overlap between USB app updates, USB OS updates, LAN app updates, and LAN OS updates (see below), we had to do a lot of work to not paint ourselves into a corner.

If we first focused narrowly on LAN app updates, we might have deemed limited Internet connectivity a fair assumption. But this can’t be assumed for USB updates where a computer may literally never reach the Internet in its lifetime. And, in fact, we had a bug in development where an app install from a USB disk would fail if the computer were offline. Flatpak assumed it required Internet access to download a very small amount of app data. (We’ve fixed this).

So, we’ve worked in multiple directions at once and now the first of those is complete: USB app installs.

USB app installs

We’ve worked closely with upstream to solve the various problems that crop up when you break assumptions of “always-on” Internet. We sign app updates so you can get a USB disk full of apps and know they came from us. Or Flathub. Or an independent app vendor (which you may, for good reason, not automatically trust).

The majority of our work will be invisible to users but we also implemented the UI to copy apps to an inserted USB disk and to install from a USB disk. Some of that, as Matthias described, includes command line tools. But we also implemented UI in Gnome Software to be accessible to everyone. We also automatically open Gnome Software to a new “USB” category when you insert the disk so it’s clear they can be installed right now. No Internet required.

Gnome Software showing the new “Copy to USB” button which appears when a USB disk is inserted.
When an app can be installed locally, Gnome Software’s “Download” button instead says “Install”.

Caveats

When we copy apps to a USB disk, we also copy all their dependencies. If you copied an app to a friend’s computer, it wouldn’t be very useful if it were missing its content or translations they required. And if they didn’t have the app’s runtime, it wouldn’t run at all.

Say computer A installs app Z, copies it to USB, and goes offline. New versions of the OS get released which include a newer version of app Z’s runtime. Computer B gets this newer version of the OS clean-installed and is offline. If you plug the USB disk into computer B, it will fail to install app Z because it thinks it’s already got app Z’s runtime installed but app Z actually requires the older version of the runtime. The one that’s conveniently sharing the USB disk with it. We’ll be fixing this soon.

If that seems complicated, realize that it’s just one of the many (and not the hardest) corner cases my colleagues and upstream maintainers have solved in developing this feature.

Credits

I’m really proud of all the work that’s been done by my colleagues at Endless and upstream in Gnome to make these features a reality. I want to make it clear that I’ve only done a small part of the work and want to make sure I give credit where it’s due.

Matthew Leeds and Philip Withnall did most of the hard work in Flatpak and OSTree to figure out appropriate plumbing and API. Joaquim Rocha implemented Gnome Software changes to detect and display apps on USB disks. Alex Larsson, Matthias Clasen, and Colin Walters did a lot of work helping us integrate our OSTree and Flatpak changes upstream. Richard Hughes helped with questions about Gnome Software. I created the UI in Gnome Software to copy apps to USB disks.

What’s next

Most of our changes to OSTree and Flatpak are already upstream. Most of our changes to Gnome Software are downstream-only but we definitely want to upstream as much of the UI and Flatpak-related changes as possible. I’ll be analyzing the work required there and working with the maintainers and Gnome UX designers to make sure we’ve got designs suitable for everyone.

Apps can be copied to and from USB disks but we still need to finalize the work for OS updates as well. We’re fairly close but it needs UI in Gnome Software. (And I’m sure we’ll find some exciting bugs with wider testing).

LAN updates

USB updates are essential for users without Internet connections but they’re also useful for users with “partial” Internet connections which are limited by quotas/throughput/reliability. Another update solution for people with “partial” Internet access is what we’re calling “LAN updates” and which we usually describe in terms of a classroom. In this scenario, a “teacher” computer has Internet access to receive OS and app updates normally. But it also serves those updates to other “student” computers on its LAN. These “student” computers are intentionally blocked from the Internet to avoid them contributing toward a monthly data cap, using some of the limited bandwidth, etc.

Imagine a 200-MiB app contains a number of books (as PDFs). A newer version of the app adds a book that adds 1 MiB to the total size. For a classroom of 30 computers and a naive app distribution system that does a full download per update, 201 MiB/computer * 30 computers = 5.9 GiB. This single update would blow past the monthly data caps for some of our customers (which can go as low as 350 MiB).

Flatpak is really fine-grained with its updates so this would drop down to a 30 MiB. That’s far more efficient but it’s already a non-trivial portion of a 350 MiB cap and it’s just for a single app update. All these numbers need to be scaled up when you factor in many app updates and an OS update or two.

With LAN updates, this app update would drop down to 1 MiB for the entire classroom. And Endless OS already detects data-metered Internet connections and turns off automatic updates to further reduce data usage in extra-constrained environments. (This setting can be overridden in the system settings at any time).

LAN updates are also nearly ready for deployment in Endless OS.

Error

default userpic

Your reply will be screened

Your IP address will be recorded 

When you submit the form an invisible reCAPTCHA check will be performed.
You must follow the Privacy Policy and Google Terms of use.