We'll see | Matt Zimmerman

James Westby: Ubuntu Distributed Development

James gave a great overview of the Ubuntu distributed development project, which has the ambitious goal of providing a homogeneous view of all of the source code for Ubuntu packages using Launchpad and Bazaar. This includes modeling the relationships between the versions of the code in Debian and further upstream, which is complicated by the use of different revision control systems, patch systems, and so on.

At this stage, most of the source code for Ubuntu and Debian is available through the system, and developers can freely branch from it and request merges. It works the same way for all packages, so developers only need to learn one toolset and workflow. We hope that this will lower the barrier to entry for contributing to Ubuntu, as well as make it easier to share patches between Ubuntu, Debian and upstream.

Timo Hoenig: Extending the scope of mobile devices

Timo reviewed how mobile devices have evolved over the past 40 years, citing dramatic improvements in compute power, memory, bandwidth and so on, but comparatively small improvement in battery life (several orders of magnitude less). Thus, he sees power management and related technologies as important to the further advancement of the category. He specifically identified network links as a key consideration, as they consume a great deal of power, and have continued to do so with newer generations of technology. Local power management, he says, is not sufficient, and we need to take a network-aware view.

He introduced the concept of an “early bird” connector, which acts as a supervisor for a mobile device. It communicates with remote network nodes on its behalf, and takes decisions about when and whether to wake up the mobile device. He estimated a 12% power savings by offloading processing to such a device, using a simple model of power consumption. The early bird would run on another system on the network, presumably without the same power constraints (like a proxy server).

Sam Vilain: Linux Containers

Sam detailed the LXC implementation of containers for Linux. In contrast with vserver, it seems to offer a much simpler interface. Because of this, it has been comparatively straightforward to merge into the Linux kernel mainline.

LXC uses existing Linux kernel facilities to group processes within containers into control groups, which can then be used to control access and scheduling of resources (network, CPU, storage, etc.). Each resource type has a namespace similar in principle to what chroot() provides for filesystems. Since all of the hardware is visible to a single kernel, there can be a great deal of flexibility in how resources are allocated. For example, a given network device and CPU can be dedicated to a container.

Usefully for system administration and diagnostics, all of these resources can be directly accessed from the host without stopping or shutting down guests.

After lunch, I had a three-way conflict, as I wanted to attend talks on SVG as an alternative to Flash (Libre Graphics Day), gnucash and domain-specific package management.

Sometimes, when I have a conflict like this, I try to attend the talk whose material is less familiar to me (in this case, probably the SVG/Flash one). However, since the talks are being recorded and made available on the Internet, this changes the dynamic a bit. I don’t have to miss out on watching anything, as I can download it later. So, it makes more sense for me to go where I can best participate, taking advantage of my presence at the conference.

Distro summit: Package management

So, I chose to attend the package management talk, as I might have something to contribute. It was about how to harmonize general distribution packaging mechanisms (dpkg, RPM, etc.) with special-purpose ones like those used by Ruby (gems), Lua (rocks), Perl (CPAN modules) and so on. The solution described employed a set of wrapper scripts to provide a standard API to these systems, so that they could be used by the distribution package manager to resolve dependencies.

Due up next was Scott James Remnant’s talk on booting faster, but due to travel difficulties, he hadn’t arrived yet. Instead, we had a free-form discussion on various other challenges in the area of package management.

I took the opportunity to put forward an idea I had been thinking about for some time, which is that we may need to move beyond a “one size fits all” or “everything is a package” approach to package management. Modern package management systems are very capable, and solve a range of difficult problems with a high degree of reliability. The cost of all of this functionality is complexity: making packages is difficult. The system can be made to work for a wide range of software, but the simple case is often not very simple.

I also made the point that there are non-technical challenges as well: it is difficult for developers and ISVs to understand the ecosystem of distributions, and even more difficult to succeed in making their software available in “the right way” to Linux users. The obstacles range from procedural to cultural, and if we only approach this as a technical problem, we risk adding more complexity and making the situation worse.

The opportunity to have this kind of participatory discussion really validated my decision about how to choose which talk to attend.

Liz Henry: Code of our Own

Back in the Haecksen/LinuxChix miniconf, Liz Henry presented an action plan
for increasing the involvement of women in open source, with many well-considered “dos” and “don’ts” based on observations of what has and has not worked for open source communities.

It was the first opportunity I’ve had to attend a free software conference session which went beyond the typical “yes, this is important” and “yes, there really is a problem here” content which is unfortunately as necessary as it is commonplace.

I won’t attempt to summarize it here, but I can definitely recommend Liz’ presentation to anyone who is looking for concrete, actionable methods to promote gender diversity in their technical communities.

Lucas Nussbaum: The Relationship between Debian and Ubuntu

Historically, in Lucas’ assessment, many Debian developers have been unhappy with Ubuntu, feeling that it had “stolen” from Debian, and was not “giving back”. He said that bad experiences with certain people associated with Canonical and Ubuntu reflected on the project as a whole.

However, he says, things have improved considerably, and today, most Debian developers see some good points in Ubuntu: it brings new users to free software and Debian technology, it provides a system which “just works” for their (less technical) friends and family, and brings new developers to the Debian community.

There are still some obstacles, though. Lucas says that many bugfix patches in Ubuntu are just workarounds, and so are not very helpful to Debian. He gave the example of a patch which disabled the test suite for a package because of a failure, rather than fixing the failure.

He felt that Canonical offered few “free gifts” to Debian, citing as the only example the website banner on ubuntu.com which was displayed for Debian’s 15th anniversary. I felt this was a bit unfair, as Canonical has done more than this over the years, including sponsoring DebConf every year since Canonical was founded.

It occurred to me that the distinctions between Canonical and Ubuntu are still not clear, even within the core free software community. For example, the “main” package repository for Ubuntu is often seen to be associated exclusively with Canonical, while “universe” is the opposite. In reality, Canonical works on whatever is most important to the company, and volunteers do the same. These interests often overlap, particularly in “main” (where the most essential and popular components are).

Lucas speculated that more mission-critical servers run Debian pre-releases (especially testing) than Ubuntu pre-releases. It would be interesting to test this, as it’s rare to get sufficient real-world testing for server software prior to an Ubuntu release.

Lucas presented a wishlist for Ubuntu:

• more technical discussions between Ubuntu and Debian (particularly on the ubuntu-develdebian-devel mailing list
• easier access to Launchpad data
• Launchpad PPAs for Debian

The prominence of Launchpad in these discussions spawned a number of tangential discussions about Launchpad, which were eventually deferred to tomorrow’s Launchpad mini-conf. One audience member asked whether Debian would ever adopt Launchpad. The answer from Lucas and Martin Krafft was that it would definitely not adopt the Canonical instance, but that a separate, interoperating instance might eventually be palatable.

I made the point that there is no single Debian/Ubuntu “relationship”, but a large number of distinct relationships between individuals and smaller groups. Instead of focusing on large-scale questions like infrastructure, I think there would be more mileage in working to build relationships between people around their common work.

After maintaining an ad hoc Linux distribution for myself for several years, I replaced it with Debian and have never looked back. One of the main reasons for this has been the mind-boggling quantity of applications and tools which are available from its repositories. Given couple of keywords, or a good guess at the name of the application, APT fetches and installs the necessary packages in a matter of seconds. After years of compiling free software programs from source code, this profoundly changed the way I thought about finding and obtaining software.

Over 10 years later, the speed and convenience of this system still occasionally leaves me awestruck. As a typical example, on one occasion, I was using a pastebin to share the output of a program I was discussing with someone online by copying and pasting it.. The output was fairly long, and it was inconvenient to copy and paste, so I wanted a tool which would read the output from a pipe and upload it directly to the pastebin, without a human in the middle.

Before I fired up an editor to write such a tool, I did a quick search to check if any such thing existed already, and found Stéphane Graber’s pastebinit, which did exactly what I wanted (and more). Not only had someone else had the idea first, they had implemented, released and packaged it over a year earlier. The end result, for me, was that within 30 seconds of discovering that I needed such a tool, I had it installed and working.

Experiences like the above still make me feel like I’m living a scene from the 1989 film Bill & Ted’s Excellent Adventure, where the protagonists discover that they have already traveled back in time to anticipate their own needs. They merely think about what they need, and there it is. The fact that I am still amazed by this probably makes me sound like a dinosaur to other free software enthusiasts, but this kind of instant gratification is something which is only just beginning to emerge in proprietary systems like the iPhone. The web was designed from the start to work this way, of course, but there is much I can do with free software that I can’t do with web applications (at least for now). The web also doesn’t give me that feeling of personal connection with the creator of the software, or (generally) the opportunity to tailor it for my needs.

The development of Ubuntu 10.04 has been underway for nearly two months now, and will produce our third long-term (LTS) release in April. Rick Spencer, desktop engineering manager, summarized what’s ahead for the desktop team, and a similar update will be coming soon from Jos Boumans, our new engineering manager for the server team.

What I want to talk about, though, is not the individual projects we’re working on. I want to explain how the whole thing comes together, and what’s happening behind the scenes to make 10.04 LTS different from other Ubuntu releases.

Changing the focus

Robbie Williamson, engineering manager for the foundations team, has captured the big picture in the LTS release plan, the key elements of which are:

Merge from Debian testing

By merging from Debian testing, rather than the usual unstable, we aim to avoid regressions early in the release cycle which tend to block development work. So far, Lucid has been surprisingly usable in its first weeks, compared to previous Ubuntu releases.

Add fewer features

By starting fewer development projects, and opting for more testing projects over feature projects, we will free more time and energy for stabilization. This approach will help us to discover regressions earlier, and to fix them earlier as well. This doesn’t mean that Ubuntu 10.04 won’t have bugs (with hundreds of millions of lines of source code, there is no such thing as a bug-free system), but we believe it will help us to produce a system which is suitable for longer-term use by more risk-averse users.

Avoid major infrastructure changes

We will bring in less bleeding-edge code from upstream than usual, preferring to stay with more mature components. Where a major transition is brewing upstream, we will probably opt to defer it to the next Ubuntu cycle. While this might delay some new functionality slightly, we believe the additional stability is well worth it for an LTS release.

Extend beta testing

With less breakage early in the cycle, we plan to enter beta early. Traditionally, the beta period is when we receive the most user feedback, so we want to make the most of it. We’ll deliver a usable, beta-quality system substantially earlier than in 9.10, and our more adventurous users will be able to upgrade at that point with a reasonable expectation of stability.

Freeze with Debian

With Debian “squeeze” expected to freeze in March, Ubuntu and Debian will be stabilizing on similar timelines. This means that Debian and Ubuntu developers will be attacking the same bugs at the same time, creating more opportunities to join forces.

Staying on course

In addition, we’re rolling out some new tools and techniques to track our development work, which were pioneered by the desktop team in Ubuntu 9.10. We believe this will help us to stay on course, and make adjustments earlier when needed.  Taking some pages from the Agile software development playbook, we’ll be planning in smaller increments and tracking our progress using burn-down charts.  As always, we aim to make Ubuntu development as transparent as possible, so all of this information is posted publicly so that everyone can see how we’re doing.

Delivering for users

By making these changes, we aim to deliver for our users the right balance of stability and features that they expect from an Ubuntu LTS release. In particular, we want folks to feel confident deploying Ubuntu 10.04 in settings where it will be actively maintained for a period of years.

At DebConf 9 this week, the Debian release team proposed a new approach to Debian’s release cycle, which was then announced on the Debian web site. Both the Debconf presentation and the announcement were quite clear, but a number of news articles and blog posts on the subject seem to have misinterpreted them:

• Debian Adopts Two-Year Time-Based Release Cycle on OSNews
• Debian to adopt time-based releases on Linux Today
• Debian to Adopt Time-based Release Cycle on Jonathan Carter’s blog
• Debian to adopt time-based releases on a blog masquerading as a news site

Debian is not switching to time-based releases. I’m glad they aren’t, because I don’t think it would be the right choice for the project at this time. Time-based releases are the approach used by Ubuntu, where we plan to release on a specific date. Instead, they will use the same approach as in previous releases, where they set criteria for release-critical bugs, and release when all release-critical bugs are closed.

The difference is that they will schedule the freeze date in advance. This means that there is a bounded time period available for new development, where things sometimes need to be broken in order to make progress. Once the freeze point is reached, Debian developers will minimize breakage and focus on stabilization. Once the RC bug count drops to zero, they’ll release as usual. That could happen soon after the freeze, or it could take a long time, depending on how many bugs are introduced during development.

This hybrid approach seems like a good balance, giving the release team the ability to guide the project toward a more predictable release cadence, without sacrificing their uncompromising approach to quality. Having a predictable freeze date will help Debian and Ubuntu developers to work together this winter to fix many of the same bugs in Ubuntu 10.04 and Debian squeeze.

From time to time, I see someone remark that Ubuntu uses packages from Debian unstable, and that they don’t think this is a very good idea. I would like to explain why we do this and how it works, and hope that this will enable a less one-sided view of the subject.

Debian is well known and regarded as a product, the Debian system itself. In addition to this end product, the Debian community produces a variety of useful by-products. One of the most celebrated is Debian “unstable”, which despite its unglamorous name is an incredible achievement. It represents the collective work of a vast number of developers, who package and maintain essentially all of the free software available.

Put simply, Debian unstable provides immediate access to anything you could want from the free software world. It’s all there, updated daily with the latest and greatest releases. As if that weren’t great enough, the components are continuously integrated, so that problems become evident right away and can be fixed quickly.

Debian doesn’t “release” unstable directly; it’s an intermediate stage in their process. The Debian release team selects components which they believe are ready to release, and place them in “testing”, which is the next stage in the process. However, many people, developers especially, use unstable as their primary operating environment. They drink from the firehose of free software, delivered daily by Debian.

Ubuntu was created in a similar spirit, but with significantly different goals. We wanted to release a complete product much more frequently and regularly. We wanted to focus on desktop users, and provide a system which met their specific needs, even if this meant serving some other needs less well. In this way, we created a complement to Debian, as many others had done before us.

What we did differently was to choose unstable as the basis for this work. There are many other distributions based on Debian, but at the time, they were mostly derived from a particular release. Debian would make a release, then someone else would take this and produce a customized version of it. The problem was that these customized versions would become outdated as Debian continued to develop. Bringing them up to date with Debian’s latest work required a lot of effort, particularly as Debian’s releases were sometimes years apart with massive changes between them.

Basing Ubuntu on Debian unstable meant keeping up. We would continually merge in the latest code from Debian, keeping Ubuntu closely in sync. While this was sure to be a lot of work, it provided us with access to all of the pre-packaged software we would need to produce Ubuntu, freshly delivered every day. Instead of duplicating this work, we could build on it.

We would, of course, need to do our own release management, and make decisions about what was ready to ship, just as Debian’s release team did. We couldn’t reuse their work, because we needed to make different decisions in order to suit our distinct release cycle and provide the desktop experience we wanted.

These decisions reflected some of the key advantages of free software. We could get the benefits of a custom solution without having to reinvent every wheel. We could focus on the pieces which mattered most to us, and use stock components everywhere else. We showed that this applied to open processes as well as open source code.

This is why, and how, Ubuntu was based on Debian unstable. It was, quite simply, the best tool for the job, and still is today.

Free software is a remarkable phenomenon.  It is a highly evolved form of collaboration: compared to other creative endeavors, free software developers all over the world are able to work together on a project with surprisingly little friction.  It is a grassroots political movement which has grown from small online communities to span geographical and national boundaries.  It is a multicultural social group with unique and diverse characteristics.  It has spawned a variety of self-governing organizations and successful corporations.

It is also an interesting example of a gift economy.  In general, participants in free software donate their work to the greater good, with no expectation of an exchange of value.  Some contributions are rewarded by social or professional recognition, where the author achieves standing among their peers or receives gratitude from recipients of their work.  Others indirectly evoke rewards in kind, such as where the creator of a program is rewarded by contributions from others which improve it further.  Some are works for hire, where a corporation commissions a contribution through its employees, in pursuit of its own aims.  There are other types of exchanges where I do not personally understand what motivates the contributor.

There are many recognized roles in free software, but they can be broadly classified into three types:

Developers are the heart of this economy.  They are continuously creating and improving free software technology, and publishing their source code for other developers to use and learn from.  Some developers write one program and then vanish from the community, while others contribute to many different projects over the course of decades.  Highly effective developers are celebrities in the free software community.

Users, also known as “people”, are the reason why software is written.  Their needs and wants determine which software is considered valuable.  Many of them also contribute directly to free software in one way or another, by promoting awareness, providing testing and feedback, supporting other users, writing documentation, or building communication links.  Historically, most users of free software were also its developers, but today, this is no longer true, and millions of people use free software who are not developers.

Packagers connect these two groups.  They gather up the source code produced by developers, wrap it in standardized packaging, and bundle it into collections (distributions) designed to meet the needs of users.  Users experience free software almost exclusively through a distribution.  As the free software stack has grown in size and complexity, so have distributions, and the maintenance of a modern distribution is a large-scale development project in itself: selecting appropriate software and versions, getting the lot working smoothly together, and releasing it in the form of a product which is accessible to users.  Packagers create only a small fraction of the software included in their distribution, but they define several key aspects of “what it is like” to use it.  Users most strongly associate their experience of free software with a distribution.

If free software were film, developers would be some combination of writer and crew, creating and expressing characters and a story.  Users would be viewers (including critics and fans), who receive and interpret the work.  Packagers would be the film crew, realizing the production on film so that it can be seen.

If free software were food, developers would be chefs, developing recipes and cooking.  Users would eat and critique the dishes.  Packagers would be restaurateurs, serving customers and creating an environment in which they can experience the food.

Neither of these analogies are very complete.  In particular, these analogies fail to capture the strange loops of free software.  Every developer is also a user, reliant upon of thousands of other programs which they receive in packaged form order to do their development work.  Every team of packagers develops some software in order to make their distribution work, and they use the distribution itself in order to do so.

Distributions, and the integration work that they do, are a critical part of this ecosystem.  Many of us would not be using free software today if not for the efforts of projects like Debian, whose mission is to produce a complete system out of free software created by others.  In my case, Debian provided both the means and the motivation for my contributions to free software, and later made Ubuntu possible.

Strong, productive relationships between these groups are essential to continuing the growth and development of free software.  Whichever groups you’re part of, get to know your counterparts in other groups.  Talk to the people who are packaging your software, writing the software you package, using your software or packages.  Learn about the problems they face and how you can help each other.  Don’t assume that this communication is someone else’s job: reach out and make it happen.