Community inertia in Debian and Ubuntu
At Debconf 9, Steve Langasek delivered a talk on pam-auth-update, which included a tongue-in-cheek slide about how it had come to be. He pointed out that this project had required about six years of “thinking” and two months of actual development, and that this seemed to be a pattern exhibited by a number of notable Debian projects in recent years, such as using dash as the default shell, or supporting package Recommends in APT.
This fostered a lively discussion with several members of the audience about how large changes are discussed, decided and implemented in Debian. Often, this process involves extensive mailing list threads, bikeshedding, hemming and hawing. Sam Hartman suggested that Debian developers could do better at facilitating consensus. Joey Hess pointed out that several of the example projects Steve mentioned were actually implemented first in Ubuntu, then brought into Debian.
This isn’t because Ubuntu was particularly innovative with regard to these projects. Many of them had been discussed and even designed in Debian before Ubuntu even existed. However, Ubuntu was able to take those crucial first steps before Debian was.
Why was this the case?
Debian is a good example of a large open source project with well established patterns of work. Most of the work in Debian is divided up according to packages, and a designated person or team is responsible for each package. This autonomy is a part of what makes Debian development particularly rewarding, as it fosters a sense of personal ownership. This arrangement also scales well: people working within well-defined areas are less likely to step on each other’s toes, and most decisions can be made quickly by a single person or a small group. There are close to 1000 individual developers actively contributing to Debian today.
However, it doesn’t work so well for projects and decisions which span these boundaries. The most problematic projects require buy-in from many individual developers to pursue a common course of action. This clashes with the autonomy which works so well for smaller scale projects.
So, why are things different in Ubuntu?
First, Ubuntu is smaller than Debian, in terms of the developer community. The development team is still small enough (less than 150 people) that consensus is more easily achieved.
Second, Ubuntu developers are loosely organized into teams around common interests, rather than by package. They don’t think of packages as “belonging” to a particular developer, and multiple developers routinely work on the same packages. Even where, in practice, only one person ever touches a particular package, that is seen as largely incidental: that person is simply doing more work on that bit of code than anyone else. It’s more similar to how things work in a shared source tree, where some developers know some parts of the code better than others, but they are fundamentally working together on the whole tree.
Areas of responsibility in Ubuntu tend to be more emergent (“it’s my responsibility because I’m involved”) than prescribed (“it’s my responsibility because I am the designated maintainer”). While there is much more team structure in Debian than there once was, the most important organizational structure in Debian is still the Maintainer field. If something is not clearly the responsibility of a particular maintainer, it can be unclear where to start, or who (if anyone) has the autonomy to decide how to proceed.
Finally, Ubuntu is a branch of Debian. Once a project grows beyond a certain point, it becomes difficult to make major changes on the spot. Major changes rarely come out perfect the first time, so they may need intensive regression testing before they can be considered stable enough even for development purposes. The changes also need to be coordinated with other developers, because they may interfere with their work.
Revision control offers a solution to this problem in the form of branches. Developers can make sweeping changes freely in a branch, work out the worst of the problems there, and then merge their branch into mainline. Rather than first persuading other developers to buy into an idea, someone can go off and implement a prototype, prove that it works, and then persuade others to adopt the working code (which is often much more compelling than an idea or even a design).
This is exactly what has happened with several major features which were developed in Ubuntu first.
It turns out that branching an entire distribution is much more complex than a single source tree (just ask the Launchpad team). Branching Debian is very heavyweight, and merging the changes back is a lot of work as well, as most of the work still must be done at a package level. Without branches, though, it is very difficult to implement large scale changes without hindering the progress of other developers.
So, what can we learn from all of this? I drew the following conclusions:
- Branches help alleviate blockage when implementing major changes
- Debian may benefit from more use of branches for major changes
- Ubuntu may be an appropriate branch for some types of changes
- Ubuntu is surely not sufficient, though, and Debian probably needs a more flexible way of branching the distribution