We'll see | Matt Zimmerman

At DebConf 9 this week, the Debian release team proposed a new approach to Debian’s release cycle, which was then announced on the Debian web site. Both the Debconf presentation and the announcement were quite clear, but a number of news articles and blog posts on the subject seem to have misinterpreted them:

• Debian Adopts Two-Year Time-Based Release Cycle on OSNews
• Debian to adopt time-based releases on Linux Today
• Debian to Adopt Time-based Release Cycle on Jonathan Carter’s blog
• Debian to adopt time-based releases on a blog masquerading as a news site

Debian is not switching to time-based releases. I’m glad they aren’t, because I don’t think it would be the right choice for the project at this time. Time-based releases are the approach used by Ubuntu, where we plan to release on a specific date. Instead, they will use the same approach as in previous releases, where they set criteria for release-critical bugs, and release when all release-critical bugs are closed.

The difference is that they will schedule the freeze date in advance. This means that there is a bounded time period available for new development, where things sometimes need to be broken in order to make progress. Once the freeze point is reached, Debian developers will minimize breakage and focus on stabilization. Once the RC bug count drops to zero, they’ll release as usual. That could happen soon after the freeze, or it could take a long time, depending on how many bugs are introduced during development.

This hybrid approach seems like a good balance, giving the release team the ability to guide the project toward a more predictable release cadence, without sacrificing their uncompromising approach to quality. Having a predictable freeze date will help Debian and Ubuntu developers to work together this winter to fix many of the same bugs in Ubuntu 10.04 and Debian squeeze.

On a recent trip abroad, I brought my laptop and planned to work online at the airport while waiting for my flight. I found a power socket, opened my laptop, and got connected with no fuss at all. I was even able to connect my G1 phone at the same time, since the WiFi was free, and sync my calendar and RSS feeds.

After 15 minutes or so, my laptop fell off the network. I couldn’t even see the access point anymore. I spent the next few minutes grumbling and running terminal commands to try to understand what was wrong, and decided that the problem must be with the access point. Then I switched on my phone and noticed that it connected without a problem!

Grumbling louder, I spent quite some time trying to figure out the problem. The crucial clue was provided by Android, which provides a configuration setting for the number of WiFi channels to use. It can be set to 11, 13 or 14. I tried setting the phone to 11 channels, to match what was reported by my laptop:

iwlagn 0000:03:00.0: Tunable channels: 11 802.11bg, 13 802.11a channels

…and then the phone couldn’t connect either! I couldn’t get Android to show me the channel information for the access point, but by trying the settings, I could infer that it was using channel 12 or 13, and must have switched channels since I started working.

So, how to get my laptop to see these extra channels? I immediately suspected the regulatory settings, since a new regulatory framework had recently been developed and added to Ubuntu. I dug through man pages, I read database dumps, and I searched Google using my phone. Everything I found told me that I was doing the right thing already, and that channels 12-14 (which were perfectly legal in the country where I was) should be available.

Then I came across a few references which told me that my WiFi card (Intel Corporation PRO/Wireless 4965 AG or AGN [Kedron] Network Connection (rev 61)) was believed to be hardware crippled to support only 11 channels. I had purchased this laptop in the US, but don’t live there anymore, and certainly didn’t want to concern myself with US regulations while outside of the US.

Tough luck for me, it seems. If I want to be able to get online in other countries reliably, it seems I’ll need to buy a new card.

At Debconf 9, Steve Langasek delivered a talk on pam-auth-update, which included a tongue-in-cheek slide about how it had come to be.  He pointed out that this project had required about six years of “thinking” and two months of actual development, and that this seemed to be a pattern exhibited by a number of notable Debian projects in recent years, such as using dash as the default shell, or supporting package Recommends in APT.

This fostered a lively discussion with several members of the audience about how large changes are discussed, decided and implemented in Debian.  Often, this process involves extensive mailing list threads, bikeshedding, hemming and hawing.  Sam Hartman suggested that Debian developers could do better at facilitating consensus. Joey Hess pointed out that several of the example projects Steve mentioned were actually implemented first in Ubuntu, then brought into Debian.

This isn’t because Ubuntu was particularly innovative with regard to these projects.  Many of them had been discussed and even designed in Debian before Ubuntu even existed. However, Ubuntu was able to take those crucial first steps before Debian was.

Why was this the case?

Debian is a good example of a large open source project with well established patterns of work. Most of the work in Debian is divided up according to packages, and a designated person or team is responsible for each package. This autonomy is a part of what makes Debian development particularly rewarding, as it fosters a sense of personal ownership. This arrangement also scales well: people working within well-defined areas are less likely to step on each other’s toes, and most decisions can be made quickly by a single person or a small group. There are close to 1000 individual developers actively contributing to Debian today.

However, it doesn’t work so well for projects and decisions which span these boundaries. The most problematic projects require buy-in from many individual developers to pursue a common course of action. This clashes with the autonomy which works so well for smaller scale projects.

So, why are things different in Ubuntu?

First, Ubuntu is smaller than Debian, in terms of the developer community. The development team is still small enough (less than 150 people) that consensus is more easily achieved.

Second, Ubuntu developers are loosely organized into teams around common interests, rather than by package. They don’t think of packages as “belonging” to a particular developer, and multiple developers routinely work on the same packages. Even where, in practice, only one person ever touches a particular package, that is seen as largely incidental: that person is simply doing more work on that bit of code than anyone else. It’s more similar to how things work in a shared source tree, where some developers know some parts of the code better than others, but they are fundamentally working together on the whole tree.

Areas of responsibility in Ubuntu tend to be more emergent (“it’s my responsibility because I’m involved”) than prescribed (“it’s my responsibility because I am the designated maintainer”). While there is much more team structure in Debian than there once was, the most important organizational structure in Debian is still the Maintainer field. If something is not clearly the responsibility of a particular maintainer, it can be unclear where to start, or who (if anyone) has the autonomy to decide how to proceed.

Finally, Ubuntu is a branch of Debian. Once a project grows beyond a certain point, it becomes difficult to make major changes on the spot. Major changes rarely come out perfect the first time, so they may need intensive regression testing before they can be considered stable enough even for development purposes. The changes also need to be coordinated with other developers, because they may interfere with their work.

Revision control offers a solution to this problem in the form of branches. Developers can make sweeping changes freely in a branch, work out the worst of the problems there, and then merge their branch into mainline. Rather than first persuading other developers to buy into an idea, someone can go off and implement a prototype, prove that it works, and then persuade others to adopt the working code (which is often much more compelling than an idea or even a design).

This is exactly what has happened with several major features which were developed in Ubuntu first.

It turns out that branching an entire distribution is much more complex than a single source tree (just ask the Launchpad team). Branching Debian is very heavyweight, and merging the changes back is a lot of work as well, as most of the work still must be done at a package level. Without branches, though, it is very difficult to implement large scale changes without hindering the progress of other developers.

So, what can we learn from all of this? I drew the following conclusions:

• Branches help alleviate blockage when implementing major changes
• Debian may benefit from more use of branches for major changes
• Ubuntu may be an appropriate branch for some types of changes
• Ubuntu is surely not sufficient, though, and Debian probably needs a more flexible way of branching the distribution

I just read a piece of email spam which has left me feeling hollow.

Trigger warning: this email contains references to sexual assault and/or violence against women which may be triggering to survivors.

An image of the message can be found here.

Like most people who take part in public discussion on the Internet, I get a lot of email spam. A great majority of it is filtered away without my having to even think about it. A few get through now and then, and these I can generally recognize by their subjects, so I don’t even look at the message body. I just save them to train my spam filter with.

Every once in a while, I’m unsure about whether a message is spam, or my fingers are just on auto-pilot, and I open up a piece of spam by mistake. Usually, this is just a minor nuisance, but in this case it was an emotional hazard.

In the news this week were two notable stories involving companies contributing to open source:

• Launchpad is now open source
• Microsoft Contributes Linux Drivers to Linux Community

Apart from this one essential element, these stories could hardly be more different. Canonical and Microsoft are dramatically different companies, making distinctly different types of contributions, for very different reasons. There’s not much to compare. Or is there?

Novell’s Chief Marketing Officer John Dragoon has taken the opportunity to compare Canonical with Novell partner Microsoft on his blog. As he “commend[s] Microsoft for taking this very significant step”, he points out that the 20,000 lines of source code contributed by Microsoft to the Linux kernel “will far surpass those contributed by Canoncial[sic]“.

John credits Novell colleague Greg Kroah-Hartman for helping Microsoft to achieve this historic milestone. Greg is fond of counting lines of code in the Linux kernel, and based on his commentary elsewhere, I’m sure it was his pleasure to provide this statistic. I haven’t checked the figures myself, but it’s certainly believable that our contributions to the Linux kernel haven’t amounted to 20,000 lines of code.

Before we congratulate Microsoft and Novell too heartily, though, let’s get beyond the numbers, and look at what those 20,000 lines of code actually do. What can Linux do now that it couldn’t do before Microsoft’s contribution? According to Microsoft’s press release, it’s a device driver which enables Linux to run much faster—on Windows servers. That’s right, it helps us to get more value out of our expensive Windows Server 2008 licenses by consolidating our Linux servers into Windows Hyper-V virtual machines. It lets us put Windows in control of our hardware, and rely on Microsoft to allow it to perform well, for as long as that makes sense for them strategically.

Launchpad, on the other hand, is a hosting platform designed to accelerate free software development. Any free software developer in the world can use it, for free, to manage their source code, bug reports, packages, translations, and more. Canonical was founded to advance the state of the art in free software, and Launchpad and Ubuntu both represent massive investments in that effort. Now that Launchpad is open source, developers can contribute to it, and those contributions benefit other open source developers by making the Launchpad service better.

Microsoft’s contribution to Linux creates new business opportunities for Microsoft by locking customers into their technology. Canonical’s contribution of Launchpad helps free software developers do what they do best, and benefits Canonical by making it easier for us to package, distribute, maintain, and provide services for free software.

Oh, and if we want to compare numbers, Launchpad is well over 200,000 lines of Python code, or more than 10 times larger than Microsoft’s contribution to Linux (in C).

This blog is syndicated to Planet Ubuntu, and from time to time, one of my posts ends up getting duplicated there. This seems to happen whenever I touch an old post (for example, to add it to a new category I’ve created, or to enable or disable comments as in the most recent case).

If anyone knows how to fix WordPress (or Planet Ubuntu) to avoid doing this, please let me know in a comment. Meanwhile, apologies for the noise.

Apologies are something I still have trouble with from time to time.  I’m going to be prescriptive in this article, though, because I think I’ve learned a thing or two about the right and wrong ways to do it, thanks to some of the people I’ve hurt who have cared enough to tell me how they felt.

I was taught from a young age that the words “I’m sorry” were the cure for hurt feelings, whether they resulted from harsh words, physical pain or thoughtlessness.  While I learned the ritual of it from this training, I had to learn separately what it really meant, and develop the emotional aptitude to put it into practice.

I wonder whether this ritualization of apology makes it more difficult for us to learn to do it properly.  We seem to confuse the language of apology with the feeling of regret, or the act of reconciliation.  Apologies are only words, unless they successfully convey regret for our actions, and a desire to do better in the future.

For example, consider the phenomenon of the non-apology apology.  This happens when someone feels the need to fulfill the ritual of apology with words, but doesn’t seem to regret their actions or wish to truly reconcile.  They may recognize that the other party is hurt, but not accept responsibility for it.  They may conditionalize the apology so as to avoid even acknowledging the hurt.  They may even blame the injured party for their own hurt, while still participating in the ritual and satisfying their trained impulses.

“I’m sorry if…”

This type of non-apology fails to acknowledge the consequences of our actions.  Maybe we hurt someone, or maybe we didn’t.  Who is to say?  In the hypothetical scenario wherein we hurt someone, we suppose we would be sorry.  Who’s to say?  When it’s clear that someone has been hurt, and that our actions caused them to be hurt, a conditional apology is tantamount to a denial.

• I’m sorry if I hurt your feelings
• I’m sorry if I crossed a line
• I’m sorry if I was an asshat

“I’m sorry, but…”

These “apologies” are really excuses in disguise.  They justify our actions, and thereby absolve us of responsibility.  They fail to recognize that, regardless of what our intentions were, we hurt someone.  It’s sometimes appropriate to offer an explanation, but that’s a separate matter from the apology, and the two don’t mix.  Attaching an explanation to an apology dilutes the apology to the point of meaninglessness.

• I’m sorry, but I didn’t mean it (that way)
• I’m sorry, but I was only joking
• I’m sorry, but I’m just speaking my mind
• I’m sorry, but you must have misunderstood me
• I’m sorry, but I don’t know what I did wrong

“I’m sorry [that] you…”

These insidious non-apologies deflect responsibility away from ourselves and onto the injured party.  They imply that they hurt themselves, and that our actions were incidental.  They often focus on the symptoms of distress, rather than the cause.

• I’m sorry you feel that way
• I’m sorry you are upset
• I’m sorry you were offended

A proper apology

• Validates the hurt
• Takes responsibility for having caused it, regardless of intent
• Expresses regret for one’s actions

It will also convey a desire to avoid repeating the same mistake in the future.  I think this is usually implicit, though, and doesn’t need to be stated unless it requires clarification.  If you do something wrong and clearly regret it, it will be inferred that you’ll avoid doing it in the future.

Apologies are easiest when the mistake is a simple one: a trodden foot, a late appearance, a forgotten courtesy.  The injury and the cause are both obvious in these cases.  We usually get into trouble where there is a more complex emotional conflict involved.  In such situations, we’re not always sure what happened or why, and may need to explore that before we can even begin to apologize.  This may require cooperation with the injured party, which can be especially difficult in a conflict situation.

Nevertheless, even in an emotional conflict, it can help to draw parallels with a simple apology.  It’s been years since I last made an HTML table, but here goes:

Of course, this is still “just” language, and needs to be supported by authenticity of expression (i.e. you need to believe it for the magic words to work).  An apology may be only the last step of a long process of understanding and reconciliation.

Still, watching our words helps us to notice when we’ve got it wrong, which may be half the battle.

I rebooted my laptop today, to effect the latest updates to Karmic. Well, actually, that was a side effect.  I had to reboot because my Intel i965 hung (again).  I’ve written a watchdog script which detects this condition and attempts to gather debug information, but it hung trying to read some files in /proc/dri.

After the reboot, new problems so far include:

• My keymap is set to USA at the gdm greeter (rather than USA/dvorak as was correctly done before): bug 401047 (hey, we’ve passed bug 400000 recently)
• I found myself unable to launch any X applications from a terminal, because I was unconditionally setting the XAUTHORITY environment variable in my ~/.zshenv, and apparently gdm is now putting it in /var/run/gdm (this one was my fault)
• I’ve ended up with two volume control applets with different behavior. One is gnome-volume-control-applet, and the other seems like it might be built into gnome-panel
• indicator-applet crashed: bug 401051
• Reporting the above bugs required several attempts, because Firefox 3.5 crashed intermittently trying to open a new tab:
  Gdk-ERROR **: The program 'firefox-3.5' received an X Window System error: bug 401055
This probably reflects a bug in the program.
The error was 'BadWindow (invalid Window parameter)'.

Things which have improved:

• My monitor’s graphics modes are now detected correctly
• 2.6.31 seems to be stable for me now

Problems which are still present:

• I get a dialog box upon logging into GNOME which informs me that the GDM greeter won’t save its position: bug 395324
• My console keymap is not set to dvorak, though it’s still configured that way in /etc/default/console-setup and running setupcon fixes it (and changes the font): bug 392218

We have a problem in the way that women in free software are regarded and treated. If this is news to you, I encourage you not to take my word for it, but read what women in the community are saying about it. Ask women you know about their experiences.

What I want to discuss here, though, is how people are received when they speak up about this, for example by criticizing sexist behavior they have observed. Often, the problem is denied, the critic themselves is personally attacked, and the victims are blamed. In short, there is a backlash.

Lefty’s account of Richard Stallman’s keynote at the Gran Canaria Desktop Summit offers an excellent case study of this pattern.

In the talk, Stallman incorporated a “joke” which Lefty described thus:

The nadir for me was Richard's explanation of "EMACS virgins" as "women who had not been introduced to EMACS" along with the advice that "relieving them of their virginity" was some sort of sacred duty for members of "The Church of EMACS".

The very first comment (anonymous) denied that Lefty had actually heard what he said he had:

Why do you automatically think of girls and woman when somebody talks about virgins? Richard did not mention the sex of the virgins.

However, many members of the audience (including myself) recall it quite clearly, and I expect a video will be published at some point to provide an objective account of what was said.

The second comment (also anonymous) denied that there is a general problem with how women are regarded in free software, citing female disinterest as the true cause:

This "include women" is all but nonsense. Women is[sic] uninterested - as a group, as you can find exceptions *everywhere* - in programming as men are uninterested at another areas where woman reigns.

Lefty received some support from other commenters, but the backlash went further downhill from there.

One anonymous commenter suggested that women should be actively prevented from participating in open source:

Blah blah blah. This kind of whiny bullshit about unimportant details is exactly why women should be left out in the cold.

Another mocked Lefty for daring to speak out:

Thank goodness a big hairy chested man like you was there to protect any helpless women that were about. [...] In other words grow some balls you silly pseudo feminist

For speaking up, he’s been accused of being a disingenuous “pseudo feminist”. Why? I didn’t see any reason in Lefty’s writings to question the validity of his objection, and the anonymous commenter presumably doesn’t know him personally. Plenty of other people were much more vocally offended by Stallman’s comments on Mono, for example, but they haven’t been accused in this way. What is different about women’s issues which trigger this response?

Another anonymous commenter shares his view that women (as a class) do not deserve to be heard, in this classic finger-pointing exercise:

Respect is a two-way street. I'm tired of hearing the so-called feminists yelling out to sexist claims towards women. You know what? I'll take their complaints seriously when most women stop their openly stereotypical view towards men.

From there, the anonymous comments descend into sexual metaphors:

Lefty, go back to sucking deicaza's cock YOU MUPPET

This led to more overt, but blessedly brief, misogynist hatespeech…

bitch, STFU

…and calls for Lefty to leave the community for having the audacity to criticize Stallman…

People like you should be ashamed of such behavior and words about other people. Moreover, people like you must quit doing whatever at Open Source.

While these are (almost exclusively) anonymous comments, and there were many others which were more reasonable, this is a pattern of backlash. In this case, it was Lefty taking offense at a single comment in a presentation at a conference which spawned this chain of vitriol. There have been entire presentations which were much worse, and whose critics were also met with rejection and insults. Lefty’s experience is not atypical.

Backlash itself isn’t the root of the problem, of course. It’s just another symptom. What I’m hoping to do is to recognize it for what it is, and question the feelings and beliefs behind it. I want to know how others feel about it. Do you see this happening as well? How does it make you feel?

Technical conferences are a great way for me to get myself thinking about technology in different ways. I spend my working days immersed in it, but primarily from the “inside”, thinking about what we’re doing in Ubuntu and what’s happening in free software today.

At conferences, on the other hand, I get the chance to think freely about what’s possible, and what might be coming. I find this very difficult to do at work, but a free software conference, with plenty of lively chats with colleagues and other participants, always seems to do the trick. I return home with a notebook full of ideas, including plenty of topics which aren’t directly relevant to the conference topics.

Here’s some of what I took home from GCDS 2009. The conference is still going on as I write this, but I’ve returned home after spending four days there.

The keynotes

• r0ml is one of my favorite speakers. His sense of humor, timing, gesture, and general exuberance really bring his material to life. This particular talk got me thinking about free software communities in terms of Aristotelian epistemology. I think my relatively sparse classical education makes me a sucker for this kind of thing.
• Walter Bender spoke about Sugar. Despite having had an XO laptop sitting on my desk for a period of months, this talk taught me most of what I know about the principles and possibilities of the Sugar platform. It really is pretty cool. My only worry is that it’s so ambitious, it might be eclipsed by less powerful, but more pragmatic systems before it reaches its potential.
• Then, of course, there was Richard Stallman. This keynote was the least interesting of the talks I attended at GCDS, yet I feel compelled to write the most about it:
  • Much of the talk was devoted to his standard general-audience material, for example, explaining the four freedoms (to an audience of free software enthusiast developers)
  • He also took the time to explain the history of KDE and GNOME (to an audience of KDE and GNOME developers)
  • He then warned of the dangers of Mono, alienating the application developers in the crowd who happened to prefer it.
  • He did his Saint IGNUcius routine, throwing in a sexist joke for good measure.

The talks

The summit was organized into two parts: a cross-desktop segment, and a GUADEC/aKademy segment. This gave GNOME and KDE developers the chance to attend some common talks, but also have talks separately. I’m not sure how well this worked out for everyone. Superficially, it seemed like we would have benefited from more cross-pollination (GNOME folks attending KDE-oriented talks and vice versa) in addition to the cross-desktop content (which seemed to focus on common code).

• In a cross-desktop audio talk, Lennart Poettering gave an overview of the audio stack and some of the problems that each of its layers have. I felt like this overview was a useful starting point, but there wasn’t much opportunity to discuss solutions and how we could work together, so I’m not sure what everyone took away from it.
• The GeoClue talk got me thinking, particularly about location awareness in more sophisticated desktop applications (i.e. not just mobile phone applications):
  • We could use GeoClue in the Ubuntu installer to more intelligently guess the user’s language, time zone and keyboard layout based on their location, or at least narrow the range of options a bit. Even country-level information would be accurate enough to be useful.
  • Because some devices have more accurate location data than others, it would be interesting to share this data between them. For example, your mobile phone could tell your laptop where it is.
  • With so much location sharing going on, I started to wonder about the apparent implicit assumption in many of these systems that the user and the device are in the same location. Programs like Google’s Latitude present this assumption very strongly, with a picture of the person placed on a map wherever their location-aware device is. This will start to get confusing as people accumulate more location-aware devices.
  • Telepathy apparently supports passing location data, which opens up all sorts of interesting possibilities. It should be possible to write very straightforward applications which do disproportionately cool things using this combination of technology.
  • It should be possible to automagically select a download mirror for Ubuntu installation media or packages using GeoClue, without writing a lot of code.
• A talk on videoconferencing centered on Farsight 2, which seems to be coming along nicely. It provides, through the familiar and flexible GStreamer API, access to a host of audio and video conferencing systems. Take your VNC screen sharing session, connect it up to a GStreamer pipeline, and have it come out the other end as an MSN webcam stream with accompanying audio. NAT traversal and all the other gory details are taken care of for you. Very cool.
• The GNOME Shell represents a small revolution in the design of the GNOME user interface. The talk demonstrated a number of its features and general princples. I found myself wanting to understand more about why those particular features and principles were chosen, but I guess that didn’t fit into the talk slot.
• The Zeitgeist talk was inspirational. It gave me all sorts of ideas about how I would organize my system differently if I had more information about how I was using it. By collecting data on what the user is doing and when, Zeitgeist opens up the possibility of more adaptive user interface techniques, as well as simple and useful tools like a journal of activity.
• Rob Bradford presented Mojito, a gateway for desktop applications to access web services data from social networking sites. It takes care of managing authentication tokens, and presenting the data from different services in similar and predictable ways. I’m definitely into the idea of socializing the desktop, though I’m not yet clear on where Mojito fits in relative to Gwibber, which provides a very functional API for several of the same web services.
• Rick Spencer, my colleague and the leader of the Ubuntu desktop team, gave a talk about the different varieties of programming, focusing on the “opportunistic” type. In this state of mind, a programmer isn’t thinking systematically, or developing for long-term goals, but just experimenting with something new or trying to solve an immediate and short-term problem. They have very different needs and expectations than someone who is approaching programming more systematically. This nicely encapsulates the subtle difference in behavior and perspective between someone who is developing the system they’re using (e.g. GNOME or Ubuntu developers) and someone who is developing on that system. The latter type of developer doesn’t care how the system itself works, and just wants to solve their problem. I don’t think they are very well served by free software yet, but Rick has created a tool called Quickly which aims to help opportunistic programmers get what they need…quickly.

The people

I was able to match a number of new faces to names, and make some new connections as well. The conference format offered ample opportunities for this, which I appreciated. Even if someone lives in my home town, I think I’m more likely to run into them at a conference on another continent, because the global community is much more cohesive.

There was a lot of Twitter and identi.ca activity during the event, which helped me to make some new connections there, as well as to involve people who couldn’t attend the event.

It was great to be able to spend some time with colleagues from Canonical in small groups and under more relaxed circumstances. Too often, I only see them all at once, and when we all have too much to do. This alone would have made it worth going.