Posts Tagged ‘Debian’
DEX finishes first batch of derivative patches for Debian
It’s been a few months since Zack and I announced the DEX project, which aims to improve the Debian ecosystem by working jointly with derivative distributions.
Our first milestone
The goal of our first project, nicknamed ancient-patches, was to clear out an old batch of a few hundred Ubuntu patches whose status was unclear. We couldn’t tell which ones had been merged into Debian, which were waiting in the BTS, and which had yet to be submitted to Debian. All of them were several years old.
I’m pleased to announce that this project is now complete. Thanks to help from David Paleino, Colin Watson, Nathan Handler and Steve Langasek, we were able to clear over 95% of the patches in a matter of days. These were the easy ones: patches which were obsolete, or had already been applied. We discussed the remainder, and resolved all of the patches whose status was still unclear. This left the harder ones: patches stalled in the BTS, and patches where there was no consensus about what to do with them.
One of the stalled patches was merged into Debian via an NMU, eliminating the delta between Debian and Ubuntu. Another had been submitted to Debian by a third party, but was no longer shipping in Ubuntu, so we considered it obsolete for purposes of this project.
This has left only two patches out of the original list of 277. Both of them are filed in the BTS and have been discussed with the relevant maintainer team. One of them is expected to be obsoleted when a new upstream version is packaged, which implements similar functionality. The other is being discussed with the upstream developers, but there is no conclusion yet about whether it can be merged upstream or in Debian.
Conclusions
Although we weren’t quite able to clear the whole list, we still consider the project to be a success because:
- We ensured that all of the patches received due consideration for inclusion in Debian
- We proved the concept of DEX, with developers from Debian and derivatives cooperating on a common goal and sharing tools
- Most importantly, we learned from the experience
What’s next
In the most recent DEX update on debian-derivatives, I highlighted a few important events for DEX:
- Our second major project, nicknamed “big-merges”, will begin soon. Our goal is to identify the few packages which are most diverged between Debian and Ubuntu, and work to get them as close to identical as possible. If you have suggestions for packages to focus on, let us know!
- Allison Randal is beginning a DEX project to implement the Python 2.7 transition across Debian and Ubuntu
- Nathan Handler is working on a Summer of Code project to develop specialized tools to help with this kind of cross-distribution teamwork
- Zack is organizing a derivatives BoF at DebConf 11
We’re looking forward to seeing DEX develop further. If you’d like to get involved, come and join us on the debian-derivatives mailing list or IRC (#debian-derivatives on freenodeOFTC).
Matt Zimmerman and Stefano Zacchiroli
DEX: Debian and its derivatives, getting things done together
Since I resumed active status in Debian, I’ve been thinking about how to bridge the gap between Debian and its derivatives*. I’ve spoken at length with Zack, the attendees of the Derivatives BoF at DebConf 10, and the fine folks at the Derivatives Front Desk about the technical and social issues affecting derivative projects, and could probably write a very thorough series of blog posts on the subject.
Instead, Zack and I decided to try doing something about it: we have begun a project to test out a new approach to the problem.
Introducing DEX
DEX is all about action: merging patches, fixing bugs, crunching data, whatever is necessary to get changes from derivatives into Debian proper. DEX doesn’t try to change the way any existing project works, but adds a “fast path” for getting code from one place to another.
DEX is a joint task force where developers from Debian and its derivatives work together on this common goal. As a pilot project, we’ve established an Ubuntu DEX Team focused on merging code from Ubuntu into Debian. With members from both projects, we hope to be able to resolve blockage anywhere in the pipeline. Whatever needs to get done in order to merge an Ubuntu patch, someone in the Ubuntu DEX team will know what to do. If we get good results with Ubuntu, we hope that other derivatives will follow. With thanks to David Paleino, we’re excited that the Utnubu project is merging into DEX as it aligns well with their goals. I’m very grateful to have Colin Watson and James Westby signed up to contribute as well.
Our first project is simple: turn this list green. This is an archive of quite old patches from Ubuntu, most of which have probably been merged already or made obsolete, but they pre-date any kind of tracking system so they need to be verified. Once that’s done, we’ll move on to a new project with a new todo list.
If you want to see Debian benefit from technical work done in derivatives, DEX is a chance for you to act together to make it happen. If you work on a derivative and want to carry a smaller delta, come and join us. I’m sure we’ll learn a lot from this experience.
* There are many instances of great cooperation between Debian and derivative distributions, including joint package maintenance teams, and some derivatives are even part of the Debian project. Nonetheless, there are areas were most people I’ve spoken to agree that we need to do better. This is what I’ve referred to as the “gap”.
Rejoining Debian
A couple of months ago, Debian project membership voted, after extensive discussion, to implement a fundamental change in the Debian community: to welcome as members people who make a valuable contribution to the project, even if they are contributing something other than source code.
This was a tremendous milestone for Debian, and one which made me feel proud to have been a part of the project. Historically, only developers had been eligible for membership, including voting and other formal privileges. Although other kinds of contributions were welcome, this disparity gave the impression that they were less valued than code contributions. It seemed to me at the time that Debian’s mission was to package all of the free software in the world, and if one’s efforts didn’t go directly to improving packages, they just weren’t as important.
I don’t remember when I first installed Debian, but I made my first contributions to the project in 1999, and officially joined as a developer in 2000. After several fun and rewarding years of packaging and development, I started a very demanding day job, and spent more and more of my energy into that, and less and less coding for Debian as a volunteer. However, my job with Canonical involved working with Debian, and that was a primary reason why it was interesting to me. It was an opportunity to introduce a whole new population of people to the things I loved about Debian.
The reality, of course, was more complicated. Following the launch in 2004, Ubuntu grew quickly in popularity and scope, diverged from Debian in significant ways, and relations between Debian and Ubuntu became strained. Canonical grew quickly as well, and the combination of a growing community, a growing company and growing user adoption was a challenge for everyone concerned. As a Canonical manager and a Debian developer, I felt the strain as much as anyone.
Meanwhile, and I felt more and more alienated from Debian. Debian developers who had been friendly in the past became suspicious of Ubuntu—and me—and I quickly became an outsider. My code contributions to Debian continued to decline, and I was no longer maintaining any packages. In Debian at the time, that meant that I didn’t exist. I saw it as an important part of my job to work with my counterparts in Debian, in a coordinating role, but found this increasingly impractical. In 2007, I received an inquiry from the Debian Account Manager, who had noticed I wasn’t actively involved in packaging, and wished to disable my account for security reasons if I wasn’t using it. Although I wanted to remain active in the Debian community, I had to agree that it wasn’t good security practice for me to hold onto my developer privileges. I relinquished my upload rights, with the option to come back if I resumed my development work, and officially became a nobody: I lost the right to vote, my email address and mailing list subscriptions, and all other official ties with Debian, except for the record of my GPG key in a special “emeritus” keyring for informational purposes.
Last month, Enrico Zini announced instructions for contributors to apply for membership under the new guidelines, which recognize many kinds of contributions, not only code. Today, after a three year hiatus, I am proud to be the first Debian member to be accepted through this new process. I expect to continue to submit the occasional patch, but my primary interest is in healing the rift which still exists between Debian and Ubuntu by contributing in a more personal way. Please feel free to contact me if you’d like to work together on this. You can reach me as mdz at either debian.org or ubuntu.com, or on IRC.
I would like to thank Stefano Zacchiroli, for proposing the General Resolution which enabled Debian to make this transition, and for all of his other work as Debian Project Leader to help Debian grow and improve. I also appreciate Enrico Zini, Jonathan McDowell and Martin Zobel-Helas for expediently processing me and working through the technical changes needed to implement the resolution correctly.
It’s good to be back.
Ubuntu: Project, Platform, Products
When most people talk about Ubuntu, they usually mean our flagship product, Ubuntu Desktop Edition. Sometimes, they might mean the Ubuntu project, or the community of people who work on it, or various other things.
Similarly, Debian might mean the Debian operating system, or the package repositories, or the project, and so on.
This gets a little confusing sometimes. When I’m talking about Ubuntu, I’ve started to use more specific terminology to explain what I mean, and this seems to help people understand better the nature of the whole Ubuntu. In particular, I use the three Ps:
- a portfolio of products, including Desktop Edition, Server Edition, Netbook Edition, Kubuntu and more. These are software bundles which can be downloaded, pre-installed on retail computers, and so on. Each one is designed to meet a certain set of user needs, and to work on a specific form factor of computer.
- a technology platform, which can be used to build a wide range of products. It is primarily of interest to developers, who build derivative distributions, OS products, applications and infrastructure using Ubuntu packages. This platform is the common foundation of the Ubuntu products above, and includes things like the global package repository. Joel Spolsky does a good job of explaining why platforms are distinctly different from products, and should be treated as such.
- an open community project, which collectively produces, distributes, promotes and supports the products and the platform. The Ubuntu project has a philosophy, a government, and various tools and processes to help contributors work together. Canonical supports the Ubuntu project by providing resources and infrastructure, and also directly participates in Ubuntu at various levels.
This breakdown may seem a bit obvious to those of us “on the inside”, but it’s confusing to people who are encountering it for the first time. I’m sharing this in the hope that if more people start using the same words, it will get easier for people to understand how these pieces fit together. I’ll also be linking to it a lot, to help put things into context using this framework.
Back to the future
In my professional role as Ubuntu CTO, I take on a number of different perspectives, which sometimes compete for my attention, including:
- Inward – supporting the people in my department, alignment with other departments in Canonical and reporting upward
- Outward – connecting with customers, partners and the free software community, including Debian
- Forward – considering the future of the Ubuntu platform and products, based on the needs of their users, our customers and business stakeholders within Canonical
- Outside-in – taking off my Canonical hat and putting on an Ubuntu hat, and looking at what we’re doing from an outside perspective
My recent work, as Canonical has gone through a period of organizational growth and change, has prioritized the inward perspective. I took on a six-month project which was inwardly focused, temporarily handing off many of my day-to-day responsibilities (well done, Robbie!). I’ve grappled with an assortment of growing pains as many new people joined Canonical over the past year.
With that work behind me, it’s time to rebalance myself and focus more outside of Canonical again. It’s good to be back!
In my outward facing capacity, I’ll shortly be attending Web 2.0 Summit in San Francisco. I attend several free software conferences each year, but this is a different crowd. I hope to renew some old ties, form some new ones, and generally derive inspiration from the people and organizations represented there. Being in the San Francisco Bay area will also give me an opportunity to meet with some of Canonical’s partners there, as well as friends and acquaintances from the free software community. With my head down, working hard to make things happen, it’s easy to lose perspective on how that work fits into the outside world. Spending more time with people outside of Canonical and Ubuntu is an important way of balancing that effect.
Looking forward, I’ll be thinking about the longer term direction for the Ubuntu platform. The platform is the layer of Ubuntu which makes everything else possible: it’s how we weave together products like Desktop Edition and Server Edition, and it’s what developers target when they write applications. Behind the user interfaces and applications, there is a rich platform of tools and services which link it all together. It’s in this aspect of Ubuntu that I’ll be investing my time in research, experimentation and imagination. This includes considering how we package and distribute software, how we adapt to technological shifts, and highlighting opportunities to cooperate with other open source projects.
My primary outside-in role is as chair of the Ubuntu Technical Board. In this capacity, I’m accountable to the Ubuntu project, the interests of its members, and the people who use the software we provide. Originally, the TB was closely involved with a range of front-line technical decisions in Ubuntu, but today, there are strong, autonomous teams in place for the most active parts of the project, so we only get involved when there is a problem, or if a technical question comes up which doesn’t “fit” the charter of an established team. It’s something of a catch-all. I’d like to re-establish the TB in a more central role in Ubuntu, looking after concerns which affect the project as a whole, such as transparency and development processes. I’m also re-joining Debian as a non-uploading contributor, to work on stimulating and coordinating cooperation between Debian and Ubuntu. I’m looking forward to working more with Zack on joint projects in this area.
This change will help me to support Canonical and Ubuntu more effectively as they continue to grow and change. I look forward to exercising some mental muscles I haven’t used very much lately, and facing some new challenges as well.
DebConf 10: Last day and retrospective
DebConf continued until Saturday, but Friday the 6th was my last day as I left New York that evening. I’m a bit late in getting this summary written up.
Making Debian Rule, Again (Margarita Manterola)
Marga took a bold look at the challenges facing Debian today. She says that Debian is perceived to be less innovative, out of date, difficult to use, and shrinking as a community. She called out Ubuntu as the “elephant in the room”, which is “‘taking away’ from Debian.” She insists that she is not opposed to Ubuntu, but that nonetheless Ubuntu is to some extent displacing Debian as a focal point for newcomers (both users and contributors).
Marga points out that Debian’s work is still meaningful, because many users still prefer Debian, and it is perceived to be of higher quality, as well as being the essential basis for derivatives like Ubuntu.
She conducted a survey (about 40 respondents) to ask what Debian’s problems are, and grouped them into categories like “motivation” and “communication” (tied for the #1 spot), “visibility” (#3, meaning public awareness and perception of Debian) and so on. She went on to make some suggestions about how to address these problems.
On the topic of communication, she proposed changing Debian culture by:
- Spreading positive messages, celebrating success
- Thanking contributors for their work
- Avoiding escalation by staying away from email and IRC when angry
- Treating every contributor with respect, “no matter how wrong they are”
This stimulated a lot of discussion, and most of the remaining time was taken up by comments from the audience. The video has been published, and offers a lot of insight into how Debian developers perceive each other and the project. She also made suggestions for the problems of visibility and motivation. These are crucial issues for Debian devotees to be considering, and I applaud Marga for her fortitude in drawing attention to them. This session was one of the highlights of this DebConf, and catalyzed a lot of discussion of vital issues in Debian.
Following her talk, there was a further discussion in the hallway which included many of the people who commented during the session, mostly about how to deal with problematic behavior in Debian. Although I agreed with much of what was said, I found it a bit painful to watch, because (ironically) this discussion displayed several of the characteristic “people problems” that Debian seems to have:
- Many people had opinions, and although they agreed on many things, agreement was rarely expressed openly. Sometimes it helps a lot to simply say “I agree with you” and leave it at that. Lending support, rather than adding a new voice, helps to build consensus.
- People waited for their turn to talk rather than listening to the person speaking, so the discussion didn’t build momentum toward a conclusion.
- The conversation got louder and more dense over time, making it difficult to enter. It wasn’t argumentative; it was simply loud and fast-paced. This drowned out people who weren’t as vocal or willful.
- Even where agreement was apparent, there was often no clear action agreed. No one had responsibility for changing the situation.
These same patterns are easily observed on Debian mailing lists for the past 10+ years. I exhibited them myself when I was active on these lists. This kind of cultural norm, once established, is difficult to intentionally change. It requires a fairly radical approach, which will inevitably mean coping with loss. In the case of a community, this can mean losing volunteer contributors cannot let go of this norm, and that is an emotionally difficult experience. However, it is nonetheless necessary to move forward, and I think that Debian as a community is capable of moving beyond it.
Juxtaposition
Given my history with both Debian and Ubuntu, I couldn’t help but take a comparative view of some of this. These problems are not new to Debian, and indeed they inspired many of the key decisions we made when founding the Ubuntu project in 2004. We particularly wanted to foster a culture which was supportive, encouraging and welcoming to potential contributors, something Debian has struggled with. Ubuntu has been, quite deliberately, an experiment in finding solutions to problems such as these. We’ve learned a lot from this experiment, and I’ve always hoped that this would help to find solutions for Debian as well.
Unfortunately, I don’t think Debian has benefited from these Ubuntu experiments as much as we might have hoped. A common example of this is the Ubuntu Code of Conduct. The idea of a project code of conduct predates Ubuntu, of course, but we did help to popularize it within the free software community, and this is now a common (and successful) practice used by many free software projects. The idea of behavioral standards for Debian has been raised in various forms for years now, but never seems to get traction. Hearing people talk about it at DebConf, it sometimes seemed almost as if the idea was dismissed out of hand because it was too closely associated with Ubuntu.
I learned from Marga’s talk that Enrico Zini drafted a set of Debian Community Guidelines over four years ago in 2006. It is perhaps a bit longand structured, but is basically excellent. Enrico has done a great job of compiling best practices for participating in an open community project. However, his document seems to be purely informational, without any official standing in the Debian project, and Debian community leaders have hesitated to make it something more.
Perhaps Ubuntu leaders (myself included) could have done more to nurture these ideas in Debian. At least in my experience, though, I found that my affiliation with Ubuntu almost immediately labeled me an “outsider” in Debian, even when I was still active as a developer, and this made it very difficult to make such proposals. Perhaps this is because Debian is proud of its independence, and does not want to be unduly influenced by external forces. Perhaps the initial “growing pains” of the Debian/Ubuntu relationship got in the way. Nonetheless, I think that Debian could be stronger by learning from Ubuntu, just as Ubuntu has learned so much from Debian.
Closing thoughts
I enjoyed this DebConf very much. This was the first DebConf to be hosted in the US, and there were many familiar faces that I hadn’t seen in some time. Columbia University offered an excellent location, and the presentation content was thought-provoking. There seemed to be a positive attitude toward Ubuntu, which was very good to see. Although there is always more work to do, it feels like we’re making progress in improving cooperation between Debian and Ubuntu.
I was a bit sad to leave, but was fortunate enough to meet up with Debian folk during my subsequent stay in the Boston area as well. It felt good to reconnect with this circle of friends again, and I hope to see you again soon.
Looking forward to next year’s DebConf in Bosnia…
DebConf 10: Day 3
How We Can Be the Silver Lining of the Cloud (Eben Moglen)
Eben’s talk was on the same topic as his Internet Society talk in February, which I had downloaded and watched some time ago. He challenges the free software community to develop the software to power the “freedom box”, a small, efficient and inexpensive personal server.
Such a system would put users more in control of their online lives, give them better protection for it under the law, and provide a platform for many new federated services.
It sounds like a very interesting project, which I’d like to write more about.
Statistical Machine Learning Analysis of Debian Mailing Lists (Hanna Wallach)
Hanna is bringing together her interests in machine learning and free software by using machine learning techniques to analyze of publicly available data from free software communities. In doing so, she hopes to develop tools for studying the patterns of collaboration, innovation and other behavior in these communities.
Her methodology uses statistical topic models, which infer the topic of a document based on the occurrence of topical words, to group Debian mailing list posts by topic. Her example analyzed posts from the debian-project and debian-women mailing lists, inferring a set of topics and categorizing all of the posts according to which topic(s) were represented in them.
Using this data, she could plot over time the frequency of discussion of each topic, which revealed interesting patterns. The audience quickly zoned in on practical applications for things like flamewar and troll detection.
Debian Derivatives BoF (Matt Zimmerman)
I organized this discussion session to share perspectives on Debian derivatives, in particular how we can improve cooperation between derivatives and Debian itself. The room was a bit hard to find, so attendance was relatively small, but this turned out to be a plus. With a smaller group, we were able to get acquainted with each other, and everyone participated.
Unsurprisingly, there were many more representatives from Ubuntu than other derivatives, and I was concerned that Ubuntu would dominate the discussion. It did, but I tried to draw out perspectives from other derivatives where possible.
On the whole, the tone was positive and constructive. This may be due in part to people self-selecting for the BoF, but I think there is a lot of genuine goodwill between Debian and Ubuntu.
Stefano Zacchiroli took notes in Gobby during the session, which I expect he will post somewhere public when he has a chance.
DebConf 10: Day 2
Today was the first day of DebConf proper, where all of the sessions were aimed at project participants.
Bits from the DPL (Stefano Zacchiroli)
Stefano delivered an excellent address to the Debian project. As Project Leader, he offered a perspective on how far Debian has come, raised some of the key questions facing Debian today, and challenged the project to move forward and improve in several important ways.
He asked the audience: Is Debian better than other distributions? Is Debian still relevant? Why/how?
Having asked this question on identi.ca and Twitter recently, he presented a summary. There was a fairly standard list of technical concerns, but also:
- A focus on quality, as defined by Debian’s highly modular approach. Each package maintainer is an expert on the software they package, and Debian as a whole offers a superior repository of packages.
- The principles of software freedom, as embodied in Debian’s Social Contract. The Debian community’s current interpretation is a purist one, and Stefano cited the elimination of non-free firmware as a milestone in the upcoming Squeeze release. I wonder, though, how many of the audience, tapping away on WiFi-connected laptops, were able to do so without such firmware.
- The project’s independent status, supported by donations and volunteers, which empowers it to make its own decisions, free of external impositions.
- Debian’s ability to make decisions, as embodied in the constitution. This happens mostly through do-ocracy (individuals are empowered to decide questions concerning their own work), though larger scope issues are decided democratically. This one evoked a bit of a chuckle, as decision making in Debian is not always perceived as fully effective.
He pointed out some areas which we would like to see improve, including:
- Developers accepting shared responsibility for the release as a whole. Making one’s own packages ready for release is necessary, but not sufficient. He cited evidence that the culture around NMUs is changing: historically, due to the do-ocratic system mentioned above, Debian developers have been somewhat territorial about their packages, and non-maintainer uploads were seen as stepping on their toes. However, recent experiments have indicated that this may no longer be the case, and Stefano encouraged more developers to help each other through NMUs.
- When making decisions, we should seek consensus, not unanimity. In a project with thousands of contributors, whose operations are open to the public, there will never be unanimous support for a proposal, and seeking unanimity leads to stalled decisions.
- In order to gain more contributors, Debian needs to welcome new and inexperienced contributors, as well as users (who can grow into contributors. He suggested reaching out to derivatives to find more of both. He decried the conventional wisdom that a “thick skin” should be a prerequisite for joining the project, pointing out that this attitude simply leads to fewer contributors. This point was met with applause by the DebConf audience.
All in all, I thought this was an accurate, timely and inspirational message for the project, and the talk is worth watching for any current or prospective contributor to Debian.
Debian Policy BoF (Russ Albery)
Russ facilitated a discussion about the Debian policy document itself and the process for managing it. He has recently put in a lot of time working on the backlog (down from 160+ to 120), but this is not sustainable for him, and help is needed.
There was a wide-ranging discussion of possible improvements including:
- Editing the policy manual so that it is more readable start to finish as a document, rather than a reference
- Creating a closer linkage between lintian and the policy manual, so that best practices from lintian get documented, and policy changes are accompanied by new checks
- Separating the normative and informative parts of the policy manual
There was also some discussion in passing of the long-standing confusion (presumably among people new to the project) with regard to how policy is established. In Debian, best practices are first implemented in packages, then documented in policy (not the reverse). Sometimes, improvements are suggested at the policy level, when they need to start elsewhere. I’m not very familiar with how the policy manual is maintained at present, but listening to the discussion, it sounded like it might help to extend the process to include the implementation stage. This would allow standards improvements to be tracked all the way through from concept, to implementation, to documentation.
The Java Packaging Nightmare (Torsten Werner)
Torsten described the current state of Java packaging in Debian and the general problems involved, including licensing issues, build system challenges (e.g. maven) and dependency management. His slides were information-dense, so I didn’t take a lot of notes.
His presentation inspired a lively discussion about why upstream developers of Java applications and libraries often do not engage with Debian. Suggested reasons included:
- They are not interested in Linux as a target platform
- Although their code is released under a free license, they are not interested in meeting Debian standards for freedom and license correctness
- They use Java because it is cross-platform, and so do not want to concern themselves with platform-specific issues
- Because Java applications are easy to download and run manually, they perceive relatively little value in the Debian packaging system
Collaboration between Ubuntu and Debian (Jorge Castro)
Jorge talked about the connections between Debian and Ubuntu, how people in the projects perceive each other, and how to foster good relationships between developers.
He talked about past efforts to quantify collaboration between the projects, but the focus is now on building personal relationships. There were many good questions and comments afterward, and I’m looking forward to the Debian derivatives BoF session tomorrow to get into more detail.
Tonight is the traditional wine and cheese party. When this tradition started, I was one of just a handful of people in a room with some cheese and paper plates, but it’s now a large social gathering with contributions of cheese and wine from around the world. I’m looking forward to it.
DebConf 10: Day 1
This week, I am attending DebConf 10 at Columbia University in New York.
The first day of DebConf is known as Debian Day. While most of DebConf is for the benefit of people involved in Debian itself, Debian Day is aimed at a wider audience, and invites the public to learn about, and interact with, the Debian project.
These are the talks I attended.
Debian Day Opening Plenary (Gabriella Coleman, Hans-Christoph Steiner)
Hans-Christoph discussed Debian and free software from a big picture perspective: why software freedom matters, challenging the producer/consumer dichotomy, how the Debian ecosystem hangs together, and so on.
Steps to adopting F/OSS in government (Andy Oram)
Andy discussed FLOSS adoption in governments, drawing on examples from Peru, the city of Munich, the state of Massachusetts. He covered the reasons why this is valuable, the relationship between government transparency and software freedom, and practical advice for successful adoption and deployment.”
Pedagogical Freedom (panel, Jonah Bossewitch et al)
The panelists discussed the use of technology in education, especially free software, some of the parallels between free software and education, and what these communities could learn from each other. This is a promising topic, though the perspectives seemed to be mostly from the education realm. There is much to be learned on both sides.
Google Summer of Code 2010 at Debian (Obey Arthur Liu)
This talk covered the student projects for this year’s Summer of Code. Most of the students were in attendance, and presented their own work. They ranged from more specialized projects like the Hurd installer, to core infrastructure improvements like multi-arch in APT.
Beyond Sharing: Open Source Design (Mushon Zer-Aviv)
Mushon gave an excellent talk on open design. This is a subject I’ve thought quite a bit about, and Asheesh validated many of my conclusions from a different angle. I’ve added a new post to my todo list to go into more detail on this subject.
Some points from his talk which resonated with me:
- When collaborating on code, everyone must reason with one collaborator: the computer. This forces a level playing field and a common encoding.
- Collaborating on other types of creative work is more difficult in part because of the differences encoding/decoding information between different individuals
- Making this easier for design work requires improving motivational factors and language as well as tools and processes
- Many design decisions are actually rational, and are compatible with a group consensus project. Too often, I hear that design can’t be done collaboratively, citing “too many cooks in the kitchen” analogies, but I have never believed it.
- Mushon’s own project, shiftspace.org, seems to be a browser-plugin-based system for collaboratively remixing web applications. I haven’t looked at it yet.
- Leadership and openness are not mutually exclusive. This is another pet peeve of mine, and there are so many examples of open leadership in the free software community that I don’t see how anyone can think otherwise.
- Mushon’s presentation is available in revision control so that it can be freely used and improved
How Government can Foster Freedom in Technology (Hon. Gale Brewer)
Councillor Brewer paid a visit to DebConf to tell us about the work she is doing on the city council to promote better government through technology.
Brewer seems to be a strong advocate of open data, saying essentially that all government data should be public. She summarized a bill to mandate that New York City government data be public, shared in raw form using open standards, and kept up to date. It sounded like a very strong move which would encourage third party innovation around the data.
She also discussed the need for greater access to computers and Internet connectivity, particularly in educational settings, and a desire to to have all public hearings and meetings shared online.
Why is GNU/Linux Like a Player Piano? (Jon Anderson Hall, Esq.)
Jon is a very engaging speaker. He drew parallels between the development of player pianos, reproducing pianos, reed organs, pipe organs…and free software. He even tied in Hedi Lamarr’s work which led to spread spectrum wireless technology. To be quite honest, I did not find that these analogies taught me much about either free software or player pianos, but nonetheless, I couldn’t help but take an interest in what he was saying and how he presented it.
DebConf Opening Plenary (Gabriella Coleman)
Biella and company explained all the ins and outs of the event: where to go, what to do (and not do), and most importantly, whom to thank for all of it. Now in its 11th year, DebConf is an impressively well-run conference.
I’m looking forward to the rest of the week!
We’ve packaged all of the free software…what now?
Today, virtually all of the free software available can be found in packaged form in distributions like Debian and Ubuntu. Users of these distributions have access to a library of thousands of applications, ranging from trivial to highly sophisticated software systems. Developers can find a vast array of programming languages, tools and libraries for constructing new applications.
This is possible because we have a mature system for turning free software components into standardized modules (packages). Some software is more difficult to package and maintain, and I’m occasionally surprised to find something very useful which isn’t packaged yet, but in general, the software I want is packaged and ready before I realize I need it. Even the “long tail” of niche software is generally packaged very effectively.
Thanks to coherent standards, sophisticated management tools, and the principles of software freedom, these packages can be mixed and matched to create complete software stacks for a wide range of devices, from netbooks to supercomputing clusters. These stacks are tightly integrated, and can be tested, released, maintained and upgraded as a unit. The Debian system is unparalleled for this purpose, which is why Ubuntu is based on it. The vision, for a free software operating system which is highly modular and customizable, has been achieved.
Rough edges
This is a momentous achievement, and the Debian packaging system fulfills its intended purpose very well. However, there are a number of areas where it introduces friction, because the package model doesn’t quite fit some new problems. Most of these are becoming more common over time as technology evolves and changes shape.
- Embedded systems need to be pared down to the essentials to minimize storage, distribution, computation and maintenance costs. Standardized packaging introduces excessive code, data and interdependency which make the system larger than necessary. Tight integration makes it difficult to bootstrap the system from scratch for custom hardware. Projects like Embedded Debian aim to adapt the Debian system to be more suitable for use in these environments, to varying degrees of success. Meanwhile, smart phones will soon become the most common type of computer globally.
- Data, in contrast to software, has simple requirements. It just needs to be up to date and accessible to programs. Packaging and distributing it through the standardized packaging process is awkward, doesn’t offer tangible benefits, and introduces overhead. There have been extensive debates in Debian about how to handle large data sets. Meanwhile, this problem is becoming increasingly important as data science catalyzes a new wave of applications.
- Client/server and other types of distributed applications are notoriously tricky to package. The packaging system works within the context of a single OS instance, and so relationships which span multiple OS instances (e.g. a server application which depends on a database running on another server) are not straightforward. Meanwhile, the web has become a first-class application development platform, and this kind of interdependency is extremely common on both clients and servers.
- Cross-platform applications such as Firefox, Chromium and OpenOffice.org have long struggled with packaging. In order to be portable, they tend to bundle the components they depend on, such as libraries. Packagers strive for normalization, and want these applications to use the packaged versions of these libraries instead. Application developers build, test and ship one set of dependencies, but their users receive a different stack when they use the packaged version of the application. Developers on both sides are in constant tension as they expect their configuration to be the canonical one, and want it to be tightly integrated. Cross-platform application developers want to provide their own, application-specific cross-platform update mechanism, while distributions want to use the same mechanism for all their components.
- Virtual appliances aim to combine application and operating system into a portable bundle. While a modular OS is definitely called for, appliances face some of the same problems as embedded systems as they need to be minimized. Furthermore, the appliance becomes a component in itself, and requires metadata, distribution mechanisms and so on. If someone wants to “install” a virtual appliance, how should that work? Packaging them up as .debs doesn’t make much sense for the same reasons that apply to large data sets. I haven’t seen virtual appliances really taking off, but I expect cloud to change that.
- Runtime libraries for languages such as Perl, Python and Ruby provide their own packaging systems, which manage dependencies and other metadata, installation, upgrades and removal in a standardized way. Because these operate independently of the OS package manager, all sorts of problems arise. Projects such as GoboLinux have attempted to tie them together, to varying degrees of success. Meanwhile, each new programming language we invent comes with a different, incompatible package manager, and distribution developers need to spend time repackaging them into their preferred format.
Why are we stuck?
I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.
– Abraham Maslow
The packaging ecosystem is very strong. Not only do we have powerful tools for working with packages, we also benefit from packages being a well-understood concept, and having established processes for developing, exchanging and talking about them. Once something is packaged, we know what it is and how to work with it, and it “fits” into everything else. So, it is tempting to package everything in sight, as we already know how to make sense of packages. However, this may not always be the right tool for the job.
Various attempts have been made to extend the packaging concept to make it more general, for example:
- Portage, of Gentoo fame, offers impressive flexibility by building packages with a custom configuration, tailored for the needs of the target system.
- Conary, from rPath, offers finer-grained dependencies, powerful revision control and object-oriented build recipes.
- Nix provides a consistent build and runtime environment, ensuring that programs are run with the same dependencies used to build them, by keeping the relevant versions installed. I don’t know much about it, but it sounds like all dependencies implicitly refer to an exact version.
Other package managers aim to solve a specific problem, such as providing lightweight package management for embedded systems, or lazy dependency installation, or fixing the filesystem hierarchy. There is a long list of package managers of various levels which solve different problems.
Most of these systems suffer from an important fundamental tradeoff: they are designed to manage the entire system, from the kernel through applications, and so they must be used wholesale in order to reap their full benefit. In other words, in their world, everything is a package, and anything which is not a package is out of scope. Therefore, each of these systems requires a separate collection of packages, and each time we invent a new one, its adherents set about packaging everything in the new format. It takes a very long time to do this, and most of them lose momentum before a mature ecosystem can form around them.
This lock-in effect makes it difficult for new packaging technologies to succeed.
Divide and Conquer
No single package management framework is flexible enough to accommodate all of the needs we have today. Even more importantly, a generic solution won’t account for the needs we will have tomorrow. I propose that in order to move forward, we must make it possible to solve packaging problems separately, rather than attempting to solve them all within a single system.
- Decouple applications from the platform. Debian packaging is an excellent solution for managing the network of highly interdependent components which make up the core of a modern Linux distribution. It falls short, however, for managing the needs of modern applications: fast-moving, cross-platform and client/server (especially web). Let’s stop trying to fit these square pegs into round holes, and adopt a different solution for this space, preferably one which is comprehensible and useful to application developers so that they can do most of the work.
- Treat data as a service. It’s no longer useful to package up documentation in order to provide local copies of it on every Linux system. The web is a much, much richer and more effective solution to that problem. The same principle is increasingly applicable to structured data. From documents and contacts to anti-virus signatures and PCI IDs, there’s much better data to be had “out there” on the web than “down here” on the local filesystem.
- Simplify integration between packaging systems in order to enable a heterogeneous model. When we break the assumption that everything is a package, we will need new tools to manage the interfaces between different types of components. Applications will need to introspect their dependency chain, and system management tools will need to be able to interrogate applications. We’ll need thoughtfully designed interfaces which provide an appropriate level of abstraction while offering sufficient flexibility to solve many different packaging problems. There is unarguably a cost to this heterogeneity, but I believe it would easily outweigh the shortcomings of our current model.
But I like things how they are!
We don’t have a choice. The world is changing around us, and distributions need to evolve with it. If we don’t adapt, we will eventually give way to systems which do solve these problems.
Take, for example, modern web browsers like Firefox and Chromium. Arguably the most vital application for users, the browser is coming under increasing pressure to keep up with the breakneck pace of innovation on the web. The next wave of real-time collaboration and multimedia applications relies on the rapid development of new capabilities in web browsers. Browser makers are responding by accelerating deployment in the field: both aggressively push new releases to their users. A report from Google found that Chrome upgrades 97% of their users within 21 days of a new release, and Firefox 85% (both impressive numbers). Mozilla recently changed their maintenance policies, discontinuing maintenance of stable releases and forcing Ubuntu to ship new upstream releases to users.
These applications are just the leading edge of the curve, and the pressure will only increase. Equally powerful trends are pressing server applications, embedded systems, and data to adapt as well. The ideas I’ve presented here are only one possible way forward, and I’m sure there are more and better ideas brewing in distribution communities. I’m sure that I’m not the only one thinking about these problems.
Whatever it looks like in the end, I have no doubt that change is ahead.
We'll see | Matt Zimmerman 
