Equilibrium in free software testing
When a bug is filed in a free software project’s bug tracker, a social exchange takes place. Bug reporters give their time and attention to describing, debugging and testing, in exchange for a fair chance that the problem will be fixed. Project representatives make the effort to listen and understand the problem, and apply their specialized knowledge, in exchange for real-world testing and feedback which drive improvements in their software. This feedback loop is one of the essential benefits of the free software development model.
Based on the belief that this exchange is of mutual benefit, the people involved form certain expectations of each other. When I report a bug, I expect that:
- the bug will be reviewed by a project representative
- they will make a decision about the relative importance of the bug
- project developers will fix the most important bugs in future releases of the software
When I receive a bug report, I expect that:
- if more information is needed, the bug reporter will supply it
- if I can’t diagnose the problem independently, the bug reporter will help with the analysis
- if I need help to test and verify a fix for the bug, the bug reporter will provide it
Naturally, everything works best when the system is in equilibrium: there is a steady flow of testing and bug reports, and users feel satisfied with the quality of the software. Everybody wins. Ideally, much of this activity takes place around pre-release snapshots of the software, so that early adopters experience the newest features and fixes, and developers can fix bugs before they release a new version for mainstream use. This state produces the best quality free software.
Unfortunately, that isn’t always the case. When our expectations aren’t met, or sufficient progress is not made, we feel misled. If a bug report languishes in the bug tracker without ever being looked at, the bug reporter’s time and effort have been wasted. If the report lacks sufficient detail, and a request for more information goes unanswered, the developer’s time and effort have been wasted. This feeling is magnified by the fact that both parties are usually volunteers, who are donating their time in good faith.
The imbalance can often be seen in the number of new (unreviewed) bug reports for a particular project. At one extreme (“left”) is a dead project which receives a flood of bug reports, which are never put to good use. At the other extreme is a very active project with no users (“right”), which suffers from a lack of feedback and testing. Most projects are somewhere in the middle, though a perfect balance is rare.
Ubuntu currently receives too many bug reports for its developers to effectively process, putting it well left of center. It has a large number of enthusiastic users willing to run unstable development code, and actively encourages its users to participate in its development by testing and reporting bugs, even to the point of being flooded with data. A similar distance to the right of center might be the Linux kernel, which receives comparatively few bug reports. Kernel developers struggle to encourage users to test their unstable development code, because it’s inconvenient to build and install, and a bug can easily crash the system and cost them time and work. There are a huge number people who use the Linux kernel, but very few of them have relationships with its developers.
So, what can a project do to promote equilibrium? Users and developers need to receive good value for their efforts, and they need to keep pace with each other
The Linux kernel seems to need more willing testers, which distributions like Fedora and Ubuntu are helping to provide by packaging and distributing snapshots of kernel development to their users. The loop isn’t quite closed, though, as bug reports don’t always make their way back to the kernel developers.
Ubuntu, perhaps, needs more developers, and so we’ve undertaken a number of projects to try to make it easier to contribute to Ubuntu as a developer, and to help our developers work more efficiently. Soon, it should be possible to commit patches directly into Launchpad without any special privileges, so that they can be easily reviewed and merged by project developers. This isn’t a fix, but we hope it will help move us closer to a balance.
What else could we try? I’m particularly interested in approaches which have worked well in other projects.