Ubuntu quality: or, “but what about my bug?”
Leading up to the Ubuntu 8.10 release, Ubuntu developers and quality assurance engineers have been very busy sorting bugs, deciding what can and should be fixed for the final release, and what cannot. They make these decisions by estimating the importance of each bug, identifying whether it is a regression, assessing the risk of potential fixes, and by applying their best judgement. Developers can then focus their efforts where they are most needed.
On the whole, I think that we do remarkably well at this. In September, for example, the total number of open bugs in Ubuntu increased by only 70. This doesn’t sound like much of an achievement, if not for the fact that in the same time period, 7872 new bug reports were filed. The remaining 7802, or over 99%, were resolved (some duplicates, some invalid, some fixed, etc.).
The news isn’t all good, of course. There are currently over 46000 open Ubuntu bug reports in Launchpad. Even at this impressive rate of throughput, and even if we were to freeze all development and stop accepting new bug reports entirely, I estimate it would take over half a year just to sift through the backlog of reports we have already received. There is a lot of noise in that data.
When 8.10 is released, as with each previous release, some users will be disappointed that it has a bug which affects them. This is regrettable, and I feel badly for affected users each time that I read about this, but it is unlikely to ever change. There will never be a release of Ubuntu which is entirely free of bugs, and every non-trivial bug is important to someone.
So, what do we do? What should be our key goals where quality is concerned? We don’t currently have a clear statement of this, but here’s a strawman:
Prioritize bug reports effectively. It’s usually difficult to say whether a bug report is valid or serious until a human has reviewed it, so this means having enough people to review and acknowledge incoming bug reports, and helping them to work as efficiently as possible. The Ubuntu bug squad is a focal point for this type of work, though a great deal of this is done by developers in their everyday work as well. Projects like 5-a-day are a good way to get started.
Measure our performance objectively. By tracking metrics for each part of the quality assurance process, we can understand where we need to improve. The QA team has been developing tools, such as the package status pages, to collect hard data on bugs.
Improve incrementally. By minimizing regressions from one release to the next, and making some progress on old bugs, we can hope to make each Ubuntu release better, in terms of overall quality, than the one before it (or at least no worse). The regression tracker (which will hopefully move to the QA site soon) will help to coordinate this effort.
Ensure that the most serious bugs are identified and fixed early. It’s important to the success of the project that we continue to produce regular releases, and showstopper bugs risk delaying our release dates and adversely affecting the next release cycle. The release management weather report is one tool which helps monitor this process, though a great deal of coordination is required in order to provide it with useful data.
Communicate about known bugs. It is inevitable that there will be known bugs remaining in each release, and we should do our best to advise our users about them, including any known workarounds. The Ubuntu 8.10 beta release notes are a good example of this.
I think that to do well in all of these areas would be a good goal for quality in Ubuntu.
We'll see | Matt Zimmerman 
Ubuntu is buggy as always.
*sigh*
Since started using ubuntu, every release had real bad bugs in various aspects of the OS. Windows and Mac are much better.
verb
October 29, 2008 at 12:34
What about merging efforts from Debian? Like basing releases on Debian testing instead of Debian sid.
foo
October 29, 2008 at 13:06
Hi,
I think you do a great job. I upgraded two computers to Intrepid already and it was smooth. (Ubuntu very smooth, Xubuntu fairly smooth.)
I don’t think 8.10 will be a very exciting release but it seems _very_ solid. So I think you are doing great on the bug side of things. Thanks!
Prioritizing bugs is great. Maybe each bug should get a “Digg” button. That way people could express that the bug bugs them without having to comment and creating noise and you would get a priority list for free.
Only thing I might add is: Get bigger machines and faster pipes for launchpad. It is getting slow(At least for me.) That would be a great investment.
Keep up the good work!
T0m
October 29, 2008 at 13:08
Please do all of that, but also, please stop hunting furiously for stuff to close wrongly or mark incorrectly as duplicates just to fill your five-a-day.
Not looking at you MDZ, but at some people running around launchpad (esp. som Pablo or Paolo I think it is does this A LOT but is not alone). Why it’s more important to improve statistics than to actually fix things is way beyond me, keep that behavior on Windows where it belongs.
It’s a problem not because it can’t be reverted (if you are lucky, try get some bug unduplicated if you like), but because it sends very unpleasant signals to people who have actual problems and don’t want dimissive “no recent action, closing” messages in their inbox.
It’s really respectless and infuriating. Just stop it. An open bug is better than an unfixed, hidden ones. Let politicians play games with “open unemployment” and “official unemployment” et al.
Stoffe
October 29, 2008 at 13:19
@T0m: we already do have a “digg” like button. See http://news.launchpad.net/general/launchpad-2110-faster-branch-uploads under “This bug affects me too”.
mdz
October 29, 2008 at 13:49
@foo: we already do merge efforts from Debian, at the beginning of every release cycle. There are many reasons why we use unstable rather than testing, including the fact that our release schedule is radically different from Debian’s and that we generally require current upstream versions of software.
mdz
October 29, 2008 at 13:50
Wow, cool!
Thanks for answering.
I wanted to check, but Launchpad gave me time out errors .. that is why I complained about the slowness ;)
Maybe a more “Digg button”-like solution might be even better. It is not really obvious what the “change” link will do IMHO. A number of “affects” might also help. I guess stick with what people already know.
T0m
October 29, 2008 at 14:11
@T0m, there’s currently (mostly today) a temporary performance problem with Launchpad, which the developers are working on. See https://bugs.edge.launchpad.net/malone/+bug/290668
If you have feedback about the “me too” functionality, please file a wishlist bug or email the launchpad-users mailing list.
mdz
October 29, 2008 at 14:28
Almost every release brought regressions for me: one time burning cd’s stopped working, another time my graphics card didn’t work anymore without resorting to hacking xorg.conf, and yet another time all video playback was suddenly flickering. These are really unpleasant things to deal with (if you can even work around them yourself, which most people won’t be able to) and I think regressions should be much higher priority.
Wouter
October 29, 2008 at 15:33
Also an interesting observation if you look at the bug statistics ( http://people.ubuntu.com/~brian/graphs/ ) :
During some periods the number of open bugs is somewhat stable, increasing only moderately. Then in the first two months after a release, there is a sudden surge in the number of open reports. A first thought would be that many more bugs are reported as people move to a new version. But interestingly if you look at the total number of reported bugs, this follows almost a perfect straight line, meaning no more bugs are reported than during other periods. So this must be due to less bugs getting fixed/closed in that period.
Perhaps it would be an idea to open the repositories for the next version as soon as possible (the day/week after release?), allowing developers to fix bugs during the whole release cycle instead of having a very silent period. Of course emphasis is not at bugfixing at that point, but I think it would improve on the current situation.
Wouter
October 29, 2008 at 15:41
Stoffe: do you have examples of bugs where things went wrong? I believe that this happens, but right now I believe it’s a small minority of bugs.
I’m happy to help to improve documentation with the Bug Squad and the QA team and talk to Bug Triagers who get it wrong in the first run.
Daniel Holbach
October 29, 2008 at 15:46
And as for marking duplicates…I vote for sound bugs, let’s let Crimsun mark the duplicates himself. So many sound bugs have like 30 people latching on, or triagers go “oh, crackling, this is all one bug,” but in reality they’re all very different bugs that just have one symptom in common. He finally posted an “everyone out of the pool” comment yesterday saying he’d only talk to the reporter and everyone else popping up on that bug had to go file their own bugs.
Stoffe:
“No recent activity” closings only happen when they’re Incomplete with no response. Meaning we asked you for information to try to fix it, and you didn’t seem to care enough to ever respond.
Mackenzie
October 29, 2008 at 15:58
I look forward to the effects of more thorough testing.. Perhaps one person can test x, y, and z every release because they have the hardware with which to do so.. Perhaps another can test a, b, and c..
If we can cover every single bit of the basic functionality of ubuntu with every .iso we put out, and yes that’s a lot of user involvement, then perhaps we can put out a distro where the average user encounters a lot less in the way of bugs. Automated testing and ‘me too’ should also help a good deal.
ethana2
October 29, 2008 at 16:27
Ubuntu-vm-builder needs to be updated every 6 months to be able to build a VM of the current unstable. Using it to install Hardy then do-release-upgrade from there to Intrepid is too many steps. The desktop applications can be tested very easily and very widely like this, so the people who are afraid of running it on their hardware but want to help are able to do so more easily. Or the people like me that went “but I finally have a really-well-working system!”
Mackenzie
October 29, 2008 at 16:45
@Mackenzie,
I agree this type of testing is useful, but most of the interesting bugs (the ones which aren’t efficiently found during the normal course of the release) only turn up in real-world scenarios: upgrading systems with real data, working with applications on a day-to-day basis, etc.
I expect that most people won’t do that in a VM, but only “smoke test” that the basic functionality is there.
For those who will, though, sandbox-upgrader in Intrepid makes it easy to do low-risk, reversible testing of your system upgraded to Intrepid. You should always take a backup of course, but it’s designed to enable this type of testing without needing to change your native OS.
mdz
October 29, 2008 at 17:01
I think the duplicate bug finder that comes up when you issue a bug could be improved. I have had numerous cases where I put the exact text of the crash in the bug, and then when I get a list of possible duplicates none of them match, so I continue to opening a new bug report. Then it turns out a bug was already opened that also had the exact error message in the title, but the duplicate bug finder did not catch it. Improving this tool could help keep duplicates from being opened.
Maxo
October 29, 2008 at 17:06
I stoped using Ubuntu because of persistent bugs. *sigh* Bugs which I didn’t find on Fedora. Fedora is more stable and faster than Ubuntu. I can barely wait for Fedora 10.
Claudio
October 29, 2008 at 19:59
How about some social bug-triaging?
http://brainstorm.ubuntu.com/idea/9624/
5-a-day may be too much for many people. And I think that automating, focusing and socializing bug-management would really help.
Thanks for all the great work,
Anders
Anders
October 29, 2008 at 20:44
>OS. Windows and Mac are much better.
But you aren’t paying money for Ubuntu. *AND* you can contribute to making it better. Microsoft and Apple (AFAIK) do not have a launchpad or bugzilla.
Felipe Alvarez
October 30, 2008 at 06:29
I have always wondered the LTS version is not a “bugs only” release. Add all the features you like, but if they don’t work then whats the point? The number of growing bug reports is horrendous. If the LTS version was intended just to catch up on bugs from the last release, I think you would see a large increase in the quality of the distribution.
clifford
October 30, 2008 at 08:42
FUBAH 46000 bugs, if thats not FUBAH then i dont know what is.
oh yes i do, nearly 8000 bugs in a single month is
OMFG, at least my WinXP is stable and reliable, and well just works
Darryl
October 30, 2008 at 09:18
FUBAH 46000 bugs, if thats not FUBAH then i dont know what is.
oh yes i do, nearly 8000 bugs in a single month is
OMFG, at least my WinXP is stable and reliable, and well just works
Darryl
October 30, 2008 at 09:18
How about taking a look at bugs that are still unresolved after 4 years? Some bugs seem to just be abandoned by everyone.
Ramon Casha
October 30, 2008 at 09:25
Although I think the Ubuntu team, along with the community do a great job at trying their hardest to find and fix bugs, I still think we should look into the fundamental way we sort and fix bugs. Because although many people work very hard at fixing bugs, it’s just no acceptable to launch with that many bugs.
Either we need to look at how bug reporting and the likes are done, or if a 6-month release is too short to pump out something stable enough for people to work with.
LostOverThere
October 30, 2008 at 10:04
I have to say that I have been very impressed with Intrepid so far. Hardy had a number of fairly persistent “bugs”:
-wifi connected poorly, if at all
-running flash player and any other sound application at the same time consistently turned off my sound card
-getting my tablet to work was a full evening’s worth of effort.
Since upgrading to Intrepid, all of the above have “just worked”. The only issue I’ve had was that my wifi stopped working when I upgraded my kernel from 2.6.27 4 to 2.6.27 7. After a few days that was fixed.
So I just want to say, the effort put into resolving bugs and building a more stable and friendly OS have been noticed, and very much appreciated. Thanks.
Erik Wiffin
October 30, 2008 at 15:02
[…] Ubuntu quality: or, “but what about my bug?” Leading up to the Ubuntu 8.10 release, Ubuntu developers and quality assurance engineers have been very busy sorting […] […]
Top Posts « WordPress.com
October 31, 2008 at 01:27
I disagree with this message. The idea is flawed from the very beginning: You are talking about bugs, but where do they com from? Obviously these bugs are introduced by YOURSELVES, EVERY time that a new release version you prepare. By the way I do not believe in regular upgrades of the OS, nor I need them. Why? Just a bit of tinkering and there it is: it must be released as a new, point something version! This is ridiculous! Stop that non-sense franctic release and sit seriously and work on ONE operating system version to make it bugs-free, solid rock stable, and with outstanding harware support. I just care for that and security fixes and avoiding crap software that is immature but nontheless included as if something finished and polished. Far from the truth. Look Mac and windows. These OSs stay for years in a same computer! No need upgrades, but updates, so users dont break their system. Learn from your competitors. They are here from long ago and will stay. LEARN FROM THEM
Max
October 31, 2008 at 19:52
[…] a comment » This report is impressive… receiving 7,872 new reports and resolving 7,802 bug reports in a single month […]
Credit to Ubuntu « Poelcat
November 1, 2008 at 00:32
[…] Zimmerman, Canonical CTO has an interesting post about […]
Ubuntu Podcast Episode #11 | Ubuntu Podcast
November 8, 2008 at 01:15
What I’d really like to see is an easier way for non-canonical users to be able to join in and fix Ubuntu’s bugs.
As someone who knows how to fix 4 or 5 bugs that are in launchpad right now I haven’t got the faintest clue where to start and I have no interest in joining the “canonical club” (I’m interested in fixing my bugs only, not becoming part of ubuntu’s MOTU).
With that said Canonical should think about more ways users can join in with little to no effort on the part of users. With the barrier to entry lowered I think you’d see more bugs fixed.
The biggest problem of course is with bugs that are in Debian. There are 4 bugs that I know filed in launchpad which are in fact because Debian built the package wrong. It’s asking too much of your users to go to the Debian mailing list, argue with the package maintainers there and then have to wait until Ubuntu syncs the package with Debian again. That’s too much bureaucracy to get anything done.
So that is where I am at, I could fix most of the bugs I file but I’m not going to because there’s too much communication that needs to be done between Ubuntu and Debian, so my bugs will sit there in Launchpad for many years to come and all because there is no direction given on how to contribute effectively on Debian or Ubuntu packages.
Andrew Fenn
November 10, 2008 at 03:02
Speaking as someone who does 2nd line support professionally.. this site – http://smoogespace.blogspot.com/2008/04/fedora-bug-tracking-incident-tracking.html
– seems to raise issues that I’ve been trying to raise. That is that raising a Bug is counterintuitive to people who want to raise what in ITIL terms is called an Incident. Namely – a support ticket.
Incidents become bugs later…
But then – that’s me.
Alex Cockell
November 15, 2008 at 18:55
I’m having problems with your metrics. Just stating 46K thousand open bugs is just asking for a negative reaction – it also isn’t even an accurate picture of what is happening.
Those who haven’t worked in software development don’t realize that all software has an endless list of bugs – period. That won’t change ever. Windows and Mac are no different, it’s just less transparent.
So when I look at measurements I don’t see wishlist items separated from bugs. While the end user doesn’t care, recognizing where there are quality issues in the code versus where there is a quality issue in the spec (gap in requirements) is significant.
Also – the graphs show number of open bugs. In a stable system (like your 7K bug turn over) those graphs don’t really change so they aren’t that interesting. Why not report on bug status against date reported. Then you might see that there are not a lot of bugs that were reported recently that are still open, and a lot of bugs closed that were reported recently.
Not only will a change in reporting help you feel good about turnover (and communicate it effectively), it might be obvious that a slew of bugs appeared in a specific time period in a specific package and then focus on reviewing that block in a given sprint.
But I also think a session on vision is important as well – with 46K reported, most people probably don’t remember the issue and don’t care anymore. Only some aggressive decisions will get you somewhere. OR if you want to keep even the smallest issue open, a proper vision will help you modify reporting to communicate that vision.
So don’t start the sentence with 46K open bugs, start with what’s important, and communicate that information.
Craig73
December 22, 2008 at 21:29
We are working on better metrics, some similar to those you describe. You can also run simple reports for yourself at https://bugs.edge.launchpad.net/ubuntu. The idea here was to give a sense of the scale of the problem of processing this quantity of bug reports.
I would love to see some proposals for good visualization techniques to help analyze this data more effectively.
We have experimented with expiring old issues, particularly those which have remained incomplete for too long.
I don’t know that any single change will make a drastic difference here, but we’re exploring several approaches in parallel to learn what works. I’d be interested to hear from folks who have dealt with this on a similar scale.
FYI, there are presently 5711 open wishlist reports.
mdz
December 23, 2008 at 17:21
@Alex, we have a similar concept in Launchpad called “Answers” (see https://answers.edge.launchpad.net/ubuntu/+questions). These are like support tickets, and track a user’s problem or question until the point where it can be determined whether it is a bug or not. There’s a simple procedure for filing a bug report based on the ticket, and Launchpad maintains a link between them for future reference.
Both systems are open to everyone, and bug reporting seems to be more popular at the moment. One possibility would be to restrict bug filing and direct more traffic to Answers, but this runs the risk of discouraging knowledgeable users from filing otherwise valuable bug reports.
mdz
December 29, 2008 at 16:20
I’ve seen it.. and yes.. it’s a start.
However, when reqesting new apps etc (Enhancement Requests, I’d call them), maybe more use could be made of forms to capture the info.
Just that it seems to be “Ask a question of the community”, rahter than if someone is a rawer beginner… instead of telling them to raise the big, have the Triage team look at stuff which might be worded more as a support ticket would be.. and then be able to generate a “bug” from it..
However, if it was to become more of a “support ticket” and Knowledgebase mechanism.. would also mean you immediately have metrics…
Also – there was talk around about getting end-users to talk to Upstream more.. surely you want experienced triagers and liaison people doing that…
Alex Cockell
February 26, 2009 at 21:00
@mdz … Amusingly, the new Brainstorm is great in that it directs users to file package requests and bugs (to reduce it’s spam) but it doesn’t mention “answers” as a possible destination.
——————–
I was wondering about wishlist requests – perhaps the way to deal with it is to
a) Define the vision for the platform and understand the vision of the application team.
b) Review the wishlist items at a high level to identify trends in the gaps identified by users to help clarify future objectives (how are we not meeting their needs)
c) Review the objectives for coming couple of release cycles
d) Then either add the wish list item to the specs, or close them as won’t fix.
note: the coming release could be a maintenance release (thus small gaps, minor issues might be fixed), or new functionality (major gaps, trend issues, could be addressed).
Either way, there shouldn’t be wishlist items over a year old. The world and the context of the issues will just keep changing, so why hold onto old issues?
(This is really an optimist and perhaps naive stance, but one that is based on just making a decision and moving on. If something is truly a gap it will come up again. Oerhaps someone with large application experience could weigh in)
Craig73
January 15, 2009 at 00:07
It is the 9.04 release and not the mentioned 8.04. Ubuntu 8.04 was released in April 2008.
Michael
February 28, 2009 at 19:42
s/8.04/8.10.
s/April/October
Michael
February 28, 2009 at 19:44
I’m not sure what you’re referring to here. This post was written in October 2008 and refers to the 8.10 release.
mdz
February 28, 2009 at 21:11
Oh, I see. I noticed this old post was uncategorized, and I added it to a category. This seems to have caused it to appear on Planet Ubuntu again. Sorry about that.
mdz
March 1, 2009 at 23:04
Amen to this.
I would add that the most irritating, annoying, and utter show stopper issues are regressions.
Canonical is doing tremendous innovative work with evolving the collaborative / problem solving – please keep it up.
Regressions are the worst evils of this entire process. There is nothing to suggest any concept of evolution if we constantly retract every single release.
If it worked once, let’s get the damn thing to work forever. Is that too much to ask? We absolutely must stop this ‘treadmill-headed-to-nowhere’ we find ourselves in with regressions.
No easy solutions, and perhaps that is part of the problem – we haven’t yet invented a process that fixes the core issue of regressions…
How about a Bug Interchange Format while we are at it? One bug standard to rule them all so that the information can flow better?
Troy James Sobotka
February 28, 2009 at 20:15
I ve published few bugs in launchpad but i did not find anyway to close it myself easily when it was fixed i by an update or upgrade. there should be a tick box to close a bug by the one who opened it.
taiebot
March 2, 2009 at 18:20
[…] For over three years now, Ubuntu has been using Launchpad to track bugs. This has been an overwhelming success in terms of the number of bug reports filed, so much so that we have trouble keeping up with them. […]
Please don’t report Ubuntu bugs directly to Launchpad « We’ll see
March 31, 2009 at 20:51