We'll see | Matt Zimmerman

I run the CyanogenMod derivative of Android on my G1, and somehow managed to get it into a state where it had withdrawn the security permissions from my installed applications. I think this happened when I attempted to upgrade the 1GB microSD to a 4GB one, but the phone failed to boot.

I first noticed the problem when trying to refresh in NewsRob would hang the application, and adb logcat showed:

W/dalvikvm( 540): threadid=3: thread exiting with uncaught exception (group=0x4001e170)
E/NewsRob ( 540): Caught the following exception:
E/NewsRob ( 540): java.lang.SecurityException: Neither user 10039 nor current process has android.permission.WAKE_LOCK.


NewsRob clearly had had this permission before, to prevent the phone from sleeping during a sync. The Manage Applications screen still showed that it did (“System tools: prevent phone from sleeping”). Watching adb logcat while the phone was booting showed what was going on, and that many other applications had the same problem:


W/PackageManager( 138): Not granting permission android.permission.INTERNET to package com.newsrob because it was previously installed without
W/PackageManager( 138): Not granting permission android.permission.WAKE_LOCK to package com.newsrob because it was previously installed without
W/PackageManager( 138): Not granting permission android.permission.ACCESS_NETWORK_STATE to package com.newsrob because it was previously installed without
W/PackageManager( 138): Not granting permission android.permission.VIBRATE to package com.newsrob because it was previously installed without
W/PackageManager( 138): Not granting permission android.permission.WRITE_EXTERNAL_STORAGE to package com.newsrob because it was previously installed without


/data/system/packages.xml, which seems to record which applications are installed and which permissions they have, showed:

<package name="com.newsrob" codePath="/data/app/com.newsrob.apk" system="false" ts="1264200476000" version="353" userId="10039" installer="com.google.android.feedback">
<sigs count="1">
<cert index="25" key="..." />
</sigs>
<perms />
</package>


i.e. the permissions block was empty. It should have looked more like this:

<perms>
<item name="android.permission.VIBRATE" />
<item name="android.permission.WRITE_EXTERNAL_STORAGE" />
<item name="android.permission.ACCESS_NETWORK_STATE" />
<item name="android.permission.WAKE_LOCK" />
<item name="android.permission.INTERNET" />
</perms>


I tried manually hacking it, and also moving and replacing the .apk file on the phone, but packages.xml always returned to this state. Maybe it’s not the master copy of that data.

What finally fixed it for me was to re-install the applications using the package manager, by running:

cd /data/app
for app in *.apk; do pm install -r $app; done


I hadn’t known about the pm command until then, and discovered it by accident when invoking an adb command told me about it. The phone chugged along for quite a while, but eventually re-installed all of the applications, and the problem was fixed.

Web searches showed that I was not the only person to find themselves in this predicament, and did not reveal an obvious solution, so I’m documenting mine here.

This will be the end of the series, as I’m leaving for the airport this afternoon.

Rusty Russell: FOSS fun with a Wiimote

Rusty told an entertaining story about his journey to produce geeky toys for his daughter, who is too young to use a keyboard or other standard human-computer interface. I always enjoy hearing about the intermediate steps of invention, and this was no exception. After five design iterations and several long distractions, Rusty produced a couple of working applications using Python and libcwiid, and demonstrated one of them.

Ariel Waldman: Space hacks

Ariel’s talk explained the (surprisingly numerous) ways in which geeks can get actively involved in advancing space science and exploration. With budgets of zero, hundreds or thousands of USD, there are projects which are accessible to individuals and schools which offer not only fun and education opportunities, but actually contribute something to the human study of outer space.

I didn’t note them down, so please watch the talk if you’re interested.

Lindsay Holmwood: Flapjack

Lindsay introduced Flapjack, which is a monitoring system designed to meet the scalability and extensibility requirements of cloud deployments. It shares the Nagios plugin interface, and so can use the same checks. It uses beanstalkd as a central message queue to coordinate the work of executing checks, recording results and making the appropriate notifications. Each of its components (worker, notifier, database) can be extended or replaced using an API, providing a great deal of flexibility.

Jeremy Allison: Microsoft and Free Software

Jeremy took us through Microsoft’s recognition of, and response to, the threat of free software to their monopoly position. After reviewing the major legal battles of this ongoing war (and the metaphor is apt), he says that Microsoft is turning to patents in an attempt to split the free software community and to earn revenue from the use of free software. Jeremy predicts that the outcome will be a never-ending conflict.

The key conflicts are likely to be around netbooks, mobile phones and appliances. How should the free software community respond?

We could ignore it, and keep making free software under copyleft licenses. Jeremy points out that this is perhaps our most effective strategy in the long run, to stay focused on the vision of a free software world.

We can continue to pressure governments and corporations to adopt truly open standards, and to investigate and challenge monopolies. Transparency is key to these efforts, as “elephants like to work in the dark” (Microsoft being “the elephant in the room”).

By lobbying against software patents, we can hope to contain the US software patent system from the rest of the world. Otherwise, the rapidly accumulating software patents in the US can suddenly and dramatically spread.

We might even convince the likes of Microsoft that patents, and patent trolls, represent a greater harm than good.

In response to direct patent attacks, we should search for prior art and attempt to undermine unjust patents. He also suggests calling out Microsoft employees on the company’s actions, to promote awareness particularly in the context of free software conferences.

He closed with a hope that Microsoft could change, citing IBM as having been “as feared and hated as Microsoft is today”.

Neil Brown:Digging for Design Patterns

Neil explored various design patterns in the kernel in order to illustrate how they are discovered, what their important attributes are, and how to use them effectively.

His examples were a binary search, “goto err”, accessor functions and kref. Naming patterns is important, especially getting that name into the code itself, so that it helps to cross-reference use, implementation and documentation of the pattern (e.g. uses of the kref pattern are sprinkled with the word “kref”). A successful pattern can both help to find bugs (this binary search doesn’t look the same as that one…why?) and to avoid bugs (by getting it right the first time).

Benjamin Mako Hill: Antifeatures (keynote)

Mako delivered an entertaining and inspirational talk on antifeatures, those oddities which intentionally make technology less useful to its consumers (think DRM, though he provided a wide range of examples). Mako explained the main reasons why antifeatures exist, and how they are endemic to the business of proprietary software.

Mako offered a potential upside to antifeatures, which is that they can help the free software community to focus on fundamental concerns like autonomy, rather than (for example) the mechanics of licensing. Antifeatures can be used to explain to the uninitiated why software freedom is important to everyday folks, not just hackers.

Denise Paolucci and Mark Smith: Build Your Own Contributors, One Part At A Time

Denise and Mark provided a practical list of “dos” and “don’ts” for building a successful community based on respect, empowerment and collaboration. Much of this was elementary from an Ubuntu perspective, but they offered a variety of examples from Dreamwidth which were illustrative.

Their list of “three things to start right now”:

1. appoint a “welcomer” and laud newcomers’ first contributions
2. stop timing out on communication when people need responses from you
3. Have words with “that person” and let them know their behavior is not okay

Chris Double: Implementing HTML5 video in Firefox

I knew I liked this idea, but I didn’t realize how much I liked it until I watched Chris’ very informative talk. The promise of an open standard for embedding video is exciting enough, but the standard offers much more than basic embedding and playback controls. Chris demonstrated javascript-driven subtitles loaded from an SRT file, custom controls, copying and analysis of pixel data, replacing the background in a video using a chroma key technique, and even more impressive real-time special effects.

The initial implementation used the xiph.org reference libraries (libogg, libtheora, libvorbis) and PortAudio, but had some problems, including poor A/V synchronization. The second iteration used higher level libraries liboggz, libfishsound, liboggplay and libsydneyaudio, and was included in Firefox 3.1 alpha and beta, but some limitations in liboggplay (a/v sync, chained oggs, etc.) led to difficulty. There were also proof of concept implementations which used GStreamer on Linux, DirectShow on Windows, and QuickTime on MacOS, but these were hampered by codec plugin complications. In the end, they’ve gone back to using the xiph.org reference libraries (but with libsydneyaudio), though the GStreamer backend is still actively developed. Chris has published a series of articles on his blog on reading, decoding and synchronizing A/V streams using various libraries.

There are still some kinks to work out: the lack of indexes and the like in Ogg complicates seeking, calculating duration and so on, and there is no satisfactory solution for cross-platform audio. Rendering is not hardware accelerated yet because the video element is part of the HTML rendering pipeline.

It will be very powerful when it’s ready, though. Theora playback is supported in Chrome, Firefox and Opera today, and Daily Motion, Wikipedia and Archive.org are using it. I can’t wait to see the full API working well on a massive scale.

Julius Roberts: ZFS

This talk was a tour through the main features of the ZFS filesystem, showing how to work with storage pools and snapshots. It was useful to see the example commands and behavior in the context of a live command line session, as I haven’t got around to playing with it in an OpenSolaris VM yet.

Mark Atwood: memcached

I ended up in this talk in between other sessions I was attending. It outlined the sorts of places where memcached can be useful in applications. Beyond the obvious caching scenarios, Mark suggested using it to store an application’s working set (key/value dictionaries), user sessions and distributed rate limiting scoreboards.

Josh Berkus: Relational vs. Non-relational

I’ve followed with interest the NOSQL movement, and was interested to hear from Josh (of PostgreSQL Experts Inc.) what I expected would be a “relationalist” point of view.

He began by addressing some database myths. He stated that the “revolutionary new database designs” lauded by the non-relationalists are actually just new implementations of old ideas (e.g. CouchDB vs. Pick). He dismissed the “NoSQL” moniker as the wrong distinction to make: there are much more important and basic differences between database implementations than whether or not they use SQL. He explained that the relational model is orthogonal to support for ACID transactions, though I didn’t realize that was an active misconception. He also reminded us that we do not need to choose a single database for all of our possible needs, and should focus on choosing the system which fits our current application goals, or choosing multiple databases (e.g. MySQL and memcached, or [interestingly] PostgreSQL and CouchDB).

Relational OLTP databases tend to have more mature implementations of transactions, the ability to enforce data constraints and consistency, support complex reporting, and vertical scaling (but not horizontal).

SQL itself doesn’t map very well to many programming languages, but promotes application portability and enables the management of schema changes over time. “No-SQL” provides a more natural mapping, and allows programmers to act as DBAs.

His main reason to use an SQL-RDBMS was when the data will outlive current application implementations, because having a looser coupling between the database and application helps support a migration to a new implementation.

He compared the applicability of embedded key-value, distributed key-value, flat file, object, and document databases but I wasn’t able to take notes quickly enough through this segment.

His conclusion was to reiterate that “relational vs. non-relational” is the wrong way to look at the problem, and that we should instead choose the right tool for the job.

After maintaining an ad hoc Linux distribution for myself for several years, I replaced it with Debian and have never looked back. One of the main reasons for this has been the mind-boggling quantity of applications and tools which are available from its repositories. Given couple of keywords, or a good guess at the name of the application, APT fetches and installs the necessary packages in a matter of seconds. After years of compiling free software programs from source code, this profoundly changed the way I thought about finding and obtaining software.

Over 10 years later, the speed and convenience of this system still occasionally leaves me awestruck. As a typical example, on one occasion, I was using a pastebin to share the output of a program I was discussing with someone online by copying and pasting it.. The output was fairly long, and it was inconvenient to copy and paste, so I wanted a tool which would read the output from a pipe and upload it directly to the pastebin, without a human in the middle.

Before I fired up an editor to write such a tool, I did a quick search to check if any such thing existed already, and found Stéphane Graber’s pastebinit, which did exactly what I wanted (and more). Not only had someone else had the idea first, they had implemented, released and packaged it over a year earlier. The end result, for me, was that within 30 seconds of discovering that I needed such a tool, I had it installed and working.

Experiences like the above still make me feel like I’m living a scene from the 1989 film Bill & Ted’s Excellent Adventure, where the protagonists discover that they have already traveled back in time to anticipate their own needs. They merely think about what they need, and there it is. The fact that I am still amazed by this probably makes me sound like a dinosaur to other free software enthusiasts, but this kind of instant gratification is something which is only just beginning to emerge in proprietary systems like the iPhone. The web was designed from the start to work this way, of course, but there is much I can do with free software that I can’t do with web applications (at least for now). The web also doesn’t give me that feeling of personal connection with the creator of the software, or (generally) the opportunity to tailor it for my needs.

The development of Ubuntu 10.04 has been underway for nearly two months now, and will produce our third long-term (LTS) release in April. Rick Spencer, desktop engineering manager, summarized what’s ahead for the desktop team, and a similar update will be coming soon from Jos Boumans, our new engineering manager for the server team.

What I want to talk about, though, is not the individual projects we’re working on. I want to explain how the whole thing comes together, and what’s happening behind the scenes to make 10.04 LTS different from other Ubuntu releases.

Changing the focus

Robbie Williamson, engineering manager for the foundations team, has captured the big picture in the LTS release plan, the key elements of which are:

Merge from Debian testing

By merging from Debian testing, rather than the usual unstable, we aim to avoid regressions early in the release cycle which tend to block development work. So far, Lucid has been surprisingly usable in its first weeks, compared to previous Ubuntu releases.

Add fewer features

By starting fewer development projects, and opting for more testing projects over feature projects, we will free more time and energy for stabilization. This approach will help us to discover regressions earlier, and to fix them earlier as well. This doesn’t mean that Ubuntu 10.04 won’t have bugs (with hundreds of millions of lines of source code, there is no such thing as a bug-free system), but we believe it will help us to produce a system which is suitable for longer-term use by more risk-averse users.

Avoid major infrastructure changes

We will bring in less bleeding-edge code from upstream than usual, preferring to stay with more mature components. Where a major transition is brewing upstream, we will probably opt to defer it to the next Ubuntu cycle. While this might delay some new functionality slightly, we believe the additional stability is well worth it for an LTS release.

Extend beta testing

With less breakage early in the cycle, we plan to enter beta early. Traditionally, the beta period is when we receive the most user feedback, so we want to make the most of it. We’ll deliver a usable, beta-quality system substantially earlier than in 9.10, and our more adventurous users will be able to upgrade at that point with a reasonable expectation of stability.

Freeze with Debian

With Debian “squeeze” expected to freeze in March, Ubuntu and Debian will be stabilizing on similar timelines. This means that Debian and Ubuntu developers will be attacking the same bugs at the same time, creating more opportunities to join forces.

Staying on course

In addition, we’re rolling out some new tools and techniques to track our development work, which were pioneered by the desktop team in Ubuntu 9.10. We believe this will help us to stay on course, and make adjustments earlier when needed.  Taking some pages from the Agile software development playbook, we’ll be planning in smaller increments and tracking our progress using burn-down charts.  As always, we aim to make Ubuntu development as transparent as possible, so all of this information is posted publicly so that everyone can see how we’re doing.

Delivering for users

By making these changes, we aim to deliver for our users the right balance of stability and features that they expect from an Ubuntu LTS release. In particular, we want folks to feel confident deploying Ubuntu 10.04 in settings where it will be actively maintained for a period of years.

I have a couple of USB hard disks which each have a single GPT partition on them. I recently moved them to an embedded server, and discovered that its Linux kernel lacked support for GPT.

For various reasons, it isn’t practical for me to replace its kernel right now, but I still wanted to be able to use the disks, and to have them automount by UUID.

…some time later…

A set of udev rules:

# Import variables from devkit-disks-part-id on the *parent* device
# devkit-disks-part-id looks at $DEVPATH regardless of the argument passed to
# it, so we need to override that
ATTR{partition}=="1", IMPORT{program}="/usr/bin/env DEVPATH=%p/.. /lib/udev/devkit-disks-part-id /dev/%P"

# If this partition is on a disk using GPT, fake it
ATTR{partition}=="1", ENV{DKD_PARTITION_TABLE_SCHEME}=="gpt", RUN+="/sbin/losetup -o 16896 -f /dev/%k"

This code uses a tool from devicekit-disks to detect when a GPT partition table is present. If so, it sets up a loop device at the appropriate (hardcoded) offset corresponding to the GPT partition.

It only works for a single partition, and it’s not exactly pretty, but it solved my problem. The loop devices generate their own uevents, the generic udev rules detect the UUID, and everything works.

This week, I’m in Dallas, Texas working at the Ubuntu Developer Summit.  Hundreds of Ubuntu developers and other community members are gathered to discuss the future of the project, particularly the 10.04 release.  Developers are engaged in technical discussions about how to implement new features for the release next April.

Obviously, a week is not enough time to decide, design and plan half a year of work, but we try to fit as much as possible into the week, because it is such a rare opportunity for us to work together face to face.  In order to make the best use of our time, there is a very full schedule of sessions, and we do a great deal of advance preparation.

There is a persistent rumor that UDS is where we decide what to do in the next cycle, but this isn’t quite accurate.  UDS is where we primarily figure out how to do what needs to be done.  Naturally, UDS is a sea of ideas, thanks to all of the creative thinking which happens among attendees, and we do dream up and decide to do new things there.  However, most of this is determined well before we all board airplanes to travel to UDS.

Brainstorm is constantly collecting and ranking suggestions from Ubuntu users.  Ubuntu development teams hold public meetings on IRC where they discuss ideas and plans.  Canonical stakeholders submit requirements for their needs.  All of this information is aggregated, sorted, evaluated and prioritized, largely by the heroic engineering managers at Canonical, who then develop the core of the agenda for UDS.  Additional sessions are then added as they come up during the week, when there is space.

At this particular UDS, I am moderating the server track, where we’re hashing out the details of our projects for Ubuntu Server Edition 10.04.  Being a UDS track moderator makes for a very busy week, with back-to-back sessions all day for five days straight.  It’s only Wednesday, and I’m feeling a bit fried already, having been away from home for over two weeks.

In each session, there is a discussion between the developers working on the project, the other UDS attendees who are interested in it, and any random folk who listen in on the audio stream and add questions or comments via IRC.  The participants take notes using Gobby and then publish them in the Ubuntu wiki, where they are developed into specification documents tracked in Launchpad.

Those specifications are further broken down into work items, which we can use to maintain a burn down chart.  Rick Spencer, our desktop engineering manager, gave a presentation this afternoon about how that process will work.  The burn down chart will give us a tool for establishing whether we are on track to complete our work, or if we are under or over committed, and make adjustments to our plans as needed.

I have a sense of tremendous momentum going into this release cycle, which will culminate in our third LTS (long-term support) release of Ubuntu.

In the 9.10 release of Ubuntu Server Edition, we introduced something new for people who are exploring cloud computing using Amazon’s Elastic Compute Cloud (EC2).

If you haven’t tried it yet, EC2 is essentially an API for managing virtual servers. Using a command line tool or script, you request a new VM, and moments later, it is ready for you to ssh in, preconfigured with your ssh public key. When you’re finished, you shut it down, and receive a bill for only the time and Internet bandwidth you used (about $0.10 per hour and $0.10-$0.17 per gigabyte). There is no downloading, no installing, and no hardware required for you to set up a server. The first step is to boot it up.

Starting with release 9.10, every release of Ubuntu Server Edition is simultaneously available on EC2. This means you can have a new Ubuntu server up and running using your EC2 account with a single command. Ready-to-run Ubuntu machine images are published on EC2 whenever a new Ubuntu release or milestone is available. All you need to know is the AMI, a short string which uniquely identifies the image you want. The AMIs for the 9.10 release are on the download page, in 32- and 64-bit versions for US and Europe zones.

Similarly, all Server Edition development builds are available on EC2 as well. When the first builds of Lucid (Ubuntu 10.04) are created, there will be AMIs for those as well. If you want to test drive a new feature, or check compatibility with your application, just fire up a new instance on EC2, do your work, and then terminate it. The whole process can be completed in less than a minute. If you find a problem in our development builds, just run ubuntu-bug on the virtual machine as you normally would, and apport will automatically attach the relevant EC2 details to your bug report.

As I mentioned above, EC2 does charge for Internet bandwidth. It does not charge for local bandwidth within your EC2 zone. For this reason, Canonical has set up Ubuntu archive mirrors within EC2, so that you can download all Ubuntu packages and updates for free. Ubuntu virtual machines inside EC2 are automatically configured to use the appropriate mirror, so you don’t need to think about it.

This is an exciting step forward in making Ubuntu more convenient and powerful to use on EC2, and I encourage you to give it a try. If you’ve never used EC2 before, just follow our Starter’s Guide to get set up.