Building a personal data locker
If you were building a digital container to store your personal data, what would it look like?
Personal data being information associated with you: your contacts, your photos, the web pages you’ve visited, the places you’ve been, the messages you’ve sent and received, and so on. In short, your stuff.
Here’s my personal wish list of technical requirements:
- It has to be made of free software, of course
- It must keep my data secure, while allowing me to share it when and how I want to
- It needs to handle a range of different data types natively, and be extensible to new types, from photos to real-time sensor data
- It should be able to collect my data from many different places where it is being created and stored
- It should have a rich API, so that I can create applications which access my data
- If I want to, I should be able to host it myself, on my own hardware, without compromising my ability to access and share it
Of course, this isn’t merely an academic exercise, as my new day job at Singly is about building exactly this type of system. With a technical team including Jeremie Miller of Jabber and XMPP fame, our goal is to develop a personal data platform which meets these criteria and more.
There’s a lot of work to do, but today, you can check out the code and run a locker of your own, which can sync in data from Facebook, Twitter, Google, Foursquare, Github and dozens of other services. It’s a bit of a bear to set up, particularly if you don’t already have API keys for these services, but that’s fairly normal at this early stage of development.
If you try it, or have thoughts about what we’re doing, please let me know in the comments.
We'll see | Matt Zimmerman 
It would be nice if elasticsearch was installed with the locker so that search is an integral part of the locker.
spara (@spara)
August 15, 2011 at 16:29
Looks like you’ve been reading the code. :-)
A global search of all the data in your locker is very close! http://blog.lockerproject.org/a-deeper-dive-on-locker-search
Matt Zimmerman
August 15, 2011 at 16:38
it will be great if the API are extended to pretty much every single service out there…
also the web interface could improve way more inline frames dose not work in every browser (opera user here)
search it’s almost here but what about the possibility to search from a user id through services.
if i forgot already about my profile in myspace for example searching by user id, email, bits of info can bring the info it’s scatter on the wild west internet space.
that way i can reclaim back my info.
lauratika
August 15, 2011 at 18:13
I’m puzzled that you didn’t include:
– Safe. Backed up and integrity checked; guaranteed to resist the sands of time. Fx 1 bit corruption on compressed files like jpg can lead to big loss – and is a very real issue on conventional drives.
– Available. *Always* remotely available. No upgrade or maintenance downtime for services.
kamstrup
August 15, 2011 at 18:18
Those are definitely important attributes for a data storage service, and we care about that too! I didn’t include them because this particular post was mainly about the technology itself, and what attributes it should have, rather than a service.
Matt Zimmerman
August 15, 2011 at 19:16
It would be nice if it did data deduplication.
Folkert Van Heusden
August 15, 2011 at 19:04
It does!
Matt Zimmerman
August 15, 2011 at 19:14
[…] An open source project that helps one collect one’s personal data, from wherever (will need API keys), and store it locally. locker demo on youtube source on github via […]
locker project at dig
August 15, 2011 at 19:41
I’d add something in there about offline access which implies dealing with concurrent updates and conflict resolution.
Items also have varying levels of importance and immediacy with this mattering because there is not an infinite amount of storage and network available. (Done right it is a pleasant user experience, done wrong it is enough to abandon the whole thing.)
Roger
August 16, 2011 at 09:07
So can I spin up an ec2 instance today and run this? Any readme on doing that?
Ahmed Kamal
August 16, 2011 at 13:20
There’s no prepared recipe for this today, but I would be happy to see one in http://wiki.lockerproject.org/ if you do it!
Our production environment is based on the Ubuntu Server AMIs, so everything should work.
Matt Zimmerman
August 16, 2011 at 17:29
Available from all operating systems. Not only when using a *x box.
mikko
August 16, 2011 at 16:51
Definitely. The idea is that your locker will be hosted somewhere, and you’ll access it through a browser or through applications which access its API.
Matt Zimmerman
August 16, 2011 at 17:29
Free software FTW!! I’m interested in how you could host it yourself easily.
don
August 17, 2011 at 20:30
We’re very eager to create small virtual machine images that have a locker and everything needed bundled to make it easy, but in the meantime there’s some instructions at http://lockerproject.org/ to get anyone ambitious enough up and running :)
Jeremie Miller
August 19, 2011 at 00:37
I wonder what the relationship to the FreedomBox will be…
Simon Richter (@GyrosGeier)
August 18, 2011 at 17:32
I’m a technical advisor to the FreedomBox project, and I hope that the developers working on it will find the locker project very interesting and ask me lots of questions about it. :-)
Matt Zimmerman
August 18, 2011 at 18:31
I’d add this requirement:
It should allow me to remove data from services I no longer want to have my data. And keep removing it, when they “accidentally” put it back.
Lars Wirzenius
August 19, 2011 at 17:06
I think there might be a good opportunity for collaboration with Mozilla’s Open Badges project. http://openbadges.org/ The DML competition includes helping to build the infrastructure: http://www.dmlcompetition.net/Competition/4/badges-competition-cfp.php Seems like there might be some overlap with a digital locker system.
Belinda Lopez
October 11, 2011 at 23:46