waving android

I am currently a software engineer at Google, where as a member of the Android platform team I build frameworks and user interfaces.

The blog here at is mostly historical; you can find more recent posts on .

On Twitter as a research problem

December 8th, 2008

Recently, I’ve spent quite a bit of time thinking about microblogging, exemplified of course by the very popular Twitter service. While I finish preparing a technical report describing the FETHR system we’ve developed1, I thought I’d take a few minutes to discuss why I find this topic compelling.

The microblog is something of an odd bird, defying easy classification or even explanation (as evidenced by the many attempts to articulate its purpose and properties). Originally conceived as a way for strongly-connected friends2 to keep track of one another’s whereabouts and present activities using SMS, it’s become a kind of ultra-lightweight conversational tool that doesn’t map exactly onto any existing publishing or communication system. Twitter’s single prompt—“What are you doing?”—is almost vestigial at this point; it is evolving into a micropublishing platform, and as such has become an important and interesting topic of research.

Microblogging has already had impact. We can speculate—from the amount of news and media coverage that the phenomenon has earned3—that it is poised to take its place next to the blog as a prominent method of publishing and interacting online. Many now rely on services like Twitter for business and personal interaction; beyond the “ambient awareness” of physically distant friends and neighbors, microblogging now finds use in business networking, customer service, national politics, journalism, and general lazyweb4-style requests. Microblogging has unquestionably become a part of the everyday lives of active users, and in extreme situations—such as the recent Sichuan earthquake—it can literally be a lifeline.5 Encouraging and improving these services is therefore an important and valuable goal.

As a computer scientist, I’m interested in microblogging systems in part because of their unique properties. They are remarkably spam-free, mostly due to the way in which users explicitly select those senders whose messages they wish to see (by “following” them). The many subscription links between microbloggers forms an interesting social graph: can we put this network to use in some way?6 Microblogging is also an unusual mode of communication, falling somewhere between blogs, chat, IM and BBSes in terms of how, when, and to whom messages are distributed. Even the way users consume messages is peculiar: a constant stream of updates commingling their own messages with those from their friends, ensuring that no two users have the same view of the system.7

Microblogging is every bit as compelling as research because of its limitations—specifically, the limitations of the flagship microblog service, Twitter. First, it is a large but entirely centralized system; there are currently a few million registered user accounts8, of which maybe half a million are active9. How much larger can it get? The folks at Twitter are, by all accounts, barely keeping up with their own success; in fact, scaling problems have been at various times the subject of much public frustration. As a result, Twitter is also fragile; its users are all unavoidably bound to Twitter’s robustness and reliability. When Twitter goes down, service is completely interrupted for everyone. It is a closed system, by which I do not necessarily mean that the source code is not available, but rather, the ways in which the system functions are not up for debate or (easy) amendment by the community.10 Twitter, Inc. is a dictator, albeit a benevolent one, and while users may wish to switch to another service—and there are plenty of Twitter-alikes to choose from, many promising more advanced features or better reliability—powerful network effects prevent users from leaving. After all, everyone you know is already on Twitter, and they’re not on, for example, Pownce.11

The potential is certainly there for Twitter to become, as its founders style it, a “communication utility”; whether or not Twitter can actually achieve this aim depends in large part on its technical evolution. That’s where my recent work fits in the microblogging timeline; watch this space.

  1. The work is also currently under submission to a competitive conference; a technical report is useful in this case to gain additional feedback from the broader microblogging community and to spur experimentation in the wild. 

  2. Presumably those in urban areas where “I’m at the coffeehouse” is an actionable piece of data. Some still use Twitter for this kind of lazy real-world rendezvous, but most have adopted a more blog-like approach to the system. One of the noteworthy things about microblogging is that it supports all these modes of interaction simultaneously. 

  3. The New York Times seems particularly smitten. 

  4. la•zy•web n. The collected wisdom of millions of internet users, mythical solver of problems cosmic and quotidian. First supplicated by name in 2005 or thereabouts by @jwz

  5. Could Twitter actually save lives? Andy Carvin speculates that with a few sophisticated group features it might be a tool for mobilizing relief efforts. The work I’m doing would help to enable this. 

  6. Some examples taken from the FETHR paper: abuse/spam detection; recommendations/introductions; and, of course, update distribution. 

  7. In this way it might be said to be most similar to Facebook’s News Feed, introduced to early controversy, due in large part to the dramatically—for some, shockingly—increased importance and persistence of formerly ephemeral bits of Facebook data like the personal status. 

  8. According to TwitDir, which is having load issues as I write this; TwitterFacts reports TwitDir’s estimation of the Twitter community as of October to be just over 3 million. 

  9. Based on our measurements during a three-week period in September, during which we observed 4,917,042 public messages from 472,735 users. 

  10. The Twitter API allows third parties to develop software that talks to the existing Twitter service, including desktop clients, search engines, and, yes, research code such as my own. What’s not possible at this point is to change the way Twitter works; for example, external developers can’t turn off Twitter’s use of TinyURL to shorten URLs (causing an unfortunate fate-sharing between Twitter and TinyURL, which has suffered its own reliability problems). More to the point, the Twitter microblogging network is isolated: there’s no way for a third party to offer Twitter users the ability to follow users of other systems or vice versa. 

  11. Not least because it’s been bought and shut down

16 responses

  1. Matt  

    The monetization of twitter may make or break its utility for some. If the level of intrusion for the average user is low I suspect most people will be okay with it.

    I would prefer to use a decentralized non-commercial service like Identi.ca/Laconica but a mass exodus of interesting people would have to do the same.

    I wrote a short post about it back in August.

    comment posted at 1:51 pm on 11 Dec 2008

  2. dsandler  

    …but a mass exodus of interesting people would have to do the same.

    This gets back to the network effects of Twitter. Any solution that doesn’t accommodate the existing userbase is in serious trouble.

    comment posted at 3:04 pm on 11 Dec 2008

  3. jstewart  

    You should talk to amislove about the social networking aspect, if you haven’t already.

    comment posted at 12:15 pm on 12 Dec 2008

  4. Rod Begbie  

    TL;DR. Can you summarize in 140 characters or less?

    (Also, minor etymological nitpick: Lazyweb predates 2005. I first heard of it via Matt Jones & Ben Hammersley’s lazyweb.org site in late 2002/early 2003: http://www.boingboing.net/2003/01/04/lazyweb-ideas-have-a.html)

    comment posted at 3:23 pm on 16 Dec 2008

  5. dsandler  

    @RodBegbie Fair enough. An answer in 139 chars (including the “why research Twitter?” prompt).

    comment posted at 3:50 pm on 16 Dec 2008

  6. Matt  

    Isn’t the question really, how best to implement a messaging or micro-blogging standard that functions something like Twitter?

    Who is to say that I couldn’t have a 300 character microblog post, while unsuitable for Twitter, another service or hub in a decentralized model would gladly accept.

    Twitter seems to be the most popular micro-blogging service right now but it’s not the only game in town.

    comment posted at 4:30 pm on 16 Dec 2008

  7. dsandler  

    Matt, I completely agree. Twitter (including its restrictions on content) should be considered one special case of the more general decentralized micropublishing problem.

    comment posted at 10:54 am on 19 Dec 2008

  8. Todd  

    “…there’s no way for a third party to offer Twitter users the ability to follow users of other systems or vice versa..”

    Slightly inaccurate. Anyone can construct their own version of Twitter, for almost nothing, using Laconica’s Identica ( http://identi.ca ) and Zeep ( http://zeepmobile.com ). Once up, interop is fairly straight forward using the Twitter API…But that assumes Twitter will support OAuth, which they do not at the moment ( Grrr! )

    comment posted at 12:28 pm on 22 Dec 2008

  9. dsandler  

    Todd: That’s not quite the same thing. You haven’t given a Twitter user connectivity to other systems, you’ve made another system that carbon copies things to and from Twitter. In this case you still maintain two identities and shuttle data between them.

    By analogy: a multiprotocol IM client (Trillian, Adium, iChat, etc.) doesn’t solve the problem of non-interoperability of IM services—it just makes it slightly less insane to use all of them at once. I’m trying to solve the interop problem.

    comment posted at 8:08 pm on 11 Jan 2009

  10. PJ  

    So FETHR:Twitter :: XMPP:AIM ?

    I hope. About time. Though I’m not exactly sure why FETHER != XMPP.

    comment posted at 3:28 pm on 19 Jan 2009

  11. dsandler  

    A very good question. FETHR bears some resemblance to server-to-server XMPP. However, XMPP has a lot of weight I’m looking to avoid (very large specification, streaming XML stanzas, persistent server-to-server connections etc.). FETHR is something you can implement pretty much from scratch in a few hundred lines of code.

    comment posted at 12:14 am on 23 Jan 2009

  12. orb  

    holy christ that (your plot of followers vs. rank) is an awesome power-law. also, happy new year! (hope all’s well with you.)

    comment posted at 10:45 pm on 26 Jan 2009

  13. dsandler  

    Indeed! The Year of the Ox, as I was informed today.

    comment posted at 11:47 pm on 26 Jan 2009

  14. Shriram Krishnamurthi  

    Dan — don’t turn into an architecture astronaut!

    comment posted at 9:57 am on 19 Feb 2009

  15. dsandler  

    Shriram: Don’t worry! The design is quite straightforward; in many places I tried to find the simplest approach that would work in a decentralized system.

    (One small example: using RESTful HTTP APIs and gossip instead of a complex, higher-maintenance P2P overlay that doesn’t play nicely with NATs and firewalls.)

    comment posted at 10:01 am on 19 Feb 2009

  16. links for 2009-03-10 « Where is my towel?  

    [...] On Twitter as a research problem (tags: research twitter communication ****) [...]

    comment posted at 1:02 am on 10 Mar 2009

Add a comment

html help (show)

newer: older: