If I were looking to sneak up behind someone pumping gas and hold them up or steal their car, I’d definitely do it at a gas station with NBC At The Pump television screens embedded in each island. I spent about 5 minutes at the Natick Service Plaza in the slack-jawed thrall of one of these bastards—completely and utterly oblivious to my surroundings—before I realized what was going on. [Photo credit: Adam “Pass the Goddamn Butter” Lisagor.]

It has been six months since my last post. So much has happened since then that I have wanted to tell you about.

And I tried!

Oh, I tried, several times—I have the WordPress drafts to prove it. But somehow anything too large for a tweet ended up a five-page essay with footnotes.¹ (I would have written less, but you know how it goes.)

So. Let me be very brief:

• I finished my doctoral studies and got my degree. You can read my thesis if such things interest you.

• My son turned one year old. He is awesome.

• My wife, The Boy, and I now live in the Boston area. We love it here.

• I now work at Google (with some dear old friends) on the Android operating system. It’s pretty exciting. There is a lot of very hard work yet to do.

• I am still interested in research topics, including distributed systems, security, and social networks. I also continue to maintain my Mac software on the side. These pursuits have taken something of a back seat to the above bullets, however.

Whew.

Now I can start making notes (of the longer-than-140-character variety) again. Thanks, Internet, for your kind indulgence.

Abstract

People are suddenly concerned about @reply spam on Twitter. It’s not here yet, exactly, but it will show up—along with its worse cousin, search spam—and then we’ll be in trouble unless Twitter stops showing us messages from strangers quite so readily.

Here comes everybody else

Via Scoble, Loïc Le Meur last week sounded the alarm about Twitter @reply spam:

It had to happen. I just got my first @loic spam and there is no way to filter it. Spammers got it. It’s very simple for them, they just tweet your @username, show up in your replies timeline and bingo that’s the one you pay so much attention at so you won’t miss it. There is no way for you to filter it. None. Yet.

So, it’s true: anyone can create a message that appears in your “Mentions” view, just by, er, mentioning you. Let’s see what that means, and if it’s really that dire.

Asking for it

First, let me say that I don’t think spam will be the death of Twitter; all complex ecosystems have parasites, and spam is a sign that Twitter is a success. In fact, abuse of a public communication system is like crime (or terrorism) in the physical world: you can’t eliminate it entirely, but you can reduce it from a crisis to a nuisance.

This has been a bitter struggle in the world of email, which was forged in the good old days of mutually-trusting research, military, and corporate networks and is therefore every bit as happy to deliver you a message from a deposed prince of Nigeria or vendor of pharmaceuticals as it is from a close friend or colleague.

Twitter, on the other hand, is remarkably resistant to abuse because of its subscription-based information flow: users get only what they ask for. In general, you don’t see messages from strangers, so there’s no way for spammers to get arbitrary text in front of your eyeballs.¹

No way, that is, unless you’re in the habit of routing around the subscription system and explicitly looking for messages from strangers. You can do this in one of three ways:

1. Viewing the public timeline (which nobody actually does);
2. Searching across Twitter; or
3. Looking at your mentions (messages containing @username, which—when they were confined to messages beginning with @username—used to be called “replies”).

So far, Twitter spam has been safely held at bay—a nuisance, not a crisis. Mention/reply spam is just not that common. Poking through my handy Twitter data set from last September², I didn’t find much in the way of @replies that obviously looked like spam.³ I admit to being surprised that this kind of spam isn’t more common, but I think my colleague Mike has hit upon the reason: you can’t reach very many people at once. There’s a finite amount of space available in a tweet, and the spammer has to divide it between recipients and payload. Assuming the average twitter nickname is 9 characters⁴ (plus a space and an @) and the average TinyURL is 25, a single spam tweet⁵ can target at most 140 – 25 – 1 = 114 / (9+2) = about 10 recipients.

Rate limiting by Twitter caps the number of messages that can be sent out by a particular user to just over one per minute. As a result, in order to reach a million Twitter users, a spammer would need to send out 100 messages/hour, each mentioning 10 users, from 1000 junk accounts. This is certainly practical, although not quite the same scale as botnet-fueled email junk. As far as I know, it’s not in common use just yet (although there exist apps like TwitterHawk that occupy a gray area between obviously legitimate and spammy use of Twitter).

Twitter search (and spam) for everyone!

The most effective vector (most eyeballs; fewest effort) attacks user behavior #2 above by spamming search terms; when Twitter launched its election.twitter.com live feed, it wasn’t long before people figured out that this was a soft target. And URL shorteners obfuscators exacerbate the problem by making it very difficult to know whether to click a URL provided by a stranger in search results.

This is where I think we may now be in for some real trouble. On Thursday, Twitter added real-time search (including a smattering of trending topics) to the main web interface. In a blog post entitled Twitter search for everyone!, Biz Stone explains:

What was that loud noise outside your apartment? Did you just feel an earthquake? What do people think about your company, your product, or your city? With this newly launched feature,⁶ Twitter has become something unexpectedly important—a discovery engine for finding out what is happening right now.

Twitter teaches us new and amazing things every day and a big lesson learned is that search is so much more than a box and a button. As public tweets fly in from around the globe, we analyze them to detect when certain words or phrases occur with higher frequency. These trending phrases are surfaced in the Twitter home page just under the new search box and they’re updated throughout the day. Built on our search technology, trends are a compelling if rudimentary way to explore a collective global consciousness.

Real-time search, and its potential to rapidly spread very, very current information (whether accurate or not, and usually lacking in analysis) is without a doubt one of the most interesting⁷ features of Twitter. By choosing to feature it so prominently in the main interface, Twitter has dramatically raised the profile of search results, and therefore considerably upended the economics of search spam (in favor of spammers).

A plan for Twitter spam?

Loïc proposes a solution (that applies equally well to search spam and @reply spam):

… add a “report as spam” button. This button would report the spammer to Twitter (or to a separate database of users) that we could exclude from the clients after a sufficient number of users report them as such and maybe some manual checking. Twitter could then just delete those users.

Others propose creating a large central tweet-filtering system akin to Akismet, the blog comments spam clearinghouse.

The problems with these approaches are manifold. Blocking/reporting suspicious users doesn’t scale (requiring users to take the time to manually identify and report junk) and opens the door for abuse (how many spurious reports would be necessary to get a legitimate user canned?). The Akismet approach—content-based filtering—continues to be an arms race, and even when you have access to millions of different users’ messages (as Twitter would, and Gmail does) it’s still hard to know what’s legit and what isn’t. And even if you take a guess (er, a statistical inference), do we now have to create a “Junk” folder for each Twitter user, including the option to “Report as not spam” to move things back into the inbox? I already have one overgrown garden to tend, thanks.

Flees a crowd

Security geeks are notorious for complaining about others’ proposals without offering a better one. Here’s my take: I think the only long-term, scalable approach is to remove “strangers” from common views of the system, forcing the user to take extra steps to go beyond his own subscriptions. In this world, the only junk most users will ever see is junk from friends and family, which can safely be classified as “not spam.” The mentions/replies view, which has become an important Twitter inbox (particularly for users following more than a handful of others), would show a subset of your main view—messages from random users don’t belong in either place. I believe this is in the spirit of how most individuals use Twitter: primarily, to stay in contact with a limited number of friends and colleagues, and only secondarily to engage the public at large.

So how do you meet new people on Twitter if you can’t see their messages? First, it should be possible to see tweets from strangers—just not the default. A “show messages from everyone” checkbox in the mentions or search views ought to suffice. Second, Twitter users are pretty good at passing along interesting stuff (via retweets), so there still ought to be plenty of cross-pollination between circles.

Finally, Twitter can exploit the social graph to show us messages from friends-of-friends (or friends-of-friends-of-friends, and so on, to any desired depth). Messages from users with some indirect connection to me are still unlikely to be spam, and moreover, if the goal is to discover new users I’m most likely to be interested in people connected to my existing contacts (crowdsourcing the task of crowd-finding, if you will). The CS community is really starting to get into this sort of thing, and although I haven’t yet published my work along these lines, it is this approach that I plan to take in FETHR, my proposed distributed microblogging platform.

Somewhat ironic postscript

As usual, blog comments and Twitter comments are open below. Of course, culling responses from Twitter relies on searching the entire unfiltered public timeline, which I just burned about a thousand words inveighing against. As I mentioned when I started the experiment, this is easily abused; for now I’m explicitly courting tweets from strangers, and we’ll just have to see how long it lasts.

The FETHR slide deck has been particularly well-received. In it I compare microblogging in 2009 to email in 1983, an analogy which seems to resonate with people (at least, people of a certain age); I think it succinctly summarizes where we are today and what needs to happen in order for microblogging to become, in fact, a communication utility alongside email, IM, blogs, and so on.

FETHR, Laconica, and OpenMicroBlogging

I’ve gotten a lot of useful criticism and feedback as well. Several people asked whether I was aware of the open-source Laconica project and how what I’m doing differs. In short: yes, I’m aware of Laconica and OpenMicroBlogging (OMB). I started my work in April ’08, and when Laconica launched in July I was gratified to see someone else pursuing open microblogging.

I think FETHR and OMB are cousins. Evan Prodromou, the creator of Laconica and the owner of identi.ca (a Twitter-like multi-user microblogging service) and I both see the same need: for an ecosystem of microblogging systems that seamlessly² interoperate. Each is a RESTful protocol designed to be used exclusively over HTTP; participants are uniquely identified by a canonical URL, which is used as a rendezvous point for API calls.

Substantial technical differences exist between the two protocols, however:³

• Message distribution. A lesson I took away from my MS work on RSS feeds is that being popular is a curse: you have to satisfy all those hungry new readers. Unlike RSS, FETHR (and OMB) are push protocols, so those readers aren’t periodically making (redundant, useless) requests, but a popular microblogger (e.g. Heather Champ) might still have to make half a million HTTP requests every time she wants to post a message. FETHR addresses this problem by allowing a publisher to ask subscribers to assist with message dissemination by gossiping updates among one another.

• Security. Each applies security techniques to prevent abuse, particularly spam (messages from sources the user isn’t subscribed to). OMB uses OAuth⁴ to secure individual connections, but this won’t fly in a true p2p environment like FETHR (in which you might receive a new message from someone other than its author). FETHR therefore secures data rather than channels; that is, individual messages are signed and hash chained in order to authenticate messages and a publisher’s timeline as a whole. (There are some other useful properties of this approach; see §3.2 of the IPTPS paper for details.)

• Message content. Finally, as currently written, OMB is naturally focused on replicating the Twitter experience, including some maximum field lengths set at 140 characters (e.g. the user’s “location”) and specifying the size (96x96 pixels) of avatar images. It expands on Twitter in certain areas (including a sorely-needed “seealso” property to allow a message to refer to some external resource), but these too are fixed in the protocol. FETHR attempts to remain agnostic on these data, choosing instead to specify a bare minimum of necessary properties and allowing applications to superimpose arbitrary key-value data (read: a JSON object). Essentially, FETHR anticipates that application developers will create their own micro-formats⁵ that best describe the type of micropublishing service model that best suits them.

Birdfeeder roadmap

There’s lots of work to do on FETHR, and now that I’ve gotten another large blocking project out of the way, I can resume work on both the protocol and the Birdfeeder prototype. In rough priority order:

1. Protocol docs. Many (including Dave Winer in a comment on Messina’s FriendFeed) have asked for a public specification of the FETHR protocol. I absolutely intend to provide this, although the exact wire protocol (as spoken by Birdfeeder) is something of a moving target at the moment. If you’re desperate, you can suss it out from the source, which brings me to…

2. Refactoring. There’s a lot of work to do on the code; I grew it organically and unpredictably as my ideas about FETHR developed, and as a result, parts of it bear a strong resemblance to a pasta dinner. Aside from general cleanups (more comprehensive tests, documentation, etc.) I’d like to better separate the Birdfeeder front-end from the FETHR transport so that it’s easier to develop new applications without all the HTML.

3. Local Twitter support in Birdfeeder. Currently, Twitter connectivity is handled by a single FETHR node, twittergw, which acts as a gateway between the two networks. (I first described the gateway in a previous blog post.) Long-term, Twitter should support the FETHR API directly; twittergw was created to provide a crude replica of such functionality. I’m running up against the limitations of this approach, however; I now plan to move Twitter support into the Birdfeeder client itself, overcoming the limitations of twittergw as well as facilitate the out-of-box experience for people looking to try out the system.

4. More refactoring. Birdfeeder makes use of the web.py application framework, but it currently uses a number of work threads, which requires that web.py be used as a standalone server. I want Birdfeeder to operate as a stateless CGI, which means moving background tasks out of threads and into special URL handlers (suitable for tickling with a cron job). This would pave the way for…

5. Google App Engine. The above improvements will allow Birdfeeder to run on GAE, which is a big step toward a large multi-user system that non-technical users can sign up (similar to identi.ca). Note, however, that the point of this project is not to create a public microblogging service, but to lay the foundation for many public micropublishing services; still, it can be beneficial to have a public example as an “ambassador” for the technology.

Research

This is just the “development” side of the R&D equation. I’ve got a stack of research topics to tackle as well (some of which can be found toward the end of my slide deck); I expect to pick these threads back up a little later in the summer.

Thoughts? Requests? I’ll leave the regular blog comments open so you can write longer notes than the Twitter watercooler will allow (although tweets are welcome as well).

Subject: [cs-grad-l] FW: Found iPod

I found an iPod in Duncan hall on the third floor. […] If you can identify the color and make of the iPod, I will get it back to you.

This is the standard proof-of-knowledge test applied in lost-and-found situations: Can you describe the appearance of the item in a way that uniquely identifies you as its owner?

In the case of an iPod, this sort of works: there are many different models spread out over several generations (multiplied again by a variety of colors). But what if the lost object had been, say, an iPhone? There are just a few of those, and a betting man (or woman) would guess “iPhone 3G, black” and have just under a 50% shot at it.¹

OK, you could drop it off with AT&T (or O2 or Rogers or whatever) and allow them to find the proper owner via SIM, IMEI, and so forth. But an iPod touch lacks these identifying features, as do many other portable (and lose-able) gadgets.

I argue that there’s still an effective way to identify the original owner, even in situations where the exact specifications of the device are easily guessed by anyone. It’s afforded by the increasing personalization of these gadgets: They may all look alike on the outside, but on the inside they’re totally different.

So, if I were to come across a lost iPod, I would instead issue this challenge:

“If you can identify 5 albums, podcasts, or games on this iPod, it’s yours.”

A couple of weeks ago I explored the idea of using Twitter as the primary forum for blog discussion. Multiple-round conversation is hard to sustain on a personal blog, but seems to appear effortlessly on Twitter, so went the thinking. Many people responded on Twitter (and elsewhere) with their thoughts, both on the general idea and the particular implementation; they were captured and displayed along with the original post using some client-side JavaScript called watercooler.js that I built for the experiment.

Below, I’ll share some of the points raised by those responses, along with some observations and data of my own.

Discussion, I haz it

155 Twitter messages matched either the URL, TinyURL, or TinyURL path-component of the original post as of Friday the 6^th of March, making it by far the most-discussed article ever on this site. For comparison, the closest competitor (using conventional blog comments) is my 2006 gripe about the iTunes 7 user interface, with 86 comments. Of those 86, 49 were actual comments (not trackbacks or pingbacks), of which 12 were mine. Five readers posted more than once—twice, in fact, in every case—and the remaining 43 comments were hand grenades (that is: fire and forget).

First: this is a tremendous level of interest for little old dsandler.org. I don’t have clear numbers about the readership of my blog—let’s put it at around 400¹—but it’s on the same order of magnitude as my Twitter subscribership.² So we can say that the post had roughly the same audience on Twitter that it would have had otherwise, and yet, it received twice as many responses as the reigning champ on the blog.³

I believe that the ease of responding via Twitter, rather than

1. clicking through to the blog (if reading it via RSS)
2. finding the comments form
3. entering a bunch of orthogonal information (your email address, name, some sort of anti-spam challenge)

before finally being able to enter a comment, was indeed conducive to discussion. Additionaly, Twitter facilitates the second hop (that is, reaching readers of my readers) thanks to the ease with which messages may be forwarded along (retweeted in the lingo).

Digging into the data

One of the concerns raised by several commentators was that, while potentially valuable for increasing the reach of a blog post, these RTs would clog the discussion. Being an ABD doctoral student in computer science, my conditioned response is to disprove this hypothesis with a graph:

Fig. 1.   Classification of Twitter replies over time.

See that big spike (Figure 1, at T+24 hours)? That’s when John Gruber mentioned the experiment on Daring Fireball, which proliferated the idea much more widely than my own immediate circle of attentive contacts (the spike at T=0). In fact, the orange retweet bar makes its first (and nearly only) appearance at T+24; in all, retweets made up only 3% (51 messages) of the weeklong discussion. @replies, a robust indicator of discussion rather than simply undirected opinion, represented 39% of the traffic.

So who was doing all this talking? Well, I was certainly responsible for a lot of it (more than a quarter), particularly at the beginning, where I was responding individually to responses from my friends. As mentions of the post broadened, there were fewer questions, so I had fewer responses (see Fig. 2).

Fig. 2.   Author (28%) vs. audience (72%).

Much of this was multiple-round discussion; 13 users had two tweets, several had 3, and some offered as many as 6 or 7 responses.

Which way to the forum?

Beyond simply sparking conversation (easy enough to do with a single Twitter message), I wanted to be able to discover that conversation and point to it from the blog. It’s not a discussion if you can’t join in, and without a way to link all the relevant tweets together, there’s no way to collect them on the blog for easy perusal (which would be no better than having no comments facility at all).

In short, this technique needs a rendezvous point. The most obvious Twitter-compatible approach is to require respondents to embed in their messages a unique string that I could send to search.twitter.com to collect all the relevant messages at any time. I chose the TinyURL of the original post (later simplified to just the path part, d2nrsv) for two reasons:

1. Anyone making mention of the blog post in a Twitter message but oblivious to the conversation protocol would still have his message included in the conversation.
2. It’s unique across any site using the same technique.

So did I actually pick up any random messages (case #1 above)? Figure 3 tells the story:

Fig. 3.   TinyURL: fragments vs. full URLs.

Indeed, there were plenty of messages that included the full TinyURL. Those that did not either inferred the protocol from the messages of others, or used the “click here to comment” link at the end of the blog post (which used the Twitter ?status= query string in order to pre-populate the input box with the “d2nrsv” fragment, like so).

Guessing what the “d2nrsv” meant, however, wasn’t easy; in general, this approach suffers badly from a lack of self-documentation. It might have been better to satisfy goal #2 (globally unique) by generating a symbol of the form #dsandlerdotorg_twittercomments⁴ that has some hope of being deciphered. A related approach might be to create a Twitter account for the blog and require discussion of the form “@dsandlerdotorg twittercomments Nice idea!”—a two-part hierarchical conversation identifier.⁵

It’s also worth noting that relying on TinyURL introduces fragility; such an external dependency can go down or change its behavior at any time, with huge impact to this technique. (Twitter has the same fate-sharing problem with TinyURL, which is why I’ve advocated for some time that Twitter needs to run its own URL-shortening service.)

Perhaps the best approach, as in so many situations, is to be liberal in what is accepted: search for (1) the TinyURL, (2) some sort-of-unique keyword referring to the post, possibly (3) directed as an @reply to a Twitter account for the blog. Maybe the unadorned @replies to the blog’s Twitter account (i.e., those without other identifying tokens) could be automatically associated with the latest prior post at that time. I’ll probably try something along these lines in my next revision of the watercooler code.

Other important considerations

All this hassling over exactly which nonce to include in responses avoids the very real problem (again, pointed out by many) that to include a tag burns precious bytes, of which Twitter message text may only use 140. Of course, there’s plenty of metadata outside those 140 bytes that is nevertheless associated with each tweet (such as time, author, status ID of an antecedent message, and so forth), so we can imagine that an in_reply_to_url datum might not be an unreasonable addition to the overall structure. ⁶

A number of people mentioned that sharing fate with Twitter search is also potentially problematic. It might not survive as long as your blog, or its API might change (forcing you to upgrade your blog to recover the comments), or it might start returning inconsistent or undesirable data (for example, truncating result sets during peak load). More generally, there was a general sense (with which I agree) that while Twitter might be a really interesting way to discuss something, if you want to collect that discussion in one place, you really ought to be copying it there. This has the unfortunate side-effect of obviating the cleverness of the watercooler.js implementation, which is entirely client-side and therefore puts plenty of load on Twitter but none on the original author’s blog. A future implementation should create a permanent local copy of all relevant public tweets, so that they persist as long as the antecedent article does.⁷

Once more unto the data, my friends

Finally, if you’d like to see what you can make of all this, I’ll save you the trouble of fetching the data from the Twitter Search API: here is JSON for the tweets used to produce the graphs above. If you see any interesting trends in there that I didn’t catch, please share it with the class; as before, include in your message the TinyURL path, dc946u or full URL.

Blog comments don’t really encourage robust discussion. (The only people who look twice at the comments are the original author and readers with an axe to grind.) But ad hoc multi-party discussion does happen on Twitter. I’m experimenting with promoting Twitter to a first-class blog comments system.

No Comment

I’m not the first to notice this, but blog comments are awkward. Many excellent blogs have eschewed them for various reasons; Jason Kottke doesn’t want the babysitting chore of dealing with a large, vocal community. Alex Payne doesn’t think good conversations emerge from blog comments because they encourage “hit-and-run attacks, unintelligible ramblings, and truckloads of spam.” John Gruber simply prefers a streamlined reading environment.¹ It’s not that they’re uninterested in discussion; rather, they’d prefer it happen elsewhere:

1. On your own time and property (viz., on your blog), or
2. In some neutral forum (Reddit, Twitter, etc.).

Option #1—the promise of a blogosphere criscrossed with scintillating conversation—is the reason TrackBack exists: without it, the author and readers alike have no way of knowing where on the web to find the next response. Unfortunately, because it allows untrusted parties to post links and content to your blog automatically, TrackBack is an enormous spam magnet and has therefore fallen into disuse.² A number of ad-hoc techniques replace it, exemplified by the convention of promulgating choice responses (received out-of-band, e.g. by email or IM) by appending them, simply, under the heading UPDATE.

A few conscientious bloggers—even those that, nominally, want you to Get Your Comments Off My Lawn—carefully comb their referrer logs for links to well-considered responses from other bloggers, which they will then compile into a “Responses” post. Twitter hacker Alex Payne, for example, recently rounded up the rebuttals to his “ranty”³ article of a few weeks prior. It is his claim that, because each of those bloggers chose to respond on his own blog—taking full responsibility for his opinions under his own masthead and in front of his own readers, rather than leaving a comment “off the record” on a distant site—those perspectives were more thoughtful and more clearly articulated.

This leaves us with option #2: Comments in the Neutral Zone. Assuming you have a high-profile discussion-free blog⁴ and also read Reddit, this works pretty well: your adherents and detractors have a sandbox in which to have it out, and you can drop in occasionally to poke the discussion. Unfortunately, the stars must align for this technique to result in interesting conversation; the likelihood is that some essential piece of the conversation—either author or readers—is not looking at your shiny new Reddit thread.⁵ In fact, the only thing that ties the group together is the original blog post, which puts us right back where we started: blog discussion, it would seem, must occur—or at least appear—on the blog itself.

But this is still insufficient, because while the ideal discussion group for a given blog is defined as a blog’s readership, we have only solved the problem in space, not in time. How often do you check the same blog post to see if there are exciting new comments? As a reader, I hardly have the time to leave a single comment, let alone check back to see if there are interesting responses. I find this to be true of readers of dsandler.org as well: it’s fruitless for me to respond to your comment with one of my own, because you’re unlikely ever to visit this post again to read it.

This is what I think of when Alex Payne refers to “hit-and-run attacks”—a natural side-effect of the fact that most readers will look at a blog post (and the current state of the discussion) at most once. The only people who look twice at the same blog entry’s comments are the author (who has the luxury—or compulsion—to respond to each and every one) and anyone with an axe to grind. All others, if they comment at all, will leave a single comment and then never return. This is not conducive to discussion.

And yet, discussion happens

I’m increasingly getting feedback about blog posts on Twitter (with good reason: I mention the post on Twitter myself, explicitly inviting a conversation). For example, when I finally deployed the recent redesign of dsandler.org late last year, I mentioned it on Twitter (where a few people had already been offering complaints about my intentionally awful interim design). I received a few comments on the blog post itself, but just as many via Twitter, so I decided to (laboriously!) copy some of those messages into the text of the post (see UPDATE).

Wouldn’t it be nice if such a thing were done automatically? For bloggers whose controversial articles (picking on @al3x again here) spawn a lot of Twitter discussion, could we actually promote Twitter to a first-class discussion mechanism?

See below

Yesterday I threw together some code to go search Twitter for any messages including the URL of the current page and display them as if they were comments. I had considered creating a unique keyword for each post on the fly (possibly using a hashtag, like “#dsandler_post_5″) that would string the conversation together, but the URL seems more natural (because people will put that into responses anyway, and besides, the TinyURL version is usually every bit as short as a hashtag).

Blindly including the results of a search on your blog is an easy way to get spammed, of course,⁶ so we’ll have to see how this goes. Note, however, that because the tweets are fetched by the client, they never actually appear in the blog when it’s crawled by search engines, so you can’t mess with my PageRank (or boost your own) by abusing this system—all you can do is cause grief. (So. Don’t.)

So, let the experiment begin: follow the link below to pre-populate a Twitter post about this entry (or simply include the URL or TinyURL in your message). Try to include the (Tiny)URL in any tweets that have to do with this post, insofar as that’s practical.

Oh, and: regular blog comments are turned off.

Update

5 March: I’m working on a new post to document the results of this little experiment. For now, I’m turning off the live Twitter search results and ossifying the state of the conversation as it stands today by including the results statically below.

I’m no expert on sneaking things through airport security, but apparently I’ve been endangering passengers for a couple of trips now with my dangerous prohibited items.

TSA agent: Sir, is this your bag?

Me: Yes…?

TSA agent: What do you plan to do with your Swiss Army knife?

Me: [immediately flushed] Uh…leave it here, I guess.

Indeed, my tiny pocket knife (a gift from Jeremy—well, he was throwing it away and I salvaged it it—visible at left in this photo), was in my bag, and the TSA agent had spotted its tiny profile on his screen. It had been living in the bottom of my pencil/pen case for quite some time; it tucks away there perfectly—being smaller than most multitools—and I can’t count the number of times it’s come in handy for:

• Gently prying apart seams on electronics (using the large blade)
• Extracting stubborn jumpers (w/ tweezers)
• Clipping/cutting clothing tags, zip-ties, twist-ties, etc (scissors)
• Assorted screw-driving
• Other tasks that require both urgency and a small blade: breaking down boxes, opening envelopes, dispatching errant threads

(Apparently, Victorinox could just leave off the nail file.)

I really had forgotten that I’ve been carrying it there—in fact, I hadn’t been able to find it in the last few weeks, so I wasn’t entirely sure where it was. (This prompts me to consider a new use for the TSA screening process: locating lost items. “Uh, mister X-ray technician, can you tell me if my flash drive is in there somewhere?”)

Conclusions:

1. BWI TSA agents are careful and thorough.
2. IAH, not so much.
3. This corroborates the findings of the Atlantic article: preventing contraband from entering an airplane is, at best, probabilistic, like any kind of random audit.
4. I am not a terrorist—and what’s more, I didn’t even realize I had the knife with me—so I can guarantee that the plane was no safer without my particular contraband than with it.
5. If, on the other hand, your goal is to confiscate knives, then that’s pretty much the best way to go about it.
6. I guess I’m in the market for a new pocket knife.

Fig. 1. A common sequence of tweets this morning.

Quick explanation of today’s “Don’t Click” attack

This viral Twitter prank uses a pretty standard trick to get you to post something to Twitter, using no JavaScript and only a very little bit of CSS. It requires only that you’re first logged in to twitter.com (and that your browser’s fonts are pretty close to standard so that things line up correctly).

Fig. 2. The “Don’t Click” page.

The attack page creates a button labeled “Don’t Click” that does nothing at all, but it also loads twitter.com in an <IFRAME> directly on top of the button. That IFRAME is then made completely transparent using CSS.

Fig. 3. The hidden IFRAME, exposed.

When you click the button, you’re actually clicking on the (now invisible) ‘Update’ button on Twitter’s web interface instead; assuming you’re logged in to Twitter, you’ll immediately post whatever’s in the form input box. Thanks to Twitter’s ?status= URL feature (that allows Twitter to be pre-loaded with a message), it’s very easy for the attacker to drop the following message in:

Don’t Click: http://tinyurl.com/amgzs6

That TinyURL expands to http://www.umoor.eu/blog/yes-we-can.php, the attack page; in this way your followers are also enticed to propagate the “attack” (which has proven quite successful).

No accounts were compromised by this prank; you don’t need to change your Twitter password. For more information, check out the source to this page, which includes the attack (but makes the Twitter IFRAME partially visible so you can see it; it also puts another <DIV> on top of it to protect you from accidentally clicking the “Submit” button).

See also: an explanation (en Français) by the author of this attack, originally launched in January 2009. (There is some speculation that the code was taken directly from James Padolsey’s proof-of-concept, owing to the similarity of the code.)

As of 1:40PM EST, twitter.com is using some JavaScript to try to detect when it’s being loaded in an IFRAME:

if (window.top !== window.self) {
    window.top.location.href = window.self.location.href; }

If you try to load Twitter in an IFRAME you’ll find the browser redirected to Twitter automatically (in this case, you’ll never have an opportunity to click on “Don’t Click”). I’ve therefore removed the attack from this page, but you can still see it if you view source.

Fig. 1.

The microblog is something of an odd bird, defying easy classification or even explanation (as evidenced by the many attempts to articulate its purpose and properties). Originally conceived as a way for strongly-connected friends² to keep track of one another’s whereabouts and present activities using SMS, it’s become a kind of ultra-lightweight conversational tool that doesn’t map exactly onto any existing publishing or communication system. Twitter’s single prompt—“What are you doing?”—is almost vestigial at this point; it is evolving into a micropublishing platform, and as such has become an important and interesting topic of research.

Microblogging has already had impact. We can speculate—from the amount of news and media coverage that the phenomenon has earned³—that it is poised to take its place next to the blog as a prominent method of publishing and interacting online. Many now rely on services like Twitter for business and personal interaction; beyond the “ambient awareness” of physically distant friends and neighbors, microblogging now finds use in business networking, customer service, national politics, journalism, and general lazyweb⁴-style requests. Microblogging has unquestionably become a part of the everyday lives of active users, and in extreme situations—such as the recent Sichuan earthquake—it can literally be a lifeline.⁵ Encouraging and improving these services is therefore an important and valuable goal.

As a computer scientist, I’m interested in microblogging systems in part because of their unique properties. They are remarkably spam-free, mostly due to the way in which users explicitly select those senders whose messages they wish to see (by “following” them). The many subscription links between microbloggers forms an interesting social graph: can we put this network to use in some way?⁶ Microblogging is also an unusual mode of communication, falling somewhere between blogs, chat, IM and BBSes in terms of how, when, and to whom messages are distributed. Even the way users consume messages is peculiar: a constant stream of updates commingling their own messages with those from their friends, ensuring that no two users have the same view of the system.⁷

Microblogging is every bit as compelling as research because of its limitations—specifically, the limitations of the flagship microblog service, Twitter. First, it is a large but entirely centralized system; there are currently a few million registered user accounts⁸, of which maybe half a million are active⁹. How much larger can it get? The folks at Twitter are, by all accounts, barely keeping up with their own success; in fact, scaling problems have been at various times the subject of much public frustration. As a result, Twitter is also fragile; its users are all unavoidably bound to Twitter’s robustness and reliability. When Twitter goes down, service is completely interrupted for everyone. It is a closed system, by which I do not necessarily mean that the source code is not available, but rather, the ways in which the system functions are not up for debate or (easy) amendment by the community.¹⁰ Twitter, Inc. is a dictator, albeit a benevolent one, and while users may wish to switch to another service—and there are plenty of Twitter-alikes to choose from, many promising more advanced features or better reliability—powerful network effects prevent users from leaving. After all, everyone you know is already on Twitter, and they’re not on, for example, Pownce.¹¹

The potential is certainly there for Twitter to become, as its founders style it, a “communication utility”; whether or not Twitter can actually achieve this aim depends in large part on its technical evolution. That’s where my recent work fits in the microblogging timeline; watch this space.

The day after Thanksgiving, November 28, has been declared National Day of Listening by the StoryCorps oral history project and NPR (among others). The idea: With family in town, bellies full, and (hopefully) a little time off work, we might all take an hour to sit down and tell each other stories. An essential part of the NDoL is to record those stories, because they probably don’t already exist on blogs or email or Twitter—particularly if the storyteller isn’t of the Internet generation.

We realized this is a perfect use for TapeDeck, our fast, fun, and foolproof audio recording software for Mac OS X. So this week we’re taking 20% off the price of TapeDeck with the hope that you’ll use it to capture your family’s stories this Friday. Just download TapeDeck and press the big red REC button to get started. By purchasing a registration you’ll remove TapeDeck’s time limits (so you can record for as long as you like); do so before Sunday, November 30 to get the discounted price.

We sincerely hope this helps you and your family start and preserve your own oral traditions. Happy Thanksgiving!   —Dan & Chris

I went to post a remark about this article from yesterday’s New York Times on the computer science gender gap (particularly how this graph shows how the tech boom utterly failed to inspire young women) and found that Twitter was down:

Actually, that’s a lie; at first, I didn’t notice anything, because this is what I see when I use Twitter:

This is a screenshot of Birdfeeder, a prototype implementation of a distributed and secure microblogging protocol called FETHR, which operates independently of (but connects to) Twitter. I posted my comment to a personal instance of Birdfeeder, which happily accepted it, digitally signed it, entangled it with other messages in my timeline, and forwarded it on to my FETHR subscribers—one of which is a Twitter gateway, which takes care of forwarding my messages on to my Twitterstream. It’s also responsible for sending me messages from people I follow, so (as you can see from the screenshot) I never have to leave my Birdfeeder interface—and therefore can occasionally miss minor Twitter glitches. (Or, as in this case, major hour-long outages. Nothing yet on the status blog, either, so it must really be all-hands-on-deck over at Twitter HQ. Update: Over an hour in, there’s a small note on the status blog.)

When Twitter eventually comes back up, the gateway will busy itself with the task of forwarding along my queued messages and fetching news from my Twitter subscriptions. I can continue to tweet in the meantime and even page back through my entire history and the archived messages of my friends. This is the fundamental benefit of decentralized micropublishing: independent providers may experience local failures without bringing the whole damn thing to a screeching halt.

So, in something of a twist on the usual, Twitter is down for everyone—but not me.

This is fun and exciting new research (currently under submission) and I hope to push out a technical report version soon so that others may begin to evaluate and improve the system design. Hit me at dsandler@ (or @dsandler, once Twitter wakes up again) if you can’t wait and would like to know more.

Photo credit: Doug Mills / The New York Times

So I’m firing up my first draft of a long, long, long-awaited dsandler.org refresh. At first glance it probably looks like just another tightly-tracked Helvetica design; while I do believe that that particular Max Miedinger Old Style look is timeless, I’m actually going for something a little different. Font nerds with Avenir (or Gotham, a slightly boxier substitute) installed will see more of what I’m going for: something a little more open and friendly. [Update: See a screenshot of the Avenir version.]

Feedback is welcome in the comments or shouted to @dsandler.

Twitter feedback

al3x @dsandler I’d click it.

boredzo @dsandler Much improved; thank you.

dangerdave @dsandler v. pretty. But the Avenir family is expensive: I can’t afford Sandler Platinum.

davidrperry @dsandler new layout = elegant & clean. I did get a kick out of the temp version, though!