dsandler.org/wp

Above: The crowning achievement of my graphic design career (edging out the Slashdot thing).

More →

If I were looking to sneak up behind someone pumping gas and hold them up or steal their car, I’d definitely do it at a gas station with NBC At The Pump television screens embedded in each island. I spent about 5 minutes at the Natick Service Plaza in the slack-jawed thrall of one of these bastards—completely and utterly oblivious to my surroundings—before I realized what was going on. [Photo credit: Adam “Pass the Goddamn Butter??? Lisagor.]

Forgive me, Internet, I have sinned.

It has been six months since my [last post](http://dsandler.org/wp/archives/2009/05/05/twitter-reply-spam). So much has happened since then that I have wanted to tell you about.

And I tried!


Oh, I tried, several times—I have the WordPress drafts to prove it. But somehow anything too large for a [tweet](http://twitter.com/dsandler) ended up a five-page essay with footnotes.[^foot] (I would have written less, but you know [how it goes](http://en.wikiquote.org/wiki/Blaise_Pascal).)

[^foot]: No, seriously.

So. Let me be very brief:

* I finished my doctoral studies and got my degree. You can read my [thesis](/pub/thesis) if such things interest you.

* My son turned [one year old](http://www.flickr.com/photos/dsandler/sets/72157621420922432/). He is awesome.

* My [wife](http://dsandler.org/erinmak/), The Boy, and I now live in the Boston area. We love it here.

* I now work at Google (with some [dear old friends](http://dsandler.org/gruntle/been)) on the [Android](http://www.android.com/) operating system. It’s pretty [exciting](http://droiddoes.com). There is a lot of very hard work yet to do.

* I am still interested in [research](/research) topics, including distributed systems, security, and social networks. I also continue to maintain my [Mac software](http://toastycode.com) on the side. These pursuits have taken something of a back seat to the above bullets, however.

_Whew._

Now I can start making notes (of the longer-than-140-character variety) again. Thanks, Internet, for your kind indulgence.

### Abstract

People are suddenly concerned about @reply spam on Twitter. It’s not here yet, exactly, but it will show up—along with its worse cousin, search spam—and then we’ll be in trouble unless Twitter stops showing us messages from strangers quite so readily.

### Here comes everybody else

Via [Scoble](http://friendfeed.com/scobleizer/c57b1869/watch-out-twitter-spammers-got-replies-and), Loïc Le Meur last week [sounded the alarm](http://www.loiclemeur.com/english/2009/04/watch-out-twitter-spamers-got-the-replies-and-a-solution.html) about Twitter @reply spam:

> It had to happen. I just got my first @loic spam and there is no way to filter it. Spammers got it. It’s very simple for them, they just tweet your @username, show up in your replies timeline and bingo that’s the one you pay so much attention at so you won’t miss it. There is no way for you to filter it. None. Yet.

So, it’s true: anyone can create a message that appears in your “Mentions” view, just by, er, mentioning you. Let’s see what that means, and if it’s really that dire.

### Asking for it

First, let me say that I don’t think spam will be the death of Twitter; [all complex ecosystems have parasites](http://craphound.com/complexecosystems.txt), and [spam is a sign that Twitter is a success](http://twitter.com/dsandler/status/854001752). In fact, abuse of a public communication system is like crime (or [terrorism](http://www.nytimes.com/2004/10/10/magazine/10KERRY.html?_r=1&pagewanted=print)) in the physical world: you can’t eliminate it entirely, but you can reduce it from a crisis to a nuisance.

This has been a bitter struggle in the world of email, which was forged in the good old days of mutually-trusting research, military, and corporate networks and is therefore every bit as happy to deliver you a message from a deposed prince of Nigeria or vendor of pharmaceuticals as it is from a close friend or colleague.

Twitter, on the other hand, is remarkably resistant to abuse because of its subscription-based information flow: users get only what they ask for. In general, you don’t see messages from strangers, so there’s no way for spammers to get arbitrary text in front of your eyeballs.[^EYEBALLS]

[^EYEBALLS]: Except [new-follower notifications](http://twitter.com/dsandler/status/914149020). But because the attacker can’t control the content of this message (beyond [his own username](http://twitter.com/dsandler/status/1156201363)), you (the recipient) have to do some extra work to actually see spam content: click through to the user’s page and possibly on through to some arbitrary URL. This is known as [follow spam](http://blog.twitter.com/2008/08/making-progress-on-spam.html).

No way, that is, unless you’re in the habit of *routing around the subscription system* and explicitly looking for messages from strangers. You can do this in one of three ways:

1. Viewing the **public timeline** (which [nobody actually does](http://twitter.com/nevenmrgan/status/1661651225));
2. **Searching** across Twitter; or
3. Looking at your **mentions** (messages containing `@username`, which—when they were confined to messages _beginning_ with `@username`—used to be called “replies”).

So far, Twitter spam has been safely held at bay—a nuisance, not a crisis.
**Mention/reply spam is just not that common.**
Poking through my handy Twitter data set from last September[^DATA], I didn’t find much in the way of @replies that obviously looked like spam.[^CONTENT]
I admit to being [surprised](http://twitter.com/dsandler/status/1035057424) that this kind of spam isn’t more common, but I think my colleague Mike has hit upon the reason: [you can’t reach very many people at once](http://twitter.com/mdietz/status/1697025157).
There’s a finite amount of space available in a tweet, and the spammer has to divide it between recipients and payload. Assuming the average twitter nickname is 9 characters[^9CHARNICK] (plus a space and an @) and the average TinyURL is 25, a single spam tweet[^SWEETS] can target at most 140 – 25 – 1 = 114 / (9+2) = about 10 recipients.

[Rate limiting](http://apiwiki.twitter.com/Rate-limiting) by Twitter caps the number of messages that can be sent out by a particular user to just over one per minute.
As a result, in order to reach a million Twitter users, a spammer would need to send out 100 messages/hour, each mentioning 10 users, from 1000 junk accounts.
This is certainly practical, although not quite the same scale as botnet-fueled email junk.
As far as I know, it’s not in common use just yet (although there exist apps like [TwitterHawk](http://news.cnet.com/8301-17939_109-10158914-2.html) that occupy a gray area between obviously legitimate and spammy use of Twitter).

[^9CHARNICK]: Based on my data, the average is 8.96 chars.

[^SWEETS]: It is standard procedure to coin an awkward portmanteau for every new type of spam encountered ([spim](http://en.wikipedia.org/wiki/Messaging_spam), [spit](http://en.wikipedia.org/wiki/VoIP_spam), [spamdex](http://en.wikipedia.org/wiki/Spamdexing), to name a few). It is also _de rigueur_ to create Twitter-related neologisms in a similar fashion (typically centering around use of “Tw” as a sort of [charming speech impediment](http://www.twibes.com/)). Convention would therefore dictate that we call spam tweets something like “twam” or “sweets.” Instead, I will break with established practice and propose that we simply refer to Twitter spam as “bird poop.”

[^DATA]: I still intend to make that data set available; watch this space.

[^CONTENT]: I looked for spammy words and URLs addressed as @replies; there were a few hits, but nowhere near the overwhelming proportion you’d need to declare spam a real problem. However, as discussed elsewhere in this piece, content-based spam identification is a real challenge, and I might certainly be missing something. You might be able to do better; see previous footnote.

### Twitter search (and spam) for everyone!

The most effective vector (most eyeballs; fewest effort) attacks user behavior #2 above by spamming search terms; when Twitter launched its [election.twitter.com](http://blogs.zdnet.com/feeds/?p=250) live feed, it wasn’t long before people figured out that this was a [soft](http://twitter.com/dsandler/status/935477015) [target](http://www.soitscometothis.com/2008/09/27/twitter-election-spam/).
And URL shorteners obfuscators [exacerbate the problem](http://twitter.com/dsandler/status/947348989) by making it very difficult to know whether to click a URL provided by a stranger in search results.

This is where I think we may now be in for some real trouble.
On Thursday, Twitter added real-time search (including a smattering of trending topics) to the main web interface. In a blog post entitled [Twitter search for everyone!](http://blog.twitter.com/2009/04/twitter-search-for-everyone.html), Biz Stone explains:

> What was that loud noise outside your apartment? Did you just feel an earthquake? What do people think about your company, your product, or your city? With this newly launched feature,[^NEWLY] Twitter has become something unexpectedly important—a discovery engine for finding out what is happening right now.
>
> Twitter teaches us new and amazing things every day and a big lesson learned is that search is so much more than a box and a button. As public tweets fly in from around the globe, we analyze them to detect when certain words or phrases occur with higher frequency. These trending phrases are surfaced in the Twitter home page just under the new search box and they’re updated throughout the day. Built on our search technology, trends are a compelling if rudimentary way to explore a collective global consciousness.

[^NEWLY]: It’s hardly “newly-launched”—many of us have been using [search.twitter.com](http://search.twitter.com) since before Twitter [bought it from Summize](http://blog.twitter.com/2008/07/finding-perfect-match.html). But it’s still new to many users, so I’ll let this slide.

Real-time search, and its potential to rapidly spread very, very current information (whether accurate or [not](http://www.npr.org/templates/story/story.php?storyId=103562240), and usually lacking in [analysis](http://twitter.com/dsandler/statuses/1629496264)) is without a doubt one of the most interesting[^SEARCHBIZ] features of Twitter.
By choosing to feature it so prominently in the main interface, Twitter has dramatically raised the profile of search results, and therefore considerably upended the economics of search spam (in favor of spammers).

[^SEARCHBIZ]: Interesting for users, of course, but possibly also for business reasons: (1) it might be something Twitter could [charge for in some form](http://twitter.com/dsandler/statuses/1669239499); (2) it might make Twitter [worth buying](http://twitter.com/dsandler/statuses/1668995046). This may explain why search has been pushed in front of users’ noses: it’s useful for users, but it’s _valuable_ to the company. (Too cynical?)

### A plan for Twitter spam?

Loïc proposes a solution (that applies equally well to search spam and @reply spam):

> … add a “report as spam” button. This button would report the spammer to Twitter (or to a separate database of users) that we could exclude from the clients after a sufficient number of users report them as such and maybe some manual checking. Twitter could then just delete those users.

Others propose [creating a large central tweet-filtering system](http://www.techcrunch.com/2009/04/26/here-comes-twitter-spam-and-how-to-fight-it/) akin to [Akismet](http://akismet.com/), the blog comments spam clearinghouse.

The problems with these approaches are manifold. Blocking/reporting suspicious users doesn’t scale (requiring users to take the time to manually identify and report junk) and opens the door for abuse (how many spurious reports would be necessary to get a legitimate user canned?).
The Akismet approach—content-based filtering—continues to be an arms race, and even when you have access to millions of different users’ messages (as Twitter would, and Gmail does) it’s still hard to know what’s legit and what [isn’t](http://twitter.com/gruber/status/1650295214).
And even if you take a guess (er, a statistical inference), do we now have to create a “Junk” folder for each Twitter user, including the option to “Report as not spam” to move things back into the inbox? I already have one overgrown garden to tend, thanks.

### Flees a crowd

Security geeks are notorious for complaining about others’ proposals without offering a better one. Here’s my take: I think the only long-term, scalable approach is to remove “strangers” from common views of the system, forcing the user to take extra steps to go beyond his own subscriptions.
In this world, the only junk most users will ever see is junk from friends and family, which can safely be classified as “not spam.”
The mentions/replies view, which has become an important Twitter inbox (particularly for users following more than a handful of others), would show a subset of your main view—messages from random users don’t belong in either place.
I believe this is in the spirit of how most individuals use Twitter: primarily, to stay in contact with a [limited](http://en.wikipedia.org/wiki/Dunbar’s_number) number of friends and colleagues, and only secondarily to engage the public at large.

So how do you meet new people on Twitter if you can’t see their messages?
First, it should be *possible* to see tweets from strangers—just not the default. A “show messages from everyone” checkbox in the mentions or search views ought to suffice.
Second, Twitter users are pretty good at passing along interesting stuff (via retweets), so there still ought to be plenty of cross-pollination between circles.

Finally, Twitter can exploit the _social graph_ to show us messages from friends-of-friends (or friends-of-friends-of-friends, and so on, to any desired depth).
Messages from users with some indirect connection to me are still unlikely to be spam, and moreover, if the goal is to discover new users I’m most likely to be interested in people connected to my existing contacts (crowdsourcing the task of crowd-finding, if you will).
The CS community is really starting to [get into](http://www.mpi-sws.org/~amislove/publications/Ostra-NSDI.pdf) this [sort of thing](http://conferences.sigcomm.org/sigcomm/2009/workshops/wosn/), and although I haven’t yet published my work along these lines, it is this approach that I plan to take in [FETHR](http://fethr.com), my proposed distributed microblogging platform.

### Somewhat ironic postscript

As usual, blog comments and Twitter comments are open below.
Of course, culling responses from Twitter relies on searching the entire unfiltered public timeline, which I just burned about a thousand words inveighing against.
As I mentioned when I [started the experiment](http://dsandler.org/wp/archives/2009/02/26/twitter-comments), this is easily abused; for now I’m explicitly courting tweets from strangers, and we’ll just have to see how long it lasts.