Wednesday, April 22, 2009

The Curse of E-Mail Filtering

Yesterday I described what makes Internet e-mail so awkward: it’s built for letters, but few of the messages it carries are really letters. Most notes written from one person to another go through the messaging systems of web sites, especially social networking sites. Those messages, although they may be read and written on a web page, don’t go through the Internet e-mail system. Most messages that go through Internet e-mail are spam, and the biggest challenges in e-mail are separating the spam from the legitimate e-mail, and then among the legitimate messages, separating the messages of value from those that can be ignored.

In other words, the heart of effective Internet e-mail is filtering. E-mail filters are like the Santa Claus of messaging, separating good messages from bad. Yet even though this is the most important thing to happen with e-mail, it is not part of the e-mail standards at all. The core of the current e-mail system is an add-on, with no two engineers agreeing on how it should work.

Worse, e-mail filtering is done based on the outdated assumption that an e-mail message should contain a letter. Most of the rules in e-mail filtering are performing a simple style check, to see how closely the message resembles a letter. Other rules check for nearly identical messages being sent to many people at once. This catches the most obvious spam efforts, but it is a tricky combination. It results in a test that the most important messages will tend to fail. There are perfectly legitimate notifications that might be sent to a million people at a time — “There is an electric bill ready for your review,” for example, or, “Madonna concert tickets are now on sale.” The people tasked with sending these messages are forced to dress them up with all sorts of superfluous verbiage just to get them past the e-mail filters. It’s an inefficient use of engineering talent, and it is also a waste of the customers’ time, as you could read the actual one-line message faster than the dressed-up one-page message that is more likely to make it through the filters.

Most e-mail filtering is designed to let most legitimate messages through, but there are exceptions. Many corporate e-mail servers are protected by firewall e-mail filters that freely delete messages they can’t figure out. And a few Internet service providers, notably Verizon, have adopted the same kind of filter for their paying customers, gleefully deleting the note from your Aunt Brittany because she sent it to a list of 15 recipients, and sent it from her husband’s computer rather than her own. The vagaries of Internet e-mail filtering mean that you can never really be sure that any message gets through. Of course, this just adds to the e-mail mess, as people who really need to make sure their message gets through will send it several times.

It would be nice if messages from people you like could bypass all the e-mail filters. Unfortunately, you control e-mail filtering only after a message reaches your own account, and to get that far, it may have to pass through three to six e-mail servers along the way. And even when a message gets to you, you are taking chances by filtering it, because astonishingly, the Internet e-mail system does not record any reliable information about the source of a message.

Information that could vouch for the source of a message could not come from the e-mail system. That information would have to come from some kind of social network. It makes sense that most personal e-mail has moved to social network web sites, because there, when you receive a message, it comes with some assurance that its sender is someone you have heard of before. Quite possibly, no filtering needed.

To strengthen and extend this side of e-mail would take at least these two steps:

  1. Stronger passwords. Many people use simple one-word passwords, such as “hi,” to sign into a social network. These passwords are simply too easy for spammers to guess. If a spammer can sign into your account and send a bunch of advertising messages to everyone you know, making the messages appear to come from you, it largely defeats the identity protections that a social network provides. Worse, the social network might boot you off if that happened.
  2. Exchange of messages between social networks. You wouldn’t have to join every social network to get the benefit of identity protection for e-mail messages if social networks could work out a way to exchange messages. The messages would have to come with a few measures to help you judge the validity of the identity of the sender. Then, for example, you can skip over the message from the person who has been in the social network for 2 days, has 2 contacts, and is writing about “discount prescriptons.”

I have just scratched the surface here, but it is already clear that if you set out to design a new e-mail system that starts with the need for filtering, you get a completely different approach than what you arrive at by trying to fix the current e-mail system. One of the things you realize when you look at e-mail this way is that it’s not really about the messages. The messages themselves are small, even inconsequential, compared to the totality of what a messaging system has to do. It is ultimately about the people you want to connect to and the things you want to do.