In My Experience... | Scale. (1519)

BBedit shovelware please. « prev
Radio coverage of blogging. » next

Scale.
[ Posted by Dan on March 31, 2003 | 12 Comments ]

Working on something that will be on the AOL Welcome Screen, or something linked from it, is sort of stressing. You simply can not fuck up, and you have to scale like a mo-fo. I've just completed an item (with tons of help from many other people) that will appear in that type of location. Now, I'm just a cog in the wheel, but I happen to be the cog where people actually see the spokes turning (metaphorically and realistically) and if it breaks, I'll be getting the 3am phone call.

It's a ticker of the amount of spam emails that have been killed (before they got to a member's mailbox) since midnight. It's a huge number, and ticks along in real time. The systems by which that data gets from one place to another, and finally gets represented on screen for the user, is equally big. This scale issue has been one of the things that has been hard to get used to over the last year, but I think I'm finally getting it.

One of the cool things is the architecture by which data is cached (or not) depending on the responsibilities of that data and how it will be consumed (all in an effort to conserve network resources). Also, equally cool is the ability to embed web stuff in the proprietary AOL screens and make it look like it's totally normal. That sort of thing makes me hopeful that more "normal" web technologies will be used to present users with the content they are there to consume, and thus make my skills more useful :)

Excuse my ignorance, but I'm not following what you mean by "scale" or the "scale issue." Do you mean layer?

-Posted by Joshua Kaufman on March 31, 2003 05:38 PM

AOL suffers from too-many-users syndrome.
Everything they do has to scale to handle more load than any other ISP has to deal with.
That's what he means.

-Posted by nobody important on April 1, 2003 11:26 AM

The question is, what mail is being killed ? Is it
mail I want ? I hear constant complaints that mail
from outside locations does not arrive in AOL mailboxes and there is no return to sender. Wahazzupwidat ?

-Posted by Bob Lynch on April 1, 2003 11:32 AM

Yeah, by scale I mean that some of the projects you work on will be seen by millions of people, and possibly be seen by millions within a tight time frame (like, an hour). Thus you have to build robust systems that can take a serious beating.

-Posted by Dan on April 1, 2003 11:36 AM

Bob: I haven't had that happen to me yet, but I have seen email take some extra time to get thru the system due to (and this is a guess) the anti-spam processing. I've created emails and had them get trashed in the system due to the odd headers I was writing out ("from" one address, that has a diff domain from the box sending the mail, with a "reply-to" to another address, and "to" a third address).

Is there's a specific example you can cite of emails not being delivered? (like an article or something, or a message you've sent, etc).

-Posted by Dan on April 1, 2003 11:43 AM

As the E-mail manager for a large 'anonymous' ISP for
many years, I have rec'd several hundred examples. I am
currently assisting a user who has (reportedly) 95% of their E-mails not delivered and not returned. She took out her signature, which had two URLs, and the messages were then delivered. This was 'pen-pal' E-mail, not junk.

AOL appears to have a policy of dropping mail based on 'global' algorithms that individual customers
cannot override. Maybe I have this wrong. Since the E-mail is not returned to sender, it just disappears, legit mail and junk mail both. If that was my mail being dropped, I'd jump ugly about it. I'd imagine AOL blames it on the sending side, but I don't have to ... I've had prior discussions with AOL and they DO blame it on the sending side. All this needs is a little more publicity, like the 800 Harvard E-mails that were dropped and not returned to sender a while back, and AOL's response was it was going to change it's spam controls. I think the only change was to let all Harvard mail in, as I see the same old story.

I just stepped off my soap box.

-Posted by Bob Lynch on April 1, 2003 01:45 PM

Bob: feel free to stay on the soapbox :)

In the example you mentioned, you say that the mail was being dropped due to the existence of two URLs in the sig of the author. I receive similar emails thru my aol.com address, so I'm guessing their is something else, that in combination with the multiple URLs, is triggering a spam killing routine.

Being the curious person that I am, I'm hoping you are willing to point out anything else that may be odd with the sender's email...

Is their a difference between the 'reply-to' value and the 'from' value of the email?

Does the domain of the sender not match the domain of the relay?

I'm not a spam guy here, but I can imagine several ways that little oddities can add up to a threshold. If you can come up with some details, I can pass those details along to someone who might be able to help you out (but I make no promises that anyone will answer my emails).

-Posted by Dan on April 1, 2003 01:59 PM

First off, let me say I do appreciate the enormity of AOL's spam problem, nonwithstanding the running of their mail system in general with such a huge member base.

The reply to and from are the same.

The problem initially surfaced on mailers with the
same domain as the sender, and was replicated on
mailers with a different domain.

It does appear there is an 'additive' function of discrete checks that determine the likelihood of a mail item being spam. That's OK; good idea. I'd like to see 'legitimate ISP' as one of the factors.

It's not the algorithm that I object to; it's the
throw out the baby with the bathwater approach, *without*
asking Mommy first. Give the user an accept from always list, fercryinoutloud. If this exists, I'd love to know
so I can tell everyone about it.

-Posted by Bob Lynch on April 1, 2003 02:19 PM

BTW, I love the ticker idea ... it's for the
person's individual mailbox, right ? It might get
AOL users connecting the non-arrival of a mail item
and AOL's spam-killing application.

So I had 120 E-mails killed yesterday and I did not
get some I expected ? Hmmmm ....

-Posted by Bob Lynch on April 1, 2003 02:37 PM

The ticker shows the numbers of spams killed since midnight (and at this moment of time is reporting over half a billion). Like I mentioned in my original post, this ticker will be on a screen that will be linked from the Welcome Screen, and thus will be seen by millions of people, and is therfore not individualized.

I think an individualized version would create new scaling issues.

-Posted by Dan on April 1, 2003 03:41 PM

BTW, it's now up at Keyword: Safety

Also, if you upgrade to '8.0 plus', there's a 'safety and security' button in the toolbar which brings you to the right place. Broadband users get the DHTML verwsion, and narrowband users get a static 'total spams killed yesterday' version.

-Posted by Dan on April 3, 2003 09:18 AM

Wow! 1,296,826,528 yesterday. It doesn't seem to understand that I'm on broadband, so I have the yesterday version.

-Posted by JC on April 3, 2003 10:45 AM

Comment posting has been turned off because I don't have enough time and will to deal with the constant comment spamming. I'm very sorry and will fix this sometime soon (soon = before 2004 ends).