Chris-Linfoot.net

Server bug cripples Dublin law firms
Badly configured MS Small Business Server

Solicitors across Dublin fell victim to an accidental mass mailing that crippled their systems this week, clogging inboxes and causing widespread disruption.

The problem was attributed to an improper server configuration, causing five servers to send out more than half a million emails to Dublin solicitors. The deluge of mail originated with a publishing company's email marketing message, which was sent to solicitors. When some solicitors attempted to reply to the mail, a fault in the solicitors' configuration of Microsoft Small Business Server sent the original email to their entire email database tens of thousands of times.

The Register piece doesn't say what bug, but I'll tell you. It's our old friend the Microsoft POP3 fetch.

Those words "a fault in the solicitors' configuration of Microsoft Small Business Server" do not really do justice to the bathos of the piece and so I'll attempt redress.

First, how the bug works (or fails to work).

POP fetching to SMTP is an intrinsically broken process made much, much worse by the MS bug. It normally would involve writing additional header fields into the MIME source of a message in transit to preserve volatile information normally lost when a message is delivered to a POP mailbox. Specifically, this volatile information which would normally be lost and needs to be preserved is the message envelopes, which say respectively who sent the message (SMTP MAIL FROM) and who this particular copy is for (SMTP RCPT TO). Without this information, a downstream SMTP attempting to deliver a message retrieved from a POP mailbox simply cannot know to whom it is supposed to be delivering the message. It can't just rely on the RFC2822 "To" field as that may include non-local recipients who already have their copy, or the name of the mailing list that the message was sent to and may not contain the intended local recipient at all if, for another example, the message is a bcc. The MS POP fetch does indeed work alongside a mechanism intended to preserve SMTP envelope information, but breaks (due to a buffer overrun) in an unsafe way.

• On encountering too many bytes in the RFC2822 "To" field (i.e. too many recipients, all named in "To"), it simply aborts having failed to extract the recipient envelope.
  
• But, it leaves the entire message intact in a local delivery queue.
  
• The local mail router picks it up and treats it as an undelivered, locally originated message.
  
• It is sent out again, without any of the original "Received" headers (a locally originated message would have no received headers at this point), and copies are sent to every named recipient in either "To" or "cc".
  
• Every recipient, including the POP mailbox being used by the broken POP fetch, receives their second copy.
  
• The process loops indefinitely and mail routers in the message path never spot the routing loop because the "Received" headers are dropped and recreated every time.

Now what happens where more than one broken POP fetch is participating?

Here's a table showing the effects of multiple broken POP fetches in the same loop. The numbers in the table show the total number of copies of the message delivered to each recipient at the end of each iteration of the loop.

						Loop Iterations
Number of fetches	Original	1	2	3	4	5	6	7	8	9
1	1	2	3	4	5	6	7	8	9	10
2	1	3	7	15	31	63	127	255	511	1,023
3	1	4	13	40	121	364	1,093	3,280	9,841	29,524
4	1	5	21	85	341	1,365	5,461	21,845	87,381	349,525
5	1	6	31	156	781	3,906	19,531	97,656	488,281	2,441,406

The Register article says there were (count them) five POP fetches were in the loop on this occasion, though it does not say how many recipients victims there were (I have shown that a comparatively low number of recipients can trigger the bug). You can clearly see that even where there are only two broken fetches the exponential growth is fairly rapid. Given the closed nature of the user group affected here, it is reasonable to assume that many of them share similar infrastructure including MS Small Business Server and so more than two broken POP fetches in the loop seems inevitable. Five broken systems would indicate that the message reached perhaps its 6th or 7th iteration before everything ground to a halt. That would seem to tally with "a fault in the solicitors' configuration of Microsoft Small Business Server sent the original email to their entire email database tens of thousands of times."

Oh, and that original mass email? Who knows? Probably an invitation to someone's Christmas bash.

Category: Dumb and Dumber
Technorati: Dumb and Dumber

1. Eric Parsons12/12/2005 11:54:13
Homepage: http://startingblockcomputing.com

Oops! Was the list, at least, opt in?

2. Chris Linfoot12/12/2005 11:55:10

I suspect not.

Unable to post a comment? Please read this for a possible explanation...