Chris-Linfoot.net

• Domino 7 servers as MX
• Standard Domino server features (blacklists, whitelists and related policy controls)
• Some very minor tweaks to Domino Directory functionality
• ... and one third party application, Trend ScanMail, which is there for another purpose (anti-virus) but does offer some spam filtering too, though that is tuned to very low sensitivity

Maintenance overhead is almost nil. We now rarely think about these features - they just sit there quietly working. This is just as well as we don't have the time to spend on constant fine tuning of anti-spam settings. We have more than enough real work these days.

To be honest, I didn't expect it to work out this way. I had expected, by 2006 or 7, to have succumbed to the necessity of implementing some other (1) anti-spam solution. This might have been an appliance or open source based solution, replacing Domino as MX, or it might have been some solution implemented in the cloud as a service, again replacing Domino as MX.

That this has not been necessary says something for the capabilities of the Domino server.

So, why did a man in my position wade into the problem of spam? Do I have nothing better to do? And how did I arrive at an almost pure Domino solution?

<history lesson>

Back in around 2002, during a bit of a lull in business activity here, I had started to notice the problem of spam. Because I had time available to spend on the problem, but budgetary constraints limited my options to spend any money solving it, the obvious course was to see what tools were already available to me and how they might be adapted to address the problem.

We were running D5 at the time, but D6 was in beta and it had this new feature - the ability to reject SMTP clients based on DNSBL hits - the so-called blacklist feature.

The problem of spam was beginning to have such an impact that I did something you should never do, gentle reader, I implemented the beta builds of D6 at the MX servers in order to gain access to the blacklist feature.

I had previously become aware that blacklists were there when I had witnessed a relay test conducted against my own servers by the DNSBL formerly known as ORBZ (2).

ORBZ was immediately one of the more effective DNSBLs here. I don't recall all of the others we were using at that time, but Spamcop was there for sure as well as ORDB.

ORBZ and ORDB were effective because so much spam was coming via open relays at that time.

Ironically, it was a Domino shop that caused the death of ORBZ. The relay tests that ORBZ used to do included a range of weird and wonderful constructions for sender and recipient envelopes, intended to confuse vulnerable MTAs into believing either that an externally originated message originated locally or that its non-local recipient was in fact local. One of those tests involved the use of @[127.0.0.1] in the sender envelope and, due to a bug long since fixed, this caused the Domino router task to go into a loop, consuming all available CPU, and effectively creating a denial of service condition on the server.

When this happened in Battle Creek, Michigan, instead of applying the patch or strengthening the Domino server's policy to reject at source messages with literal sender envelopes, the administrator of that Domino server called the police.

Back to the story.

Because D6 had blacklists but no whitelist, we had to invent a variety of funky tricks to get around the problem of false positives (3). This was eventually solved with the invention of a third party add-in which replaced the native blacklist feature and added a whitelist feature.

The use of very strong blacklisting, along with strategic whitelisting, is actually about 80-90% of the solution to spam, providing blacklists are well chosen and the whitelist is pro-actively maintained.

I was reporting most spam to Spamcop at that time and became accustomed to reading the headers. Having gained a degree of familiarity with spam headers, I started to see patterns emerging. This led me to speculate that it might be possible to identify spam messages during or after transmission through examination of headers and raw MIME and this ultimately resulted in backstop #1.

This had a further dramatic impact on spam.

After D7 was released, we reverted to native functionality for the blacklist and migrated our whitelist to the new D7 whitelist feature.

And some time along the way, because we already had Trend ScanMail licensed for anti-virus, we switched on the anti-spam feature at a very low sensitivity so as to avoid false positives and this gave us our second backstop.

And that is essentially the prevailing set-up here. The single most effective anti-spam tool we have in all of this is our own local blacklist, built up over several years with the aid of trapped and reported spam. On any given day, our local list takes out around half of all inbound SMTP sessions.

This is what happened at one of our MXes yesterday.

Having expended all that effort on identifying a sustainable way of killing spam using largely native Domino functionality, it seemed churlish not to write it up.

I'm not sure why I concluded that a blog would be a good way of doing this. A wiki would probably have been better, or I could even have posted it all over at notes.net LDD developerWorks (4) - come to think of it, I did that quite a lot.

But that is why, dear reader, you find yourself sitting reading a blog that rarely makes any mention of spam despite its reputation.

</history lesson>

1. I'd say stronger but that would be mistaken.
2. We passed.
3. Example.
4. Surely overdue for another change of name - I suggest Notes.Net

Category: Spam miscellany
Technorati: Domino IBM Lotus Spam

1. Dennis28/09/2007 14:58:06

Excellent. I've followed your blog for years and used some of your techniques. For the month to-date my accepted for delivery is 6% with 86% as blacklisted (varoius), 8% as signature based and less than 1% as virus/phishing/etc. We outsourced some of the task simply due to the sheer volume. The CPU cycles used for SPAM were taking their toll on the server. Thanks again...

2. Charles Robinson28/09/2007 16:15:06
Homepage: http://cubert-codepoet.blogspot.com

I used the D6 DNSBL for a while, but we ended up needing whitelisting and like Dennis, spam processing was eating up far too much CPU. We eventually implemented a Barracuda Networks spam firewall and it has worked beautifully. It handles over 85% of all the inbound SMTP traffic and keeps the workload off our Domino servers.

Recently a whitelisted supplier was infected with a virus that blasted us with porn image spam and gave the spammers all the e-mail addresses in the suppliers' address books. How do you combat that?

3. Walter Mobach28/09/2007 21:14:22

I have read and appreciated all your articles on preventing spam and configuring the server.
FYI some statistics on our server:
Blacklists 13000-20000 each and every day
Statistics at this moment since 03.00 this morning
bl.spamcop.net.Hits = 138
cn.countries.nerd.dk.Hits = 76
dnsbl.sorbs.net.Hits = 231
kr.countries.nerd.dk.Hits = 24
TotalHits = 11604
virbl.dnsbl.bit.nl.Hits = 30
zen.spamhaus.org.Hits = 11105
External whitelist
nlwhitelist.dnsbl.bit.nl.Hits = 41

Virus? just one every other day or so
spam trapped by trendmail 200
mail received 1400, of which 50% on private whitelist
I have no information on the amount of spam that got through

4. Peter von Stöckel28/09/2007 21:43:09
Homepage: http://www.bananahome.com/

Whenever I need some info on spam, this is the place I go to first, and it hasn't failed me yet. This really IS the spam bible. I also direct other people here when they have questions in this area. The key isn't really about how many posts you do about spam, but all the spam information you have gathered over the years. It really is invaluable to all of us. If you ever decide to close this blog, please let me have a copy of the .nsf.

5. Chris Linfoot28/09/2007 22:53:31

@2: it is regrettable that IBM took so long to give Domino a whitelist. Without it, Domino sites could only ever use weak blacklists and would inevitably have ended up accepting and having to filter a lot of spam. This is probably why it was necessary for you to take a different path.

But the load caused by spam filtering at the protocol (DNSBL) level is so low that even a relatively low powered server can handle it, providing whitelisting is also in place.

Our primary MX is a Domino server running on a slow (<1GHz, if memory serves, about 3-400MHz) CPU with 512 MB memory. CPU utilisation is typically <1% and virtual memory utilisation is also quite low.

Spam filtering is very efficient if dealt with at the protocol level.

@4: I have no intention of retiring just yet. But you can have this blog's .nsf if I ever do.

6. Matthias Lesii29/09/2007 09:01:47
Homepage: http://www.dnswl.org/

Chris: I assume that the blacklist you show in your stats are roughly used in the order shown, and a single blacklist hit will be sufficient to reject a mail?

If my assumption is correct, then the stats above may not really show the relative effectiveness of each list, but is rather a statistical artefact. Or in other words: how effective would your private blacklist be if it were used later in the "chain"?

@3: shameless plug: you're missing one whitelist

7. Chris Linfoot29/09/2007 09:46:55

@6 - You are right. It all has to do with the order in which lists are tested and, in Domino, the local blacklist is always tested first.

This does have the effect of making the local list appear more effective than it probably is - it mainly lists addresses that are apparently dynamically assigned, many of which would probably also cause a hit on the PBL element of the Spamhaus list or the DUHL element of the SORBS list. However, we don't give the benefit of the doubt, preferring to blacklist aggressively and make exceptions on request, so I am fairly sure that our list takes out SMTP sessions that would not be taken out if relying solely on public DNS lists.

And there is some performance benefit in having the local list queried first. Not one of those sessions rejected by the local list caused any DNSBL lookup activity. By contrast, a non-whitelisted system which is not listed in the local blacklist will cause a DNSBL lookup on every DNSBL zone we use until there is a hit or all zones test negative.

8. Conrad Longmore01/10/2007 10:16:17
Homepage: http://www.dynamoo.com/

I'm a convert to external filtering services that take the load off your servers. We use Postini (other spam filtering services are available), and it works pretty well as all the spam goes through there servers first and we get the cleaned up mail.

One reason that an external solution works well is the increasing problem of directory harvesting attacks (DHAs). Before we turned on our optional DHA protection in Postini, 98% of our inbound mail was going to invalid addresses. Put that another way, and basically the mail load was 50 times higher than is should have been and parts of the infrastructure could no longer cope with the load.

Of course, there's a cost involved.. but actually it was cheaper and easier to implement spam filtering that way than having to cope with what was basically turning into a DDOS attack.

Oh yes.. I check your blog regulary and I don't even run Notes

9. Ben Rose03/10/2007 14:47:01
Homepage: http://www.jaffacake.net

Interesting stuff.

Our main problems come from doing increasing business in Poland, Russia, Italy and China...it's led to many false positives.

Or rather, genuine positives that affected a business partner and so had to be whitelisted anyway.

Unable to post a comment? Please read this for a possible explanation...