Chris-Linfoot.net

1. Richard Schwartz27/04/2005 16:32:39
Homepage: http://smokey.rhs.com/web/blog/rhs.nsf

This is great stuff. Thanks! I've been piecing these things together from your earlier posts for the past few days, but so far was only getting about 1/3 of messages that got past my DNSBLs and static blocks. About 90% is caught by DNSBLs and static blocks, 3% by enhanced rules, 3% by ordinary server-wide sender or content rules, and 4% is getting through, with some of what gets through being caught by user rules. Small sample size in this configuration, still I'll see what I get from adding the rest of the rules. The 4% that is getting through is still way too much in actual volume.

-rich

2. Michael G27/04/2005 16:38:23

Chris, thank you so much for this. Just a quick ? Shouldn't When Mailer contains The Bat move to database spamtrap.nsf be X-Mailer?

Thanks!

3. Chris Linfoot27/04/2005 16:42:33

@Michael: The way I tweaked the code in the rule form and script library is internally inconsistent (I don't do code). So while the drop down on the rule box itself does indeed say X-Mailer, the rule as built in the UI and saved in the rules list just says Mailer.

Still works though - these fields are not actually used for anything but display purposes.

4. Richard Schwartz27/04/2005 17:09:16
Homepage: http://smokey.rhs.com/web/blog/PowerOfTheSchwartz.nsf

Hey, since this post is likely to attract rules hackers... I thought of something last night. Something sneaky. It might work. Haven't tried it yet.

I inadvertantly entered "" in the value for an "Is Not" test instead of just leaving it blank. The error message I got indicated to me that a the UI is simply dropping quotes around whatever you put in the field and letting the formula engine do the rest. So... it occurs to me that it might be possible to do two types of hacks:

1. Enter a" : "b as the value to do a shorthand OR test. This would be useful in order to be able to control precedence of ORs within ANDs in rules.

2. Enter " + @GetField(some-field) + " as the value. If this works, it might open up some really interesting possibilities!

I gotta make some time to run some tests on this!

-rich

5. Chris Linfoot27/04/2005 17:18:19

I too will test this. Looks almost too good to be true.

6. 27/04/2005 18:21:14

Chris,
Another question. How do you deal with quarantined messages being released to the user. One thing that I have to deal with in the spamtrap.nsf database is releasing messages to users. Any advice?

Thanks!

7. Richard Schwartz27/04/2005 23:14:21
Homepage: http://smokey.rhs.com/web/blog/PowerOfTheSchwartz.nsf

It looks like the behavior I was seeing is somehow specific to the hacked rules. If you try entering a " character into a subject or sender condition, you get an error message. If you enter it into any of the extended rules, it seems to allow it. I haven't tested the effect of using any tricks based on this behavior in the extended rules yet. If that works, though, then it could be worth looking into creating a new set of extensions. I'm thinking that an extended rule that works on the sender field or sourceIP but doesn't error check the formula might allow people to write their own custom whitelisting formula using an @DbLookup. Performance could be an issue for such a thing, of course, but first I'll worry about whether it's even going to be possible.

8. Michael G28/04/2005 03:13:33

Chris, Just and FYI but in my first day, I had 2 false positives with the X-Virus-Scanned contains Symantec AntiVirus Scan Engine. I have removed this from my set.

9. Richard Schwartz28/04/2005 03:46:41
Homepage: http://smokey.rhs.com/web/blog/PowerOfTheSchwartz.nsf

I've now got all of the extended rules implemented. Time to reset stats

One sugguestion: you should move the russian rule down the priority list, or split it into two rules, one of which checks the subject and the other checking the body -- and move the latter down to the lowest priority. Body checking is far more expensive than any of the other tests.

-rich

10. Chris Linfoot28/04/2005 08:33:23

@Michael - you sure? Real Symantec AV headers do not look like that so it is forged even if you think the email is benign. Ask yourself why they would forge something like an AV header? Smells abit funny to me.

Send me the headers of a sample false positive if you like and I'll do some further digging.

11. Chris Linfoot28/04/2005 10:32:27

@Anonymous poster (I believe that is you, Michael G, again) - On the subject of releasing quarantined messages to users - a couple of ways this can be done - pick one depending on how big an issue it is for you.

1. I so rarely see a false positive that I deal with exceptions the hard way.

Open quarantined message. View MIME source. Copy to clipboard and paste to Notepad. Edit source to remove item that caused the problem (e.g. X-Mailer).

Telnet into port 25 of one of your servers, issue on separate lines HELO whatever, MAIL FROM:<whoever>, RCPT TO:<whoever>, DATA then paste the entire MIME source. Enter a . on a line by itself to complete the transaction the enter QUIT. This will deliver the message to the intended recipient, bypassing the rule that blocked it.

Be sure to tune the rule so that it doesn't happen again - e.g. if the trigger was X-Mailer, change the rule to include "AND Internet domain is not example.com".

This is fine if called upon at the most once every two or three of months as it is here.

2. Do it programmatically

Because messages are written to the trap database by a rule, not the mail router, all information needed to route correctly is still in these documents.

You could add an action button on the in-box and/or all documents view of your trap database. The action would:

a) grab the original sender and intended recipients from the message
b) set the Notes sender and recipient fields to be those original sender and recipient fields
c) use @MailSend to requeue the message

Obviously, you would probably need to tune the mail rule that caused the false positive in the first place before clicking the button or the message would just be trapped again.

I have not done this myself - as I said it is not really needed here. If you try it, let me know how you get on.

12. Steve Dionne28/04/2005 14:01:12
Homepage: http://www.canamgroup.ca

@Michael

If you are like me, I got a lot of Domino Servers.

I have created a server group configuration document just for our MX Servers and this configuration document include all the Mail Rules.

Our internal mail server have another server group configuration document that doesn't contains any mail rule.

So when a false positive happen, I just cut the message from the SpamTrap db and copy it to one of the mail box(mail1.box) of an internal server. The router then process this message as usual and the recipient receive the message.

Very simple.

13. Scott Iver28/04/2005 15:00:13

@Chris and Michael,

I too have false pos. on the Symantec line. Here is the source of one asking about one of our products....


Received: from ms-smtp-04.rdc-kc.rr.com ([24.94.166.116])
by My.Server.us (Lotus Domino Release 6.5.3FP1)
with ESMTP id 2005042719040751-465 ;
Wed, 27 Apr 2005 19:04:07 -0500
Received: from hppav (CPE-24-209-155-157.wi.res.rr.com [24.209.155.157])
by ms-smtp-04.rdc-kc.rr.com (8.12.8p1/8.12.7) with SMTP id j3RNoARq001381
for <info@mydomain.com>; Wed, 27 Apr 2005 18:50:12 -0500 (CDT)
Message-ID: <000501c54b86$402fc320$6401a8c0@wi.rr.com>
From: "USER" <USER AT wi.rr.com>
To: <info@mydomain.com>
Subject: Senior26 (15+ cats)
Date: Wed, 27 Apr 2005 19:07:19 -0500
MIME-Version: 1.0
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1409
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409
X-Virus-Scanned: Symantec AntiVirus Scan Engine
X-MIMETrack: Itemize by SMTP Server on MY SERVER (Release 6.5.3FP1|December
15, 2004) at 04/27/2005 07:04:07 PM,
Serialize by Notes Client on ME (Release 6.5.3|September
14, 2004) at 04/28/2005 08:53:45 AM,
Serialize complete at 04/28/2005 08:53:45 AM
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset="iso-8859-1"

Is there any way to order this brand of cat food in larger bags? or directly
from the you instead of a retailer? If so, please let me know how to go
about it. Thanks.

USER

Very strange isn't it... I'm curious as to if Michael's false pos. were also from RR, as all of mine have been.

14. Michael G28/04/2005 15:00:14

Thanks for the tips...I should have thought of this!

15. Chris Linfoot28/04/2005 15:12:46

Et voila!

I suggest you reinstate the rule on Symantec AV and add a modifier "AND HELO does not contain rr.com"

Here is why.

16. Eric Parsons29/04/2005 15:57:50
Homepage: http://StartingBlockComputing.com

Getting lots of false positives with the "HELO contains [ and contains ]" rule, so I'll have to reduce on that one. It is a common paranoia in my industry to "hide" the domain (I'll never figure that out, but...)

Is there a way to compare the helo to the ip? As example, we get this ""from [192.168.1.2] ([66.38.30.14]) by ..." where the helo IP doesn't match the actual IP. Bad setup, but certainly a possiblity for bad mail.

17. Chris Linfoot29/04/2005 16:10:04

@Eric - See Rich's comment above re rules hackers. With a bit of thought I reckon you could devise a rule to do that.

And I warned you about the address literal thing didn't I. Works well here but we've had to extend it with a few exceptions.

TrackBack From startingblockcomputing.com30/04/2005 02:04:43

Monitoring rules before stopping messages

Journaling gives you a way to monitor rule effectiveness, and decide on its effectiveness.

19. Steve Dionne02/05/2005 17:01:18
Homepage: http://www.canamgroup.ca

Over the weekend, some emails have been put in our spam trap.
I checked all of them this morning, and I did a few modifications to the mail rule.

I got 6 false positive with “X-Virus-Scanned contains Symantec AntiVirus Scan Engine”.
As Michael and Scott told us, mine was coming as well from “rr.com”. So, this time I have added "AND HELO does not contain rr.com" as Chris wrote.

Up to now I got 7 False Positive with the Domain Literal “[“ and “]” mail rule.
A company using Lotus Notes has sent 4 of them. I have added “AND Mailer does not contains Lotus Notes” to the domain literal rule.
5 spams have been caught with this rule.

Over all 18 spams has been identified and moved in the Spamtrap database with the entire mail rule in place.

20. Scott Iver02/05/2005 20:50:26

Whew! I was just waiting on confirmation on the RR.com thing... Wanted to make sure I wasn't crazy!

21. Rob Kirkland25/12/2005 18:36:46
Homepage: http://www.takeawalk.com

When I view the page source of a message (Alt+v,s,s) there is no "HELO" field. I guess that the text between "from" and "(" in the first listed "Received" header is the content received with HELO. Correct?

22. Chris Linfoot26/12/2005 19:42:35

Correct.

23. Rob Kirkland27/12/2005 23:11:35
Homepage: http://www.takeawalk.com

I finally got around to implementing Daniel Koffler's mail rule enhancements, thanks, Chris, to your making your implementation of them available. I'm embarrassed to admit that doing this has been on my to-do list since Daniel's articles were first published.

Having done so, however, and having then impemented many of the above rules (thank you, Chris, again), I would make the following suggestions to other readers of Chris's post above (or, Chris, perhaps you'd like to edit your post):

- Read carefully. There is a big difference between "is" and "contains". Make sure that, if Chris's rule says "is", you choose "is" and not "contains", which is the default choice.

- Read carefully. Everything in Chris's rules between the words "is" or "contains" and "move to database" or "don't accept message" is part of the condition. You can set whatever condition you want, of course; but if you are copying Chris's conditions, do it with care.

24. Rob Kirkland27/12/2005 23:22:16
Homepage: http://www.takeawalk.com

Chris, regarding your comment posted above dated 28/04/2005, about moving false positives out of the spamtrap database, I have been using a "Copy to database" simple view action to move the false positive message to the mail.box database. It seems to work great. I don't see any difference between the headers of the pasted message and those in the original in the spamtrap. I'm wondering why you would go to all the trouble of telneting as in your post.

Any comment?

25. Chris Linfoot28/12/2005 14:07:38

Because I never thought of it

I always tried moving to recipient's mail file and that doesn't work.

Unable to post a comment? Please read this for a possible explanation...