Attempting To Improve Accuracy

[bcw]

I am attempting to provide my users with the best possible spam blocking without blocking ham, I am playing with the options available. Unfortunately, there are several issues I have not been able to resolve on my own. The following is the list followed by the details of each:

1. Interaction between MagicSpam and Plesk spam protection based on DNS black hole list.
2. Use of best practises rules
3. Email from sources on IP reputation lists not identified as spam
4. White listing major ISP mail servers IP
5. Format of rejection message text

MagicSpam and Plesk
Is there a value in using the Plesk DNS black hole protection in conjunction with MagicSpam? I am less concerned about the added resource overhead than blocking the most spam. As I can add the lists from SpamHaus and others using this feature, I assume that this is worth using. My consideration in using these other lists is that I am seeing spam flagged as ham from sources on these other lists.
Also, is MagicSpam invoked before or after the Plesk DNS black hole protection?

Best Practises Rules
Many of my users are home/small businesses so that for those using the Plesk server for their SMTP/outgoing server, I must be certain not to reject their attempts. There are three best practises rules that appear to be excluded by my requirements:

check_dynamic_reverse_dns – The IP is always dynamic and of the host name specified is the ISP’s configuration not my user’s domain.
resolve_helo_domain – The helo is always the computer name
valid_helo_domain – The helo is always the computer name

Am I correct in my understanding? If not, please elaborate.

There is a fourth rule, valid_from_domain, on which I am uncertain. For my users the from address will always be well formed and the domain portion will resolve to their domain on the server so it sounds like this should be a safe rule to implement. Still, the explanation of this rule seems to imply that there is also a relationship to the IP that is sending the email. Am I safe to put this rule in use?

IP Reputation Accuracy
I have noticed a number of messages marked as ham with IPs that are black listed by the lists I have in use. Currently I am using all lists with the exception of SORBS-DUL and the following IPs are not marked as spam even though they are on the lists shown:

RATS-Dyna - 222.155.105.236 and 212.30.223.103
UCEPROTECTL1 - 195.249.91.122, 149.156.87.107, 212.30.223.103, and 195.158.103.157

This is not an exhaustive listing, just a quick look over a few hours of the logs and while I appreciate that the MagicSpam lists are not dynamic, I have a difficulty explaining that this many errors can be attributed to the list being a few hours out of date. Do I have something set wrong or is there another explanation.

White Listing Mail Servers
This week, one of the major ISPs in my area had one of their mail servers show up on the UCEPROTECTL1 list. In short order, this started causing havoc for my users as ham was rejected. White listing this one IP solved the immediate problem and I really do not want to have this grief again should this or another of their servers make some list. Are their any cautions against listing all the mail server IPs for specific ISPs? Also, is there a good way to get a listing of these IPs?

Rejection Message Format
It appears that there is a problem with the formatting of the text as the control character \n is being displayed rather than generating a new line. How can this be resolved? The following is an example:

The reason for the problem:
5.1.0 - Unknown address error 550-'Your message was rejected by this system and was not delivered.\nReason: This system will not accept messages from you\nProtection provided by: MagicSpam 1.0.4-6 (http://www.magicspam.com/)\nFor more information, please visit the URL:\nhttp://www.linuxmagic.com/empowering_the_user\nor contact your ISP or mail server operator.'

PS It would be very helpful if the dashboard or setting pages indicated which version is installed.

[magicspam]

If you are seeing false positives when you use the other lists and one of your primary concerns is to avoid false positives, you may want to avoid using those lists. As to whether MagicSpam is invoked before or after the Plesk protection, that depends on which MTA you're running. For Postfix, MagicSpam is run after, and for Qmail-SPP MagicSpam is run before.

I will briefly cover the rules you mentioned:

1) check_dynamic_reverse_dns: The reverse DNS entry for the IP matches a known-pattern of dynamic-style addresses, most often used by ISPs to denote home connections. Legitimate mail servers should have a proper reverse DNS that complies with the generally accepted best practices (e.g: mail.example.com). If your users are triggering this, they either need to get their reverse DNS changed or use SMTP authentication.

2) resolve_helo_domain: Does the domain that the server HELO/EHLOs with resolve? Some systems will HELO as "server.local", which will not resolve, and therefore triggers this rule.

3) valid_helo_domain: This checks to see if the HELO domain is in a proper domain format (e.g: xyz.com, rather than just xyz).

4) valid_from_domain: If this rule is enabled, it ensures that the domain of the listed sender (e.g: user@domain.com) actually exists. If the domain does not exist, the rule is triggered and the message rejected. This is similar to the resolve_helo_domain, except that it ensures the sender's domain exists. This rule is off by default primarily because of the additional overhead it requires (essentially, a DNS query for every attempted message).

IP Reputation Accuracy:

The lists are updated every 6 hours, which is more than enough time for several new IPs to be added to the lists. If those IPs continue to be a problem even after your lists have been updated, it may warrant further investigation. However, the most likely explanation is the time between updates.

White Listing Mail Servers:

If you whitelist all the IPs for an ISP, you increase the risk of allowing any spam that potentially comes from those IPs to enter your system unrestricted. Beyond that there is no danger. To get a listing of their mail server IPs, you can try doing a host lookup on their primary domain and limit the results to MX records. Not all companies are set up that way, however, so it may not work for you.

Rejection Message Format:

How are you viewing the rejection message? It's possible that the backslashes are being escaped, resulting in the \n showing up in your output.

[bcw]

The rejection message is directly from the email client receiving the failure response.

[magicspam]

Which email client is being used to view the rejection message? Have you tried other email clients or webmail providers? Some of them may render the \n characters differently.

[bcw]

I have seen this with Outlook and Outlook Express.

As there is no control over the client being used, I would expect that the message is constructed in such a way as to independent of the receipients method of viewing.

[magicspam]

Most mail clients will correctly interpret the \n character as a newline. For whatever reason, either a mail server or the mail client is escaping the newline, resulting in the printed \n.

However, I will bring this up with management to see if there are any plans to move to a message format that does not include newlines.

[bcw]

With luck, I can get things working such that no user ever needs to see the message

Regarding the check_dynamic_reverse_dns rule, I may not have been quite clear on the situation.

Because, users connect directly from their PC over the Internet to the server for outgoing mail, they will always fail this rule and be rejected if the rule is enabled. This makes the rule unusable and therefore leaves a segment of spam not covered.

If it were possible to excempt authorised users or something similar, this would open the use of this rule to a wider audience.

Perhaps I am misunderstanding the process and if so, please let me know.

[bcw]

I reread the original reply and saw that I had overlooked that SMTP authorised traffic is excluded from the check_dynamic_reverse_dns rule. I have enabled the rule, tested that in fact this is true, and am now seeing the rule do its work.

[bcw]

I have one entry in my log that while marked as ham, looks to me as something that the MIPS list should catch. Doing what I can to determine the information, ip=[72.55.149.57:cheapquotesonline.com] appears to be a server dedicated to UBE. Is it a candidate for the MIPS list?

This raises two additional questions.

The first is whether we should report suspect servers and if so, what is the correct method?

The second is whether there is a way to allow an individual user to receive UBE without excluding that email address from all spam checking?

[bcw]

I am still seeing entries marked as ham that should have been flagged as spam based on the IP reputaation lists. This has me wondering if my lists are being updated so can you tell me where to look on my server to confirm that the update is happening.