Hello,
In the past 12 months, we have seen an increase in the amount of spam allowed through by spamassassin, along with numerous backscatter problems getting us blacklisted (even though support has told us how to stop backscatter, it still doesn't seem to work).
Most concerning is the number of phishing emails allowed through, which turn into a massive bot attack as soon as one user falls for it. This occurs every 2-3 months regardless of the number of time we tell users to NEVER release their password, and that we will NEVER ask them for it. The bots are specifically targeting Zimbra servers and using SOAP calls to send thousands of emails in a very short time. We usually find out when we get a blacklist alert, or the user notices bounces in their inbox, at which point they remember that they let their password out...
We need:
1. rate-limiting to prevent any account from sending more than x messages per y interval.
2. rate-limiting to prevent more than x SOAP connections from a single IP per y interval.
3. alerts when any of the limits are reached.
I think we can do the above, but haven't looked into it yet.
One complication is how the scenario plays out:
1. One user falls for a phishing email. Sometimes the email is sent to only a small number of users, so we don't see it right away. In the recent incident, the spammer waited at least a week (until the Easter weekend) before actually doing anything.
2. Once that user is breached, a second phishing email pretending to be from our IT Department, usually describing a Zimbra upgrade or problem, is sent to ALL local users, and others in the addressbook.
3. User 1 starts sending waves of spam from Nigerian SOAP connections.
4. Subsequent users fall for more convincing (although still with poor English) second phish from internal address with Yahoo reply-to. Yahoo is useless at dealing with these accounts, so they continue to use yahoo (does yahoo still use Zimbra?)
5. We run a script to remove all messages based on phishing subject line when we notice it.
6. Nearly every attack starts on a weekend, so our response time is longer.
What I don't know is if I should spend any more time with the Zimbra spam solutions of spamassassin or dspam. We really don't have time to constantly tweak the system, and also deal with losing settings on every Zimbra update (although we are told where to make changes to avoid this, it still seems to occur every so often, or at least the changes are rendered ineffective). Should we be looking at Barracuda, IronPort, or others? We are already using barracuda for dnsbl, and it seems to be either the fastest to respond, or the most accurate, or both:
b.barracudacentral.org 358341
zen.spamhaus.org 27991
urbl.hostedemail.com=127.0.0.100 3520
bl.spamcop.net 1030
ix.dnsbl.manitu.net 156
hostkarma.junkemailfilter.com=127.0.0.2 68
hostkarma.junkemailfilter.com 41
urbl.hostedemail.com 32
db.wpbl.info 7
bb.barracudacentral.org 4
cbl.abuseat.org 3
db.wpbl.info=127.0.0.2 2
ubl.unsubscore.com 1
b.barracudacentral.org=127.0.0.2 1
=================================================
Total DNSBL rejections: 391197
We are a school with ~700 users, 450 of them students. Surprisingly, our staff are usually the worst offenders at falling for phishing attempts. All of the attacks appear to be monitored by a human until an account is breached, then the bot takes over. The recent wave of attacks on the weekend started with test connections from a proxy/anon ip block, then the bots hit from a Nigerian ip block. We have blocked all of these at the firewall, but there will be others soon I'm sure. Attempting to contact admins for either ip block have been ignored.
Hotspot Shield Free and Elite VPN Download for Internet Privacy, Security and to Access Blocked Sites - AnchorFree
74.115.0.0 - 74.115.7.255
Visafone Communications Limited - visafone.com.ng
41.71.128.0 - 41.71.255.255
Spam, bot attacks - is it time to get an antispam appliance?
-
swrightsls
- Advanced member
- Posts: 66
- Joined: Sat Sep 13, 2014 12:07 am
Spam, bot attacks - is it time to get an antispam appliance?
Hi,
[quote user="swrightsls"]Hello,
In the past 12 months, we have seen an increase in the amount of spam allowed through by spamassassin, along with numerous backscatter problems getting us blacklisted (even though support has told us how to stop backscatter, it still doesn't seem to work).
Most concerning is the number of phishing emails allowed through, which turn into a massive bot attack as soon as one user falls for it. This occurs every 2-3 months regardless of the number of time we tell users to NEVER release their password, and that we will NEVER ask them for it. The bots are specifically targeting Zimbra servers and using SOAP calls to send thousands of emails in a very short time. We usually find out when we get a blacklist alert, or the user notices bounces in their inbox, at which point they remember that they let their password out...
We need:
1. rate-limiting to prevent any account from sending more than x messages per y interval.
2. rate-limiting to prevent more than x SOAP connections from a single IP per y interval.
3. alerts when any of the limits are reached.
-cut-
[/QUOTE]
Rate-limit sending message could be achieved within Zimbra by using PolicyD :
[HowTo] Enabling CBPolicyD in Zimbra 7.1.1
or
[HowTo] Enabling CBPolicyD in Zimbra 8.0.0 and 8.0.1
Tips Zimbra : Cara Mengamankan Akses PolicyD Web Administration | PT. Excellent Infotama Kreasindo
Tips Zimbra : Membatasi Pengiriman Email/Rate Limit Sending Message dengan PolicyD | PT. Excellent Infotama Kreasindo
Sorry, the last two articles written in Bahasa Indonesia, I haven't convert it to English
Using anti spam appliance both software and hardware would be great. Those are some cloud-based anti spam appliance that easy to be integrated with Zimbra by only modify your mx records, such as spamtitan, mailcleaner, spamhero, etc.
[quote user="swrightsls"]Hello,
In the past 12 months, we have seen an increase in the amount of spam allowed through by spamassassin, along with numerous backscatter problems getting us blacklisted (even though support has told us how to stop backscatter, it still doesn't seem to work).
Most concerning is the number of phishing emails allowed through, which turn into a massive bot attack as soon as one user falls for it. This occurs every 2-3 months regardless of the number of time we tell users to NEVER release their password, and that we will NEVER ask them for it. The bots are specifically targeting Zimbra servers and using SOAP calls to send thousands of emails in a very short time. We usually find out when we get a blacklist alert, or the user notices bounces in their inbox, at which point they remember that they let their password out...
We need:
1. rate-limiting to prevent any account from sending more than x messages per y interval.
2. rate-limiting to prevent more than x SOAP connections from a single IP per y interval.
3. alerts when any of the limits are reached.
-cut-
[/QUOTE]
Rate-limit sending message could be achieved within Zimbra by using PolicyD :
[HowTo] Enabling CBPolicyD in Zimbra 7.1.1
or
[HowTo] Enabling CBPolicyD in Zimbra 8.0.0 and 8.0.1
Tips Zimbra : Cara Mengamankan Akses PolicyD Web Administration | PT. Excellent Infotama Kreasindo
Tips Zimbra : Membatasi Pengiriman Email/Rate Limit Sending Message dengan PolicyD | PT. Excellent Infotama Kreasindo
Sorry, the last two articles written in Bahasa Indonesia, I haven't convert it to English
Using anti spam appliance both software and hardware would be great. Those are some cloud-based anti spam appliance that easy to be integrated with Zimbra by only modify your mx records, such as spamtitan, mailcleaner, spamhero, etc.
Spam, bot attacks - is it time to get an antispam appliance?
We have a cluster of commercial antispam appliances in our deployment and these phishing attempts still get through. After submitting some samples to the provider, it looks like the number may have decreased.
I noticed what looked like Zimbra-targeted phishing/spam attempts start ~02/2012 and they've been coming off and on since then.
You're already doing some of the stuff below, but I'll go ahead and list it all...
Things I've done to try to mitigate these attacks:
- Ratelimit users: We deployed policyd as the poster above recommends and that helps a bit. I have noticed that some of the spammers will try to figure out what your rate limit is by trial and error. It's also important to make sure that you understand how policies are processed by policyd and that you have policies with higher sending limits that you can put power users into.
- Monitor logs realtime: I wrote a daemon that we use to monitor the mail log (File::Tail) and generate alarms based on policyd limits being reached. This daemon also looks for traffic from specific netblocks (e.g. Nigeria).
- Mine the policyd session data: I also wrote a perl script that runs as a cron job to mine recent data from the policyd session table every 5 minutes looking for traffic outside of specific regions (geocoded) to identify malicious traffic. Long term, this code will probably be modified to store usage profiles (src network, number of messages in given windows) for users to help reduce false positives.
- Monitor RBLs for your MTA IPs: Another good thing to do is monitor the popular RBLs for your MTA IPs. It's a bit late when this happens, but if all the other checks miss something, at least you can catch it as quickly as possible--hopefully before customers start calling. Google 'nagios check_rbl' for an example that works with nagios.
- Monitor your queues: We graph and monitor our MTA queues. If they exceed certain thresholds, we page someone. Typically the other methods catch compromises first, but it's another good fallback.
- Remove phishing attempts from mailboxes: We're using some in-house scripts to parse mail logs (after we find out about a phishing attempt) that allow us to pull the messages out of the users inboxes to help reduce the number of users that are exposed to the phishing attempt.
- Report!: When there are URLs (e.g. form to enter their username/password) in the emails, report them to the provider ASAP. We also try to submit abuse complaints to the providers who the netblocks are allocated to (for the SMTP/SOAP compromises). For the phishing messages, report abuse to the email service provider the messages came from and in the reply to if it's different. I have observed the same pattern of inaction that you have with some providers. We had a phishing form on zapsurvey.com that was left up for 5 months. I finally escalated it to their hosting provider and it was removed fairly quickly.
It's pertinent to note that you always want to "Invalidate sessions" (the ZM_AUTH_TOKEN) after you change a users password from the web admin GUI. We've tried to automate as much of this as we could for our operations group using the API.
It seems obvious, but also helps to search for logins for other users from the same offending IPs when you have a compromised account. The spammers seem to be lazy enough to test the accounts from one place. On the other side, we've seen exploited accounts get used from hundreds of different IPs before we had some of the mechanisms mentioned above in place when they were just relaying via SMTP.
There seems to have been a pattern to the SOAP-based compromises that I haven't spent much time looking at. They seem to like to put their message in the users signature (meaning you see an ldap modify for the signature) before the spam starts. There may be a way to try to use this to detect these types of account compromises. Again, I haven't spent enough time looking at them. The other pattern that I've started to see is in the recipient list. I haven't had time to look at the full dataset, but it appears that the phishing messages mostly go to a small-ish subset of our users in somewhat consistent groups. Analyzing inbound mail flow for messages sent to the same recipient groups may be useful.
Best of luck!
I noticed what looked like Zimbra-targeted phishing/spam attempts start ~02/2012 and they've been coming off and on since then.
You're already doing some of the stuff below, but I'll go ahead and list it all...
Things I've done to try to mitigate these attacks:
- Ratelimit users: We deployed policyd as the poster above recommends and that helps a bit. I have noticed that some of the spammers will try to figure out what your rate limit is by trial and error. It's also important to make sure that you understand how policies are processed by policyd and that you have policies with higher sending limits that you can put power users into.
- Monitor logs realtime: I wrote a daemon that we use to monitor the mail log (File::Tail) and generate alarms based on policyd limits being reached. This daemon also looks for traffic from specific netblocks (e.g. Nigeria).
- Mine the policyd session data: I also wrote a perl script that runs as a cron job to mine recent data from the policyd session table every 5 minutes looking for traffic outside of specific regions (geocoded) to identify malicious traffic. Long term, this code will probably be modified to store usage profiles (src network, number of messages in given windows) for users to help reduce false positives.
- Monitor RBLs for your MTA IPs: Another good thing to do is monitor the popular RBLs for your MTA IPs. It's a bit late when this happens, but if all the other checks miss something, at least you can catch it as quickly as possible--hopefully before customers start calling. Google 'nagios check_rbl' for an example that works with nagios.
- Monitor your queues: We graph and monitor our MTA queues. If they exceed certain thresholds, we page someone. Typically the other methods catch compromises first, but it's another good fallback.
- Remove phishing attempts from mailboxes: We're using some in-house scripts to parse mail logs (after we find out about a phishing attempt) that allow us to pull the messages out of the users inboxes to help reduce the number of users that are exposed to the phishing attempt.
- Report!: When there are URLs (e.g. form to enter their username/password) in the emails, report them to the provider ASAP. We also try to submit abuse complaints to the providers who the netblocks are allocated to (for the SMTP/SOAP compromises). For the phishing messages, report abuse to the email service provider the messages came from and in the reply to if it's different. I have observed the same pattern of inaction that you have with some providers. We had a phishing form on zapsurvey.com that was left up for 5 months. I finally escalated it to their hosting provider and it was removed fairly quickly.
It's pertinent to note that you always want to "Invalidate sessions" (the ZM_AUTH_TOKEN) after you change a users password from the web admin GUI. We've tried to automate as much of this as we could for our operations group using the API.
It seems obvious, but also helps to search for logins for other users from the same offending IPs when you have a compromised account. The spammers seem to be lazy enough to test the accounts from one place. On the other side, we've seen exploited accounts get used from hundreds of different IPs before we had some of the mechanisms mentioned above in place when they were just relaying via SMTP.
There seems to have been a pattern to the SOAP-based compromises that I haven't spent much time looking at. They seem to like to put their message in the users signature (meaning you see an ldap modify for the signature) before the spam starts. There may be a way to try to use this to detect these types of account compromises. Again, I haven't spent enough time looking at them. The other pattern that I've started to see is in the recipient list. I haven't had time to look at the full dataset, but it appears that the phishing messages mostly go to a small-ish subset of our users in somewhat consistent groups. Analyzing inbound mail flow for messages sent to the same recipient groups may be useful.
Best of luck!
Spam, bot attacks - is it time to get an antispam appliance?
We've been hammered recently with accounts getting compromised and using submission port to spam out. We do have appliance type service for incoming mail but phishing attempts still come in. And I still have an 'in-between' amavisd/spamassassin setup in between the applicance and zimbra. Those clearly have not been effective lately.
Looking at getting policyd setup now but would love to get some of these scripts as I do have power users and don't understand how to be able to allow power users some leeway.
I never did the jump from sendmail to postfix - worse case, I'm thinking on setting up a separate smtp server that would only allow submission and would rate control everything there. Internal users would still use zimbra (we split dns so only external users would be affected).
Worse is f'ing senderbase - once you hit that poor reputation, it's impossible to get off and these spammers are relentless, therefore we fall back to poor and are blocked by most appliances out there. Even senderbase blocks poor reputation - you cannot even send them a support request.
Please contact off-list and send me copies of those scripts. TIA.
Looking at getting policyd setup now but would love to get some of these scripts as I do have power users and don't understand how to be able to allow power users some leeway.
I never did the jump from sendmail to postfix - worse case, I'm thinking on setting up a separate smtp server that would only allow submission and would rate control everything there. Internal users would still use zimbra (we split dns so only external users would be affected).
Worse is f'ing senderbase - once you hit that poor reputation, it's impossible to get off and these spammers are relentless, therefore we fall back to poor and are blocked by most appliances out there. Even senderbase blocks poor reputation - you cannot even send them a support request.
Please contact off-list and send me copies of those scripts. TIA.
Spam, bot attacks - is it time to get an antispam appliance?
[quote user="su_A_ve"]
Looking at getting policyd setup now but would love to get some of these scripts as I do have power users and don't understand how to be able to allow power users some leeway.
[/QUOTE]
If you're still running the version of Zimbra in your signature, don't use the version of policyd that ships with it.
There's a bug in the connection code that delays messages sent by 1 second for each recipient. I think this was fixed in 7.2.2 (see the bug below).
I">https://bugzilla.zimbra.com/show_bug.cgi?id=77747
I spun up a separate version of policyd/mysql on a dedicated VM and just pointed MTA nodes at it.
This is the official Zimbra howto:
How-to for cbpolicyd - Zimbra :: Wiki
I used mysql to allow us to share session data across MTA nodes.
Make doubly sure that you're running enough policyd sessions to handle your inbound mail load. If you don't have enough, you'll see errors like this from postfix:
"451 4.3.5 Server configuration problem"
This isn't the only reason you could get the error, but it is a good indicator.
Finally, I'd recommend using syslog for policyd logging.
Looking at getting policyd setup now but would love to get some of these scripts as I do have power users and don't understand how to be able to allow power users some leeway.
[/QUOTE]
If you're still running the version of Zimbra in your signature, don't use the version of policyd that ships with it.
There's a bug in the connection code that delays messages sent by 1 second for each recipient. I think this was fixed in 7.2.2 (see the bug below).
I">https://bugzilla.zimbra.com/show_bug.cgi?id=77747
I spun up a separate version of policyd/mysql on a dedicated VM and just pointed MTA nodes at it.
This is the official Zimbra howto:
How-to for cbpolicyd - Zimbra :: Wiki
I used mysql to allow us to share session data across MTA nodes.
Make doubly sure that you're running enough policyd sessions to handle your inbound mail load. If you don't have enough, you'll see errors like this from postfix:
"451 4.3.5 Server configuration problem"
This isn't the only reason you could get the error, but it is a good indicator.
Finally, I'd recommend using syslog for policyd logging.
Spam, bot attacks - is it time to get an antispam appliance?
[quote user="pointer"]If you're still running the version of Zimbra in your signature, don't use the version of policyd that ships with it.
There's a bug in the connection code that delays messages sent by 1 second for each recipient. I think this was fixed in 7.2.2 (see the bug below).
I">https://bugzilla.zimbra.com/show_bug.cgi?id=77747
I spun up a separate version of policyd/mysql on a dedicated VM and just pointed MTA nodes at it.
[/QUOTE]
GREAT - not only I get 'Access denied' when trying to view the bug, I'll simply get zimbra's canned response "You need to update" - with 12K accounts this doesn't just happen in a day - let alone a month. This simply pushes me more towards recommending bailing out of Zimbra.
Always the same - release something - discover bugs, wait till next major release to fix - back to step one
There's a bug in the connection code that delays messages sent by 1 second for each recipient. I think this was fixed in 7.2.2 (see the bug below).
I">https://bugzilla.zimbra.com/show_bug.cgi?id=77747
I spun up a separate version of policyd/mysql on a dedicated VM and just pointed MTA nodes at it.
[/QUOTE]
GREAT - not only I get 'Access denied' when trying to view the bug, I'll simply get zimbra's canned response "You need to update" - with 12K accounts this doesn't just happen in a day - let alone a month. This simply pushes me more towards recommending bailing out of Zimbra.
Always the same - release something - discover bugs, wait till next major release to fix - back to step one