Torrid Networks was recently engaged by a large software company to investigate the rising spam issues with their corporate mail server. Company had over 2500 mailboxes with a software based anti-spam solution. The same solution was running from quite a while without any trouble, however spam levels soared recently. Moreover, the spam coming to user inboxes was not actually in terms of spam emails rather were bounced messages with mailbox owner as sender and some unknown email address as recipient to which owner never sent any email. Customer was anticipating a serious breach with its mail server or virus outbreak in the network and wanted an expert information security team to investigate.

The Problem

Torrid Networks started off with a routine SMTP tests to check if the server was open relay, missing spoofing protection or SMTP vulnerabilities. Things were well in place and server was configured pretty neat. Next step was to check the effectiveness of anti-spam solution, which was tested by submitting recent spam samples and were caught by the anti-spam engine. We quickly arrived at a conclusion that there weren’t any configuration issues neither with Mail Transport Agent (MTA) nor with the underlying spam engine.

As obvious, the investigation went towards the log analysis of the mail server and the problem was caught within few minutes. Emails were being sent using authentic mail accounts to external users from outside IP addresses. It was clearly indicating a compromise of email passwords to launch spam from customer’s email server. Our team checked with the customer on the compromised email accounts and customer agreed on carrying default password for newly created mailboxes. Moreover, it was an open source solution based email server without any mechanism in the email server to expire the passwords or to only allow enforcement of complex passwords. In the first step, customer was advised to change the passwords for all the users whose accounts were being abused by the spammers.

Problem deepened when the investigation team found the repeating pattern of the problem and another set of mailboxes got abused in the same way as previously. Logs were picked up once again to drill it to the bottom and the second round of the analysis wasn’t a surprise, it was a brute force attack against the mail server to guess the credentials and send bulk emails after successful compromise.

The Solution

It was time to work on brute force protection against mailboxes in the server utilizing the limited resources available with the customer. Mail server software being used was postfix and it by default creates a log file for every successful and failed login attempt at /var/log/maillog in the Linux server. Analysis suggested that the brute force attack was coming from distributed IP addresses and IP ranges so it was hard to identify which IP address or range to block. Fail2ban software, a free and open source software, was identified to protect the server from brute force attacks. It checks for the failed attempts in the log files using its inbuilt database of regular expressions and creates a TCPWrapper or iptables rule to block the source IP address. Fail2ban rules were enabled to block IP addresses brute forcing for email passwords or SSH passwords with a threshold of 2 failed attempts and limit of 10hrs to keep the IP blocked.

In couple of hours it was a long list of blocked IP addresses and everything looked perfect with SPAM reduced to almost zero. However, the server was still under the close observation. After a day or so, investigation team realized that the attack was targeted one and attackers were changing the strategy to counter every protection our team was suggesting to the customer. It was revealed during the third round of investigation that now the spammers were making only single attempt from one IP address bypassing the fail2ban rule to block an IP after 2 failed attempts and moreover the IP addresses were changing rapidly as opposed to the previous round of investigation. We couldn’t suggest to block the IP after single attempt because that way even legit users could get blocked leading to Denial of Services (DoS).

We started figuring out the patterns of the IP addresses which we thought of skipping by relying simply on fail2ban just to get a clue on which way to proceed with. Team analyzed nearly 100 IP addresses from where the brute force was being attempted. All those IP addresses were found to be Chinese IP addresses. Customer was using CISCO firewall but it was outsourced to external party for management and wanted us to derive a solution on the server itself rather coordinating with firewall management team to block Chinese IP addresses.

Torrid Networks’ team quickly gathered chinese IP address ranges from the public but authentic sources and prepared a list for iptables rules. SPAM came to halt by using the combination of denying all the Chinese ranges and implementing fail2ban rules. Customer is also recommended to update the mailing solution to the latest version to enforce password policies.

Click here to download the list of Chinese IP ranges used.