This document summarizes Otto Kekäläinen's talk about investigating and recovering from a WordPress security breach at his company Seravo. On November 9th, 2018 four WordPress sites hosted by Seravo were compromised due to a vulnerability in the WP GDPR Compliance plugin. Seravo's security team launched an investigation that uncovered malicious user accounts, identified the vulnerable plugin as the entry point, and cleaned up the sites. The experience highlighted the importance of having an incident response plan even when security best practices are followed.
Automate your Kamailio Test Calls - Kamailio World 2024
How to Investigate and Recover from a Security Breach
1. How to Investigate and
Recover from a Security Breach
Real-life Experiences with WordPress
Otto Kekäläinen
@ottokekalainen
WordCamp Nordic
March 8, 2019
2. ● A CEO who codes at Seravo.com
● Written WP themes and plugins,
contributed to WordPress Core,
MySQL, MariaDB, Debian, Ubuntu,
Linux kernel, AppArmor…
● Linux and open source advocate
Otto Kekäläinen
3. I’ve spoken many
times about what
WordPress site
owners should
focus on to keep
their site secure...
wordpress.tv/?s=otto+kekäläinen
9. Weird siteurl – on all 4 sites!
Mistake by site admin? – No way
Targeted attack on one and same company? –
Plausible, but weird modus of operandi
Security breach? – Definitely!
$ wp option get siteurl
http://erealitatea.net
10. High alert – 4 sites down for investigation
1. First responder notifies security officer on-call
2. Process list saved and further PHP execution frozen
3. Customer notified about on-going security incident
4. Response escalation: 3 investigators working in parallel
12. Security breach investigation questions
● What is happening? Is it stopped?
● What happened before? When did this start?
● Is there malicious code somewhere? Backdoors
planted?
● What files or database contents has changed? Which
changes are malicious?
● Who did what? What IP addresses and other
identifiers are linked to what actions?
13. Security breach investigation questions
● How did they get in?
● What level of access did they gain?
● What data could have leaked?
● What was their motive?
● What damage was caused?
14. Investigation and recovery steps
1. Make a new backup
2. Compare backups
wp-backup-list-changes
diff -ur wordpress backup/wordpress
3. Check last WP and SSH logins
Store current state
Reveal file and database
changes
Detect unauthorized use based
on anomalies in timestamps or IP
geolocation
15. Investigation and recovery steps
4. wp core verify-checksums
wp plugin verify-checksums --all
wp package install seravo/wp-checksum
wp checksum all --details
Compare WordPress core,
plugin and theme files to
their original versions as
downloaded from wp.org
16. Modified plugin code found
..but was a false alert, modification most likely a mistake
by real plugin author who released two plugin variants
published with same version number.
$ wp checksum diff plugin entry-views inc/widget-entry-views.php
Executing diff /tmp/1541763665-4CBDYu.tmp
wordpress/htdocs/wp-content/plugins/entry-views/inc/widget-entry-views.php
49c49
< $this->WP_Widget(
---
> parent::__construct(
18. Investigation and recovery steps
5. wp user list
6. wp db query
'SELECT post_modified, id, post_title,
post_name, post_type FROM wp_posts
ORDER BY id DESC LIMIT 50;'
View recent new users
View recent new contents
19. Two suspected attacker user accounts
Variants of trollherten and different .ru email addresses
found on multiple of the investigated sites.
$ wp user list
+----+---------------+--------------+----------------------+---------------------+---------------+
| ID | user_login | display_name | user_email | user_registered | roles |
+----+---------------+--------------+----------------------+---------------------+---------------+
| 12 | t2trollherten | | trollherten@mail.com | 2018-11-08 13:36:03 | administrator |
| 13 | t3trollherten | | t3trollherten@bk.ru | 2018-11-08 14:42:09 | administrator |
+----+---------------+--------------+----------------------+---------------------+---------------+
20. Bingo!
usernames, timestamps,
IP addresses, email
+----+---------------+--------------+----------------------+---------------------+---------------+
| ID | user_login | display_name | user_email | user_registered | roles |
+----+---------------+--------------+----------------------+---------------------+---------------+
| 12 | t2trollherten | | trollherten@mail.com | 2018-11-08 13:36:03 | administrator |
| 13 | t3trollherten | | t3trollherten@bk.ru | 2018-11-08 14:42:09 | administrator |
+----+---------------+--------------+----------------------+---------------------+---------------+
These can be given to grep /data/log for log data mining
22. POST /wp-admin/admin-ajax.php
● Seravo does not log POST requests for good reasons
● So what was the payload that granted magic powers
to the attacker?
● Luckily we have other PHP and database logs...
23. Anomalies in database use
● Weird empty WordPress options value updates
● Unusual requests to database table wpgdprc_access_requests
● What plugin does that belong to?
$ grep -rF wpgdprc_access_requests wp-gdpr-compliance/
wp-gdpr-compliance/Includes/AccessRequest.php:
return $wpdb->base_prefix . 'wpgdprc_access_requests';
wp-gdpr-compliance/uninstall.php:
$wpdb->query("DROP TABLE IF EXISTS
`{$wpdb->base_prefix}wpgdprc_access_requests`");
26. Point of entry known
● The plugin WP GDPR Compliance Plugin most likely route
● Fix: remove it from all 4 sites
$ wp plugin deactivate --uninstall wp-gdpr-compliance
28. More information started coming in
● When the US woke up (in European afternoon) and published blogs the
Sucuri RSS feed we subscribe showed interesting stuff:
blog.sucuri.net/2018/11/erealitatea-net-hack-corrupts-websites-wit
h-wp-gdpr-compliance-plugin-vulnerability.html
● Then more and more other reports were found:
a. www.wordfence.com/blog/2018/11/privilege-escalation-flaw-in-
wp-gdpr-compliance-plugin-exploited-in-the-wild/
b. vitalisec.blogspot.com/2018/11/wp-gdpr-plugin-attack.html
c. wpvulndb.com/vulnerabilities/9144
29. Vulnerability details
● A SQL injection flaw in WP GDPR Compliance allowed a remote
attacker to set arbitrary WP option values
a. First allow anybody to register with users_can_register=1
b. Then set default_role=”administrator” for all new users
c. Register an account, log in and do whatever an admin can do
● Reported to wpvulndb.com by Adrian Mörchen / moewe.io
● Fixed in WP GDPR Compliance version 1.4.3
30. Fix issue globally for all our customers
commit 2ffb891415628ead16263e1fa09d78dac9e5dcdd
Author: Ville Korhonen
Date: Fri Nov 9 14:51:18 2018 +0200
Add WP GDPR Compliance plugin to urgent updates
WP GDPR Compliance < 1.4.3 has critical SQL injection
flaw which allows simple privilege escalation.
<https://plugins.trac.wordpress.org/changeset/1970313>
Added to Seravo’s update systems as an urgent update
32. Investigation and recovery steps
7. Based on findings, clean up the site
a. Recover clean version from backups
b. Remove malicious code and content
manually
8. As a precaution, reset all WordPress user
sessions and passwords
wp-reset-all-passwords
In this case option A was not
possible, but luckily option B
was quite easy as backups
showed only one potential
malware file was injected.
33. Investigation and recovery steps
9. As extra precaution, scan the site for
malware one more time when it is
otherwise deemed to be clean
Using Seravo’s custom
made WordPress/PHP
malware scanner
34. 2018-11-09 15:26:22 <redacted>.seravo.com RESOLVED
2018-11-09 15:17:49 <redacted>.seravo.com RESOLVED
2018-11-09 15:29:03 <redacted>.seravo.com RESOLVED
2018-11-09 15:20:24 <redacted>.seravo.com RESOLVED
All sites clean and finally back online
35. Investigation and recovery steps
10. Elevated monitoring and follow-up for
site once it has been re-opened, just in
case there was more attack avenues not
discovered during the investigation.
During the investigation Seravo sent 8 status update e-mails to the site
owner and the customer mobilized their own team to support the effort and
they also sent us valuable additional information. A few additional emails
from Seravo to the customer followed over the weekend and next week to
confirm all necessary measures had been completed.
36. Notification e-mail from new registration of
‘trollherten’ users
● Later we found out the site owner did get an email
notification from WordPress about the new user
named “trollherten” but since the e-mail was vague
and did not contain any alarming information, the
person who read the e-mail ignored it.
37. Luckily this was not a targeted attack
● Most likely the attacker just wanted to own the site
and use it to redirect traffic, spam, mount more
attacks against other sites etc.
● The site itself or the data it had was not the target and
most likely not used.
38. Be prepared: no security is perfect
● No plugin author makes perfect code.
● All plugins on the site were updated a
week earlier, the vulnerability was used
close to zero-day.
● Unreasonable for site admin to read
deeply all notification e-mails.
● Fact: sometimes even good security isn’t
enough. One also needs to have a
security incident response plan.
● We do. Do you?