Detecting headless browsers

Headless Browser
Hide & Seek
Sergey Shekyan, Bei Zhang
Shape Security

Who We Are
• Bei Zhang
Senior Software Engineer at Shape Security, focused on analysis
and countermeasures of automated web attacks. Previously, he
worked at the Chrome team at Google with a focus on the Chrome
Apps API. His interests include web security, source code analysis,
and algorithms.
• Sergey Shekyan
Principal Engineer at Shape Security, focused on the development of
the new generation web security product. Prior to Shape Security, he
spent 4 years at Qualys developing their on demand web
application vulnerability scanning service. Sergey presented
research at security conferences around the world, covering various
information security topics.

What Is a Headless Browser and How it Works
Scriptable browser environment that doesn’t require GUI
• Existing browser layout engine with bells and whistles
(PhantomJS - WebKit, SlimerJS - Gecko, TriﬂeJS - Trident)
• Custom software that models a browser (ZombieJS,
HtmlUnit)
• Selenium (WebDriver API)

What Is a Headless Browser and How it Works
Discussion will focus on PhantomJS:
• Backed by WebKit engine
• Cross-platform
• Popular
• True headless

PhantomJS World
PhantomJS
JavaScript Context
QWebFrame
QtWebKit
Web Page
JavaScript
Context
Control
Callback
Injection
PageEvent
Callbacks are
serialized
var page = require('webpage').create();
page.open(url, function(status) {
var title = page.evaluate(function() {
return document.title;
});
console.log('Page title is ' + title);
});

Legitimate uses and how you can beneﬁt
• Web Application functional and performance
testing
• Crawler that can provide certain amount of
interaction to reveal web application topology,
Automated DOM XSS, CSRF detection
• SEO (render dynamic web page into static
HTML to feed to search engines)
• Reporting, image generation

Malicious Use of Headless Browser
• Fuzzing
• Botnet
• Content scraping
• Login brute force attacks
• Click fraud
• Bidding wars
Web admins tend to block PhantomJS in production,
so pretending to be a real browser is healthy choice

How It Is Different From a Real Browser
• Outdated WebKit engine (close to Safari 5 engine, 4 y.o.)
• Uses Qt Framework’s QtWebKit wrapper around WebKit
• Qt rendering engine
• Qt network stack, SSL implementation
• Qt Cookie Jar, that doesn’t support RFC 2965
• No Media Hardware support (no video and audio)
• Exposes window.callPhantom and window._phantom
• No sandboxing

Headless Browser Seek
• Look at user agent string
if (/PhantomJS/.test(window.navigator.userAgent)) {
console.log(‘PhantomJS environment detected.’);
}

Headless Browser Hide
• Making user-agent (and navigator.userAgent) a
“legitimate” one:
var page = require(‘webpage').create();
page.settings.userAgent = ‘Mozilla/5.0 (Macintosh; Intel Mac
OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0';

Score board
Phantom Web site
User Agent String Win Lose

• Sniff for PluginArray content
if (!(navigator.plugins instanceof PluginArray) ||
navigator.plugins.length == 0) {
   console.log("PhantomJS environment detected.");
  } else {
   console.log("PhantomJS environment not detected.");
  }

• Fake navigator object, populate PluginArray with whatever values you need.
• Spooﬁng Plugin objects inside the PluginArray is tedious and hard.
• Websites can actually create a plugin to test it.
• CONCLUSION: Not a good idea to spoof plugins.
page.onInitialized = function () {
   page.evaluate(function () {
   var oldNavigator = navigator;
   var oldPlugins = oldNavigator.plugins;
   var plugins = {};
   plugins.length = 1;
   plugins.__proto__ = oldPlugins.__proto__;
   window.navigator = {plugins: plugins};
   window.navigator.__proto__ = oldNavigator.__proto__;
   });
};

Score board
Phantom Web site
Inspect PluginArray Lose Win

• Alert/prompt/conﬁrm popup suppression timing
detection
var start = Date.now();
  prompt('I`m just kidding');
  var elapse = Date.now() - start;
  if (elapse < 15) {
   console.log("PhantomJS environment detected. #1");
  } else {
  }

• Can’t use setTimeout, but blocking the callback by
all means would work
page.onAlert = page.onConﬁrm = page.onPrompt = function ()
{
   for (var i = 0; i < 1e8; i++) {
   }
   return "a";
};

Score board
Phantom Web site
Timed alert() Lose Win

• Default order of headers is consistently different
in PhantomJS. Camel case in some header
values is also a good point to look at.
PhantomJS 1.9.7
GET / HTTP/1.1
User-Agent:
Accept:
Connection: Keep-Alive
Accept-Encoding:
Accept-Language:
Host:
Chrome 37
GET / HTTP/1.1
Host:
Connection: keep-alive
Accept:
User-Agent:
Accept-Encoding:
Accept-Language:

• A custom proxy server in front of PhantomJS
instance that makes headers look consistent with
user agent string

Score board
Phantom Web site
HTTP Header order Win Lose

• PhantomJS exposes APIs:
• window.callPhantom
• window._phantom // not documented.
if (window.callPhantom || window._phantom) {
} else {
}

• store references to original callPhantom, _phantom
• delete window.callPhantom, window._phantom
page.onInitialized = function () {
   page.evaluate(function () {
   var p = window.callPhantom;
   delete window._phantom;
   delete window.callPhantom;
   Object.deﬁneProperty(window, "myCallPhantom", {
   get: function () { return p;},
   set: function () {}, enumerable: false});
   setTimeout(function () { window.myCallPhantom();}, 1000);
   });
};
page.onCallback = function (obj) { console.log(‘proﬁt!'); };
Unguessable name

Score board
Phantom Web site
window.callPhantom Win Lose

• Spooﬁng DOM API properties of real browsers:
• WebAudio
• WebRTC
• WebSocket
• Device APIs
• FileAPI
• WebGL
• CSS3 - not observable. Defeats printing.
• Our research on WebSockets: http://goo.gl/degwTr

Score board
Phantom Web site
HTML5 features Lose Win

• Signiﬁcant difference in JavaScript Engine: bind() is
not deﬁned in PhantomJS prior to version 2
(function () {
   if (!Function.prototype.bind) {
   return;
   }
  })();
Function.prototype.bind = function () {
var func = this;
var self = arguments[0];
var rest = [].slice.call(arguments, 1);
return function () {
var args = [].slice.call(arguments, 0);
return func.apply(self, rest.concat(args));
};
};

• Detecting spoofed Function.prototype.bind for
PhantomJS prior to version 2
(function () {
   return;
   }
   if (Function.prototype.bind.toString().replace(/bind/g, 'Error') !=
Error.toString()) {
   return;
   }
  })();

• Spooﬁng Function.prototype.toString
function functionToString() {
   if (this === bind) {
   return nativeFunctionString;
   }
   return oldCall.call(oldToString, this);
}

• Detecting spoofed Function.prototype.bind for
PhantomJS prior to version 2
(function () {
   return;
   }
   if (Function.prototype.bind.toString().replace(/bind/g, 'Error') !=
Error.toString()) {
   return;
   }
   if (Function.prototype.toString.toString().replace(/toString/g, 'Error') != Error.toString()) {
   return;
   }
  })();

• Spooﬁng Function.prototype.toString.toString
function functionToString() {
   if (this === bind) {
   return nativeFunctionString;
   }
   if (this === functionToString) {
   return nativeToStringFunctionString;
   }
   if (this === call) {
   return nativeCallFunctionString;
   }
   if (this === apply) {
   return nativeApplyFunctionString;
   }
   var idx = indexOfArray(bound, this);
   if (idx >= 0) {
   return nativeBoundFunctionString;
   }
   return oldCall.call(oldToString, this);
}

Score board
Phantom Web site
Function.prototype.bind Win Lose

• PhantomJS2 is very EcmaScript5 spec compliant,
so checking for outstanding JavaScript features is
not going to work

• Stack trace generated by PhantomJs
• Stack trace generated by SlimerJS
• Hmm, there is something common…
at querySelectorAll (phantomjs://webpage.evaluate():9:10)
at phantomjs://webpage.evaluate():19:30
at phantomjs://webpage.evaluate():20:7
at global code (phantomjs://webpage.evaluate():20:13)
at evaluateJavaScript ([native code])
at global code (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14)
Element.prototype.querySelectorAll@phantomjs://webpage.evaluate():9
@phantomjs://webpage.evaluate():19
@phantomjs://webpage.evaluate():20

var err;
try {
null[0]();
} catch (e) {
err = e;
}
if (indexOfString(err.stack, 'phantomjs') > -1) {
} else {
console.log("PhantomJS environment is not detected.");
It is not possible to override the thrown TypeError
generic indexOf(), can be spoofed, deﬁne your own

• Modifying webpage.cpp:
QVariant WebPage::evaluateJavaScript(const QString &code)
{
QVariant evalResult;
QString function = "(" + code + ")()";
evalResult = m_currentFrame->evaluateJavaScript(function,
QString("phantomjs://webpage.evaluate()"));
return evalResult;
}
at
at
at
at
at
• Produces

• Spoofed PhantomJs vs Chrome:
at querySelectorAll (Object.InjectedScript:9:10)
at Object.InjectedScript:19:30
at Object.InjectedScript:20:7
at global code (Object.InjectedScript:20:13)
at
TypeError: Cannot read property '0' of null
at HTMLDocument.Document.querySelectorAll.Element.querySelectorAll [as
querySelectorAll] (<anonymous>:21:5)
at <anonymous>:2:10
at Object.InjectedScript._evaluateOn (<anonymous>:730:39)
at Object.InjectedScript._evaluateAndWrap (<anonymous>:669:52)
at Object.InjectedScript.evaluate (<anonymous>:581:21)

Score board
Phantom Web site
Stack trace Lose Win

Score board
Phantom Web site
Stack trace Lose Win
SCORE: 4 4

How to turn a headless browser against the
attacker
The usual picture:
• PhantomJS running as root
• No sandboxing in PhantomJS
• Blindly executing untrusted JavaScript
• outdated third party libs (libpng, libxml, etc.)
Can possibly lead to:
• abuse of PhantomJS
• abuse of OS running PhantomJS

• In a lot of cases --web-security=false is used in
PhantomJS
var xhr = new XMLHttpRequest();
xhr.open('GET', 'ﬁle:/etc/hosts', false);
xhr.onload = function () {
console.log(xhr.responseText);
};
xhr.onerror = function (e) {
console.log('Error: ' + JSON.stringify(e));
};
xhr.send();

• Obfuscate, randomize the output, randomize the modified API
call
var _0x34c7=["x68x74x6D
x6C","x71x75x65x72x79x53x65x6Cx65x63x74x6F
x72x41x6Cx6C","x70x72x6Fx74x6F
x74x79x70x65","x73x74x61x63x6B","x70x68x61x6E
x74x6Fx6Dx6Ax73","x61x70x70x6Cx79"];var
html=document[_0x34c7[1]](_0x34c7[0]);var
apsdk=document[_0x34c7[1]];Document[_0x34c7[2]]
[_0x34c7[1]]=Element[_0x34c7[2]][_0x34c7[1]]=function ()
{var _0xad6dx3;try{null[0]();} catch(e)
{_0xad6dx3=e;} ;if(indexOfString(_0xad6dx3[_0x34c7[3]],_0
x34c7[4])>-1){return html;} else {return apsdk[_0x34c7[5]]
(this,arguments);};};

Tips for Using Headless Browsers Safely
• If you don’t need a full blown browser engine, don’t
use it
• Do not run with ‘- -web-security=false’ in production,
try not to do that in tests as well.
• Avoid opening arbitrary page from the Internet
• No root or use chroot
• Use child processes to run webpages
• If security is a requirement, use alternatives or on a
controlled environment, use Selenium

Tips For Web Admins
• It is not easy to unveil a user agent. Make sure you want
to do it.
• Do sniff for headless browsers (some will bail out)
• Combine several detection techniques
• Reject known unwanted user agents (5G Blacklist 2013
is a good start)
• Alter DOM API if headless browser is detected
• DoS
• Pwn

Start Detecting Already
Ready to use examples:
github.com/ikarienator/phantomjs_hide_and_seek

References
• http://www.sba-research.org/wp-content/uploads/publications/jsfingerprinting.pdf
• https://kangax.github.io/compat-table/es5/
• https://media.blackhat.com/us-13/us-13-Grossman-Million-Browser-Botnet.pdf
• http://ariya.ofilabs.com/2011/10/detecting-browser-sniffing-2.html
• http://www.darkreading.com/attacks-breaches/ddos-attack-used-headless-browsers-
in-150-hour-siege/d/d-id/1140696?
• http://vamsoft.com/downloads/articles/vamsoft-headless-browsers-in-forum-spam.pdf
• http://blog.spiderlabs.com/2013/02/server-site-xss-attack-detection-with-modsecurity-
and-phantomjs.html
• http://googleprojectzero.blogspot.com/2014/07/pwn4fun-spring-2014-safari-part-
i_24.html

Thank you!
@ikarienator
@sshekyan

Detecting headless browsers

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Detecting headless browsers

Ähnlich wie Detecting headless browsers (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Detecting headless browsers