How to overcome an HTMLUnit ScriptException?

Solution 1:

HtmlUnit does not play well with JavaScript. It will frequently throw errors complaining about variables or functions not defined.

In that sense, real life browsers (FireFox, Internet Explorer, Chrome, etc) are much more flexible. That means that they will allow syntactically incorrect pieces of HTML and JavaScript (eg: not defining functions or not ending HTML tags).

HtmlUnit expects everything to be (almost) perfect. Although, it will fix some missing ending HTML tags, in general, it expects the code in the pages not to contain any kind of error. Furthermore, even if everything looks correct HtmlUnit might even complain.

Some items for you to think about are:

• The most important one is switching between different BrowserVersions. You can set them when creating the WebClient object. Internet Explorer (ironically) has proven to give me the best results when it comes to interpreting JavaScript
• Make sure your HTML and JavaScript code are both correct
• Avoid using of complex libraries (jQuery seems to be properly supported)
• Try to use non-minimized versions of libraries
• If you happen to be using jQuery (or other similar libraries) avoid complex jQuery methods (eg: dynamically adding events to elements)

Of course, those comments would apply if you have control over the source code you're fetching from the server. Sometimes, this is not the case. In this situation your hands are even more tied.

One option, would be to suppress the exception with:

webClient.getOptions().setThrowExceptionOnScriptError(false);

Although, this will get you through the exception won't correct any JavaScript error. That means that if the JS piece of code that is throwing this exception happens to be crucial in your logic, I mean, you absolutely depend on the result of the execution of that code then you can not let HtmlUnit handle your JS. If this happens to be the result of an AJAX request then you can issue the request yourself manually instead of letting HtmlUnit do so.

On the other hand, if the JS code that is giving you trouble is not critical in your logic, I mean, it might just be hiding an element or changing a color that you don't care about, then suppressing the exception would be the way to go.

There aren't many options left.

Solution 2:

Try setting your web client not to throw exception:

client.getOptions().setThrowExceptionOnScriptError(false);

Solution 3:

I had this same problem when I get some website with method:

webClient.getPage("http://somepage.com");

If you don't need use JavaScript to working with websites you can write:

webClient.getOptions().setJavaScriptEnabled(false);

In my case it works well and script is performing immediately (when I used only webClient.getOptions().setThrowExceptionOnScriptError(false) then the script always tried perform bad JavaScript code and for about 10 seconds writing out exception messages in the console, so I don't recommend to use it).