Using Multiple page.open in Single Script

Solution 1:

The problem is that the second page.open is being invoked before the first one finishes, which can cause multiple problems. You want logic roughly like the following (assuming the filenames are given as command line arguments):

function handle_page(file){
    page.open(file,function(){
        ...
        page.evaluate(function(){
            ...do stuff...
        });
        page.render(...);
        setTimeout(next_page,100);
    });
}
function next_page(){
    var file=args.shift();
    if(!file){phantom.exit(0);}
    handle_page(file);
}
next_page();

Right, it's recursive. This ensures that the processing of the function passed to page.open finishes, with a little 100ms grace period, before you go to the next file.

By the way, you don't need to keep repeating

page = require('webpage').create();

Solution 2:

I've tried the accepted answer suggestions, but it doesn't work (at least not for v2.1.1).

To be accurate the accepted answer worked some of the time, but I still experienced sporadic failed page.open() calls, about 90% of the time on specific data sets.

The simplest answer I found is to instantiate a new page module for each url.

// first page
var urlA = "http://first/url"
var pageA = require('webpage').create()

pageA.open(urlA, function(status){
    if (status){
        setTimeout(openPageB, 100) // open second page call
    } else{
        phantom.exit(1)
    }
})

// second page
var urlB = "http://second/url"
var pageB = require('webpage').create()

function openPageB(){
    pageB.open(urlB, function(){
        // ... 
        // ...
    })
}

The following from the page module api documentation on the close method says:

close() {void}

Close the page and releases the memory heap associated with it. Do not use the page instance after calling this.

Due to some technical limitations, the web page object might not be completely garbage collected. This is often encountered when the same object is used over and over again. Calling this function may stop the increasing heap allocation.

Basically after I tested the close() method I decided using the same web page instance for different open() calls is too unreliable and it needed to be said.