What Does 'Then' Really Mean in CasperJS

I'm using CasperJS to automate a series of clicks, completed forms, parsing data, etc through a website.

Casper seems to be organized into a list of preset steps in the form of then statements (see their example here: http://casperjs.org/quickstart.html) but it's unclear what triggers the next statement to actually run.

For example, does then wait for all pending requests to complete? Does injectJS count as a pending request? What happens if I have a then statement nested - chained to the end of an open statement?

casper.thenOpen('http://example.com/list', function(){
    casper.page.injectJs('/libs/jquery.js');
    casper.evaluate(function(){
        var id = jQuery("span:contains('"+itemName+"')").closest("tr").find("input:first").val();
        casper.open("http://example.com/show/"+id); //what if 'then' was added here?
    });
});

casper.then(function(){
    //parse the 'show' page
});

I'm looking for a technical explanation of how the flow works in CasperJS. My specific problem is that my last then statement (above) runs before my casper.open statement & I don't know why.

then() basically adds a new navigation step in a stack. A step is a javascript function which can do two different things:

1. waiting for the previous step - if any - being executed
2. waiting for a requested url and related page to load

Let's take a simple navigation scenario:

var casper = require('casper').create();

casper.start();

casper.then(function step1() {
    this.echo('this is step one');
});

casper.then(function step2() {
    this.echo('this is step two');
});

casper.thenOpen('http://google.com/', function step3() {
    this.echo('this is step 3 (google.com is loaded)');
});

You can print out all the created steps within the stack like this:

require('utils').dump(casper.steps.map(function(step) {
    return step.toString();
}));

That gives:

$ casperjs test-steps.js
[
    "function step1() { this.echo('this is step one'); }",
    "function step2() { this.echo('this is step two'); }",
    "function _step() { this.open(location, settings); }",
    "function step3() { this.echo('this is step 3 (google.com is loaded)'); }"
]

Notice the _step() function which has been added automatically by CasperJS to load the url for us; when the url is loaded, the next step available in the stack — which is step3() — is called.

When you have defined your navigation steps, run() executes them one by one sequentially:

casper.run();

Footnote: the callback/listener stuff is an implementation of the Promise pattern.

then() merely registers a series of steps.

run() and its family of runner functions, callbacks, and listeners, are all what actually do the work of executing each step.

Whenever a step is completed, CasperJS will check against 3 flags: pendingWait, loadInProgress, and navigationRequested. If any of those flags is true, then do nothing, go idle until a later time (setInterval style). If none of those flags is true, then the next step will get executed.

As of CasperJS 1.0.0-RC4, a flaw exists, where, under certain time-based circumstances, the "try to do next step" method will be triggered before CasperJS had the time to raise either one of the loadInProgress or navigationRequested flags. The solution is to raise one of those flags before leaving any step where those flags are expected to be raised (ex: raise a flag either before or after asking for a casper.click()), maybe like so:

(Note: This is only illustrative, more like psuedocode than proper CasperJS form...)

step_one = function(){
    casper.click(/* something */);
    do_whatever_you_want()
    casper.click(/* something else */); // Click something else, why not?
    more_magic_that_you_like()
    here_be_dragons()
    // Raise a flag before exiting this "step"
    profit()
}

To wrap up that solution into a single-line of code, I introduced blockStep() in this github pull request, extending click() and clickLabel() as a means to help guarantee that we get the expected behaviour when using then(). Check out the request for more info, usage patterns, and minimum test files.