Mocking $modal in AngularJS unit tests

When you spy on the $modal.open function in the beforeEach,

spyOn($modal, 'open').andReturn(fakeModal);

or 

spyOn($modal, 'open').and.returnValue(fakeModal); //For Jasmine 2.0+

you need to return a mock of what $modal.open normally returns, not a mock of $modal, which doesn’t include an open function as you laid out in your fakeModal mock. The fake modal must have a result object that contains a then function to store the callbacks (to be called when the OK or Cancel buttons are clicked on). It also needs a close function (simulating an OK button click on the modal) and a dismiss function (simulating a Cancel button click on the modal). The close and dismiss functions call the necessary call back functions when called.

Change the fakeModal to the following and the unit test will pass:

var fakeModal = {
    result: {
        then: function(confirmCallback, cancelCallback) {
            //Store the callbacks for later when the user clicks on the OK or Cancel button of the dialog
            this.confirmCallBack = confirmCallback;
            this.cancelCallback = cancelCallback;
        }
    },
    close: function( item ) {
        //The user clicked OK on the modal dialog, call the stored confirm callback with the selected item
        this.result.confirmCallBack( item );
    },
    dismiss: function( type ) {
        //The user clicked cancel on the modal dialog, call the stored cancel callback
        this.result.cancelCallback( type );
    }
};

Additionally, you can test the cancel dialog case by adding a property to test in the cancel handler, in this case $scope.canceled:

$scope.modalInstance.result.then(function (selectedItem) {
    $scope.selected = selectedItem;
}, function () {
    $scope.canceled = true; //Mark the modal as canceled
    $log.info('Modal dismissed at: ' + new Date());
});

Once the cancel flag is set, the unit test will look something like this:

it("should cancel the dialog when dismiss is called, and $scope.canceled should be true", function () {
    expect( scope.canceled ).toBeUndefined();

    scope.open(); // Open the modal
    scope.modalInstance.dismiss( "cancel" ); //Call dismiss (simulating clicking the cancel button on the modal)
    expect( scope.canceled ).toBe( true );
});

To add to Brant's answer, here is a slightly improved mock that will let you handle some other scenarios.

var fakeModal = {
    result: {
        then: function (confirmCallback, cancelCallback) {
            this.confirmCallBack = confirmCallback;
            this.cancelCallback = cancelCallback;
            return this;
        },
        catch: function (cancelCallback) {
            this.cancelCallback = cancelCallback;
            return this;
        },
        finally: function (finallyCallback) {
            this.finallyCallback = finallyCallback;
            return this;
        }
    },
    close: function (item) {
        this.result.confirmCallBack(item);
    },
    dismiss: function (item) {
        this.result.cancelCallback(item);
    },
    finally: function () {
        this.result.finallyCallback();
    }
};

This will allow the mock to handle situations where...

You use the modal with the .then(), .catch() and .finally() handler style instead passing 2 functions (successCallback, errorCallback) to a .then(), for example:

modalInstance
    .result
    .then(function () {
        // close hander
    })
    .catch(function () {
        // dismiss handler
    })
    .finally(function () {
        // finally handler
    });

Since modals use promises you should definitely use $q for such things.

Code becomes:

function FakeModal(){
    this.resultDeferred = $q.defer();
    this.result = this.resultDeferred.promise;
}
FakeModal.prototype.open = function(options){ return this;  };
FakeModal.prototype.close = function (item) {
    this.resultDeferred.resolve(item);
    $rootScope.$apply(); // Propagate promise resolution to 'then' functions using $apply().
};
FakeModal.prototype.dismiss = function (item) {
    this.resultDeferred.reject(item);
    $rootScope.$apply(); // Propagate promise resolution to 'then' functions using $apply().
};

// ....

// Initialize the controller and a mock scope
beforeEach(inject(function ($controller, $rootScope) {
    scope = $rootScope.$new();
    fakeModal = new FakeModal();
    MainCtrl = $controller('MainCtrl', {
        $scope: scope,
        $modal: fakeModal
   });
}));

// ....

it("should cancel the dialog when dismiss is called, and  $scope.canceled should be true", function () {
    expect( scope.canceled ).toBeUndefined();

    fakeModal.dismiss( "cancel" ); //Call dismiss (simulating clicking the cancel button on the modal)
    expect( scope.canceled ).toBe( true );
});

Brant's answer was clearly awesome, but this change made it even better for me:

  fakeModal =
    opened:
      then: (openedCallback) ->
        openedCallback()
    result:
      finally: (callback) ->
        finallyCallback = callback

then in the test area:

  finallyCallback()

  expect (thing finally callback does)
    .toEqual (what you would expect)