Check if a dialog is displayed with Espresso

1. To verify if dialog appears you can simply check if View with a text that present inside the dialog is shown:

   onView(withText("dialogText")).check(matches(isDisplayed()));
   

   or, based on text with id

   onView(withId(R.id.myDialogTextId)).check(matches(allOf(withText(myDialogText), isDisplayed()));
   

2. To click on dialogs button do this (button1 - OK, button2 - Cancel):

   onView(withId(android.R.id.button1)).perform(click());
   

   UPDATE

3. I think is possible since Espresso has multi window support.
4. Not sure about clicking outside the custom dialog view but for checking if it is displaying or not you have to create your custom matcher and check inside it.

I currently use this and it seems to work fine.

onView(withText(R.string.my_title))
    .inRoot(isDialog()) // <---
    .check(matches(isDisplayed()));

If you have an AlertDialog like that:

You can check if the components are displayed:

int titleId = mActivityTestRule.getActivity().getResources()
        .getIdentifier( "alertTitle", "id", "android" );

onView(withId(titleId))
        .inRoot(isDialog())
        .check(matches(withText(R.string.my_title)))
        .check(matches(isDisplayed()));

onView(withId(android.R.id.text1))
        .inRoot(isDialog())
        .check(matches(withText(R.string.my_message)))
        .check(matches(isDisplayed()));

onView(withId(android.R.id.button2))
        .inRoot(isDialog())
        .check(matches(withText(android.R.string.no)))
        .check(matches(isDisplayed()));

onView(withId(android.R.id.button3))
        .inRoot(isDialog())
        .check(matches(withText(android.R.string.yes)))
        .check(matches(isDisplayed()));

and perform an action:

onView(withId(android.R.id.button3)).perform(click());

To answer question 4, which the accepted answer does not, I modified the following code, which I found here on Stack Overflow (link) for testing whether a Toast was displayed.

@NonNull
public static ViewInteraction getRootView(@NonNull Activity activity, @IdRes int id) {
    return onView(withId(id)).inRoot(withDecorView(not(is(activity.getWindow().getDecorView()))));
}

The id passed in is the id of a View currently displayed in your dialog. You could also write the method like so:

@NonNull
public static ViewInteraction getRootView(@NonNull Activity activity, @NonNull String text) {
    return onView(withText(text)).inRoot(withDecorView(not(is(activity.getWindow().getDecorView()))));
}

And now it's looking for a View containing a particular text string.

Use it like so:

getRootView(getActivity(), R.id.text_id).perform(click());