Non-modal dialog in full-screen Mail.app

You can use AppleScript to create a new message in a new, non-modal window, however, the window will be placed in one of your Desktop spaces, not in Mail's fullscreen space. You can save each of these as an application and access them through the Dock or the Script menu in the menu bar (enabled in AppleScript Editor's prefs).

New Message

tell application "Mail"
    make new outgoing message with properties {visible:true}
    activate
end tell

Reply to Sender

tell application "Mail"
    set theMessage to item 1 of (selection as list)
reply theMessage with opening window
    activate
end tell

Reply to All

tell application "Mail"
    set theMessage to item 1 of (selection as list)
    reply theMessage with opening window and reply to all
    activate
end tell

If you need to have the message viewer showing behind your message (e.g. to be able to copy something from an open email), then you can add the following lines to these scripts, immediately after the tell application "Mail" line:

set theMessage to selected messages of message viewer 1
set theMailBox to selected mailboxes of message viewer 1
set newViewer to make new message viewer
set selected mailboxes of newViewer to theMailBox
set selected messages of newViewer to theMessage
delay 1

This will create a new non-fullscreen mail viewer window, leaving the fullscreen window where it is. Then, after you send your message, you can simply close this new viewer.

Note: Depending on how quickly the new viewer opens, you may need to adjust the delay number. This delay (in seconds) allows the new viewer window enough time to finish opening before the script continues. If your machine is fast enough, you may be able to delete that line altogether.

I'm afraid Apple made a design choice to not allow non-modal dialogs in full-screen Mail. I believe their reasoning for this is that you would not be able to access these dialogs if you switched back to full-screen Mail, as the Mail UI would be on top of them.