Configure custom SSL certificate for RDP on Windows Server 2012 (and later) in Remote Administration mode?

It turns out that much of the configuration data for RDSH is stored in the Win32_TSGeneralSetting class in WMI in the root\cimv2\TerminalServices namespace. The configured certificate for a given connection is referenced by the Thumbprint value of that certificate on a property called SSLCertificateSHA1Hash.

UPDATE: Here's a generalized Powershell solution that grabs and sets the thumbprint of the first SSL cert in the computer's personal store. If your system has multiple certs, you should add a -Filter option to the gci command to make sure you reference the correct cert. I've left my original answer intact below this for reference.

# get a reference to the config instance
$tsgs = gwmi -class "Win32_TSGeneralSetting" -Namespace root\cimv2\terminalservices -Filter "TerminalName='RDP-tcp'"

# grab the thumbprint of the first SSL cert in the computer store
$thumb = (gci -path cert:/LocalMachine/My | select -first 1).Thumbprint

# set the new thumbprint value
swmi -path $tsgs.__path -argument @{SSLCertificateSHA1Hash="$thumb"}

In order to get the thumbprint value

1. Open the properties dialog for your certificate and select the Details tab
2. Scroll down to the Thumbprint field and copy the space delimited hex string into something like Notepad
3. Remove all the spaces from the string. You'll also want to watch out for and remove a non-ascii character that sometimes gets copied just before the first character in the string. It's not visible in Notepad.
4. This is the value you need to set in WMI. It should look something like this: 1ea1fd5b25b8c327be2c4e4852263efdb4d16af4.

Now that you have the thumbprint value, here's a one-liner you can use to set the value using wmic:

wmic /namespace:\\root\cimv2\TerminalServices PATH Win32_TSGeneralSetting Set SSLCertificateSHA1Hash="THUMBPRINT"

Or if PowerShell is your thing, you can use this instead:

$path = (Get-WmiObject -class "Win32_TSGeneralSetting" -Namespace root\cimv2\terminalservices -Filter "TerminalName='RDP-tcp'").__path
Set-WmiInstance -Path $path -argument @{SSLCertificateSHA1Hash="THUMBPRINT"}

Note: the certificate must be in the 'Personal' Certificate Store for the Computer account.

If you getting "Invalid Parameter" when you trying Ryan's solution, make sure you are using elevated command prompt (run as administrator).