Digital signature in c# without using BouncyCastle

Without using 3rd party BouncyCastle library, is there a way to read a custom private key and sign the message ? (sha256 hash+encryption using private key)

Technically, yes. Depending on what kind of key you have the answer gets more tricky.

Edit (2019-Oct): .NET Core 3.0 has built-in support for all of these formats, in their DER-encoded (vs PEM-encoded) forms. I'm adding the .NET Core 3.0+ answers after a sub-heading within each file format.

PKCS#8 PrivateKeyInfo (PEM "BEGIN PRIVATE KEY")

If you have this type of file, and you're on .NET 4.6 or higher, then yes. You need to have the DER encoded (vs PEM encoded) data blob (see below if it's PEM).

using (CngKey key = CngKey.Import(blob, CngKeyBlobFormat.Pkcs8PrivateBlob))
using (RSA rsa = new RSACng(key))
{
    return rsa.SignData(data, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
}

4.6 is required for for RSA, 4.6.1 for ECDSA, 4.6.2 for DSA.

.NET Core 3.0+ PKCS#8 PrivateKeyInfo

The ImportPkcs8PrivateKey method is declared on AsymmetricAlgorithm, and all asymmetric built-in types (RSA, DSA, ECDsa, ECDiffieHellman) support it.

using (RSA rsa = RSA.Create())
{
    rsa.ImportPkcs8PrivateKey(blob, out _);
    return rsa.SignData(data, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
}

PKCS#8 EncryptedPrivateKeyInfo (PEM "BEGIN ENCRYPTED PRIVATE KEY")

Congratulations, your private key transport is strong. Sadly, this requires the maximum amount of code to be written if you want to actually handle it. You don't want to handle it. You really, really, want to

Create a certificate for the key
Put the cert and key into a PFX file
Load the PFX into an X509Certificate2
Use cert.GetRSAPrivateKey(), cert.GetDSAPrivateKey(), or cert.GetECDsaPrivateKey() (as appropriate)

See How is a private key encrypted in a pem certificate?, and then continue to the next section for the primer on the hard way. You have a lot more work than it will talk about, though. You need to read the file, understand the encryption scheme and parameters, decrypt the blob, then use CNG for reading the PKCS#8, or just keep diving down the rabbit hole and enjoy your file parser.

.NET Core 3.0+ PKCS#8 EncryptedPrivateKeyInfo

The ImportEncryptedPkcs8PrivateKey method is declared on AsymmetricAlgorithm, and all asymmetric built-in types (RSA, DSA, ECDsa, ECDiffieHellman) support it.

using (RSA rsa = RSA.Create())
{
    rsa.ImportEncryptedPkcs8PrivateKey(password, blob, out _);
    return rsa.SignData(data, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
}

PKCS#1 RSAPrivateKey (PEM "BEGIN RSA PRIVATE KEY")

You're at the unfortunate confluence of "relatively simple" and "relatively hard" that is known to math majors as "an exercise left to the reader".

Strongly consider doing the PFX approach from EncryptedPrivateKeyInfo. Alternatively, you can do this in custom code. Custom code? Okay, let's do this. The reference texts that you need at this point are

ITU.T-REC X.680-201508.

This defines the ASN.1 language, which tells you how to read the RSAPrivateKey (et al) object structure definition.
For RSAPrivateKey this is mostly optional, since there aren't many nuances to SEQUENCE that it uses, and INTEGER is pretty straightforward.

ITU.T-REC X.690-201508

This document describes the BER (and CER) and DER encoding rules for ASN.1.
These key files are in DER. (Unless they're in PEM, but we'll fix that soon)

The RFC appropriate to your object type.

RSAPrivateKey (RFC 3447)
EncryptedPrivateKeyInfo (RFC 5208)
PrivateKeyInfo (also RFC 5208)
Other formats are in other RFCs.

Okay, let's proceed.

If the file is PEM encoded ("-----BEGIN RSA PRIVATE KEY-----" or "-----BEGIN PRIVATE KEY-----", etc) you need to "un-PEM" it.

The PEM format is
- (newline or beginning of file)
- 5 hyphens, BEGIN, space, the type identifier, 5 hyphens, a newline
- a base64-encoded payload (with newlines after every 72 text characters)
- a newline (unless you ended with a newline because you were a multiple of 72 text characters)
- 5 hyphens, END, the same type identifier as before, 5 hyphens
The part we want is the payload. Run it through Convert.FromBase64String, and now we have the DER-encoded byte[] for the key object.

Using the type definition and the ITU documents, write a parser for your key file format.
Parse the key.
Convert the parsed key to an RSAParameters object (or DSAParameters, or ECParameters, as appropriate)
Call RSA.Create() (etc)
Load the key via the ImportParameters method.
Good to go.

For step 4, there are some things to be careful about. Specifically, the ASN.1/DER INTEGER components have two rules that RSAParameters does not like.

All leading 0x00 values are removed.
If the leading byte has the high bit set (>=0x80) but the number was supposed to be positive, insert a 0x00.

.NET wants the values as big-endian byte arrays (which is the same byte order as the DER encoding) with the following relationship:

Exponent is as big as it needs to be, so long as it doesn't start with 0x00.
Modulus is as big as it needs to be, so long as it doesn't start with 0x00.
D must be the same size as modulus (insert 0x00 as necessary)
P must be "half-round-up" the size of Modulus ((Modulus.Length + 1) / 2), insert 0x00 as necessary.
Q, DP, DQ, and InverseQ must have the same length as P. (Insert 0x00 as necessary).

.NET Core 3.0+ PKCS#1 RSAPrivateKey

The ImportRSAPrivateKey method is declared on RSA, and since it parses data and calls ImportParameters it works for all RSA derived types (assuming they already supported parameter import).

using (RSA rsa = RSA.Create())
{
    rsa.ImportRSAPrivateKey(blob, out _);
    return rsa.SignData(data, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
}

Some other format

Determine what RFC defines the ASN.1 structure for your key format, then keep that in mind and evaluate the RSAPrivateKey section.

DSAParameters and ECParameters each have their own spatial expectations.