Compare password hashes between C# and ColdFusion (CFMX_COMPAT)

I looked through the Railo code as someone else here mentioned in comments.

Following is CFMX_Compat ported to C# from the Railo Java source. See below for an example of usage.

using System;
using System.Collections.Generic;
using System.Text;
using System.Security.Cryptography;

namespace RailoUtil
{
    // SOURCE: Railo Source Code License LGPL v2
    // http://wiki.getrailo.org/wiki/RailoLicense
    public class RailoCFMXCompat
    {
        private String m_Key;
        private int m_LFSR_A = 0x13579bdf;
        private int m_LFSR_B = 0x2468ace0;
        private int m_LFSR_C = unchecked((int)0xfdb97531);
        private int m_Mask_A = unchecked((int)0x80000062);
        private int m_Mask_B = 0x40000020;
        private int m_Mask_C = 0x10000002;
        private int m_Rot0_A = 0x7fffffff;
        private int m_Rot0_B = 0x3fffffff;
        private int m_Rot0_C = 0xfffffff;
        private int m_Rot1_A = unchecked((int)0x80000000);
        private int m_Rot1_B = unchecked((int)0xc0000000);
        private int m_Rot1_C = unchecked((int)0xf0000000);

        public byte[] transformString(String key, byte[] inBytes)
        {
            setKey(key);
            int length = inBytes.Length;
            byte[] outBytes = new byte[length];
            for (int i = 0; i < length; i++)
            {
                outBytes[i] = transformByte(inBytes[i]);
            }
            return outBytes;
        }

        private byte transformByte(byte target)
        {
            byte crypto = 0;
            int b = m_LFSR_B & 1;
            int c = m_LFSR_C & 1;
            for (int i = 0; i < 8; i++)
            {
                if (0 != (m_LFSR_A & 1))
                {
                    m_LFSR_A = m_LFSR_A ^ m_Mask_A >> 1 | m_Rot1_A;
                    if (0 != (m_LFSR_B & 1))
                    {
                        m_LFSR_B = m_LFSR_B ^ m_Mask_B >> 1 | m_Rot1_B;
                        b = 1;
                    }
                    else
                    {
                        m_LFSR_B = m_LFSR_B >> 1 & m_Rot0_B;
                        b = 0;
                    }
                }
                else
                {
                    m_LFSR_A = (m_LFSR_A >> 1) & m_Rot0_A;
                    if (0 != (m_LFSR_C & 1))
                    {
                        m_LFSR_C = m_LFSR_C ^ m_Mask_C >> 1 | m_Rot1_C;
                        c = 1;
                    }
                    else
                    {
                        m_LFSR_C = m_LFSR_C >> 1 & m_Rot0_C;
                        c = 0;
                    }
                }

                crypto = (byte)(crypto << 1 | b ^ c);
            }

            target ^= crypto;
            return target;
        }

        private void setKey(String key)
        {
            int i = 0;
            m_Key = key;
            if (String.IsNullOrEmpty(key)) key = "Default Seed";
            char[] Seed = new char[key.Length >= 12 ? key.Length : 12];
            Array.Copy(m_Key.ToCharArray(), Seed, m_Key.Length);
            int originalLength = m_Key.Length;
            for (i = 0; originalLength + i < 12; i++)
                Seed[originalLength + i] = Seed[i];

            for (i = 0; i < 4; i++)
            {
                m_LFSR_A = (m_LFSR_A <<= 8) | Seed[i + 4];
                m_LFSR_B = (m_LFSR_B <<= 8) | Seed[i + 4];
                m_LFSR_C = (m_LFSR_C <<= 8) | Seed[i + 4];
            }
            if (0 == m_LFSR_A) m_LFSR_A = 0x13579bdf;
            if (0 == m_LFSR_B) m_LFSR_B = 0x2468ace0;
            if (0 == m_LFSR_C) m_LFSR_C = unchecked((int)0xfdb97531);
        }
    }
}

Here is a usage example that Hex-encodes the encrypted text, and then decrypts the same thing.

RailoCFMXCompat cfmx = new RailoCFMXCompat();
UTF8Encoding encoding = new UTF8Encoding();

//encrypt my string
byte[] encrypted = cfmx.transformString("mySecretKey", encoding.GetBytes("clear text"));
string encryptedHex = BitConverter.ToString(encrypted); //72-07-AA-1B-89-CB-01-96-4F-51

//decrypt my string
byte[] encryptedBytes = HexToBytes("72-07-AA-1B-89-CB-01-96-4F-51");
byte[] decrypted = cfmx.transformString("mySecretKey", encryptedBytes);
string cleartext = encoding.GetString(decrypted);

MD5 is the default hashing algorithm for the hash(). I'm not a C# programmer, but it shouldn't be too hard to create an MD5 hash to compare to your ColdFusion result.

As for encrypt(), is there a reason you're encrypting the username before hashing it? I can't think of any benefit to doing this, but that doesn't mean there isn't one. I would simply do:

Hash( UCase( GetPass.username ) )

Which should be easier to replicate in C#.

I'll leave the original answer content below for historical reference, but it should be noted that this is NOT a working answer to the original question.

Instead, see the top-voted answer in this thread, by @Terrapin in January 2011. I hope the OP sees this and can change the accepted answer. Heck, I'll even flag the mods to see if anything can be done about this.

To build on the answer by Edward Smith, and the follow-up comments by czuroski, here is my solution.

First, you need an XOR function in C#, which I've taken from here and modified slightly.

using System;
using System.Collections.Generic;
using System.Text;

namespace SimpleXOREncryption
{    
    public static class EncryptorDecryptor
    {
        public static string EncryptDecrypt(string textToEncrypt, int key)
        {            
            StringBuilder inSb = new StringBuilder(textToEncrypt);
            StringBuilder outSb = new StringBuilder(textToEncrypt.Length);
            char c;
            for (int i = 0; i < textToEncrypt.Length; i++)
            {
                c = inSb[i];
                c = (char)(c ^ key);
                outSb.Append(c);
            }
            return outSb.ToString();
        }   
    }
}

Then, take the result of the XOR and base-64 encode it. After you have that string, MD5 hash it. The result should match the result from the original code snippet:

#Hash(Encrypt(Form.UserPassword,GetSiteVars.EnCode))#