How do I create a file with a custom file extension?

I tried to recreate your Problem and noticed that you probably have not enabled the "Show all File Extensions" option in the Finder Preferences.

You can check by selecting the Finder and selecting "Preferences" from the Menubar. A small window should appear, titled "Finder Preferences". Select the "Advanced" Icon at the Top of that window. You should now see the option "Show all File Extensions", it should be checked. All files in your finder window should now show their extensions, like .txt or in your case .crt

You can now easily change all files to another extension.

If that does not work then the Fileextension is probably linked to another application. .crt-Files should link to the Keychain-Application which handels all certificates. You can check by selecting the crt-File and pressing "command" + "i". An information window should appear with additional information to that file. At the Entry "open with" should appear "Keychain.app".

@J.C.'s answer is good advice, and will let you change the file extension, but it'll still be in RTF format and will not be useable as a certificate file. To solve this, convert the document to plain text in TextEdit before saving it (Format menu > Make Plain Text). When TextEdit has the document in plain text mode, it'll display in a monospace font, and the format bar will vanish from the window. Most importantly, it'll now default to saving with a .txt extension but let you change that in the save dialog. In the save dialog, there'll be a checkbox labelled "If no extension is provided, use '.txt'." -- with that box selected, if you try to save as .crt it'll object but give you a "Use .crt" option; with it deselected, it won't even object.