What encryption method was used in OS9 Apple File Security?

I don't know what the format is, but my understanding is that ye olde Finder uses extended attributes to determine if the file has been encrypted. You could probably manually add this data back and get it to recognize it is encrypted again.

For example, here are the extended attributes on a file I created in OS 9:

$ xattr -l helloworld.txt
com.apple.FinderInfo:
00000000  65 6E 63 32 63 72 70 32 01 00 00 00 00 00 00 00  |enc2crp2........|
00000010  00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00  |................|
00000020

Contents of the file before encryption:

$ xxd helloworld.txt
00000000: 6865 6c6c 6f20 776f 726c 64              hello world

This Terminal command can add the attribute back to the file:

xattr -w -x 'com.apple.FinderInfo' '65 6E 63 32 63 72 70 32 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00' yourfile.ext

Not sure if these commands can be done from OS 9 but you should be able to do it from a modern macOS system and then pack the file in a .hqx file with binhex encode yourfile.ext for safe transport (so the attributes are not lost).