Solution 1:

Does the RAR file format use a serious encryption (such as AES)?

The current version of the RAR file format is using AES encryption with 128-bit key length.

Will it take unfeasibly long to brute force a password?

That depends entirely on the strength of the password.

What tools do you suggest that do so automatically?

There are some shareware programs (e.g. RAR Password Cracker, RAR Password Recovery) using 'brute force' and 'dictionary-based' attacks to crack the password, the process can be very time consuming (again, depending on the password strength).

Solution 2:

Although I don't know which mechanism exactly, RAR used pretty solid encryption. Brute force attacks on it have been known to last :), for all but the simplest of passwords.

(There were some password extracting mechanisms for old versions of RAR, but I do mean oooold, ... DOS days or something from that time period).