Word for going to a university class but without being enrolled?

A common term is audit

North American - Attend (a class) informally, not for academic credit.

Oxford Dictionaries Online

If a student intends to sit in on a course without registering, this is indeed called auditing. Auditing can be formal, in which case it might not even be free and could require arrangement with the university, or it can be informal if the professor agrees to allow it (which they may or may not technically be allowed to do).

Mary wasn't sure whether to take the biology course for credit, so she decided to audit it instead.

Contrary to the dictionary definition, I once took a course in which auditors were required to do all the reading and assignments, even though they wouldn't receive a grade, because all members of the class were expected to contribute. This sort of auditing was still a formal relationship, and I think it was even indicated on their transcripts. These were matriculated graduate students, however. At the same university (not mine), I myself audited a class by casual arrangement with the professor, though his requirements for participation were rather less onerous.

On top of that, there were also "auditors" from the general public in these courses who paid dearly for the privilege of listening to our discussions but, I understand, were expressly forbidden from actually participating in those discussions themselves—they were auditors in the true sense of the word.

The practice of trying a course out, on the other hand, is usually called shopping, and many if not most US universities these days allow for a one or two week "shopping period" at the beginning of each semester, so students can do precisely this. Shopping is not a slang term for it, either, but is what the universities themselves officially call it.

Mary wasn't sure whether to take the biology course, so she tried it out during the shopping period first.

This is a little bit redundant, since you could just say she tried it out or went to check it out, but I thought you might be interested to know this other common term and that there is a distinction between auditing and shopping. The former implies a commitment to a course despite not receiving a grade, while the latter indicates a trial period.