Collection that allows only unique items in .NET?

HashSet<T> is what you're looking for. From MSDN (emphasis added):

The HashSet<T> class provides high-performance set operations. A set is a collection that contains no duplicate elements, and whose elements are in no particular order.

Note that the HashSet<T>.Add(T item) method returns a bool -- true if the item was added to the collection; false if the item was already present.

How about just an extension method on HashSet?

public static void AddOrThrow<T>(this HashSet<T> hash, T item)
{
    if (!hash.Add(item))
        throw new ValueExistingException();
}

From the HashSet<T> page on MSDN:

The HashSet(Of T) class provides high-performance set operations. A set is a collection that contains no duplicate elements, and whose elements are in no particular order.

(emphasis mine)

If all you need is to ensure uniqueness of elements, then HashSet is what you need.

What do you mean when you say "just a set implementation"? A set is (by definition) a collection of unique elements that doesn't save element order.