Unittest (sometimes) fails because floating-point imprecision

1) How can I make sure the test works?

Use assertAlmostEqual, assertNotAlmostEqual.

From the official documentation:

assertAlmostEqual(first, second, places=7, msg=None, delta=None)

Test that first and second are approximately equal by computing the difference, rounding to the given number of decimal places (default 7), and comparing to zero.

2) Is it possible to do it without testing for an approximate value?

Esentially No.

The floating point issue can't be bypassed, so you have either to "round" the result given by vec.normalize or accept an almost-equal result (each one of the two is an approximation).

By using a floating point value, you accept a small possible imprecision. Therefore, your tests should test if your computed value falls in an acceptable range such as:

theoreticalValue - epsilon < normalizedValue < theoreticalValue + epsilon

where epsilon is a very small value that you define as acceptable for a variation due to floating point imprecision.

I suppose one possibility is to apply the function to test cases for which all inputs, the results of all intermediate calculations, and the output are exactly representable by float.

To illustrate:

In [2]: import math

In [4]: def norm(x, y):
   ...:     return math.sqrt(x*x + y*y)
   ...: 

In [6]: norm(3, 4) == 5
Out[6]: True

Not sure how practical this is though...

In general, you should not assert equality for floats. Instead, ensure that the result is within certain bounds, e.g.:

self.assertTrue(abs(vec.normalize(5).length - 5) < 0.001)