What happens to a function when it is undefined?

When you cancel a factor from the numerator with one in the denominator, you are saying that the ratio of the two is the same as the number one. This is true unless the two factors you happen to be cancelling are both zero. In this case the ratio is not one, it is undefined. So technically the ratio you obtain is $x-\sqrt{2}$ if $x \ne -\sqrt{2}$.

It is important to realise what you are doing when we say that a common factor "cancels" from the numerator and denominator. We are actually dividing both the numerator and denominator by that common factor. As you have already pointed out: you can't divide by zero.

It is true that $$\frac{x^2-2}{x+\sqrt{2}} \equiv \frac{(x-\sqrt{2})(x+\sqrt{2})}{x+\sqrt{2}}$$

Neither the left hand side, nor the right hand side are defined when $x=-\sqrt{2}$. Both are indeterminate forms. However, if we assume that $x\neq -\sqrt{2}$ then we may divide both the numerator and the denominator by $x+\sqrt{2}$ to give $$\frac{x^2-2}{x+\sqrt{2}} = \frac{(x-\sqrt{2})(x+\sqrt{2})}{x+\sqrt{2}} = x-\sqrt{2}$$

The next step to understand this is the idea of the limit. If we take the limit as $x$ tends to $-\sqrt{2}$, we assume that $x \neq-\sqrt{2}$, but that it gets closer and closer. Even though the original function is not defined when $x = -\sqrt{2}$, we can look what happens to its value as $x \to -\sqrt{2}$. In that case we have $$\lim_{x \to -\sqrt{2}}\left(\frac{x^2-2}{x+\sqrt{2}}\right) = \lim_{x \to -\sqrt{2}}\left(\frac{(x-\sqrt{2})(x+\sqrt{2})}{x+\sqrt{2}}\right) = \lim_{x \to -\sqrt{2}}\left(x-\sqrt{2}\right) = -2\sqrt{2}$$