Why does Rudin define $k = \frac{y^n-x}{n y^{n-1}}$ or $h < \frac{x - y^n}{n(y+1)^{n-1}}$ when he tries to prove that every real x has a nth root?

First, Rudin is using $n\gt0$, and the inequality:

$$b^n-a^n\lt (b-a)nb^{n-1}$$

only holds for $n\in\mathbb{Z}, n\gt1$.

But this isn't massively important here.

We are trying to find a contradiction to $y^n\lt x$, where $x$ is given and $y=\sup E$.

A good place to begin is with a $y+h$, $h$ arbitrarily small and positive. In which case we get:

$$(y+h)^n-y^n \lt hn(y+h)^{n-1}$$

by the above inequality (for $n\gt1$!)

As $h$ is arbitrarily small, we can say $h\lt 1$, and so we continue:

$$(y+h)^n-y^n \lt hn(y+h)^{n-1}\lt hn(y+1)^{n-1}$$

We now use the fact that we are trying to find a contradiction, $(y+h)^n\lt x$ for example, which gives us the contradiction Rudin then uses.

So we need:

$$hn(y+1)^{n-1}\lt x-y^n$$

or:

$$h\lt \frac{x-y^n}{n(y+1)^{n-1}}$$

We can see that:

$$0\lt \frac{x-y^n}{n(y+1)^{n-1}}$$

and so we are free to pick $0\lt h\lt 1$, as required.

$k=\dfrac{y^n-x}{ny^{n-1}}$ is similar, except for $k$ is already fixed, and $k\lt y$ by the definitions given.