Why does Rudin define $k = \frac{y^n-x}{n y^{n-1}}$ or $h < \frac{x - y^n}{n(y+1)^{n-1}}$ when he tries to prove that every real x has a nth root?
First, Rudin is using $n\gt0$, and the inequality:
$$b^n-a^n\lt (b-a)nb^{n-1}$$
only holds for $n\in\mathbb{Z}, n\gt1$.
But this isn't massively important here.
We are trying to find a contradiction to $y^n\lt x$, where $x$ is given and $y=\sup E$.
A good place to begin is with a $y+h$, $h$ arbitrarily small and positive. In which case we get:
$$(y+h)^n-y^n \lt hn(y+h)^{n-1}$$
by the above inequality (for $n\gt1$!)
As $h$ is arbitrarily small, we can say $h\lt 1$, and so we continue:
$$(y+h)^n-y^n \lt hn(y+h)^{n-1}\lt hn(y+1)^{n-1}$$
We now use the fact that we are trying to find a contradiction, $(y+h)^n\lt x$ for example, which gives us the contradiction Rudin then uses.
So we need:
$$hn(y+1)^{n-1}\lt x-y^n$$
or:
$$h\lt \frac{x-y^n}{n(y+1)^{n-1}}$$
We can see that:
$$0\lt \frac{x-y^n}{n(y+1)^{n-1}}$$
and so we are free to pick $0\lt h\lt 1$, as required.
$k=\dfrac{y^n-x}{ny^{n-1}}$ is similar, except for $k$ is already fixed, and $k\lt y$ by the definitions given.