How to prove that $A_5$ has no subgroup of order 30?

Such a subgroup would have index $2$ and so immediately be normal and non-trivial. It suffices to show that $A_5$ is simple.

Alternatively, show that the class equation is $60=1+15+20+12+12$. One cannot obtain $30$ as a sum of any subcollection of the summands on the right. Since normal subgroups are composed of whole conjugacy classes, there is then no subgroup of $A_5$ of order $30$.

Assume to the contrary it has a subgroup $H$ of order 30. Then, it has an element $g$ of order 2 by Cauchy's theorem. Then considering the cycle type of $g$ and using $g \in A_5$, $g$ must be a product of two transpositions. But since $H$ is normal in $A_5$, $H$ has to contain all products of 2 transpositions: WLOG assume $g = (1 2)(3 4)$. Then setting $x = (1 2 3)$, $xgx^{-1} = (2 3)(1 4)$. You can set $x = (1 3 2)$ to get $xgx^{-1} = (3 1)(2 4)$. So $H$ contains a subgroup ($V_4$) of order 4, contradicting Lagrange's theorem.

Here's another approach: A subgroup of order $30$ is, as was noted, normal, and must contain the commutator subgroup (since the quotient would be of order $2$, hence abelian).

First, note that the commutator subgroup of $A_5$ will contain all $3$-cycles; namely, given a $3$-cycle $(ijk)$, let $r$ and $s$ be the two elements of $\{1,2,3,4,5\}\setminus\{i,j,k\}$; then $$[(rs)(ij),(rs)(ik)] = (rs)(ij)(rs)(ik)(rs)(ij)(rs)(ik) = (ij)(ik)(ij)(ik) = (ijk)\in [A_5,A_5].$$

It must also contain all permutations of the form $(ij)(rs)$ with all $i,j,r,s$ distinct. Indeed, given $(ij)(rs)$, we have $$[(irs),(ir)(sj)] = (isr)(ir)(sj)(irs)(ir)(sj) = (ij)(rs).$$ Thus, this subgroup must contain at least all $3$-cycles (there are $20$ of them, $\frac{1}{3}(5\times 4\times 3)$), and all products of two transpositions (there are $3\times\binom{5}{1}=15$ of them). Together with the identity, this gives at least $36$ elements in the commutator subgroup. In particular, there can be no subgroup of order $30$, as it would necessarily have to contain at least 36 elements...

(In fact, every element of $A_5$ is a commutator; the recently proven Ore Conjecture shows that the same is true for any finite nonabelian simple group, though the proof relies on the classification of finite simple groups).