Prove that $\operatorname{trace}(A^TA) = 0\ \iff\ A = 0$.

If $A = 0$, then $A^TA = 0$ and ${\rm trace}(A^TA) = 0$, good. For the other direction you must use the definition of matrix multiplication. $$(A^TA)_{ij} = \sum_k (A^T)_{ik}A_{kj} = \sum_kA_{ki}A_{kj}.$$ And: $${\rm trace}(A^TA) = \sum_i (A^TA)_{ii} = \sum_i\sum_k (A_{ki})^2.$$ If you have $$A_{11}^2+\ldots+A_{1n}^2+A_{21}^2+\ldots+A_{2n}^2+\ldots+A_{n1}^2+\ldots+A_{nn}^2 = 0,$$ what can you say about the $A_{ij}$?

$A^TA$ is positive semi-definite, since it is symmetric ($(A^T A)^T = A^T (A^T)^T = A^T A$) and $x^T A^T A x = (A x)^T (A x) = || A x||^2 \geq 0$ (so its Rayleigh quotient is non-negative). So, $A^T A$ has non-negative eigenvalues. $tr(A^T A)$ is simply the sum of the eigenvalues of $A^T A$, and is thus zero if and only if all the eigenvalues of $A^T A$ are zero (since they are non-negative). Now, the eigenvalues of $A^T A$ are simply the squared singular values of $A$, so this is equivalent to all the singular values of $A$ being zero. Since the rank of $A$ is simply the number of non-zero singular values, this is equivalent to $A$ having rank zero, i.e. being the zero matrix.

One line proof using SVD: $$ 0 = tr(AA^T)=tr(U\Lambda VV^T\Lambda U^T)=tr(\Lambda^2 U^TU)=tr(\Lambda^2) \implies \Lambda = 0 \implies A = 0. $$