Is the function $A \mapsto \sum\limits_{j=0}^{\infty} \langle A^j v, A^j v \rangle$ differentiable everywhere?

I believe $\{A:\rho(A)<1\}$ is the interior of $\mbox{Dom}(f)$, which means the answer is no. Since the inclusion "$\subseteq$" is pretty simple, I will only argue that for all $A\in\mbox{Dom}(f)$ with $\rho(A)\geq1$ there exists $B\not\in\mbox{Dom}(f)$ with $\|A-B\|$ arbitrarily small.

Let $A\in\mbox{Dom}(f)$ with $\rho(A)\geq1$. By changing $A$ an arbitrarily small amount we can obtain a matrix $B$ with a complex eigenbasis $\beta$, such that $\rho(B)\geq1$, and such that the representation of $v$ in terms of $\beta$ has only non-zero coordinates. Then for some eigenvector $b$ of $B$ the corresponding eigenvalue is at least $1$ in absolute value. Then the absolute value of the $b$-coordinate of $B^jv$ with respect to $\beta$ is non-decreasing. It follows that $\langle B^jv,B^jv\rangle$ does not converge to $0$, so the series $f(B)$ does not converge, so $B\not\in\mbox{Dom}(f)$.