Solution 1:

The result you found that $|G/Z(G)|$ cannot be prime is essentially the only restriction for $p$-groups:

For every $k\geq 2$ and every $p\in\mathbb{P}$ there exists a finite $p$-group $G$ such that $|G/Z(G)|=p^k$.

Proof: Let $q:=p^{k-1}$ and consider the matrix group $\begin{pmatrix}1&\mathbb{F}_p&\mathbb{F}_q \\ &1&\mathbb{F}_q \\ && 1\end{pmatrix}$ which has order $p^{2k-1}$ and center of order $p^{k-1}$ (the top right corner) which gives us a center quoient of order $p^k$ as desired. QED.

Observe that by taking direct products of such groups one can easily write down examples of $|G/Z(G)|=n$ for every natural number $n>1$ in which every prime factor that occurs, occurs at least twice. This is because $Z(G_1\times G_2)=Z(G_1)\times Z(G_2)$.

Solution 2:

In connection of this question it might be interesting to note that bounding the size of $G/Z(G)$ for a given finite group $G$ has some interesting history. For a given finite group Jordan's theorem gives an upper bound on the size of $G/Z(G)$ in terms of the degree of the smallest faithful representation of $G$.