For $p,q$ primes such that $q|p-1$, show there is only one non-abelian group of order $pq$, up to isomorphism

This question has been answered a lot of times on this site, but I'm looking for an approach that does not use Sylow theory, since this is not covered in my syllabus. All answers I read this far used material that I did not yet learn. My level this far is up to automorphisms, group actions, and the isomorphism theorems.

My syllabus uses the following construction of a non-abelian group of order $pq$ where $q|p-1$. Let $N=C_p$ such that $\operatorname{Aut}(N)$ has order $p-1$. From Cauchy's theorem we deduce that there exists a subgroup $H\subset \mathrm{Aut}(N)$ of order $q$. Let $\tau:H\to \operatorname{Aut}(N)$ be the identity map. Then $N\rtimes_{\tau}H$ has order $pq$ and is non-abelian.

This far I can follow, but now I have to show that this group is the only non-abelian group of order $pq$. A hint for this exercise is to use that $(\mathbb{Z}/p\mathbb{Z})^*$ is cyclic if $p$ is prime.

My attempt (it is not really an attempt, I just looked what I could deduce, but it led me nowhere): Let $G$ be a finite non-abelian group of order $pq$. There exists, by Cauchy's theorem, an element $g$ of order $p$, so $\langle g\rangle$ is a subgroup of $G$ of order $p$ and index $q$, and since $q$ is the smallest prime dividing $pq$, this subgroup is normal.

Furthermore, it is easy to deduce that the center is trivial, since if it is not, $G$ is abelian because then $G/Z(G)$ is cyclic.

I have the idea that in some way I have to say that we know $G$ can be written as a semi-direct product (but how do I know that this is always the case?). Then this semi-direct product should be between $C_p$ and $C_q$, which are both abelian, so the used automorphism cannot be the identity map. From there I think I have to show there is only one map possible, thus only one possible semi-direct product. Because we already found one way to write $G$ as a semi-direct product, then the result would follow. But how exactly does this work? I think they want me prove the theorem this way, but I don't see how and why this would work.

Any help is much appreciated!

There is only one non-abelian group of order $pq$ with primes $p<q$ and $p\mid q-1$, which is proved in Theorem $3.8$ of Keith Conrad's notes here. It is indeed given by a semidirect product; and as a subgroup of the affine group ${\rm Aff}(\mathbb{Z}/(q))$.

Proposition: Let $G$ be a group of order $pq$ with primes $p<q$. If $q\not\equiv 1 \bmod p$, then $G \cong C_{pq}$. If $q\equiv 1 \bmod p$, then $G$ is isomorphic to either $C_{pq}$, or to the non-abelian group $$ \left\{ \begin{pmatrix} a & b \\ 0 & 1 \end{pmatrix} \mid a\in (\Bbb{Z}/(q))^{\times}, b\in \Bbb{Z}/(q), a^p\equiv 1 \bmod q \right\}. $$

Remark: When $q \equiv 1 \bmod p$, so ${\rm Aut}(\mathbb Z/q\mathbb Z) = (\mathbb Z/q\mathbb Z)^\times$ is cyclic of order $q-1$ and thus contains a unique subgroup of order $p$, the matrix $(\begin{smallmatrix}a&b\\0&1\end{smallmatrix})$ can be written as $(\begin{smallmatrix}1&b\\0&1\end{smallmatrix})(\begin{smallmatrix}a&0\\0&1\end{smallmatrix})$. Therefore the matrix group in the proposition can be described as a semidirect product $\cong C_q\rtimes C_p$ where $C_p = \{a \in (\mathbb Z/q\mathbb Z)^\times : a^p \equiv 1 \bmod q\}$ acts as automorphisms of $C_q = \mathbb Z/q\mathbb Z$ by multiplication

Proof: Let $P$ be a Sylow $p$-subgroup of $G$, and $Q$ be a Sylow $q$-subgroup of $G$. We have $P\cong C_p$, $Q\cong C_q$ and $(G:Q)=p$, which is the smallest prime dividing $(G:1)$. So $Q$ is normal. Because $P$ maps bijectively onto $G/Q$, we have that $G=Q\rtimes P$. Since ${\rm Aut}(Q)\cong C_{q-1}$ we obtain $G=Q\times P\cong C_q\times C_p\cong C_{pq}$, unless $p\mid (q-1)$, i.e., $q\equiv 1 \bmod p$. In that case the cyclic group ${\rm Aut}(Q)$ has a unique subgroup $A$ of order $p$. In fact, $A$ consists of the automorphisms $x\mapsto x^i$ for $i\in \Bbb{Z}/q\Bbb{Z}$ with $i^p=1$. Let $a$ and $b$ be generators of $P$ and $Q$ respectively, and let the action of $a$ on $Q$ by conjugation be $x\mapsto x^j$ with $j\neq 1$ in $\Bbb{Z}/q\Bbb{Z}$. Then $$ G=\langle a,b\mid a^p=b^q=1,\; aba^{-1}=b^j\rangle, $$ which is the semidirect product $Q\rtimes P$ with this action of $P$ on $Q$ by conjugation. Choosing a different $j$ amounts to choosing a different generator $a$ for $P$, and so gives a group isomorphic to $G$. By definition, this group is non-abelian. In fact it is isomorphic to the subgroup of ${\rm Aff}(\Bbb{Z}/(q))$ given above.

Note: The proof above uses the Sylow theorems and the terminilogy of semidirect products, but the proof in the link does not.