Calculate center of circle tangent to two lines in space

What you want is the tangent, tangent, radius algorithm. One way to handle this is as follows:

1. Measure the angle $\alpha = \widehat{RQP}$. This is done using the cross product and dot product from the coordinates of the points.
2. Construct the bisector of the angle and note that if the radius is known as $h$ the distance from the vertex to the circle center $QA$ is $$s=\frac{h}{\sin \frac{\alpha}{2}}$$
3. Numerically create a vector of length $s$ along $QR$ and rotate it by $\frac{\alpha}{2}$ to find point $A$.

I would suggest something like this to find the center of your circle: Since you know the coordinates of $P$ and $Q$, you can find a normalized vector that is perpendicular to $\vec{QP}$, using the inner product. You also said that $PQ$ is a tangent line meaning, the vector $\vec{PO}$ is perpendicular to $\vec{PQ}$. Given the fact you can calculate a vector perpendicular to $PQ$ means you only have to travel along this vector over a lenght of your (known) radius. If you start in $P$ you will end up exactly at the center of your cirlce.

There is no unique solution to this problem. There are infinitely many circles which will be tangent to the two given lines. The centre's of these circles, as pointed in the solution given, will be on the angle bisector. See the animation below: