How to find the orthogonal complement of a subspace?

For a finite dimensional vector space equipped with the standard dot product it's easy to find the orthogonal complement of the span of a given set of vectors:

Create a matrix with the given vectors as row vectors an then compute the kernel of that matrix.

Orthogonal complement is defined as subspace $M^\perp = \{ v\in V\,|\, \langle v, m\rangle = 0,\forall m\in M\}$. This is really a subspace because of linearity of scalar product in the first argument. Also, it is easy to see that $M = (M^\perp)^\perp$ and that $M\dotplus M^\perp = V$ (in finite dimensional case).

So, in your example you have $$M = \{ (x,y)\in \mathbb R^2\,|\, 3x + 2y = 0\} = \{ (x,y)\in \mathbb R^2\,|\, \langle(x,y),(3,2)\rangle = 0\}$$

i.e. $M$ is orthogonal complement of subspace spanned by $(3,2)$. But, by $M = (M^\perp)^\perp$, we then have that $M^\perp = \operatorname{span}\{(3,2)\}$.

It would be overkill in this example, but in general if you can find basis for $M$, you can always find $M^\perp$. Let's say that $M=\operatorname{span}\{e_1,\ldots,e_m\}$. Then $$\langle x,e_i\rangle= 0,\quad i=1,\ldots,m$$ is a homogeneous system of $m$ linear equations which you can solve and solution subspace will be orthogonal complement.

In the plane, it's easy. There's only one line through the origin that's perpendicular to any other given line.

So if the first line is generated by $(a,b)$, you can take the second (perpendicular) line to be generated by $(b,-a)$ since $(a,b)\cdot(b,-a) = ab-ba=0$.