why do we use cosine as the expression of vector dot product?

Let $\vec{\mathbf a} = (x_1, y_1) = (a \cos \alpha, a \sin \alpha)$

Let $\vec{\mathbf b} = (x_2, y_2) = (b \cos \beta, b \sin \beta)$

Then $\theta = |\beta - \alpha|$

By definition,

\begin{align} \vec{\mathbf a} \circ \vec{\mathbf b} &= x_1x_2 + y_1y_2 \\ &= ab(\cos \alpha \cos \beta + \sin \alpha \sin \beta) \\ &= ab \cos(\beta - \alpha)\\ &= ab \cos \theta \end{align}

(Note $\cos(\theta) = \cos(-\theta)$)

The dot product of two vectors $A$ and $B$ is just the product of the magnitude of one vector with the scalar projection of the other one on itself. Hence the $cos$ term. Also, note that the $cos$ function is greater for smaller angles, and lesser for larger ones, just like the length of the projection. (both are the same thing, actually.)

The dot product is defined in that way. Note that $cos\theta$ is a suitable function; since by the Schwarz inequality: $$|\mathbf{A} \cdot \mathbf{B}| \leq |\mathbf{A}| |\mathbf{B}|$$ and thus the dot product ranges continuosly between -1 and 1, as $cos \theta$ for $\theta \in [0,\pi] $.

This is entirely determined by what we consider a rotation in the plane.

Let $u, v$ be two unit vectors. Let the angle between them be $\theta$, and we can naturally write $v$ as

$$v = u \cos \theta + u_\perp \sin \theta$$

where $u_\perp$ is a unit vector perpendicular to $u$. Then clearly, the dot product is $u \cdot v = \cos \theta$.

But, if you're not in a Euclidean plane anymore, this relationship no longer holds. For example, in a Lorentzian space, instead of cosine and sine, we get hyperbolic functions instead:

$$v = u \cosh \theta + u_\perp \sinh \theta$$

And the dot product is $u \cdot v = \cosh \theta$. A physicist should recognize that this $\theta$ is the "rapidity", and that the form of $v$ given here is exactly that of any Lorentz boost.

So the reason we use sine and cosine in Euclidean space is because they are dictated by the use of sine and cosine in rotations. In other spaces, with different rotation operators, you use the functions that are associated with those rotations instead.