What's a BETTER way to see the Gauss's composition law for binary quadratic forms?
There is a group structure of binary quadratic forms of given discriminant $d$:
Let $[f]=[(a,b,c)], [f']=[(a',b',c')],$ where $d=b^2-4 a c=b'^2-4 a' c'.$
The composition of two binary quadratic forms is defined as:
$$[f] [f']=[(A,B,C)],$$ where $A=a a',$ $0<B<2 a a',B=b \mod 2a，B=b' \mod 2a'，B^2=d \mod 4aa'，C=(B^2-d)/4 a a'.$
It is not hard to see that binary quadratic forms of discriminant $d$ form a finite abelian group. But how should one interpret the composition law? Why does it have to be this way?
I know that binary quadratic forms are closely related to quadratic number fields. Is there an explination from the pointview of $Q(\sqrt{d})$?
Perhaps the most illuminating way is to transport the class group structure from ideals to primitive binary quadratic forms. Below is a description of the standard maps from section 5.2, p. 225 of Henri Cohen's book $ $ A course in computational algebraic number theory. To me, this is one of the most beautiful examples of transport of structure.
There is a different perspective that took me a while to figure out, and this is a convenient place to write the ideas all down.
Let me rephrase your question in a way that I hope is still your intention: "Is there a sense in which Gauss's composition of forms is canonical?" The answer of KeyIdeas gives a very natural translation of the group law, but there is a sense in which it is not answering this question. The transport of structure is being carried out at the level of equivalence classes only. Gauss defined composition far more generally, at the level of forms, not just equivalence classes of forms, and he even composed forms of different discriminants. So an answer that works only at the level of equivalence classes is immediately removing some of the "non-canonical" nature of form composition.
I want to give now a sense in which composition is canonical and then exhibit the special case of composition described in the original post within that context. I will not work as generally as Gauss, but rather stick to primitive forms of the same discriminant. What follows could be done in other ways, say with Bhargava's cubes, but I will stick to Gauss.
Suppose now that $f=(a_1,b_1,c_1)$ and $g=(a_2,b_2,c_2)$ are primitive forms of the same nonsquare discriminant. Suppose there exists a third form $F$ and a pair of quaternary forms
\begin{align} B_1 (x,y,z,w) &= p_1 xz + p_2 xw + p_3 yz + p_4 yw\\ B_2 (x,y,z,w) &= q_1 xz + q_2 xw + q_3 yz + q_4 yw \end{align}
making true the identity
$$ f(x,y) g(z,w) = F(B_1(x,y,z,w), B_2(x,y,z,w)) $$
Let us say in this case that $F$ is a composition of $f$ and $g$ through the matrix
$$ M = \begin{bmatrix} p_1 & p_2 & p_3 & p_4\\ q_1 & q_2 & q_3 & q_4 \end{bmatrix}.$$
This seems to me to be a natural definition in the sense desired by the OP. Gauss showed that every pair of primitive forms of the same discriminant has some composition $(F,M)$, and he showed how to put a right-$\mathrm{SL}_2(\mathbb{Z})$ action on the set of compositions of a given pair of forms. If $\gamma$ is in $\mathrm{SL}_2(\mathbb{Z})$, then $\gamma$ acts as
$$ (F,M) \cdot \gamma = (F \cdot \gamma, \gamma^{-1} M) $$
There are thus infinitely many compositions of $f$ and $g$, and often no apparent canonical choice.
Gauss singles out four different types of compositions, and identifies one as "direct". He shows that $F=(A,B,C)$ is a direct composition of $f$ and $g$ through the matrix $M$ above if and only if the following nine equations are satisfied:
\begin{align} p_1 q_2 - p_2 q_1 &= a_1\\ p_1 q_3 - p_3 q_1 &= a_2\\ p_1 q_4 - p_4 q_1 &= \frac{b_2 + b_1}{2}\\ p_2 q_3 - p_3 q_2 &= \frac{b_2 - b_1}{2}\\ p_2 q_4 - p_4 q_2 &= c_2\\ p_3 q_4 - p_4 q_3 &= c_1\\ q_2 q_3 - q_1 q_4 &= A\\ p_2 p_3 - p_1 p_4 &= C\\ p_1 q_4 + p_4 q_1 - p_2 q_3 - p_3 q_2 &= B \end{align}
(The non-direct compositions satisfy these equations with sign changes).
My main point is the theorem:
Theorem. All direct compositions of $f$ and $g$ lie in a single $\mathrm{SL}_2(\mathbb{Z})$-equivalence class.
This shows that there is a canonical "thing" attached to forms that we may call their "composition", but the thing is an equivalence class of form-matrix pairs. (This immediately implies that the equivalence class of the composition of two forms is canonically defined.)
One may hope to find a canonical choice of matrix within each equivalence class, and thereby identify a canonically chosen form to be the composition. One candidate is a pair in which the matrix has $q_1 = 0$.
The following are readily checked:
if $F$ is a direct composition of $f$ and $g$ through $M$ as described above, then $(F,M)$ is equivalent to a pair $(F_0,M_0)$ for which the matrix has $0$ in the lower-left corner.
Because the discriminant is nonsquare, $a_1 \neq 0$. Then acting on $(F_0,M_0)$ by a matrix of the form $$ \begin{bmatrix} 1 & k \\ 0 & 1 \end{bmatrix}$$ gives another pair with $0$ in the lower-left corner of the matrix. All such pairs are produced by acting by such a matrix.
Acting on $(F_0,M_0)$ by a matrix as described in (2) takes the form $F_0 = (A_0, B_0, C_0)$ to one with $A_0$ as first coefficient, but with middle coefficient congruent to $B_0$ modulo $2A_0$.
The first three of the nine equations above show that for our $M_0$, we have $p_1 q_2 = a_1$, $p_1 q_3 = b_1$, and $p_1 q_4 = \frac{b_1 + b_2}{2}$. If we set $\mu = \gcd(a_1, a_2, \frac{b_1+b_2}{2})$, then $p_1$ divides $\mu$. Examining equations four, five, and six, using primitivity of $f$ and $g$, and with a little work, we find that $p_1 = \mu$. We then determine that
$$ p_1 = \mu \qquad q_2 = \frac{a_1}{\mu} \qquad q_3 = \frac{a_2}{\mu} \qquad q_4 = \frac{b_1+b_2}{2\mu} $$
and then from equations seven through nine, we have
$$ p_2 = \frac{b_2 - B_0}{2 a_2/\mu} \qquad p_3 = \frac{b_1 - B_0}{2 a_1/\mu} \qquad p_4 = \frac{\Delta + b_1 b_2 - (b_1 + b_2) B_0}{4 a_1 a_2/\mu^2}. $$
(here, $\Delta$ is the discriminant). Since these latter three are integers, we find that $B_0$ satisfies the congruences mentioned in the original post (in fact, the more general congruences, due to Arndt, that I put in my comment on the OP.)
This all shows that we may find a pair $(F_0,M_0)$ in any class so that $M_0$ has 0 in the lower-left corner, but this choice is not canonical. It is only canonical up to action by the subgroup of $\mathrm{SL}_2(\mathbb{Z})$ consisting of matrices as described in 2. Historically, people have accepted this level of non-canonicalness and have chosen the composition so that the form $F_0$ has middle coefficient which is as small as possible and non-negative. With this choice, we have the Gauss composition operation you describe.
This is only canonical in the sense that choosing a pair $(F,M)$ in an equivalence class with $0$ in the lower-left corner is canonical, and it isn't, really. When $\Delta$ is positive, a completely canonical choice was found by Zagier: choose $(F,M)$ so that $|F(p_1, p_2)|$ is the unique representation of $|a_1 a_2|$ by a reduced form $F$ (reduced in the sense of Zagier) by integers $p_1 > 0$ and $p_2 \geq 0$. We use a reduced $F$ when $a_1 a_2$ is positive and the negative of a reduced form when $a_1 a_2$ is negative.
The point is, at the level of equivalence classes, any such choice will give the same abelian group operation on classes. But if we are interested in what happens at the level of forms, say if we study Shanks' infrastructure, then there are other useful and perhaps more canonical choices for the operation than that given by Gauss.