How to approximate relative error further?

You got some of the details right. The error of the difference quotient is in first order $\frac12f''(x)h+O(h^2)$. The combined error of the sine evaluations is bounded by $|f(x)|δ$ if $δ\approx 2\cdot 10^{-16}$ is the floating point machine constant. In total you get an error bound of $$ \frac{|f(x)|δ}{h}+\frac12|f''(x)|h+O(h^2) $$ As for $f(x)=\sin(x)$ you get $f''(x)=-\sin(x)$, the minimum of this error bound can be found by minimizing $\fracδh+\frac h2$ which is at $h=\sqrt{2δ}=2⋅10^{-8}$.

The correctness of this bound and its minimum can be seen if you sample the formula at some more points and graph them together with the bound,

A general heuristic is that with an order $p$ approximation of the $k^{\rm th}$ derivative you get error terms $O(h^p)$ from the method itself and $O(δ/h^k)$ from the floating point evaluation. The overall error is lowest when both terms are about equal, that is at $$ h=\sqrt[p+k\,]δ $$ which gives the somewhat counter-intuitive rule that the higher these numbers the larger the optimal $h$.