Fractional part summation
Solution 1:
The definition of $S_K$ is complicated, it is hard to figure out a full asymptotic expansion for it.
Let us look at a simpler one which we do know.
For any integer $p > 0$, consider following sum:
$$F(p) = \sum_{k=1}^p \{ \sqrt{k} \} = \sum_{k=1}^p \sqrt{k} - \sum_{k=1}^p \lfloor \sqrt{k}\rfloor\tag{*1} $$
Let $x = \sqrt{p}$, $t = \{ x \}$ and $m = x - t = \lfloor x \rfloor$.
For the first term on RHS of $(*1)$, it has following asymptotic expansion:
$$\sum_{k=1}^p \sqrt{k} \asymp \frac23 x^3 + \frac12 x + \zeta\left(-\frac12\right) + \frac{1}{24} x^{-1} -\frac{1}{1920} x^{-5} + \frac {1}{9216} x^{-9} +\cdots\tag{*2} $$ See here for one way to derive this expansion.
For the second term, we can rewrite it as
$$\begin{align} \sum_{k=1}^p \lfloor \sqrt{k}\rfloor &= \sum_{k=m^2}^p m + \sum_{\ell=1}^{m-1}\sum_{k=\ell^2}^{(\ell+1)^2-1}\ell\\ &= (p - m^2 + 1)m + \sum_{\ell=1}^{m-1} (2\ell+1)\ell = (p - m^2 + 1)m + \frac16 (m-1)m(4m+1)\\ &= (x-t)\left(x^2 - (x-t)^2 + 1 + \frac16(x-t-1)(4(x-t)+1)\right) \end{align}\tag{*3} $$ Substitute $(*2)$ and $(*3)$ into $(*1)$ and simplify, we obtain
$$F(p) = \frac12 p + A(t)\sqrt{p} + B(t) + \frac{1}{24} p^{-1/2} + \cdots\quad\text{where}\quad \begin{cases} A(t) = -(\frac13 + t - t^2)\\ B(t) = \zeta\left(-\frac12\right) + \frac56 t + \frac12 t - \frac13 t^3 \end{cases} $$ Since $\displaystyle\;\int_0^1 A(t) dt = -\frac12 \ne 0$, even after we subtract away the piece $-\frac12 p$, the sum
$$\sum_{k=1}^p \left( \{ \sqrt{k} \} - \frac12 \right) = F(p) - \frac12 p$$
is still systematically biased by a term of the form $-\frac12 \sqrt{p}$. This is the source of the $O(1)$ term you notice. To see this, let us rewrite the sum at hand in terms of $F(p)$.
Let $\theta_K = \#\{ n \in \mathbb{Z}_{+} : \sqrt{K} \le n \le 2\sqrt{K} \}$, the sum at hand can be rewritten as $$\sum_{K=1}^N \left( S_K - \frac{1}{2}\sqrt{K} \right) = \underbrace{\sum_{K=1}^N \sum_{\sqrt{K} \le n \le 2\sqrt{K}}\left(\left\{ \sqrt{n^2 - K} \right\} - \frac12\right)}_{\mathcal{I}} - \underbrace{\frac12 \sum_{K=1}^N \left(\sqrt{K} - \theta_K\right)}_{\mathcal{J}} $$ For the first term $\mathcal{I}$, we can express it using at most two $F(\cdot)$. $$\begin{align} \mathcal{I} &= \sum_{n=1}^{2\sqrt{N}}\sum_{\frac14 n^2 \le K \le \min( N, n^2 )} \left(\left\{ \sqrt{n^2 - K} \right\} - \frac12\right)\\ &= \sum_{n=1}^{2\sqrt{N}}\sum_{ \max(0,n^2 - N)\le \ell \le \frac34 n^2} \left(\left\{ \sqrt{\ell} \right\} - \frac12\right)\\ &= -\frac12\lfloor\sqrt{N}\rfloor + \sum_{n=1}^{2\sqrt{N}}\sum_{ \max(1,n^2 - N)\le \ell \le \frac34 n^2} \left(\left\{ \sqrt{\ell} \right\} - \frac12\right) \\ &= -\frac12\lfloor\sqrt{N}\rfloor + \sum_{n=1}^{2\sqrt{N}}\left[ F(p) - \frac12 p \right]_{p = \max(0,n^2-N-1)}^{\lfloor \frac34 n^2\rfloor} \end{align} $$ For the second term $\mathcal{J}$, one can show $$\sum_{K=1}^p \theta_K = \sum_{K=1}^p \left(\lfloor\sqrt{K}\rfloor + \frac12\right) + \frac12 | p - m(m+1) |$$ This implies $\mathcal{J}$ falls of like $\sqrt{N}$ and one can ignore them in the final limit $$\mathcal{J} = \frac12\sum_{K=1}^N \left(\sqrt{K} - \theta_K \right) = \frac12( A(t) - \frac12|1-2t|)\sqrt{N} + O(1)\quad\text{ where }\quad t = \{\sqrt{N}\}$$
To proceed further, we will wave our hands. We will assume
- only the leading term $A(t)\sqrt{p}$ in $F(p) - \frac12 p$ matters.
- we can average out $A(t)$ over $t$, i.e. replace $A(t)\sqrt{p}$ by $-\frac12\sqrt{p}$
- we can approximate the sum over $n$ by an integral over $n$
Changing variable to $n = \sqrt{N}s$, we find
$$\begin{align}\frac1N\sum_{K=1}^N \left( S_K- \frac{1}{2} \sqrt{K} \right) = \frac{\mathcal{I}}{N} + O (N^{-1/2}) &\approx -\frac{1}{2N} \left[ \int_0^{2\sqrt{N}} \frac{\sqrt{3}}{2} n dn - \int_{\sqrt{N}}^{2\sqrt{N}}\sqrt{n^2-N} dn\right]\\ &= -\frac{1}{2} \left[ \frac{\sqrt{3}}{2} \int_0^{2} s ds - \int_1^2 \sqrt{s^2 - 1}ds \right]\\ &= -\frac14\log(2+\sqrt{3}) \approx -0.3292394742312041 \end{align} $$ A number compatible with what OP described in question.