Oleinik's entropy condition

I assume you mean with Oleinik's entropy condition this inequalities: $$ \frac{f(v) - f(u_l) }{v - u_l} \geq s \geq \frac{f(v) - f(u_r)}{v -u_r}$$

I present the proof I found in lecture notes on Numerics of Conservation Laws by Siddartha Mishra. Start with the Fundamental Theorem of Calculus $$\eta (u_r) - \eta (u_l) = \int_{u_l}^{u_r} \eta'(v) dv$$

Using integration by parts, $$\int_{u_l}^{u_r} \eta'(v) \cdot 1 dv = \eta'(v)(v - u_l) \Big \vert^{u_r}_{u_l} - \int_{u_l}^{u_r} \eta''(v) (v - u_l) dv \\ = \eta'(u_r)(u_r - u_l) - \eta'(u_l) \cdot 0 - \int_{u_l}^{u_r} \eta''(v) (v - u_l)$$

For $\psi$, we also use first the Fundamental Theorem of Calculus to obtain $$\psi(u_r) - \psi(u_l) = \int_{u_l}^{u_r} \psi'(v) dv $$

where $\psi'$ fulfills (if forming an entropy pair with $\eta$) $$\psi' = \eta' f'.$$

Thus, $$\int_{u_l}^{u_r} \psi'(v) dv = \int_{u_l}^{u_r} \eta'(v) f'(v) dv $$

Again, use integration by parts to obtain $$\int_{u_l}^{u_r} \eta'(v) f'(v) dv = \eta'(v) \big(f(v) - f(u_l) \big) \Big \vert^{u_r}_{u_l} - \int_{u_l}^{u_r} \eta''(v) \big(f(v) - f(u_l) \big) d v\\ = \eta'(u_r) \big(f(u_r) - f(u_l) \big) - \eta'(u_l) \cdot 0 - \int_{u_l}^{u_r} \eta''(v) \big(f(v) - f(u_l) \big) d v$$

Plugging this into your result: $$ \psi(u_r) - \psi(u_l) - s \big( \eta(u_r) - \eta(u_l) \big) = \eta'(u_l) \Big[\big(f(u_r) - f(u_l) \big) - s \big(u_r - u_l\big) \Big] \\ + \int_{u_l}^{u_r} \eta''(v) \Big[ s \big(v - u_l \big) - \big(f(v) - f(u_l) \big) \Big] d v \overset{!}{\leq} 0$$

By the Rankine-Hugoniot condition you also stated in the question the first summand on the RHS drops out. Since $\eta''(v)$ is a strictly convex function we have that $\eta''(v) > 0 \: \forall \: v$. Thus, we must have $$ s \big(v - u_l \big) - \big(f(v) - f(u_l) \big) \leq 0 \Leftrightarrow s \leq \frac{f(v) - f(u_l)}{v - u_l} $$

which is exactly the first inequalitiy.

To show the second equality, use the fundamental theorem of calculus with changed boundaries, i.e, $$\eta (u_r) - \eta (u_l) = - \int_{u_r}^{u_l} \eta'(v) dv$$ which adds the necessary $-$ in front of the integrals to flip the inequality and gets you the $u_r, f(u_r)$.