Hom / tensor adjunction for $O_X$ modules?

Everything actually works for arbitrary ringed spaces and arbitrary sheaves of $\mathcal O_X$-modules, cf. e.g. Tag 01CM of the Stacks Project (where the proof of this lemma is omitted).

Firstly, you should be aware of three subtleties:

• $M \otimes_{\mathcal O_X} N$ is the sheafification of the presheaf $$U \mapsto M(U) \otimes_{\mathcal O(U)} N(U).$$ To avoid confusion, I will write $M \odot_{\mathcal O_X} N$ for this presheaf tensor product.
• The sheaf $\mathscr Hom_{\mathcal O_X}(M,N)$ is given on $U$ by $\operatorname{Hom}_{\mathcal O_U}(M|_U, N|_U)$, not $\operatorname{Hom}_{\mathcal O(U)}(M(U),N(U))$. This makes it slightly harder to define the 'obvious map', as we will see below.
• For $M, N$ both quasicoherent, it is not in general true that $\mathscr Hom_{\mathcal O_X}(M, N)$ is quasicoherent (this is the remark you make on the bottom; it is enlightening to try to think of a counter-example). However, it does work when $M$ is coherent (and $X$ is Noetherian, or use the correct definition of coherent sheaves on an arbitrary scheme or even ringed space).

I will define an obvious (but not so obvious) isomorphism $$f \colon \mathscr Hom_{\mathcal O_X}(M, \mathscr Hom_{\mathcal O_X} (N, K)) \to \mathscr Hom_{\mathcal O_X}(M \otimes_{\mathcal O_X} N, K).$$ We have to construct compatible isomorphisms $$f_U \colon \operatorname{Hom}_{\mathcal O_U} (M|_U, \mathscr Hom_{\mathcal O_U} (N|_U, K|_U)) \to \operatorname{Hom}_{\mathcal O_U}((M \otimes_{\mathcal O_X} N)|_U, K|_U)$$ for all $U \subseteq X$. Let $U$ be fixed from now on. I will break down what both sides are.

Right hand side. Since sheafification commutes with restriction to opens, the right hand side is $$\operatorname{Hom}_{\mathcal O_U}(M|_U \otimes_{\mathcal O_U} N|_U, K_U) = \operatorname{Hom}_{\mathcal O_U}^{\operatorname{pre}} (M_U \odot_{\mathcal O_U} N|_U, K|_U).$$ An element of this is a compatible family of maps (for all $V \subseteq U$) $$\psi_V \colon M(V) \otimes_{\mathcal O(V)} N(V) \to K(V).$$ However, as pointed out by the OP in the comments, you cannot now use the classical tensor-hom adjunction to say that this is the same as giving compatible maps $$M(V) \to \operatorname{Hom}_{\mathcal O(V)}(N(V), K(V)).$$ See also the remark below.

Left hand side. On the other hand, the left hand side consists of giving a compatible system of maps $$\phi_V \colon M(V) \to \operatorname{Hom}_{\mathcal O_V}(N|_V, K|_V).$$ The map $f$. Thus, there is an obvious map from left to right given by taking global sections: given $\phi_V$ and $m \in M(V)$, the map $\phi_V(m)\colon N|_V \to K|_V$ induces a map $\chi_V(m)\colon N(V) \to K(V)$ by taking global sections. This gives $\chi_V \colon M(V) \to \operatorname{Hom}_{\mathcal O(V)}(N(V), K(V))$. Using the classical tensor-hom adjunction, this in turn gives a map $$\psi_V \colon M(V) \otimes_{\mathcal O(V)} N(V) \to K(V).$$ However, having a map $\chi_V(m) \colon N(V) \to K(V)$ does not give you a map $\phi_V(m) \colon N|_V \to K|_V$ in general. It works when $V$ is an affine scheme and $N$ is quasicoherent, but that is not satisfactory. Thus, defining an inverse is not quite so easy.

The inverse of $f$. This is where the compatibility of the various $\psi_V$ comes in. That is, given a system $\psi_V$ from the right hand side, for any $m \in M(V)$, we have to construct a morphism of sheaves $$\phi_V(m) \colon N|_V \to K|_V.$$ For any $W \subseteq V$, we define \begin{align*} \phi_V(m) \colon N(W) &\to K(W)\\ n &\mapsto \psi_W(m|_W \otimes n). \end{align*} One checks that these are compatible for varying $W$, so we get $\phi_V(m)$ as claimed. Then check that the $\phi_V$ are compatible for varying $V$, etc. $\square$

I also omitted the verifications that anything was $\mathcal O$-linear, or even that the maps defined above are actually inverses. All of this is relatively straightforward.

Remark. Given a commutative diagram $$\begin{array}{ccc} M(V) \otimes_{\mathcal O(V)} N(V) & \to & K(V) \\ \downarrow & & \downarrow \\ M(W) \otimes_{\mathcal O(W)} N(W) & \to &\ K(W), \end{array}$$ we do not get a commutative diagram $$\begin{array}{ccc} M(V) & \to & \operatorname{Hom}_{\mathcal O(V)}(N(V), K(V)) \\ \downarrow & & \downarrow \\ M(W) & \to &\ \operatorname{Hom}_{\mathcal O(W)}(N(W), K(W)).\end{array}$$ In fact, we don't even get the right vertical map, because of the contravariance of $\operatorname{Hom}$ is the first variable. Thus, we cannot simplify the description of the right hand side.