Solution 1:

This is only a partial answer.

If $κ>|M|^+$ and $F$ is $|M|^{++}$-complete then $|M|=|N|$, if not, let $(f_i\mid i\in|M|^+)$ be a (possibly partial) listing of representatives of elements of $N$

Let $D_{i,j}$ be the set of indices where $f_i$ and $f_j$ disagree, this is a $F$-big set, in particular $\bigcap_{i<j\in|M|^+}D_{i,j}$ is not empty, if $m\in \bigcap_{i<j\in\lambda}D_{i,j}$ we have $f_i(m)≠f_j(m)$ for all $i,j∈λ$ so $|\{f_i(m)\mid i\in |M|^+\}|=|M|^+>|M|$, contradiction.

Solution 2:

I think $|N|$ could be smaller than $\kappa$ for some trivial reason: if $F$ contains a set $X$ of size $\lambda$ such that $|M|^\lambda<\kappa$, then $N=M^\kappa/F$ is the same as $M^\lambda/F'$ where $F'$ is the restriction of $F$ on $X$, so has size at most $|M|^\lambda<\kappa$.

If $F$ is uniform then the question becomes more interesting (probably more difficult too). If $|M|$ is countable and $F$ is countably incomplete, then $|N|$ is at least continuum, see section 5 of this survey.

Here is another short note which I vaguely remember but it took some time to find it again...If $F$ is regular then $N$ has the largest size possible, namely $|M|^\kappa$. Also from here it seems the general question is indeed quite difficult.