Non-axiomatisability and ultraproducts
Let $T$ be a first-order theory over a language $L$, and let $\mathcal{M}$ be a subclass of the class of models of $T$. As I understand it, if there is no theory $\hat{T}$ over $L$ whose class of models is exactly $\mathcal{M}$, frequently, the "morally correct" reason is that $\mathcal{M}$ is not closed under ultraproducts. However, it is sometimes possible to obtain a simpler proof of non-axiomatisability by compactness or completeness considerations.
Example. Consider the first order theory of fields, and let $\mathcal{M}$ be the class of fields of positive characteristic. Then $\mathcal{M}$ is not axiomatisable in the language of fields since, for example, if we take the ultraproduct of all the finite fields $\mathbb{F}_p$, $p$ prime, we would obtain a field of characteristic 0.
We could also prove this directly: if $T'$ is the naïve axiomatisation of fields of characteristic 0 (i.e. the one with one axiom of the form $\underbrace{1 + \cdots + 1}_{n\text{ times}} \ne 0$ for every positive $n \in \mathbb{N}$), and $\hat{T}$ is any axiomatisation of $\mathcal{M}$, then $T' \cup \hat{T}$ is inconsistent, so there is some finite subset which is inconsistent, so there is some finite set $X$ for which $\hat{T}$ proves that there is an $n \in X$ such that $\underbrace{1 + \cdots + 1}_{n\text{ times}} = 0$; but there are fields of positive characteristic other than those $n \in X$ — a contradiction.
Question. Is it in fact always possible to translate a proof of non-axiomatisability using ultraproducts to one using compactness/completeness?
To start with, here is a conditional negative answer (really any compact proper extension of $\mathsf{FOL}$ can get the ball rolling, but this is a really cool one):
If we add the ability to quantify over isomorphisms of ordered fields in a precise sense we get a compact logic $\mathcal{L}(Q_\mathsf{Of})$; see Mekler/Shelah 1993. The class $\mathfrak{K}$ of rigid ordered fields is $\mathcal{L}(Q_{\mathsf{Of}})$-definable, so its non-elementarity (= non-$\mathsf{FOL}$-definability) cannot be ruled out purely on compactness grounds. Moreover, all nontrivial ultraproducts over $\omega$ are countably saturated, so assuming $\mathsf{CH}$ any nontrivial ultraproduct over $\omega$ of the rigid ordered field $\mathbb{Q}$ must be non-rigid (being countably saturated and of size $\aleph_1$). So conditionally on $\mathsf{CH}$ we have simple negative answer to your question: the non-elementarity of $\mathfrak{K}$ cannot be established by compactness alone, but can be established using ultrapowers.
OK, now let's think about getting rid of $\mathsf{CH}$ ...
One immediate idea is to use an absoluteness theorem. Specifically, the sentence
$(*)\quad $ "There is a countable non-rigid ordered field isomorphic to $\mathbb{Q}$"
is $\Sigma^1_1$, and by the argument above plus downward Lowenheim-Skolem we know that $(*)$ follows from $\mathsf{CH}$. Mostowski absoluteness then gives us $(*)$ unconditionally.
But there's a serious drawback to this approach (besides the obvious one of "but why though?"). The appearance of downward Lowenheim-Skolem should actually be very disappointing. Lindstrom's theorem says that first-order logic is maximal with respect to compactness and (a very weak form of) downward Lowenheim-Skolem. Consequently, in some sense any non-axiomatizability fact is a purely abstract consequence of compactness and downward Lowenheim-Skolem. So if we want to build a situation where ultraproducts suffice and compactness alone doesn't, we should avoid Lowenheim-Skolem.
It seems more reasonable, therefore, to try to prove directly in $\mathsf{ZFC}$ that $\mathbb{Q}$ has a non-rigid ultrapower. There's a big theorem which lets us do this in one line: non-rigidity is upwards-preserved by ultrapowers, so apply Keisler-Shelah given some non-rigid ordered field $F\equiv \mathbb{Q}$. Of course, even if we're okay bringing a big theorem like KS into the picture, there's still the issue of how we get such an $F$ in the first place. Put another way, it's neat that the sentence
$(\dagger)\quad$ "Every infinite structure has a non-rigid ultrapower"
can be proved in $\mathsf{ZFC}$ by appeal to the properties of first-order logic, but that limits the satisfyingness of $(\dagger)$ for our purposes here. So we plausibly have an unconditional ultraproduct-only construction, but in terms of an argument we still haven't gotten very far.
All of this raises the following follow-up question:
Can $\mathsf{ZFC}$ prove (without bringing first-order model theory into the picture) that every countably infinite structure in a countable language has a non-rigid ultrapower over $\omega$?
Embarrassingly, even if we ignore the parenthetical this isn't obvious to me (and indeed Shelah has a number of papers on the subtleties of $\mathsf{ZFC}$-provable facts about ultrapowers, such as the "Vive la difference" series)! But surely I'm having a silly moment and someone will point out what I'm missing in a comment below ... (EDIT: I've now asked this as a separate question.)