Non-averaging set cannot have positive measure

Let $E \subset \mathbb{R}$ be non-averaging, that is, it does not contain any midpoints, so for any $a, b \in E$, $\frac{a + b}{2} \notin E$. It's claimed that as a consequence of the Lebesgue density theorem there are no non-averaging sets with positive measure. I don't see how this follows.

Suppose that $E$ is a measurable set and $x\in E$ such that $\lim\limits_{a\to0+}\dfrac{m(E\cap[x-a,x+a])}{2a}=1$. Without loss of generality (translating everything by $-x$), assume $x=0$, so we have $\lim\limits_{a\to0+}\dfrac{m(E\cap[-a,a])}{2a}=1$. Then there exists $a>0$ such that $m(E\cap[-a,0))>\frac23a$ and $m(E\cap(0,a])>\frac23a$. Consider the set $-(E\cap[-a,0))=-E\cap(0,a]$, which has measure greater than $\frac23a$. It follows that $m(-E\cap E\cap(0,a])>\frac13a$, and in particular $(E\cap -E)\setminus\{0\}\neq\varnothing$. Let $y$ be an element of $(E\cap -E)\setminus\{0\}$. Then $y$ and $-y$ are distinct elements of $E$, and their average is in $E$.

If $E$ has positive Lebesgue measure, then such an $x$ exists by the Lebesgue density theorem.