Prove that $\vert E \vert=0$ if $\frac{x+y}{2}\notin E$ for $x,y \in E$
Let $x\in E.$ Define $A_h = E\cap (x-h,x)$ and let $B_h$ be the reflection of $A_h$ across $x.$ Note $m(B_h) = m(A_h).$ Note also that $B_h\cap E = \emptyset,$ otherwise $x$ is the average of points in E. Therefore, for all $h>0,$
$$\tag 1\frac{1}{2h}m(E\cap (x-h,x+h)) \le \frac{1}{2h}[m(A_h ) + h -m(B_h)]$$ $$ = \frac{1}{2h}[m(A_h ) + h -m(A_h)] = \frac{1}{2}.$$
Now for a.e. $x\in E,$ the Lebesgue differentiation theorem shows the limit on the left side of $(1)$ equals $1.$ We've just shown that for every $x\in E$ the limit on the left side of $(1)$ is no larger than $1/2,$ if it exists. It follows that $m(E) = 0.$
Let $I(x)$ be the indicator function of $E$. By Lebesgue's density theorem, there exists a segment $[a,b]$ such that $\mu(E\cap [a,b]) \geq \frac{9}{10} |b-a|$. Denote $L= b-a$. Then $$\int_a^b \int_a^b I\left(\frac{x+y}{2}\right) dx dy \leq \int_a^b \int_a^b dx dy - \int_{[a,b]\cap E} \int_{[a,b]\cap E} 1 - I\left(\frac{x+y}{2}\right) dx dy = \left(1- \frac{9}{10}\times \frac{9}{10}\right) L^2 = \frac{19}{100}L^2.$$ On the other hand, letting $s=(x+y)/2$ and $t=(x-y)/2$, we get $$\int_a^b \int_a^b I\left(\frac{x+y}{2}\right)\, dx dy \geq 2 \int_{(3a+b)/4}^{(3b+a)/4} \int_{(a-b)/4}^{(b-a)/4} I(s)\, ds dt $$ In the $xy$-plane, the integral on the right is over a subset of the square $[a,b]^2$; namely it is over the square with the corners in the middle points of the sides of the square $[a,b]^2$. We have, $$ 2 \int_{(3a+b)/4}^{(3b+a)/4} \int_{(a-b)/4}^{(b-a)/4} I(s)\, ds dt \geq (b-a) \int_{(3a+b)/4}^{(3b+a)/4} I(t) dt \geq (b-a)\left(\frac{b-a}{2} - \frac{1}{10} (b-a)\right) =\frac{2}{5}\,L^2.$$ We get a contradiction.