Strictly increasing, absolutely continuous function with vanishing derivative

Solution 1:

Note that $$\frac{F(x)-F(x+h)}{h}=\frac{m(A\cap [x+h,x])}{h}\le 2\frac{m(A\cap[x-h,x+h])}{2h},\ \forall h<0 \tag{1},$$

and

$$\frac{F(x+h)-F(x)}{h}=\frac{m(A\cap [x,x+h])}{h}\le 2\frac{m(A\cap[x-h,x+h])}{2h},\ \forall h>0 \tag{2},$$

Lebesgue density theorem, $(1)$ and $(2)$ implies that $$ F'(x)=0,\ a.e. \ x\in [0,1]\setminus A.$$

Edit: As @Dave pointed out in the comment, we also have that $F'(x)=1$ a.e. $x\in A$. This implies in particular that $G'(x)=1$ a.e. $x\in A$ and $G'(x)=-1$ a.e. $x\in [0,1]\setminus A$, therefore, by using the properties of $A$, it follows immediate that $G$ cannot be monotone in any interval.

Solution 2:

We have $F(x)=\int_{-\infty}^x\chi_A(t)dt$, where $\chi_A\in L^1(m)$ since $A$ is a Borel subset of $[0,1]$. Consequently (see, for example, Folland, Real Analysis, Corollary 3.33) $F$ is absolutely continuous and $F'=\chi_A$ a.e. Thus $F'=0$ a.e. on $[0,1]\backslash A$. Since $m(A)<m([0,1])$ by assumption, $F'=0$ on a set of positive measure. (If $0\le x<y\le1$, then $F(x)-F(y)=m((x,y]\cap A)>0$ by assumption, so $F$ is strictly increasing on $[0,1].$)

Likewise, $G(x)=\int_{-\infty}^x(\chi_A-\chi_{[0,1]\setminus A})(t)dt$, so $G$ is absolutely continuous and $G'=\chi_A-\chi_{[0,1]\setminus A}$ a.e. If $I$ is any subinterval of $[0,1]$, then both $I\cap A$ and $I\setminus A$ have positive measure, since $0<m(A\cap I)<m(A)$ by assumption. Thus $G'$ assumes both values $1$ and $-1$ on $I$, whence $G$ is not monotone on $I.$