Filtration and martingale from definition
Solution 1:
I'll go through your answers, which are correct , but need more detail. I provide detail with the idea of making this a "model" approach, which you can derive assistance from in approaching similar problems.
1
I think it should be justified as to why $\mathcal F_n := \sigma(Y_1,...,Y_n)$ is related to $\sigma(Y_n)$. To see this, you should actually prove that each of the $Y_i$ for $i<n$ are actually Borel functions of $Y_n$, so that they don't contribute anything to the sigma-algebra with their preimages.
We first take care of $Y_1$, by noting that $Y_1 = 1_{Y_k \neq 0}$ for all $k$, so $Y_1$ is $\sigma(Y_k)$-measurable.
Now, we inductively notice that for all $k>3$, we have $$ Y_{k-1}= Y_k1_{[0,1-\frac 1{k-1}] \cup (1-\frac 1{k},1]}+ Y_1 1_{[1-\frac 1{k-1},1-\frac 1k]} $$
Therefore, inductively, each of $Y_2,Y_3,...,Y_{k-1}$ is $\sigma(Y_1,Y_k)$-measurable, and hence $\sigma(Y_k)$ measurable. It follows that $\mathcal F_n = \sigma(Y_1,...,Y_k)=\sigma(Y_k)$.
Now, to find $\sigma(Y_k)$, for any $A$ we write $Y_k^{-1}(A) = Y_k^{-1}(A \cap [0,(1-\frac 1n)^2]) \uplus Y_k^{-1}(A \cap\{1\})$, since the range of $Y_k$ is the set $[0,(1-\frac 1n)^2] \uplus \{1\}$. (where $\uplus$ denotes the disjoint union).
The latter set in the union is either empty or $(1-\frac 1n,1]$.For the former, note that on the range $[0,(1-\frac 1n)^2]$, $Y_n$ restricts to the domain $[0,(1-\frac 1n)]$ where it is a continuous increasing bijective function. Therefore, the preimage of any subinterval of $[0,(1-\frac 1n)^2]$ is a subinterval of $[0,(1-\frac 1n)]$, and the bijective nature of the transformation implies that every subinterval of $[0,1-\frac 1n]$ is a preimage of some subinterval of $[0,(1-\frac 1n)^2]$.
Therefore, it follows that $\sigma(Y_n) = \sigma\left(\mathcal B[0,1-\frac 1n], (1-\frac 1n,1]\right)$ .
2
For this question, I am sure you assumed that $\mathbb E[X|Y_n] = g(Y_n)$ since $E[X|Y_n]$ is $\sigma(Y_n)$-measurable and any such function is a Borel function of $Y_n$. Once you do this, I'm not sure how you got the conditional expectation from the definition, but I see it as quite straightforward from the definition $$ \int_{A} 2\omega d \omega = \int_{A} g(Y_n(\omega)) d \omega \quad\forall A \in \sigma(Y_n) $$
Take $A = (1-\frac 1n,1]$, then you get $1 - (1-\frac 1n)^2 = \frac{g(1)}{n}$, therefore $g(1) = 2 - \frac 1n$. The equality above also holds for all $A \in \mathcal B[0,1-\frac 1n]$, and one can use the result that two Borel functions on a domain that have the same integral over all Borel subsets of that domain, are equal a.e. (i.e. except on a set of Borel measure zero) on that domain. Thus, $g(\omega^2) = 2 \omega$ on $[0,1-\frac 1n]$, and this leads to $g(\omega) = 2 \sqrt{\omega}$. Thus, the final answer is $g(Y_n) = 2 \sqrt{Y_n} 1_{Y_n \neq 1} + (2-\frac 1n) 1_{Y_n = 1}$.
3
I was once again unsure as to how you got the conditional expectation, but I'll write it out anyway. Indeed, to find $\mathbb E[X_n^2 | Y_{n-1}]$, we use the same approach as previously : write $X_n^2 = h_n(Y_{n-1})$, and note that $$ \int_{A} X_n^2(\omega) d \omega = \int_{A} h_n(Y_{n-1}(\omega))d \omega \quad \forall A \in \sigma(Y_{n-1}) $$
Take $A = (1-\frac 1{n-1},1]$. On this set, the equality reads :$$ \int_{1-\frac 1{n-1}}^{1-\frac 1n} 4 \omega^2 d \omega + \int_{1-\frac 1n}^1 (2-\frac 1n)^2 d \omega = \frac{h(1)}{n-1} $$
which, after the necessary algebra, gives $g(1) = (2-\frac 1{n-1})^2 + (n-1)\left(\frac 1{3(n-1)^3} - \frac 1{3n^3}\right)$. The equality taken over all $A$ in $\mathcal B[0,1-\frac 1{n-1}]$ admits a logic similar to part $2$, so that on this range , $4 \omega^2 = h_n(\omega^2)$, therefore $h_n(\omega) = 4 \omega$. That is , the final expression is $$ \mathbb E[X_n^2 | Y_{n-1}] = 4 Y_{n-1} 1_{Y_{n-1} \neq 1} + \left[(2-\frac 1{n-1})^2 + (n-1)\left(\frac 1{3(n-1)^3} - \frac 1{3n^3}\right)\right] 1_{Y_{n-1} = 1} $$
Squaring the $X_{n-1} = g(Y_{n-1})$ expression from part $2$ gives :$$ X_{n-1}^2 = 4Y_{n-1}1_{Y_{n-1} \neq 1} + (2-\frac 1{n-1})^2 1_{Y_{n-1} = 1} $$
Therefore, on the set $\{Y_{n-1} = 1\}$ which has Borel measure greater than zero, we have $X_{n-1}^2 \neq \mathbb E[X_n^2 | Y_{n-1}]$, which proves that $X_n$ is not a martingale, as desired.
Perhaps I was too nitpicky , but my intention in this case is to highlight that along the way, I used some definitions and facts in a very particular manner, and I think this is how one can tackle similar situations as well. Furthermore, doing examples rigorously can often be more beneficial than just going through the formulas, as a mathematical writing exercise.