What can be the value of the integral $\int_{3a}^{5a}f(x)\mathrm {d}x$?

The integral is equal to $4.5a^2$.

The value of $f(x)$ is positive on the given domain. So the value of $(f\circ f)(x)$ is positive too. Then the given inequality assumption is equivalent to $f^\prime(x)<0$. From the graph, it is equivalent to $1/2\, a<x<3/2\, a$ or $3a\, <x<9/2\, a$.

You can determine the value of the integral by checking the five options one by one.

For your illustration, I will show my work on option (A).

From $4.5a^2 = 4$, you get $a=\frac{2\sqrt{2}}{3}$. Substituting this value to the solution to the inequality, you get $\frac{\sqrt{2}}{3}<x<\sqrt{2}$ or $2\sqrt{2}<x<3\sqrt{2}$. So, the integer solutions to the inequality is $x=1, 3, 4$. However, the product $=12 > 4a = \frac{8\sqrt{2}}{3}$. So you can exclude the option (A).