I think this is true.

Given $K'\subseteq K\subseteq G$, one gets a fiber bundle $K/K'\rightarrow G/K'\rightarrow G/K$ where the the first map is induced by inclusion $K\rightarrow G$ and the projection is the obvious one sending $gK'$ to $gK$.

Assuming the existence of this bundle for the moment, since $G/K\cong F$ is contractible, the bundle must be trivial, so it is diffeomorphic to a product $K/K'\times G/K$. But $G/K\cong F$.

So, where does this fiber bundle come from?

Well, for starters, we have a principal $K$ bundle $K\rightarrow G\rightarrow G/K$ coming from, say, right multiplication of $K$ on $G$. (More generally, given any proper free action of a Lie group $K$ on a manifold $M$, one gets a principal bundle $K\rightarrow M\rightarrow M/K$.)

Now, note that $K$ acts on $K/K'$ by left multiplication. So, we can form the space $(G\times K/K')/K$ where $K$ acts by right/left multiplication on both factors: $k\ast( g, hK') = (gk^{-1}, khK')$. This action is free and proper, so the quotient $(G\times K/K')/K$ is a smooth manifold. Moreover, the projection $\pi:(G\times K/K')/K \rightarrow G/K$ given by $\pi( [ g, hK]) = gK$ is a fiber bundle map with fiber $K/K'$. The inclusion $K/K'\rightarrow (G\times K/K')/K$ is obtained by fixed a point of $G$ (say, the identity $e\in G$), and mapping $hK'$ to $[(e,hK')]$. This is called the associated bundle construction.

So, we have a bundle $K/K'\rightarrow (G\times K/K')/K\rightarrow G/K$. The last step is to identify $(G\times K/K')/K$ with $G/K'$ and check that the maps work like we think.

So, consider $f:(G\times K/K')/K\rightarrow G/K'$ defined by $f([g,hK']) = g h K'\in G/K'$. Let's show that $f$ is a diffeomorphism.

First note that $f$ is well defined: for any $k\in K$, $f([gk^{-1}, khK']) = gk^{-1}khK'= ghK' = f([g,hK']$. Moreover, $f$ is obviously smooth.

On the other hand, we can simply write down an inverse: $f^{-1}(gK') = [(g, eK')]$.

I'll leave it to you to verify that this is the inverse.

To finish, I'll just verify that with respect to this diffeomorphism, the projection $G/K'\rightarrow G/K$ is $gK'\mapsto gK$; I'll leave the case of the inclusion of a fiber to you. The projection $G/K'\rightarrow G/K$ is, via transport of structure, really the composition $G/K'\xrightarrow{f^{-1}} (G\times K/K')/K\rightarrow G/K$. For an element $gK'\in G/K'$, this is $$gK'\mapsto [(g,eK')] \mapsto g/K.$$