Old Mans Cooking Quest - Too late to complete?

I think I have got too far in the main quest to complete the old Mans Cooking Quest and get the warm doublet. I have made the meal, but when I go to the campsite in the woods the old man is not there. If I light the fire and then wait until night he does not appear. By this point I have already completed the four shrines.

I take it then I have gone too far in the main quest (the old man is now supposed to be waiting at the place between the four shrines so I assume he won't be anywhere else)

And if so, is there another way to get warm clothes in order to go into cold places? Do I even need to now that I have done that shrine in the mountains? (I got there using a torch)

Go ahead and meet with the old man like he asked, then after the meeting return to his cabin/house. There should be a chest there with the warm doublet, and a note.