Is it worth it to let trees grow taller?

Before going into detail, I should first point out there are five stages of tree growth. I've made a small collection of images to display each of the five stages of growth. Each stage is numbered, and some overlap into the next images for a size comparison.

Stage One

This is how the trees will appear immediately after planting. They cannot be cut. The only option you will have, should you try, is "Plough Field". You do not want to do this.

Stage Two

After approximately one day of game time, the tree will grow into its second stage. Again, in this stage, it can only be ploughed and no wood will be gained.

Stage Three

This is the first time you will be able to harvest any wood from your trees. This occurs approximately one in-game day after the tree reaches stage two. Harvesting the trees at this stage will only grant you 1 wood.

Stage Four

Once your tree passes stage three and advances to this level of growth, you will be able to harvest 3 wood. That's two extra wood for waiting just one extra day.

Stage Five

This is the final stage of your tree's growth. Occurring, again approximately, one in-game day after stage four, this plot of trees will yield 5 wood. With this being the final stage of growth, you can only receive a maximum of 5 wood per tree plot harvested.

Given the amount of time it takes for the trees to mature to this point, and the low sale value of wood, I would say it is worthwhile waiting for your trees to reach full maturity prior to harvesting, as wood is the primary ingredient for building mills, wells, and cattle pens.