Did "brushwoodsmen" exist?

The only things I could find for the word brushwoodsman was a hit Kath Trevelyan, by Jeremy Cooper, although the preview doesn't show it.

I thought I would try my luck and look for brushwood men. Surprisingly this actually worked.

The first thing I found was a bit disappointing. In the book Sir Nigel (sorry, the link highlights the wrong thing, but the words are there):

The extract distinguishes "brushwood men" and villagers, so it must mean something, but it's not clear what.

An extract from The White Company came closer:

This clearly paints brushwood men as seemlying outcast, if not beggars.

(Both the above books are by Arthur Conan Doyle.)

But then I hit gold in Tolkien the Medievalist:

So You can see that brushwood men were indeed outlaws. Escaped serfs who would live in the forest and rob people to get by.

As is often the case back in those days, what someone is known to do would become their surname, so it is entirely plausible that brushwood would originate as a surname for the folk who can't live in the village and are outlaws.

I'd say it is equally plausible that the word brushwoodsmen is used in speech, if not writing, to indicate the same people.