Sources say English was rhotic in the 17th century -- how do they know that?

Solution 1:

In addition to rhymes and dictionaries already noted, you can cite spelling shifts in which the "r" letter positioned after a vowel disappears from certain words.

The Wikipedia article about Rhotic and non-rhotic accents cites

the Oxford English Dictionary reports bace for earlier barse (today "bass", the fish) in 1440 and passel for parcel in 1468.

which is more or less the time frame in the question. This shows, assuming the spelling evolution follows the pronunciation shift, that in this words the rhotic pronunciation had disappeared as well.

Also have a look to a contemporary phenomenon named "The Great vowel shift".

Solution 2:

For one thing, there are dictionaries from that time that indicate pronunciation, and we can learn from how they write down pronunciation. For example, I have heard Walker's Critical Pronouncing Dictionary cited as a source on that matter.