What does "went through the guards of the broadsword" mean?
Solution 1:
Even with the context you provided, and being a native speaker, that sentence tripped me up a bit too, mostly because it's employing a unfamiliar sense of the word guard (which would have been more familiar to readers at the time Scott wrote Waverly).
So I'll tell you both what the word means and how I puzzled it out, which might help you in similar situations in the future.
First, to answer your question directly, in this context, guards means:
the posture of defence or readiness in fencing, boxing, cricket, etc.
Collins sense 20
The hints that this is the sense being employed are that the kid is going through the guards with a weapon, and that weapon usually hangs over the chimney (in the selfsame study), and the nephew was surprised at the entrance of his uncle, all of which tell us it is likely the kid took the sword off the wall and was practicing fencing with it.
In particular, he was practicing the guards of that kind of weapon; guards here meaning the defensive positions in fencing, similar to the images below (though these guards are specific to the longsword, not the broadsword as in the story):
image credit: Pinterest
And, in case it's not clear to non-native speakers, through here is used in the sense of "from A to Z"; there is a fixed series of guards (as in the illustration above), and the kid was practicing each, one after another: he went through the series.
In short, the kid was play-fighting with the family's heirloom sword when his uncle walked in and nearly caught him.
Solution 2:
Broadswords varied a lot, and the guard protecting the hand changed over time from simple cross bar to ones that came down to protect the hand. The more elaborate ones had loops and other embellishments which were used to trap the enemies blade or in some cases as sword breakers.
I found with broadsword the more elaborate the guard was the harder is was to use. I suspect that was because i preferred a hand and half sword and had prior sword fencing skills in epee and sabre. It was easy to catch your clothing or your own wrist with the fancy guards which im sure would go away with more practice with that style and weight of blade.
You might find that phrase implies the hero tried to block a blade with the guard ( perfectly possible ) and mistimed it so the enemy blade went through the guard. At that point so long as it's not on your fingers you could try a lock technique to trap their blade and disarm them, the instinct is to pull away, you have to train yourself through your natural instinct.