What's the best way to annihiliate an enemy's family?

The good news is that you don't have to kill absolutely everyone. Claims gradually weaken and go away as they are inherited; killing a strong claim holder gives his heirs weak claims, which are much harder to push and carry lower opinion maluses.

If you do have targets living in your realm, don't forget that you can take the Intrigue focus with Way of Life and use the Spy On action. This grants additional opportunities to abduct and murder, on top of making it more likely that you will reveal any plots they are a part of, granting you Righteous Imprisonment reasons (i.e. without tyranny).

Short of mass (read: slow, gradual) murder — including the more contrived forms of getting them killed in battle or by disease — the only other way to end a dynasty is through marriage. You can marry the males matrilineally and the females regularly, and any of their progeny will not be of the same dynasty any more. This will still allow claims to be inherited normally, but if they're marrying into your dynasty, then it's problem solved (from the game-over point of view, at least).

Unfortunately, this is ill suited to your question, as your target dynasty is probably either 1) not under your direct control or 2) unwilling to accept an invite to your court, because the claim they hold is against your realm. But it can work well for the branches of the family that aren't an immediate threat to your realm. Regular marriage to a female, even with a strong claim or actually in the line of succession, is usually accepted by the AI. But matrilineal marriage will typically only be accepted if:

• The man is your direct courtier — if they hold a title, you can't get power over them by making them a member of your council.
• The woman is important enough — basically, if you have a female relative close enough to grant an alliance to a sitting ruler through marriage, you can get non-inheriting members of other great houses to marry them matrilineally.
• The man has no claims — useful for second generations who stand to inherit a claim, but haven't gotten it yet. If they aren't in the actual line of succession, the AI will usually let them go to matrilineal marriage.
• More generally, the woman has to be equal to or greater in importance than the man. So if you have a second cousin who's not getting any claims or any inheritances, you're going to have to settle for marrying them to an unimportant member of the target dynasty as well.

One other note is that it's also possible to end progeny through poor marriages. If you have someone with low fertility or nearing the end of childbearing years, they can serve as a stop-gap to prevent further procreation. Only very old men will accept a bride out of childbearing years (typically 65+, but I've seen it as low as 50), but the AI is less careful about allowing marriages to homosexual theologians or stressed, depressed or otherwise ill people. As a bonus, with the buffed up diseases in the latest patch, an unmarried small-pox sufferer might serve as a means to assassinate multiple targets!

Also keep in mind that negative congenital traits are simulated with recessive genetics in mind. If you can invite a hunchback to your court and get them to breed, you could have a set of poor spouses to draw upon for your nefarious familial wedding. If you marry enough hunchbacks into your target dynasty, it'll eventually resurface. You could also try to kill two birds with one marriage by getting the target dynasty members to marry each other. (!)

And of course, if you're Byzantine you can take a more punitive approach to reproduction – this still requires imprisonment, but doesn't seem to incur tyranny like execution. On the plus side, sieges on castles seem to have a much higher chance to result in capturing prisoners from the court, so a well timed war on a distant branch of the family or a revolting vassal could result in rounding up a bunch of claimants.

I kinda love how realistically this game rewards being a bad person... Straight up murder is frowned upon, but misogyny, sociopathy and germ warfare can help you take over the world.