"We rarely go on holiday." Can I say "Us neither"?

As mentioned above, it's all down to usage and "us neither" is not commonly used, with people preferring the phrase "neither do we" - for that reason I would stick with saying "neither do we".

I would say the same is true of the other personal pronouns: "neither do they", etc.

There's nothing wrong with saying "us neither", but (and you mention it in your second line) I'd tend to go for "neither do we".

"neither do we" - 473k hits on Google

"us neither" - 374k

I don't agree that using neither or either in this scenario is correct: you use those in the case of a negative statement, but the statement in question is positive. If the statement was the negative version:

We don't go on vacation often.

Then the response would be "We don't either" or "Neither do we" or (colloquially) "Us neither".

But when the statement is positive:

We rarely go on vacation.

Then the response should be So do we or We do, too or (colloquially) Us too.

Granted, few will bat an eye, and everyone will know what you mean, if you respond conversationally to "We rarely go on vacation" with "Neither do we". It is easy to flip the original statement into the negative grammatical form "We don't go on vacation much".

But still, technically the reply is nonsensical, because it is effectively "Neither do we (rarely go on vacation)". The original statement is affirmative, and it doesn't make grammatical sense to append a "neither" to an affirmation.