Does one stand by or at the bar?

To answer your question directly: both by and at are viable options for setting someone's location.

at would imply a very very close proximity to the bar (think, able to touch it)

by would imply proximity, but could just be the general vicinity of it

In this particular case, I would choose at since it's likely that they're physically quite close if they're talking to the bartender (barring yelling from several feet back).

However, there are a couple other things with your phrasing that bother me. I would likely word it:

When both of you are standing at the bar ordering a drink, and a bartender asks you to pay now or open a tab.

In particular

• your use of the "you're" contraction seems alien here. I think it's because you isn't the subject, but rather the object of the preposition of.
• the verb stand should be standing
• I believe asks to pay now is incorrect (at the least, it sounds very odd). You could say asks if you want to pay now or asks you to pay now.

In British use, "at the bar" is the invariable idiom for "standing near the bar being served or waiting to be served".

If you said "by the bar" it would suggest that you are physically near the bar, but not necessarily for service (maybe you have been served, and are standing or sitting there enjoying your drink).

I don't know whether or not this distinction is familiar in N America.