Can quarry borders overlap in buildcraft?

Sharing frames is perfectly fine, as you can see by these three operational overlapping quarries:

Note, however, that stacking quarries, like the two on the left, makes one quarry ineffective. This is because quarries mine in the same pattern, so the second quarry is constantly trying to catch up with the first, not mine new blocks.

Only frames blocks themselves may overlap, tho. If the blocks don't line up, they will get stuck, as you suggest:

Here are two pairs of non-functioning quarries. Even tho neither is trying to place frame blocks in the mining area of the other, they get stuck because the are trying to place frame blocks in the frame area of the other.

Lastly, just for completeness, quarries cannot place frame blocks in the mining area of each other either:

The other anwswer said that "stacking quarries, like the two on the left, makes one quarry ineffective". While this is true the way he did it, you can stack quarries, in an effective way. You just have to make the upper one smaller by more than half. Here's an example of something I recently did in tekkit.

It's a bit of a mess, but if you examine it, you'll see 6 quarry frames. The 2 on the bottom and the one on the right we are not concerned with. Of the other 3, one is a full-blown 62x62 quarry. Then, stacked on top of it, there are two smaller quarries. One, on the right, is slightly less than half the area. I built that when the original quarry was about halfway done. Then another one, on the left, is slightly less than half of the remaining area. I built that when the other two were about 2 or 3 layers from bedrock, because I just wanted it to finish, and I had some extra quarries lying around from smaller projects.

This setup worked to speed up the process of mining that 62x62 area. You just need to be sure of 2 things:

1. Line the smaller quarry up to a corner of the larger one, that way the larger one doesn't waste time crossing empty gaps
2. Make sure the smaller quarry is less than half the size of the larger. This ensures that the smaller quarry is always as deep or deeper than the larger quarry, so the larger doesn't try to get blocks in the smaller's area.