"The place where we promised to meet"
This is talking about a promise to meet at a certain place. However, is it grammatically correct? Is it badly phrased? It seems that it can be misinterpreted to mean that at a certain place a promise was made to meet, rather than promising to meet at a certain place later.
It is grammatical (as far as it goes - I assume you're using it as part of a longer sentence!).
It is also potentially ambiguous, as you say - it could be interpreted to mean the place where we made the promise rather than the place where the meeting will take place. However, it is more likely to be interpreted the way you want; moreover in context it is unlikely to cause much confusion, especially if there are more cues in the sentence to indicate which meaning is intended, for example:
At the appointed time, I went to the place where we agreed to meet, and began to wait.
This is very unlikely to be interpreted as the place where we made our agreement.
To remove any ambiguity (although I agree with @psmears that the sentence is unlikely to be misinterpreted) you could use meeting place or meeting point.
I went to the meeting point, as promised.
However, the place that we promised to meet has some poetic vein to it that is lost in the above example.
If confusion reigns due to misunderstanding of that phrase, try changing it a little so that it retains it romantic taste while being absolutely clear:
The place we had promised we would meet.
There can be no ambiguity about the above.
For a rather archaic, but unambiguous option you could use tryst or trysting place.