Is the word "overseas" only used to country bordered by water?

From a British English perspective it is a synonym of abroad, but from a continental Europe perspective, for instance, it would not be used to refer to countries that are not divided by the sea or the ocean.

Overseas:

to or in a foreign country, especially those separated from your country by the sea or ocean (synonym abroad)

• to live/work/go overseas The product is sold both at home and overseas. He was working overseas for an oil company.

(OLD)

Overseas:

• 1580s, from over + sea. Popularised during World War I as a British euphemism for "colonial."

(Etymonline)