What’s the etymology of the word “unstable”, in the context of software?

Unstable does not have a special meaning in the context of computer software. It has the conventional meaning. Unstable describes something which is likely or prone to give way, change, or fail.¹

An early use in print of the word unstable applied to software can be found in Edsger Dijkstra’s² A Discipline of Programming (1976):³

Since then we have witnessed the proliferation of baroque, ill-defined and, therefore, unstable software systems. Instead of working with a formal tool, which their task requires, many programmers now live in a limbo of folklore, in a vague and slippery world, in which they are never quite sure what the system will do to their programs. Under such regretful circumstances the whole notion of a correct program—let alone a program that has been proved correct—becomes void. What the proliferation of such systems has done to the morale of the computing community is more than I can describe. (Emphasis added.)

The word unstable is often used to name or characterize a software package which has not been subjected to extensive testing. For example, an automated daily build which does not undergo regression testing might be labeled as the unstable build, again using the conventional meaning: there is no assurance that such a build will not give way or fail when used, and such a build will change from day to day.

For terms which are specific to the software development process, see the Wikipedia article “Software release life cycle”.⁴ Stable and unstable are employed, again using their conventional meanings, in the definitions of some of these terms.

I'm pretty certain "unstable" has always been used in the history of computing, as it has been used in science and engineering… it's the most natural word to use for something that is "not stable."

As for the second part of your question, I think "unstable" and "buggy" have slightly different uses. I would say "buggy" more often refers to incorrect output and "unstable" more often refers to catastrophic failure, where the software ceases to function, "hangs" or "freezes".

That said, they can be interchangeable.