Is not to mention correctly used in this context?

Your example (which is grammatically correct) modified in a different way:

As an active market participant for the past few years, I am amazed by how 'Algorithmic Trading' has redefined the way trading is done and by the increasing number of asset managers that are adopting this technique, not to mention the outstanding returns and ability to consistently outperform more traditional investment vehicles.

A longer sentence, but not a run-on. Including not to mention at the ending parts of a sentence emphasizes its use as a rhetorical device.

From the OED:

not to mention ——: used to refer to an additional fact or point which reinforces the speaker's case (a rhetorical device suggesting that the full strength of the speaker's argument is not being presented).

Another example:

Tempting, but wrong: The Democratic base, not to mention the large majority of Americans is not nearly as partisan as the loudest voices on Twitter or cable news. Slate Jan 27, 2019

As long as 'not to mention' is followed by a complete sentence, it's just as good a connector as 'however' or 'in addition'. Also, if 'colloquial' comes from the Latin 'colloquium', meaning conversational, I think it's safe to say 'not to mention' has made it into formal written language, at least here across the pond :). For example, Henry David Thoreau began a sentence with the phrase in "A Plea for Captain John Brown" cited in the Goldman Law Library of Yale Law School,

Not to mention his other successes, was it a failure, or did it show a want of good management, to deliver from bondage a dozen human beings, and walk off with them by broad daylight, for weeks if not months, at a leisurely pace, through one State after another, for half the length of the North, conspicuous to all parties, with a price set upon his head, going into a court-room on his way and telling what he had done, thus convincing Missouri that it was not profitable to try to hold slaves in his neighborhood?

This historical treatise was written in 1859. It's safe to say you can correctly begin a sentence with 'not to mention' as formally as you'd like here in the States.