Is there a word/phrase that can describe playing a musical instrument in a casual way?

Such a performer would be noodling. Merriam-Webster for "noodle":

intransitive verb

: to improvise on an instrument in an informal or desultory manner

He was just noodling around on the guitar.

Noodling is especially casual, and often entails breaking form in some way (not holding a guitar right, not practicing common chords or progressions on a piano). Some musicians deride it as a desultory practice, but others emphasize that it is beneficial for creativity.

Examples:

We're sitting with Hots Michaels. If you want to noodle at the piano you can, fine. ("'Working' Then And Now: A Hotel Piano Player Frustrated By His Future." NPR, 30 Sept 2016).

It was, like, pick up and noodle on a guitar. A cool thing that I get to do live with Fall Out Boy is embellish. ("Joe Trohman on the Damned Things’ Return and Fall Out Boy’s Future." Rolling Stone, 16 May 2019.)

The first word to come to my mind was dabble.

Dabble : to work or involve oneself superficially or intermittently especially in a secondary activity or interest

Merriam Webster

Although this can be used for any hobby or activity, not just music.

The word you are looking for may be plunk

Plunk means to play a musical instrument without interest or enthusiasm

It also means to play a musical instument not well but to play often loudly

• I really enjoy plunking away on the guitar.

(Cambridge Dictionary)

See also:

https://www.macmillandictionary.com/thesaurus-category/british/playing-performing-and-arranging-music

https://dictionary.cambridge.org/us/dictionary/english/plunk

The verb tootle springs to mind, if a whimisical and non-serious footling about on an instrument is what you’re after. Here is the relevant section from Collins Cobuild English Dictionary accessed 22/11/2019, where we are interested in definition 2:

In my experience, this verb is used most frequently with wind instruments, where the word has a kind of onomatopoeic effect.

In addition to the generic terms, there may also be useful ones relative to specific instruments, e.g.:

• He strummed the guitar casually
• She tickled the (piano) keys idly
• They rattled the maracas briefly

for some instruments this may seem inappropriate unless you are deliberately attempting to create a striking or paradoxical phrase:

• Edgar squawked out a few trumpet blasts on the way to the break room
• Allen gave an absentminded thwack to the gong as he sat down
• Poe's drumsticks unleashed a brief cavalcade before he stood up