use rm to remove files and directories recursively

To directly answer your question, "no - you can't do what you describe with rm".

You can, however, do it you combine it with find. Here's one of many ways you could do that:

 # search for everything in this tree, search for the file pattern, pipe to rm
 find . | grep <pattern> | xargs rm

For example, if you want to nuke all *~ files, you could so this:

 # the $ anchors the grep search to the last character on the line
 find . -type f | grep '~'$ | xargs rm

To expand from a comment*:

 # this will handle spaces of funky characters in file names
 find -type f -name '*~' -print0 | xargs -0 rm

"without using other commands"

No.

Using Bash, with globstar set, yes:

rm basedir/**/my*pattern*

Try it with e.g. ls -1 first, before rm to list the files you match.

You set options through e.g. shopt -s globstar.

Alternatively, a shorter find variant:

find -type f -name 'my*pattern*' -delete

or for GNU find:

find -type f -name 'my*pattern*' -exec rm {} +

or another alternative for non-GNU find (a bit slower):

find -type f -name 'my*pattern*' -exec rm {} \;

To also remove directories, as you ask for: just change rm into rm -r in the above commands and skip matching on only -type f in the find commands.