Can I ssh somewhere, run some commands, and then leave myself a prompt?

Probably the simplest thing is:

$ ssh -t host 'cmd1; cmd2; sh -i'

If you want to set variables, do:

$ ssh -t host 'cmd1; cmd2; FOO=hello sh -i'

Note that this is a terrible hack, and you would be much better off putting your desired initial commands in a script and doing:

$ scp setup host:~
$ ssh host
host$ . setup

You could also use the following expect script:

#!/usr/bin/expect -f
spawn ssh $argv
send "export V=hello\n"
send "export W=world\n"
send "echo \$V \$W\n"
interact

Turns out this is answered by this question:

How can I ssh directly to a particular directory?

to ssh:

ssh -t anvil "export V=hello; export W=world; bash"

followed by:

jla@anvil$ echo $V $W
hello world