Can you break out of jail and not get attacked?

Solution 1:

If you escape from jail, your bounty will remain, plus you risk an additional 100 gold bounty if you're caught while trying to get out.

Solution 2:

When you perform a jailbreak a bounty of 100 gold is added to your current bounty - this happens regardless of getting caught. After completing the jailbreak guards will not interrogate or attack you, only saying "Wait, I know you!".

There was conflicting information about this online so I tested it without Mods on a fresh character. I stole a knife in Riverwood (9 gold bounty), ran to Whiterun and talked to a guard, got sent to prison, escaped, grabbed items on way out, ran around town jumping in front of guards with 109 gold bounty.

To remove the bounty you'll need to pay it. Or,

If you are part of the thieves guild and have enough reputation in the specific hold there will be a bribe a guard option to pay the bounty at a discount. If you use this bounty payment option you can immediately steal back the bounty from the guard (so I've read online).

There's also wikia guide on how to escape each of the jails.