Is it OK to omit 'that' after 'however, in the middle of a sentence?

I know it is generally okay to omit 'that' which introduces a sentence, but I am not sure if it is OK to omit 'that' right after 'however' which is put in the middle of a sentence as follows.

I have to admit, however, that I was wrong.
I have to admit, however, I was wrong.

Is the second sentence okay without 'that'? At first, I thought it would be all right, but on second thought, it is not correct if I drop it because 'that' comes after a comma.

'That' right after a comma should not be omitted, should it?

I don't think the problem is with however. These rules will make my point clear.

You can also use however near the beginning of a sentence to mean ‘but’, ‘nevertheless’ or ‘regardless of the fact’. It is often used this way for emphasis.

The 2010–2011 Federal Budget was no fiscal revolution. It did, however, mark the first ‘real’ step towards tax reform following the Henry Review.

When you use however in the middle of a sentence to separate two clauses, it is usually separated with a semicolon and a comma (... ;however,…). Many modern writers use a comma instead of the semicolon, but the semicolon is still regarded as more correct.

The 2010–2011 Federal Budget was no fiscal revolution; however, it did mark the first ‘real’ step towards tax reform following the Henry Review.

You can avoid the semicolon if you replace however with but or yet.

The 2010–2011 Federal Budget was no fiscal revolution, but it did mark the first ‘real’ step towards tax reform following the Henry Review.

However, when used to mean ‘no matter how’ or ‘in whatever way’ doesn’t need commas.

I want you to finish your report today however busy you are.

Though you've used however in the middle of your sentence, it still means 'nevertheless' or 'in spite of a previously stated fact'.

Look at this: I have to admit that I was wrong.

By adding 'however' in the middle of the sentence, you're just adding an additional detail, that's all. This addition does not affect the 'I have to admit that...' structure.

I have to admit I was wrong. This is also right because of the reasons listed here. Now, even if you add 'however', the original structure remains the same.

The conclusion is that both the sentences are right.