"Myself" vs. "by myself"

Solution 1:

"I can't do it" - I am not capable of doing it.

"I can't do it myself" - again I am not capable of doing it, but the myself adds emphasis on the fact that I am talking about myself, hence highlighting the possibility that someone else could do it.

"I can't do it by myself" - I couldn't do it alone.

Literally "by myself" would mean "with myself", using a sense of by that we don't really have any more. This is illogical considered on its own, but if you think of it as an answer to the question "who was you with?" then it's easy enough to see how it would be sensible to say "I was with me" and not intend it to be taken literally. These days it would be a very well-known idiom to the point where we don't even think about how those two words come to that meaning, we just think of "by oneself" meaning oneself without company.

Solution 2:

The first phrase you offer "I can't do it myself." seems redundant and would probably be better phrased simply as "I can't do it."

As to meaning,

"I can't do it (myself)." means that the speaker cannot perform the action irrespective of circumstance.

"I can't do it by myself." means that the speaker likely could perform the action, but only if s/he were assisted.