Adjective "home-colleged"
Can home-colleged (or self-colleged) be used as an adjective to describe somebody who didn't go to college to study some subjects, but instead studied them himself by self-learning? Would home-colleged imply that somebody else taught him (for example his father, or he hired a professor), and self-colleged would be better?
I understand that these are not standard terms in English and can be used only in an informal manner, but do they convey the meaning I described?
Neither of those terms, 'home-colleged' nor 'self-colleged', can be relied on to convey the meaning 'somebody who didn't go to college to study some subjects, but instead studied them himself by self-learning'.
While the currency of the seemingly parallel term 'home-schooled' might result (in some few and temporally limited circumstances) in those hearing or reading the terms understanding the meaning you'd like to convey, even in those cases there would very likely be resistance to taking that meaning.
Not Taking Your Meaning
The resistance to taking the meaning would result from a discrepancy between the 'known' or guessed intended meaning and the usual or commonly understood meaning of the form 'colleged'. While the OED does list both a verb, 'college', and an adjective, 'colleged', the meanings of those do not correspond with the meaning you'd like to convey.
The verb is unsuitable because it signifies a place, the college, where one is sent or educated.
college, v.
....
nonce-wd.
trans. To send to college; to educate at college.
1819 A. Balfour Campbell I. 27 (Jam.), Now, say that the laddie's colleged, and leecenced to preach, what's he to do till he get a kirk?
1850 T. T. Lynch Memorials Theophilus Trinal xi. 211 How he was born, cradled, schooled..colleged, and the like.
[From "college, v.". OED Online. September 2015. Oxford University Press. http://www.oed.com/view/Entry/36299?rskey=M320cY&result=2&isAdvanced=false (accessed October 03, 2015). Emphasis mine.]
For this verb, your formulation 'home-colleged' is oxymoronic, and your formulation 'self-colleged' suggests the person the phrase is applied to owns or runs a college where they were educated.
The adjective also denotes that the person your formulations are applied to owns or runs a college where they were educated.
colleged, adj.
....
2. Having a college or colleges.
[From "colleged, adj.". OED Online. September 2015. Oxford University Press. http://www.oed.com/view/Entry/36300 (accessed October 03, 2015).]
Not Understanding Your Meaning
'Home-colleged', but neither 'self-colleged' nor any of the variants without the hyphens have what I would call serious currency. A web search didn't turn up any results for 'self-colleged', and the only results for variants without the hyphens were ephemeral personal sites or other insubstantial uses.
The serious currency for 'home-colleged' is scant, and the sense conveyed does not match the meaning you would like to convey. For example, an article in The Chronicle of Higher Education, "Home College: an Idea Whose Time Has Come (Again)" (February 28, 2014, by Hollis Robbins), proposes a scheme involving a highly qualified private tutor or tutors paid by one or more students or their families. This seems to be at odds with the sense you would like to convey which, as you point out, more closely matches what might be conveyed by 'self-colleged'...if that was a viable term.
Other mentions of 'home-colleged' ('home-colleging', etc.) are directly related to the proposal made by Hollis Rollins. The American Interest, in an article titled "Home-schooling for Higher Ed" (2014) picks up the proposal by Rollins and very slightly embellishes it. Marginal Revolution, a self-described "online education platform" promotes Rollins' proposal to a discussion forum. And Rollins has taken what appear to be preliminary steps toward rounding up those "highly qualified" tutors.