"high-reliable", "highly reliable", or something else?

Solution 1:

It should be highly reliable.

Reliable is an adjective which is modified by highly. Adverbs ending in -ly do not take hyphens.

highly [adverb indicating the level of reliability]
reliable [adjective describing system architecture]

You would use a hyphen if speaking of a high-maintenance system architecture:

high [adjective describing maintenance]
maintenance [noun]
high-maintenance [compound adjective describing system architecture]

Without the hyphen or -ly ending you would have a high meaning tall/up in the air.