PC uses speaker of headset instead of microphone

I need to record voice on my old desktop PC (Asus P5B-SE - Windows 7). There are three lines on back panel: two with speaker icon and one with microphone icon. Also there are two lines on front panel (one speaker and one microphone) which seems not working.

I connected a smart phone headset to microphone line. Realtek software detected it as microphone. However, it doesn't use microphone of headset; instead, it uses its speakers as microphone! I have to shout in a really near distance (less than 5cm) to record a weak sound.

On my laptop (Acer Aspire E1-572G), when I connect headset, I have three options in Realtek software and when if select headset, it uses both microphone and speaker of headset.

Is it an old incompatible hardware problem? Isn't there any software solution? I've heard about some adapters which separate microphone and speaker lines.

Yes, you need an adaptor.

These are the three main types of connector for mic/speakers/headset

^{Image from CableChick - Understanding TRRS & Audio Jacks}

These are generally referred to by the number of 'stripes'.
Left is Tip, Sleeve - TS - usually used for a mono mic
Centre is Tip, Ring, Sleeve - TRS - used for regular stereo headphones
Right is Tip, Ring, Ring, Sleeve - TRRS - used for a combination headset/mic.

Your laptop uses a single TRRS socket - headphones left/right, mic & ground.

Your desktop has two sockets
TRS for speakers/headphones - left/right, ground
TS for mic - mono, ground [These are sometimes wired with another TRS, but the functionality is the same to the end user]

When you plug your TRRS headset into the desktop's mic input, it's going to be fairly random as to which of the rings is going to connect - in your case, i'ts one of the speakers.

Just search for "headset splitter" - input will be TRRS, output often both TRS, but this should work.