How to make my own custom dialer in an Android phone

Create an app that responds to Intent.ACTION_DIAL. In the AndroidManifest.xml you need to add the following to that Activity:

<intent-filter>
    <action android:name="android.intent.action.DIAL" />
    <category android:name="android.intent.category.DEFAULT" />
</intent-filter>

You can take the official phone app as reference. But be warned, it is no trivial task.

You can replace only the Dialer that way. The actual calling thing (what you see during calls) can't be changed.

For more information, see Stack Overflow question Android dialer application.

If you want to totally replace the existing phone dialler and manage the call from within your application then the answer is that this can't be done except on custom ROM you build after changing the android source code replacing the default dialler with your own.

You can place an overlay over the actual phone call when it shows up. There are loads of phone skin apps like this