How to set multiple relayhost in postfix?

Multiple destinations are supported in Postfix 3.5 and later.

relayhost = mail1.example.com:587, mail2.example.com:587
relayhost = [mail1.example.com]:587, [mail2.example.com]:587

Postfix doesn't allow defining multiple entries for relayhosts for redundancy. Have a look at the page you have referenced:

In the case of SMTP, specify a domain name, hostname, hostname:port, [hostname]:port, [hostaddress] or [hostaddress]:port. The form [hostname] turns off MX lookups.

As you can see, there is no way to define multiple addresses/hostnames here.

if I give relayhost a made-up hostname that have multiple mx records, will it try all?

I guess so, but if it requires authentication, then it may not work with the secondary server anyway and fail.

If I give relayhost a made-up [hostname] that have multiple a records, how will it handle it?

It will use the first address (A record) returned by the dns in round robin method. But this also doesn't imply that it will try another one in case the one returned by the dns server fails.

Having said that, you may have a look at this postfix option: smtp_fallback_relay

I would recommend to go with DNS.

But you don't need to set multiple MX records - you need to resolve the given hostname with multiple A-Records (host should resolve to a given set ip addresses).

Having multiple hosts to handle traffic is quite common for mail. This is done using multiple mx records. So it's just straight forward when you'll use the same setup when sending from your own host/relay.