Allow only IPSEC outgoing traffic to a port (using IPTABLE?)

Thanks for the suggestions. It turned out to be related to how IPSEC VPN works on iOS (apple) devices.

On iOS devices, on turning the ipsec VPN ON, rule is added automatically in route table to send the IPSEC server/peer traffic directly to gateway to avoid looping (packet getting encrypted again) and rest of the traffic are sent to ipsec tunnel first to get encrypted and then onto ipsec server.

And in my case IPSEC server and the target server(where traffic is targeted) happens to be the same machine (resolves to same IP address), so when the traffic is sent to the target server (port XXXX), instead of its getting encrypted in IPSEC tunnel, it was sent directly to the server due to the special rule without getting encapsulated in ipsec packet. So when these packets arrived on server, it didn't get recognised as ipsec packets and that was the issue.

Since I couldn't avoid iOS client sending the traffic to port XXXX (though without ipsec encapsulation), I had to find a way to let only that device to have access to port XXXX and not to the whole world. So what I ended up doing was to have my own ipsec updown script on ipsec server that adds a INPUT chain rule dynamically to allow traffic from the established client's IP address to port XXXX on vpn connect and remove the rule on vpn disconnect. This way only the VPN connected clients will have access to port XXXX.