IPv6 support makes openconnect server vpn connection too slow

ULA is not for routing to the internet. It will get dropped before leaving your network resulting in a bad user experience.

Remove the ULA net from ocserv configuration and replace it with one or more /64 from your address plan. These subnets are only for the VPN. For example:

ipv6-network = 2001:db8:3025:1407::/64

On VPN clients with a web browser, verify with a dual stack test such as http://test-ipv6.com/

You have the right idea about implementing IPv6. IPv6 awareness is necessary to secure your networks. Some networks require IPv6. An IPv6 only stack without NAT is a simpler design.