AWS EC2 public IP unreachable over IPSec VPN but private IP still reachable

We have a FortiGate (FortiOS 6.0) connected to a IPSec Site-to-Site VPN (Dynamic BGP) on AWS. When the VPN is up, we can only access EC2 using the private IP, public IP no longer works. Only the public IP in the VPC linked to the VPN are unreachable, the public IP of others AWS account remains reachable.

How do we make both IPs reachable, even when connected to the VPN?

For the testing purpose i created an "allow all traffic from any ip" security group to make sure i wasn't blocked by that.

That is a limitation of AWS VPNs. They only allow traffic to/from private IPs belonging to the VPC the VPN is deployed in.