PowerShell remoting works with local IPs but not with the public IP on an Azure VM

Host is a single Windows Server 2019 running as Azure VM, no domain, no AD.

To isolate the issue, I am trying to create New-PSSession to the host doing it locally, using its local PowerShell. My command works for host's local IPs 10.0.0.4 and 127.0.0.1 but not with host's public IP. The host's public IP is via Azure, so it is not known by host directly as an interface IP

My script what I used to enable PowerShell remoting on the host are 2 years old, and those enabling methods worked successfully, but not now on this new host. I did all diagnostics what I've remembered and also googled the current common knowledge.

I am using password on the clipboard in both working and not working cases to rule out mistyping

My command which is working. Note I am establishing a remote PS session to the host itself, just for diagnostics reasons: (also with 127.0.0.1)

New-PSSession -ComputerName 10.0.0.4 -UseSSL -Credential myadminname -SessionOption $sessionOptions

My command which does not work is exactly the same, except using the public IP, note I am trying to establish remote PS session to the host itself, just for diagnostics reasons:

New-PSSession -ComputerName 137.xxx.xxx.xxx -UseSSL -Credential myadminname -SessionOption $sessionOptions

New-PSSession : [137.xxx.xxx.xxx] Connecting to remote server 137.xxx.xxx.xxx failed with the following error message : Access is denied.

What I did as diagnostics:

Checked the port is open with portqry -n 137.xxx.xxx.xxx -e 5986 It says :

TCP port 5986 (unknown service): LISTENING

Maybe I am wrong, but based on this I ruled out the connectivity issues, both host's firewall, both Azure network security group rules are configured correctly. I stopped the WinRM service, and checked there was nothing listening then, then started and was listener, so it is definitely the running WinRM service who is listening.

I've also ran Test-NetConnection -ComputerName 137.xxx.xxx.xxx -Port 5986 from a remote client computer and it gave positive result:

ComputerName     : 137.117.142.253
RemoteAddress    : 137.117.142.253
RemotePort       : 5986
InterfaceAlias   : Ethernet 2
SourceAddress    : 192.168.1.1
TcpTestSucceeded : True

I've ran winrm e winrm/config/listener:

Listener Address = * Transport = HTTPS Port = 5986 Hostname Enabled = true URLPrefix = wsman CertificateThumbprint = 4F89xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ListeningOn = 10.0.0.4, 127.0.0.1, ::1, fe80::xxxx:xxxx:xxx:xxxx%3

I've also ran winrm get winrm/config:

Config
MaxEnvelopeSizekb = 500
MaxTimeoutms = 60000
MaxBatchItems = 32000
MaxProviderRequests = 4294967295
Client
    NetworkDelayms = 5000
    URLPrefix = wsman
    AllowUnencrypted = false
    Auth
        Basic = true
        Digest = true
        Kerberos = true
        Negotiate = true
        Certificate = true
        CredSSP = false
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    TrustedHosts = *
Service
    RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
    MaxConcurrentOperations = 4294967295
    MaxConcurrentOperationsPerUser = 1500
    EnumerationTimeoutms = 240000
    MaxConnections = 300
    MaxPacketRetrievalTimeSeconds = 120
    AllowUnencrypted = false
    Auth
        Basic = false
        Kerberos = true
        Negotiate = true
        Certificate = false
        CredSSP = false
        CbtHardeningLevel = Relaxed
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    IPv4Filter = *
    IPv6Filter = *
    EnableCompatibilityHttpListener = false
    EnableCompatibilityHttpsListener = false
    CertificateThumbprint
    AllowRemoteAccess = true
Winrs
    AllowRemoteShellAccess = true
    IdleTimeout = 7200000
    MaxConcurrentUsers = 2147483647
    MaxShellRunTime = 2147483647
    MaxProcessesPerShell = 2147483647
    MaxMemoryPerShellMB = 2147483647
    MaxShellsPerUser = 2147483647

Ok, trying to summarize here.

If you get no connection at all when trying to connect from a remote machine, then it's definitely a networking and/or firewall issue. Please note that being able to connect to the machine from itself doesn't necessarily mean you'll be able to do the same from the outside, because the connection can be blocked anywhere in the path from the client to the server.
(Also, I suggest using Test-NetConnection for this.)
If you get an access denied error when trying to connect to the machine from itself, you are most likely running into this: https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/accessing-server-locally-with-fqdn-cname-alias-denied; and yes, the behavior can vary wildly depending on the host name or IP address you are using for the connection.