Access azure storage account from pipeline agent in same region with access restrictions enabled

The solution is to use private endpoints. I am dealing with the same issue and after considerable research, I found that private endpoints will facilitate access over internal IPs between the remote vnet where your agents are located and the vnet where your storage is located. I have tested this and provided details below.

After testing I found that the way this works is by creating a Private DNS in the storage vnet and setting up a vnet DNS link that allows the VMs in the remote vnet where the agents sit to resolve the storage account connection to a private IP instead of a public IP. Additionally a NIC in the remote vnet is created which provides a route to the private IPs of the storage.

So that NIC sits in the same vnet as the agents providing a route for private IP connections, and a DNS link exists to resolve storage to private IPs, and it all just works. The agents will rely on the private DNS rather than public DNS to resolve the storage hostname and so the agents will communicate with azure properly via private IPs.

Edit: I have setup a private endpoint and confirmed that it works as expected. There are some caveats to doing so with terraform which I've outlined in a related azurerm provider GitHub issue:

https://github.com/hashicorp/terraform-provider-azurerm/issues/2977#issuecomment-1011183736