How can I replace my on-premise server with something on Azure?

I have an on-premise server that's aging. It's almost purely used as a file server but requires proprietary software for some of the file sharing capabilities. I was looking for a solution available on Azure that I could use. I was thinking if I could spin up a virtual machine that used that latest edition of Windows Server then I could install my proprietary software.

Question then is how would I tell my workstations how to reach that VM to access that file share?

I do not use active directory services in my on-premise environment.

You need to create a VPN between your local site (where the clients are that need access to the file shares) and the virtual network your file server is in.

You will then need to open up SMB ports in between the sites.

Here is some reading on VPNs to Azure: https://docs.microsoft.com/en-us/microsoft-365/enterprise/connect-an-on-premises-network-to-a-microsoft-azure-virtual-network?view=o365-worldwide

Azure can do file shares as a service. Standard SMB though, does not use this proprietary whatever.

Yes you can run whatever you like on compute instances in Azure. Put the IP address of the instance, or a load balancer in front of it, in DNS. Clients use the DNS name. When ready to cut over, make the old server a CNAME to the new server.

This application may not be safe to run over the internet. Extend your organization's network to Azure with VPN or ExpressRoute peering.