How mac detects that personal hotspot is available without internet?

Apple devices talk to each other and listen without needing the internet or a server or a network.

Look up Rendezvous and Bonjour and mDNS as they are the marketing and technical terms of the first iteration of this capability. More recent devices have enhanced capabilities including NFC and Bluetooth range estimation - look up Bluetooth Beacons for that iteration of this feature.

Lastly, the most advanced tech that Apple ships is ultra wide band radio chipsets that augment the previous technologies with time of flight measurements to answer hypothetical situations like “Is your watch closer to the HomePod than your phone?” for instance.

The hardware and software are capable of mapping out space and time to figure out where they are in the world as well as how close to other devices they may be moment to moment. Apple then synthesizes a reaction based on multiple signals to let you know a hotspot may be near. This can be as simple as hearing a WiFi beacon indicating a device provides this service or a cryptographically signed two way handshake to validate your AppleID signed in recently on two devices and they can trust the identity and validity of a handshake data exchange.

This feature is marketed under the name "Instant Hotspot" as part of the Continuity framework and is implemented very using Bluetooth Low Energy (BLE).

The implementation differ slightly depending on whether or not the user is part of "Family Sharing", but the end result is the same. Various forms of encryption are used to ensure security, while being able to determine that nearby devices are signed in to the same iCloud account as the current device. Then it can send a BLE message to the iPhone to request Personal Hotspot be turned on, and then it can connect via WiFi as usual.

Bluetooth LE has a limited physical range, so only devices near by the user can "hear" these messages, and only those are able to respond to them.

If you want to know how this is implemented, I can recommend the independent article "Handoff All Your Privacy - A Review of Apple's Bluetooth Low Energy Continuity Protocol".