What factors determine whether or not you notice when another player is hacking you?

If you are being hacked, then you always get a chance to respond, but if you are being tailed, you are not.

When you try to hack someone, you have five minutes to find the other player and hack his phone. At that moment, a circle appears on your map that determines the area in which you have to stay to hack. From then on, you have one minute to initiate the actual hack and find some cover. The other player will only be notified once you initiate the hack. So it is possible, that he is already out of the circle and has to come back, but he will always be notified as soon as the download is initiated.

This is not the case when you tail someone. If everything goes smoothly, the other player won't even know you have been there. On Playstation, you can see if someone tailed you by looking at the "Players Met" section under the "Friends" tab in the main menu. I have recently seen two connections marked as "Online Tailing" although I had not started any at that time.

There is a possibility to be notified that you are being invaded (hacking and tailing), though, and that is when you attempt to start something that is not possible while you are being invaded. These things are all types of missions (campaign, phone games etc.). When you attempt to start it while invaded, you will instantly get the mission to find the invader, whether he is tailing or hacking, and even if the hack has not yet started. The same goes for your mentioned experience with trying to open the menu.

My personal experience is that I recently wanted to start a phone game and was instantly notified that I was being tailed and got this message for the first time, although the before mentioned "Players Met" menu showed previous tailings. I don't think there is another way to find out that you are being tailed.

I also had someone drive behind me like a maniac once without getting an invasion message and could identify him as an invader, so if you notice someone behaving very unusual (e.g. sprinting past you), quickly open the profiler to get a reward for spotting him. Because the tailing victim doesn't get noified, he also loses no points for it. I once tailed someone and climbed around in front of him, but because he probably didn't know about the mechanic, the just kept on walking. That is another hint that the tailing victim does not get notified.