Safari does not restore saved tabs when I exit via X icon
Solution 1:
Closing the window is not the same as quitting the app.
Clicking the red X closes the window, telling Safari that you've done with the contents - leaving nothing for it to remember.
It's not clear from your question quite what you are selecting from the right click menu, but anything other than quit - or quitting after all windows are already closed, will do the same.
Note that, if you do forget & close the windows before quitting, that you can use History menu > restore all windows from last session to get them back again.