How to mute a FaceTime iPhone call on OS X through a shortcut?

Yes you can! Here are the steps:

1. Open Automator and create an Automator Service, with a single Run AppleScript action, containing the following code:

   High Sierra and later:

   tell application "System Events" to tell process "Notification Center"
       if checkbox "Mute" of window 1 exists then
           click checkbox "Mute" of window 1
       end if
   end tell
   

   Sierra and earlier:

   tell application "System Events"
       if checkbox "Mute" of window 1 of application process "FaceTime" exists then
           click checkbox "Mute" of window 1 of application process "FaceTime"
       end if
   end tell
   

2. Set the service to receive "no input" and to work in "any application", save it with a name of your choosing.
3. Now, go into the Shortcuts tab in the Keyboard preferences in System Preferences, and click on Services. The newly created service should be at the bottom, under the name you chose. Click on the service, then click on "add shortcut" to its right, and specify a shortcut to your liking.
4. You may have to grant Automator and/or FaceTime access in Accessibility in the Privacy tab of the Security and Privacy preferences.

That's it! You should now have a service set up, tied to the shortcut you chose that can be accessed from anywhere.