Is it possible to install the GNU mt tape drive command in OSX?

macOS does not come with mt - neither the BSD nor the GNU variant.

mt requires your system to have, amongst other things, the magtape I/O-controls. The specification of those controls are made available by the system to developers via a file named mtio.h. That particular file was included with OS X before Snow Leopard, but all versions of OS X and macOS since 2009 has not included that file.

In addition to the magtape ioctls, you'll also need a driver for your actual tape drive. This is the part of the system that will set up a device for you that the magtape ioctls can be used on. This typically shows up as /dev/rstX on Unix-systems. However, such a driver has never shipped with any version of OS X or macOS.

If you want to use a tape driver according to Apple's recommendations, you'll need to use a driver and user software that uses Apple's own SCSI API.

However, third party developers have created a project for macOS to create a device driver that shows up as a /dev/rst0 device that is compatible with mt - alongside the actual mt program. Even though lots of work has been done, it is still listed as not being of "production quality" - nor does it seem to have been worked on for the some years now. You can find the project and download the software here:

https://github.com/jessepeterson/IOSCSITape

You do not list what your intentions are in getting mt working on macOS. In some cases you'll probably be better off by running FreeBSD or Linux on top of macOS using a virtualisation system (such as VMware, Parallels, VirtualBox or similar). Depending on your specific type of tape drive and SCSI interface, it could be possible to use a FreeBSD or Linux mt command along with their drivers for your tape drive.