As originally used, the phrase "mentally check out" meant "go over in one's mind [some task, problem, or procedure], as if reviewing a checklist." Thus, for example, from an unidentified article in Hospital Management, volume 101 (1966) [combined snippets]:

Let us now review these together and mentally check out our hospital pharmacy score board.

  1. There shall be a pharmacy directed by a professionally competent licensed pharmacist or a drug room under adequate supervision which means a part-time or consultant pharmacist, in my opinion.

  2. There shall be policies established to control the administration of dangerous drugs with specific reference to the direction of the order and dosage ...

And from Paul Erdman, The Billion Dollar Sure Thing (1973) [combined snippets]:

Dr. Walter Hofer's phone call was, of course, totally superfluous, but it gave him the necessary time to once again mentally check out his calculations. His male secretary had already briefed him well on the situation, following Sir Robert's phone call a few days earlier when the items on the agenda had been mentioned.

As these two examples illustrate, this version of "check out" uses the verb phrase transitively: the person is mentally checking out something else. But by the mid-1970s, the expression was also being used intransitively, on the model of someone checking himself or herself out of a hotel room or a hospital—that is, settling or signing off on the bill and leaving. From an unidentified article in Foreign Policy, issues 9–12 (1973) [combined snippets] (also posted in the Congressional Record of June 25, 1974):

In the multitude of private decisions now being made about career preferences by Americans in their twenties, the steaming domestic cauldron of urban challenge, the rebuilding of our local, state and regional institutions to brace for the twenty-first century—these crises of domestic democracy—can excite publicly-oriented young men and women and pull them away from the world scene into the domestic one. Partly for that reason and in contrast to it, there may be an abnormal proportion of other publicly-oriented Americans who may now seek refuge in a flight into foreign policy—a flight from the tempestuous egalitarianism of the American scene to the more comfortable inequalities of world affairs. Unquestionably competent, talented, often energetic, they mentally check out of our domestic crises and bring along with them an underlying set of non-democratic, even antidemocratic, attitudes precisely at the time when American foreign policy needs such attributes the least.

And from Rändi Smith, Written Communication for Data Processing (1976):

What would you say is the worst possible day? You're right if you think Friday. T.G.I.F.—a cliche, but a true one.I don't know whether people have already mentally checked out for the weekend or what the problem is, but the largest number of negative responses was evoked on Friday. How about the second worst day? Not surprisingly, it's Monday.

From these examples, it appears that "mentally check out" has been in use as a metaphorical sense for at least 45 years. A similar notion is expressed in the phrase "your mind is on vacation"—which Mose Allison used in the title and lyrics to a memorable song back in 1976:

Because your mind is on vacation and your mouth is working overtime.