Solution 1:

I believe either one is actually correct, since the thing that determines the verb's case is the noun that comes first in the predicate nominative expression (on the left side of the imaginary equals sign). In this sentence, that first noun is what, which is technically a pronoun, but stands in for the noun that comes later. But of course, at this point in the sentence, it has not yet been determined whether the predicate noun that what is referring to is singular or plural, so the verb is essentially given the benefit of the doubt and is allowed to take either case, regardless of what the predicate noun turns out to be. This flexibility really only arises out of the fact that what is naturally ambiguous in number. If the sentence had begun The things he is looking for..., the predicate would have had to have been are books. Similarly, if the sentence had begun The thing he is looking for..., the predicate would have been singular - is books.

Solution 2:

The first is grammatically correct since we'll want are to agree with books:

What he is looking for are books written by Jane Austin.

And this might be a bit awkward to say, so we can restructure it to something like the following, which makes the verb agreement a bit more obvious, I think:

The books, written by Jane Austin, are what he's looking for.

Although, we do end with a preposition in this case :-) That said, in English, it's common to end informal (verbal) sentences with a preposition, otherwise one might sound too formal for the occasion!

UPDATE

User sumelic found the article below which describes the issue at the heart of OP's question.

http://www.grammarphobia.com/blog/2012/06/subject-complement.html

To quote:

There are many good discussions of this problem. One of the more succinct can be found in The American Heritage Dictionary of the English Language (5th ed.):

“When the what in the what-clause is the object of the verb and the complement of the main clause is singular, the main verb is always singular: What they wanted was a home of their own.”

The usage note continues: “When the complement of the main sentence is plural, the verb is most often plural: What American education needs are smaller classes.”

Solution 3:

I'm a bit out of my comfort zone on this one, but I believe the sample sentence is an example of an inverse copular construction.

In the inverse copular constructions, the copula agrees with the singular predicative expression to its left as opposed to with the plural subject to its right. Interestingly, this phenomenon seems to be limited to English (and possibly French); it does not occur in related languages such as German, e.g.

however, the reverse is not necessarily true

Inverse copular constructions where the inverted predicative expression is a noun phrase are noteworthy in part because subject-verb agreement can (at least in English) be established with the pre-verb predicative NP as opposed to with the post-verb subject NP, e.g.
a. The pictures are a problem. - Canonical word order, standard subject-verb agreement
b. A problem is/??are the pictures. - Inverse copular construction, subject-verb agreement reversed in a sense

from wikipedia. https://en.wikipedia.org/wiki/Inverse_copular_constructions

Solution 4:

What he is looking for is...

Verbs need to agree with their subjects. books is not the subject—nor is it an object. It's a predicate nominative. he is also not the subject of the sentence. After all, he is not books.

The subject of the sentence is the noun phrase what he is looking for, which has a head noun of what. Personally, I would treat this as singular, as would Daily Writing Tips.