"his mother look very tired"

It doesn't sound good to me.

A very important issue that has to be kept in mind: there are several possible grammatical structures with the verb "see." I'll discuss three of them. I'm a little rusty on my terminology, but I'll try to describe them as best as I can. If you tell me what grammatical terms and concepts you are familiar with, I can do a better job of tailoring this answer to your question.

With a that-clause (which has its own subject and finite verb)

It seems the test-makers intended for you to analyse the example sentence with a "that"-clause where the "that" is omitted:

James could see [that [his mother __ very tired from all the chores]].

In a structure like this, "his mother" is actually the subject of an embedded clause that has the same structure as an independent sentence would. In other words, the verb form used here has to one that can work in the stand-alone sentence

His mother __ very tired from all the chores.

So you cannot use an infinitive. (With this overall structure.)

This leaves the three finite forms: 1) looks 3) looked 4) had looked. The best choice out of these is "looked" because of the context of "could see".

By the way, this is also the way I would analyse the structure of "I see [that] you look tired." Here, "look" is not the infinitive; it is a finite verb in the present tense that lacks an -s suffix only because the subject (of the verb "look") is the second-person pronoun "you." The reason I say this is the following evidence: if we switch the second-person pronoun to the third-person, I would say "I see [that] he looks tired."

I'll also list example sentences with a pronoun in place of "her mother," which might help because the third-person singular feminine pronoun has distinct forms for subject and object:

James could see she looked very tired from all the chores.

With an object and a participle

James could see his mother looking very tired from all the chores.

James could see her looking very tired from all the chores.

In this sentence, "his mother" is an object and "looking" is a participle. That structure would be grammatical. But as you note, it is not given as an option.

With an object and an infinitive

A third grammatical structure would call for the infinitive "look." (I didn't realize this until after I first posted my answer.) The verb "see" can sometimes be followed by an object and an infinitive, in sentences like "I could see him run." But, that structure doesn't seem right to me here for some reason. Sorry, I guess this part is actually the core of your question!

It does seem to me that the same awkwardness is present when using other copula-like verbs:

James could see she was very tired from all the chores.

is much better than

?James could see her be very tired from all the chores.

I'll try to add more about this. I hope what I have posted so far helps you to avoid confusing these grammatically different uses of the verb "see."