"One can do his homework in the library"

In my opinion, using one in this sense is grammatical but awkward. I don't think it is entirely a matter of formality or that the usage has fallen out of practice (although, comparing one can, he can, she can, they can on ngrams does tell an intriguing story). I think the use of one is comparable to the use of the passive voice: both are clear and technically correct, but simply strained.

The best solution is to use a more specific word than one.

A student can do his or her homework in the library.

A touch of awkwardness remains. An easy resolution is to replace his or her with their (if you accept the singular they).

A student can do their homework in the library.

Alternatively, we can just make the subject plural. After all, the library should accommodate multiple students.

Students can do their homework in the library.

These days, one seldom uses the subject "one", meaning the impersonal "one", though one generally still understands when others use it. It sounds stilted and old-fashioned. It's too bad that it has gone out of use. I liked it.