How to properly say that a given day/date does not exist?

"The specified date is invalid." is not only valid, but quite common.

If you want to avoid "sounding like a robot", then you could take the invalid date in question and produce a sentence like:

There is no 31^st of April.

There is no 29^th of February in 2013.

"The specified date is invalid" will be perfectly well understood by essentially everyone who encounters it as a website error message.

You'd probably get good understanding with the unusual but brief and informative

• No such day in calendar

If you wanted to be complete and/or pedantic you could say

• xxx is not a valid date

where xxx is the user input data.

This carries a very slightly stronger suggestion that what is wrong is the formatting or genuine-date nature of the entry.
Without this the entry could be a properly formatted date entry and refer to a real-world day but still be "invalid". For example,

Departure date ? : 11/11/2013 Return date ? : 12/12/2012 -> The specified date is invalid (Because return is before departure)

Date of birth ? : 11/11/1812 -> The specific date is invalid (Because not even I am that old)

Even the above is not an absolute statement about date formatting or real-world date being invalid, but making this certain would require a much more complex and pedantic sentence, which is not justified or useful in the context of an error message.