Is 'rest' singular or plural?

You are right. Both singular and plural agreement are found. The Corpus of Contemporary American English has 469 records for the rest are and 88 for the rest is. The figures in the British National Corpus are 92 and 153. The Oxford English Dictionary has 137 citations for the rest are, and 97 for the rest is. There are even cases where the singular is used where others might prefer the plural. A citation from 1990 is ‘There are competent performances from David Garrison and Becky Ann Baker as his best friends, but the rest of the cast is amateur night’, but it is possible to regard cast as plural, and, at least in the UK, it might appear as the rest of the cast are. (Amateur night here simply means ‘amateurish’.)

As for your examples, the rest are three and the rest is less than one bottle are unlikely to occur. In those circumstances a native speaker would say There are three left and There is less than a full bottle left. However, in the case of the boxes, we might say The rest are OK, and in the case of the bottles we might say The rest is not enough to fill even a single bottle.

The examples used here all seem to compare nouns which can easily determine whether they are mass or count nouns. I would think that in the case of the boxes, since a box is a count noun, one would say the rest "are." In the case of the salt, since salt is a mass noun, one would make it singular and say the the rest "is."