Dimensionality in quantum mechanics
To be quite frank, while the textbook by Griffiths may be a decent introduction to quantum mechanics, you should not rely on it for anything mathematics related.
That hermitian matrices can be orthogonally diagonalized is not a 'rather technical fact'. It is a completely standard result (known to both mathematicians and non-mathematicians), which is covered in any introductory linear algebra course. It finds application in a wide range of subjects, not just quantum mechanics.
There is a vast mathematical literature devoted to making mathematical sense of quantum mechanical concepts. Saying that this massive field, which lies in the intersection between physics and mathematics, is simply concerned with hermitian matrices, and then stating that the results do not carry over to the infinite dimensional setting... I don't know what to say. At the very least, it is an extreme misrepresentation of the actual state of affairs.
When should you expect to have a complete set of eigenvectors of a Hamiltonian? Usually, this happens if you consider a confining potential. Examples include the particle in a box, like you mentioned, and the harmonic oscillator.
When should you not expect to have a complete set of eigenvectors? Usually, this happens if your system can 'break apart', becoming essentially a non-interacting free system. Examples include the free particle.
When should you expect to have eigenvectors, but not a complete set? If your system has bound states, but it is also possible to break it apart. Examples include the hydrogen atom, with bound states below the ionization threshold.
Since you mention chemistry, it is in order to mention that for many applications, in practice one often considers only the space spanned by the bound states. In this subspace, the eigenvectors do of course constitute a complete set. This is what is usually done in introductory quantum mechanics courses, where one calculates the energy levels of the hydrogen atom. Often, it is not even mentioned that the spectrum of the hydrogen atom contains the positive real line along with these negative eigenvalues! On the other hand, this fact is not really important if you only care about the bound states.
If you are interested in the mathematics of quantum mechanics, one of the key topics is spectral theory of unbounded (self-adjoint) operators. The classics 'Perturbation Theory for Linear Operators' by Kato and 'Methods of Modern Mathematical Physics' by Reed and Simon are recommendable. For more recent texts, I would also recommend 'Unbounded Self-adjoint Operators in Hilbert Space' by Schmüdgen and 'Quantum Mathematical Physics' by Thirring. Of course, all of these texts require a firm background in mathematics, with a focus on functional analysis.
Finally there are a bunch of questions on this site which are related to your question:
When Schrodinger operator has discrete spectrum?
why is the spectrum of the schrödinger operator discrete?
Spectrum of Laplace operator with potential acting on $L^2(\mathbb R)$ is discrete
When the point spectrum is discrete?
Proving Compactness of Resolvent Of an Operator
Operators with compact resolvent
Selfadjointness of the Dirac operator on the infinite-dimensional Hilbert space
Hermitian and self-adjoint operators on infinite-dimensional Hilbert spaces
Physicists persist in writing things that do not make sense mathematically, but there is a mathematically rigorous version of it: the Spectral Theorem for densely defined self-adjoint operators on Hilbert space. Most functional analysis texts will cover it.