Why do physicists get away with thinking of the Dirac Delta functional as a function?

For instance they use it for finding solutions to things like Poisson's Equation, i.e. the method of Green's functions.

Moreover in Quantum Mechanics, it's common practise to think of the delta functions $\delta_x$ as being a sort of standard basis for the vector space of square integrable functions but $\delta$ is obviously not a square integrable function itself so how can it form a basis for something that it's not a even an element of.

(On a slightly separate issue, it also doesn't make sense to me why you can think of the functions $e^{ikx}$ (Fourier Basis) as a basis square integrable functions since $e^{ikx}$ is also not square integrable. )

Are physicists just really lucky that these things work out or is there some deeper underlying reason why it's okay to think in these terms? I've come across a lot of great books by a lot of great physicists who've simply assumed this to be the case. It's hard to believe that they were all naive enough to dismiss the mathematical fallacy in their arguments. Clearly I 'm missing something. My question is what is it?

An elaborate answer would be much appreciated.

Almost always, when dealing with Physics and Engineering, the answer to "How do they get away with doing that?" is simple: They get correct answers.

The delta function really started with Heaviside, who is considered to be the Father of modern Electrical Engineering. His version of the delta function was the derivative of the step function. The step function described what happened when you flipped a switch. Heaviside developed abstract methods of solving differential equations of Engineering by consider evolution operators before this was a well-formed idea in Mathematics, recognizing that evolution operators associated with time invariant systems such as circuits had an exponential property. Both the delta function as well as the Laplace transform directly evolved out of Heaviside's methods. (It appears that Mathematicians named the transform after Laplace, however, because Heaviside was a prick, and an integral resembling the transform had been looked at by Laplace.) Mathematicians were very troubled by his methods, especially when it came to expanding operators in a binomial series and applying that to an impulse function. Most people at the time would have dismissed Heaviside as a crackpot were it not for one important fact: he got correct answers, and he verified those answers. When Mathematicians would try to insist that Heaviside be more rigorous, he told them that it was "their job" to clean it up, which didn't make the Mathematicians any friendlier.

It appears that Dirac picked up the formalism from Heaviside's work that came several decades before Dirac's work. He too got correct answers. Mathematicians eventually came up with a rigorous formalism (actually more than one) for studying the delta function. But this work is not simple. However, it turned out to be incredibly useful on a theoretical level when it came to studying PDEs, especially elliptic operators. So Mathematicians did their job, and much (but not all) that the Physicists do with $\delta$ makes sense in that advanced context.

Are Physicists lucky that everything works out? Not really. There was great intuition behind the methods leading to answers, and people checked their answers. You can well imagine that many wrong ideas were discarded along the way, "wrong" meaning they gave wrong answers.

Why do Physicists teach an intuitive approach using $\delta$ functions? The short answer is this: there's not enough time for Physicists to become Mathematicians before they start Physics. And who would tolerate many years of tedious Mathematics before being allowed to start Physics? Life is short, and people have to get things done. And in getting things done, maybe we'll advance Science to eventually end up living long enough to do more of that sort of thing, or we'll become experienced enough to compact the Math to be taught at a lower level.