The mathematics of music - why sine waves?

It sounds like you really want to read Benson's Music: a Mathematical Offering (freely available at the link). I'm not completely sure what you're asking, but if it's anything like "why is it natural to think about music in terms of sine waves," this question is addressed right in the introduction:

...what's so special about sine waves anyway, that we consider them to be the "pure" sound of a given frequency? Could we take some other periodically varying wave and define it to be the pure sound of this frequency?

The answer to this has to do with the way the human ear works. First, the mathematical property of a pure sine wave that's relevant is that it is the general solution to the second order differential equation for simple harmonic motion. Any object that is subject to a returning force proportional to its displacement from a given location vibrates as a sine wave. The frequency is determined by the constant of proportionality. The basilar membrane inside the cochlea in the ear is elastic, so any given point can be described by this second order differential equation, with a constant of proportionality that depends on the location along the membrane.

The result is that the ear acts as a harmonic analyser [emphasis added]. If an incoming sound can be represented as a sum of certain sine waves, then the corresponding points on the basilar membrane will vibrate, and that will be translated into a stimulus sent to the brain.

Hmm, sometimes confused are normal modes of a resonant system (string or tube of air) and periodic signals. Periodic signals can be decomposed into a Fourier series with integer harmonics. The vocal cords produce a nearly periodic signal (flaps going back and forth, but it's not a sine wave). Some musical instruments have normal modes with frequencies that are harmonic. A perfect string has harmonic overtones. Wind instruments are often designed so that modes blown have frequencies in integer ratios (e.g., the trumpet). But when you blow a trumpet and you get the air inside to resonate, you get essentially a nearly periodic signal out. Same goes for blowing into a flute or didgeridu. Why do we "like" periodic signals (or those with harmonic overtones) and tend to associate them with music? That's a perception question. Not to be confused with normal modes of a vibrating system or the Fourier transform of a nearly periodic signal. Our ears and brains seem to have a pretty sophisticated system of interpreting the voice. So maybe musical sounds are similar to voice and that's why we like them.

An additional question might be why do we prefer sounds with musical intervals (harmony) or complex tones (say a well designed bell) that have overtones that are in integer ratios (musical intervals called fifths, thirds, sixths which are tones in ratios of 3:2 5:4 6:5 etc.)? This is harder to answer but may be related to our preference for periodic signals. If you combine two nearly periodic signals that have periodicity in a ratio of 3:2 (a fifth interval) then the overtones beat if they aren't in tune. If there is a beat frequency and it's noticeable then it's either pleasing (like a vibrato) or annoying like two out of tune violins playing simultaneously.

To make things more complicated it turns out that we actually don't really like to listen to exactly periodic signals. They sound electronic. As a flute player I can tell you that I adjust "volume" not by playing softer but reducing my vibrato so that people pay less attention to my sound. So slight variations in voice and differences between exact periodicity are really important for musical sounds.

You might ask do we like to listen to pure sine waves or pure tones? Not really, this also sounds electronic, and actually pretty dull too (unless it's high frequency in which case it would be pretty annoying). As Qiaochu mentions above our ear separates different frequencies into different regions on the basal membrane. That means that more neurons fire if the signal is richer (and has a wide range of spectral frequencies). This gives us more information. The spectral information is used for example to identify formants in speech (these are bright spectral bands) which vary for different vowel sounds. So maybe we really like nearly periodic signals that have rich spectral information (or lots of harmonics).

To answer your question: Is there a better way to expand musical signals than using a Fourier series? Given that we "like" nearly periodic signals, Fourier series is probably the way to go. However wavelets give alternative types of bases and are used in seismology, for example, where you have rapidly changing signals (rather than continuous nearly periodic ones).

Most of the answer has already been given with the reference to the auditory system. It dissolves the sound into a certain base functions. I don't know of a good description of their behaviour, but sine waves are a good approximation. So their choice provides – though historically inspired – a good description of the transition from air into brain.

You are asking whether there are other bases, that are useful as well. Your answer concerning acoustics is: yes. So there is no reason not to use them.

But you must always consider: Acoustics is not music. Even a pure harmonic sound is still not music. Music can not easily described by mathematical formulae. It is dependent on acoustics, physics, cultural background, your mood and many other factors.

William Sethares describes in his book “Tuning, Timbre Spectrum Scale” how scales for instruments with non-harmonic spectra can be constructed. In this book he provides examples where the harmonic analysis fails from a musical point of view. The resulting music can be judged differently. As it has to pass the bottleneck of our auditory system it will never sound like a perfect organ pipe or flute (both can produce nearly perfect sine waves).

If you want to construct a guitar then you should consider both descriptions: the plucked string and a convenient model for our auditory system, thus the harmonic dissolution.

Nevertheless our modern society has agreed to the equal temperament in a certain degree. As described in the other answers this coincides with non of the usual spectra in music. Thus, accuracy is not the only paradigm. On the other hand history has shown that not every mathematical model is suitable for music, even if it describes a certain aspect with a high precision.

The solutions to the differential equation of the wave equation in a limited size tube (basically all blowing instruments : flutes, saxophones et.c.) or string (guitar, violin, banjo, lute, mandolin et.c.) can be solved easier using Fourier Analysis which expresses the solution as sums or series of sines and cosines. This is because the differential operator translates very nicely into a multiplication with frequency in the Fourier transform domain.

Here is the dude who started decomposing functions into sums of sines and cosines (the method bears his name The Fourier Transform): Joseph Fourier