Reference request: Fourier and Fourier-Stieltjes algebras
Solution 1:
I don't know references that I'd call good beyond what I already mentioned in the comments. One problem is that most references take a thorough acquaintance with abstract harmonic analysis for granted. Edit: For a quick introduction to this covering many aspects you should be interested in I'd recommend George Willis's contribution to H.G. Dales et al. Introduction to Banach algebras, operators, and harmonic analysis, London Mathematical Society Student Texts, 57. Cambridge University Press, Cambridge, 2003, (e-book version), MR2060440. There are many references for further study in there. Classic texts such as Loomis, Hewitt-Ross or Reiter haven't lost their value over time and can be found in any library.
As I said, I think the basic theory of the Fourier and Fourier–Stieltjes algebras is well-explained in T.W. Palmer, Banach algebras and the general theory of $\ast$–algebras, Encyclopedia of Mathematics and its Applications, Vols 49 and 79. Cambridge University Press, 1994 and 2001. MR1270014 and MR1819503. My guess is that it is in volume 2 but I can't check at the moment.
The Fourier algebra $A(G)$ and the Fourier–Stieltjes algebra $B(G)$ were introduced by Pierre Eymard in L'algèbre de Fourier d'un groupe localement compact, Bulletin de la Société Mathématique de France, 92 (1964), p. 181–236, MR228628.
Besides the initial motivations by Eymard, one nice fact that one may take to illustrate the importance of these algebras is the theorem due to M.E. Walters showing that $A(G)$ and $B(G)$ are complete invariants of the group. More precisely, if $G$ and $H$ are locally compact groups then $G$ and $H$ are isomorphic if and only if $A(G)$ and $A(H)$ are isometrically isomorphic if and only if $B(G)$ and $B(H)$ are isometrically isomorphic, see Walters, $W^{\ast}$-algebras and nonabelian harmonic analysis, J. Functional Analysis 11 (1972), 17–38, MR352879. This breaks down horribly for the group $C^{\ast}$ algebra $C^{\ast}(G)$ and the group von Neumann algebra and the only other algebras I know for which a similar theorem holds are the convolution algebra $L^1(G)$ (Wendel's theorem) and the measure algebra $M(G)$ (Johnson's theorem). In fact, Wendel's, Johnson's and Walters's theorems are closely related not only in spirit but also in the techniques used in the proofs. In the $C^{\ast}$/von Neumann world much more structure needs to be introduced for such a result to hold, see e.g. Enock–Schwartz, Kac algebras and duality of locally compact groups, Springer-Verlag, Berlin, 1992, where you can find a simultaneous generalization of the three aforementioned results (Wendel's main ideas still remain the heart of the matter, though even if his use of double centralizers is obscured by the use of multipliers).
Here's one definition of the Fourier algebra $A(G)$:
Let $\lambda: G \to U(L^2(G))$ be the left regular representation of $G$ on $L^2(G)$, i.e. the representation given by $\lambda(g)\xi(x) = \xi(g^{-1}x)$. For each pair $\xi,\eta \in L^2(G)$ we obtain a bounded continuous function on $G$ by setting $\omega(g) = \langle \lambda(g) \xi, \eta\rangle_{L^2}$. Note that $|\omega(g)| \leq \|\xi\|\,\|\eta\|$ by Cauchy-Schwarz and continuity of $\omega$ is a straightforward consequence of the strong continuity of the left regular representation. Now by definition $A(G)$ is the set of all continuous functions $\omega$ of the form $\omega(g) = \langle \lambda(g)\xi,\eta\rangle$ and the norm is defined to be $$\|\omega\|_{A(G)} = \inf{\{\|\xi\|\,\|\eta\|:\omega(g) = \langle \lambda(g)\xi,\eta\rangle, \, \xi,\eta \in L^2(G)\}}$$ It turns out that $A(G)$ is a (commutative) Banach algebra with respect to this norm and pointwise multiplication. By Eymard, Théorème 3.10 the Fourier algebra $A(G)$ is in fact the pre-dual of the (left) group von Neumann algebra $VN(G)$ (often denoted by $L(G)$). Recall that for an abelian group $VN(G) = L^{\infty}(\widehat{G}) = L^1(\widehat{G})^{\ast}$. For many purposes, a good way to think about $A(G)$ is as the adequate substitute for $L^1(\widehat{G})$ in the non-commutative setting, where the dual group doesn't really make sense (the unitary dual $\widehat{G}$ creeps around in these considerations of course, but note that $\widehat{G}$ is a rather pathological space and has no group structure whatsoever if $G$ is non-commutative).
You asked about the relation to amenability of $G$. There are a few surprises here: First of all, using Johnson's classical notion of amenability of Banach algebras (discussed at length e.g. in Runde's book Lectures on amenability, Springer LNM 1774 (2004), MR1874893) it turns out that amenability of $A(G)$ is equivalent to $G$ having a closed finite index abelian subgroup, which is way too strong, of course. For instance already for the beautifully nice compact group $SO(3)$ the Fourier algebra $A(G)$ is not amenable in this sense (conclude from this whatever you want).
One of the early successes of Effros–Ruan's theory of operator spaces is Ruan's result that $A(G)$ is operator amenable if and only if $G$ is amenable: see The operator amenability of $A(G)$, Amer. J. Math. 117 (1995) no. 6, 1449–1474 MR1363075 for the original reference and Runde's book for an exposition (taking some properties of $A(G)$ for granted but introducing the necessary baggage on operator spaces). For more on operator spaces see Pisier's book Introduction to Operator Space Theory, London Mathematical Society Lecture Note Series 294, Cambridge University Press, 2003 MR2006539 or Effros–Ruan's book on Operator Spaces.
The Fourier–Stieltjes algebra $B(G)$ can be defined as the dual space of $B(G) = C^{\ast}(G)^{\ast}$ and there is also its reduced sibling $B_r (G) = C_{r}^{\ast}(G)^{\ast}$. There are different descriptions of $B(G)$ via positive-definite functions, but I'm talking way too long already. Similarly to $A(G)$ the amenability of $B(G)$ in the sense of Johnson is equivalent to $G$ compact and having an abelian closed subgroup of finite index, so again one is led to look at operator amenability and related properties.
The relation between operator amenability of $B(G)$ and amenability of $G$ is a more subtle and I refer you to the two recent works of Runde–Spronk, Operator amenability of Fourier–Stieltjes algebras, Math. Proc. Cambridge Philos. Soc. 136 (2004), no. 3, 675–686 MR2055055 and Operator amenability of Fourier–Stieltjes algebras. II., Bull. Lond. Math. Soc. 39 (2007), no. 2, 194–202. MR2323448. Note that the MathSciNet review of the second paper contains quite a few references to closely related results.
I think I have compiled a rather extensive list of references which should get you started and hope one or the other of them is useful for you.
Added:
The paper by Kaniuth–Lau, Fourier algebras and amenability, Banach algebras and their applications, 181–192, Contemp. Math., 363, Amer. Math. Soc., Providence, RI, 2004, MR2097958 looks like a nice survey. There is also the paper by P. Eymard, A survey of Fourier algebras, in Applications of hypergroups and related measure algebras 111–128, Contemporary Mathematics, 183, American Mathematical Society, Providence, RI, 1995, MR1334767.