What's 4 times more likely than 80%?

The most reasonable way to match the answer in the book would be to define the likelihood to be the ratio of success over failure (aka odds): $$ q=\frac{p}{1-p} $$ then the probability as a function of the odds is $$ p=\frac{q}{1+q} $$ In your case the odds are $4:1$ so $4$ times as likely would be $16:1$ odds which has a probability of $$ \frac{16}{17}=94.1176470588235\% $$ This matches the $3\%$ to $11.0091743119266\%$ transformation, as well.

Bayes' Rule

Bayes' Rule for a single event says that $$ O(A\mid B)=\frac{P(B\mid A)}{P(B\mid\neg A)}\,O(A) $$ where the odds of $X$ is defined as earlier $$ O(X)=\frac{P(X)}{P(\neg X)}=\frac{P(X)}{1-P(X)} $$ This is exactly what is being talked about in the later addition to the question, where it is given that $$ \frac{P(B\mid A)}{P(B\mid\neg A)}=4 $$

Daniel Kahneman's book mentions Bayesian reasoning. An answer using Bayesian reasoning is as follows:

Let $C$ be the event that Tom is compsci, $N$ be the event that he has a "nerdy" personality.

We are given $P(N|C)/P(N|\neg C)= 4$, which implies that $P(N|\neg C) = P(N|C)/4$.

By Bayes Theorem (and using the theorem of total probability to expand the denominator)

$$\begin{eqnarray*} P(C|N) &=& \frac{P(N|C) P(C)}{ P(N)} \\ &=& \frac{P(N|C) P(C)}{P(N|C)P(C) + P(N|\neg C) P(\neg C)} \\ &=& \frac{P(N|C) P(C)}{P(N|C)P(C) + 0.25 P(N|C)P(\neg C)} \\ &=& \frac{P(C)}{P(C) + 0.25 P(\neg C)} \\ &=& \frac{0.8}{0.8 + 0.25 \times 0.2} \\ &\approx& 0.9411765 \end{eqnarray*}$$

Similar reasoning in the 3% case leads to $P(C|N) = 0.03 / (0.03 + .25*.97) \approx 0.1100917$.

The statement of the context (in my words) is as follows:

If you believe that $80\%$ of graduate students are enrolled in computer science (base rate), and you also believe that the description of Tom W is four times more likely for a graduate student in computer science than for a graduate student in other fields, then Bayes’s rule says you must believe that the probability that Tom W is a computer scientist is now $\approx94.1\%$.

Here is how to perform the Bayesian reasoning. Let $\rm CS$ be the event that a student is enrolled in computer science, and $\rm desc$ the event that [description] holds true of a graduate student. Then

• The base rate says that $P(\rm CS)=80\%$.
• The relative statement says that $P({\rm desc|CS})=4P({\rm desc|\neg CS})$

Thus $P(\neg{\rm CS})=0.2$ and $P({\rm desc|\neg CS})=0.25P({\rm desc|CS})$. Bayesian reasoning says that

$$\begin{array}{ll} P({\rm desc}) & = P({\rm desc~\&~CS})+P({\rm desc~\&~\neg CS}) \\ & =P({\rm desc|CS})P({\rm CS})+P({\rm desc|\neg CS})P(\neg{\rm CS}) \\ & =(0.8+0.25\cdot0.2)P({\rm desc|CS}) \\ & =0.85P({\rm desc|CS}) \end{array}$$

Bayes rule says that

$$\begin{cases} P({\rm desc|CS})= \frac{P({\rm desc~\&~CS})}{P({\rm CS})} \\ \phantom{blah} \\ P({\rm CS|desc})=\frac{P({\rm CS~\&~desc})}{P({\rm desc})} \end{cases}$$

Therefore

$$P({\rm CS|desc})=\frac{P({\rm desc|CS})P({\rm CS})}{P({\rm desc})}=\frac{0.8}{0.85}=0.9411764705882352\dots\approx94.1\% $$

Similarly, if the base rate was $3\%$ instead of $80\%$, the calculation would go as follows:

$$\begin{array}{ll} P({\rm desc}) & = P({\rm desc~\&~CS})+P({\rm desc~\&~\neg CS}) \\ & =P({\rm desc|CS})P({\rm CS})+P({\rm desc|\neg CS})P(\neg{\rm CS}) \\ & =(0.03+0.25\cdot0.97)P({\rm desc|CS}) \\ & =0.2725P({\rm desc|CS}) \end{array}$$

$$P({\rm CS|desc})=\frac{P({\rm desc|CS})P({\rm CS})}{P({\rm desc})}=\frac{0.03}{0.2725}=0.1100917431192660\dots\approx11\% $$