Formula for number of solutions to $x^4+y^4=1$, from Ireland and Rosen #8.18.
Solution 1:
If $(a)$ is a principal ideal if $\Bbb Z[i]$, it has $4$ generators, $a,ia,-a$, and $-ia$.
And so, if $p$ is a prime congruent to $1$ modulo $4$, there are exactly $8$ solutions to $p=a^2+b^2i$, corresponding to the generators of the two primes above $(p)$ in $\Bbb Z[i]$.
The choices of $\pi$ and of $A+iB$ (or their real part) correspond to choosing a particular generator of those ideals (or a particular conjugate pair of generators of the two ideals above $(p)$) :
let $I = (1+i)$ be the ideal of norm $2$. Then, $(\Bbb Z[i]/I^3)^*$ is a group with $4$ elements, naturally isomorphic to $\{1,i,-1,-i\}$. So if $(a)$ is coprime with $(1+i)$ there is one canonical way of choosing a generator for $(a)$ : you have to take the one that is congruent to $1$ modulo $I^3$. The result of Chapter 11 section 5 says that $\pi$ is this generator for $(p)$.
Instead, the exercise wants to make this choice modulo $I^4$ : pick $A+iB$ the generator which is congruent to $1$ or $1+2i$ modulo $I^4$ (you have to make one choice for each element of $(\Bbb Z[i]/I^4)^*/\langle i \rangle$).
If you know about Artin's reciprocity map, the case in which $p$ falls corresponds to how it factors into the class field for $I^4$, which is of degree $2$ over $\Bbb Q(i)$, which is still abelian on $\Bbb Q$ (because every group of order $4$ is abelian), and is actually the ray class field of $\Bbb Q$ of conductor $(8)\infty$, which tells you that :
$a \in \{1,i,-1,-i\} \pmod {I^4}$ is equivalent to $N(a) \equiv 1 \pmod 8$ and $a \in \{3+2i,2+3i,1+2i,2+i\} \pmod {I^4}$ is equivalent to $N(a) \equiv 5 \pmod 8$
(if you don't like class field theory, you can easily verify this by a tedious computation: $(4k+3)^2 + (4l+2)^2 \equiv 3^2 + 2^2 = 13 \equiv 5 \pmod 8$, and so on ...)
Thus, the choice of $A+iB$ is the same as $\pi$ if and only if $\pi \equiv 1 \equiv A+iB \pmod {I^4}$, which is equivalent to $N(\pi) = N(A+iB) = p \equiv 1 \pmod 8$,
and the choice is not the same if and only if $\pi \equiv 3+2i \equiv -(1+2i) \equiv -(A+iB)) \pmod {I^4}$, which is equivalent to $N(\pi) = N(A+iB) = p \equiv 5 \pmod 8$.
You can also find the result without using the result from chapter $11$ (and prove it instead), simply by observing that the number of solutions to $x^4+y^4 = 1$ has to be a multiple of $8$ (in fact it is congruent to $8$ modulo $16$) :
Indeed, solutions are the $8$ couples $(i^k,0),(0,i^k)$ plus a number of solutions $(i^kx, i^ly)$, who come by bunches of $16$.
Now, since you know that for $p \equiv 1 \pmod 8$, $N = p-3-6 \Re(\pi)$, you get $0 \equiv 1-3-6\Re(\pi) \pmod 8$, hence $\Re(\pi) \equiv 1 \pmod 4$. Thus, since there is only one $A \equiv 1 \pmod 4$ such that $p = A^2 + B^2i$, $\Re \pi$ has to be this $A$.
Similarly, for $p \equiv 5 \pmod 8$, $N = p+1-2 \Re(\pi)$, you get $0 \equiv 5+1-2\Re(\pi) \pmod 8$, hence $\Re(\pi) \equiv 3 \pmod 4$. And so, $\Re(- \pi) \equiv 1 \pmod 4$. Again, since there is only one $A \equiv 1 \pmod 4$ such that $p = A^2 + B^2i$, $\Re \pi$ has to be $-A$.
Solution 2:
Yes: the problem is that you have sneakily defined $A$ twice. You first defined $A$ as the number that is 1 modulo 4 and satisfies $p=A^2+B^2$ for some $B$. Then, you defined it again as the real part of $J(\chi,\chi)=A+Bi$. Both definitions imply that $p=A^2+B^2$, but there are two integers that satisfy this condition alone, namely $A$ and $-A$. Your two definitions are actually the negatives of each other.