Confused about adding cardinalities of sets as elements

You don’t go in circles: a set has fixed elements and a fixed cardinality, so you don’t go back and forth on what its members are or what its cardinality is. The solution shows that if B has 3 elements, then B has 2 elements. From this, we can conclude that B cannot have 3 members, because it having 3 members would lead to a contradiction (and not to some kind of infinite loop). And, once you have established that, the rest follows.

And no, with A being $\{2,3\}$, $B$ does not become $\{1,2,3\}$: We know that $B$ has to contain $1$ and $2$ (the $2$ being yhe cardinality of $A$), so if the cardinality of $B$ is either $1$ or $2$, then that is not a new element. And indeed, with the cardinality of $B$ being $2$, that works out exactly right.

While the notation $\{x,y,z,\ldots\}$ is often used as a way of "building" a set by enumeration of its elements, I think it is a mistake to approach the equations \begin{align} A &= \{3, \lvert B\rvert\}, \tag1\\ B &= \{1, \lvert A\rvert, \lvert B\rvert\} \tag2 \end{align} as if this were an exercise in constructing sets by enumeration.

Instead, you are meant to deduce what sets $A$ and $B$ might possibly satisfy these two equations simultaneously. We know that \begin{align} A &= \{2,3\},\\ B &= \{1,2\} \end{align} is a solution because when we substitute $\{2,3\}$ for $A$ and substitute $\{1,2\}$ for $B$ simultaneously into Equations $(1)$ and $(2)$, we get two true equations. There is no "infinite loop" here; we just list the elements of the proposed sets $A$ and $B$, plug everything in, and (boom!) we're done. It works.

All other proposed assignments of $A$ and $B$ can be proved not to be solutions because they lead to contradictions. Again, there is no need for an "infinite loop"; once you find a contradiction you know the proposed assignment is not a solution, and you do not have to continue to look for additional contradictions to that solution.

It will not always be the case that there is a unique solution for such a set of equations. For example, consider \begin{align} C &= \{1, \lvert D\rvert\}, \\ D &= \{1, \lvert C\rvert, \lvert D\rvert\}. \end{align} One solution of this set of equations is $C = \{1,2\},$ $D = \{1,2\},$ because then $\lvert C\rvert = \lvert D\rvert = 2.$ But another solution is $C = \{1\},$ $D = \{1\},$ in which $\lvert C\rvert = \lvert D\rvert = 1.$ So we can say the system of equations has a solution, but we cannot point uniquely to the solution of the equations.

As another example, consider \begin{align} E &= \{1, 2, \lvert F\rvert\}, \\ F &= \{2, \lvert E\rvert, \lvert F\rvert\}. \end{align} Clearly $\lvert F\rvert$ can only be one of the numbers $1, 2, 3.$ But in the case $\lvert F\rvert = 1$, we see that $F$ has at least the two elements $1$ and $2,$ which contradicts $\lvert F\rvert = 1.$ In the case $\lvert F\rvert = 2$, we see that $\lvert E\rvert = 2,$ which implies that $F = \{2\},$ which contradicts $\lvert F\rvert = 2.$ And in the case $\lvert F\rvert = 3$, we see that $\lvert E\rvert = 3,$ which implies that $F = \{2,3\},$ which contradicts $\lvert F\rvert = 3.$ So there is no solution at all.

The fact that we were able to apparently "define" $A$ and $B$ according to the solution of the original problem is due merely to a "fortunate" choice of equations that happen to have a solution and happen to have only one solution.

Getting back to the original question, we have the statement, "If at the end A now has 2 elements, B should be updated as well."

In fact this is not how it works for any solution of any system of equations, whether the objects on each side of the equation are sets, or algebraic expressions of numbers such as in the equation $x = 2y - x + 3,$ or matrix expressions, or some other kind of mathematical object. A solution to a system of equations is a solution not because it results from following a correct procedure, but simply because when you plug it in, it makes the equations true. Conversely, any proposed "solution" that makes the equations false when you plug it in is not a solution. There is no "update" step; either the solution works as presented, or it doesn't.

This is a very useful fact to keep in mind when solving various problems, especially problems where there is not a direct solution method that leads unerringly to one and only one correct solution.

What might appear to be an "update" step in the given solution of the original problem is actually a dead end: having assumed that the solution has a particular property, such as $\lvert B\rvert = 3,$ we reach a contradiction, hence the statement $\lvert B\rvert = 3$ must be false. But when this case is ruled out, we learn something about the set $A.$ And that leads to further deductions, finally reducing all the candidates for the assignment of the contents of the sets $A$ and $B$ to just one possible assignment.