Penrose's remark on impossible figures
Solution 1:
To answer $\sf \color{Fuchsia}{(A)}$, one has to understand the meaning of the ambiguity group clearly first - indeed, your interpretation of the group is not entirely correct.
The punchline here is that the ambiguity group measures ambiguity, not quite impossibility (which is the job of the cohomology group defined later), of the given figure. For example, consider the tribar, and piece it up into $Q_1, Q_2, Q_3$, as done by Penrose. If you stretch $Q_1$ by a factor of $\lambda > 0$, shrink $Q_2$ by a factor of $\lambda$ (that is to say, stretch by $\lambda^{-1}$), and leave $Q_3$ as it is, the resulting object is really the same tribar, which is what Penrose was trying to convey through the following picture from the paper.
That is to say, modifying $Q_i$'s in this way gives you a symmetry of $X$. And you have such a symmetry for each $\lambda > 0$. Of course, collection of such symmetries $G$ is a group: consider one symmetry which stretches $Q_1$ by $\lambda$, shrinks $Q_2$ by $\lambda$ and leaves $Q_3$ as is and another one which does the same thing with $\lambda'$ this time instead. You can of course compose them, and the resulting symmetry stretches $Q_1$ by $\lambda \lambda'$, shrinks $Q_2$ by $\lambda \lambda'$, and leaves $Q_3$ invariant. This is the ambiguity group. This partially answers $\sf \color{Fuchsia}{(C)}$. Note that the origin of the name comes from the fact that the tribar is ambiguous in the sense that you can have different $Q_1, Q_2, Q_3$'s matching up to produce the same tribar. You can never tell the length of a bar in the tribar - it'll depend on the point of view you're seeing it from. And the "group of symmetries" $G$ captures all the ambiguity there is.
It is now clear that there is a bijection $G \to \Bbb R^+$, sending a symmetry to the corresponding $\lambda > 0$ in $\Bbb R^+$ (group under multiplication), which is in fact a homomorphism. Thus, $G \cong \Bbb R^+$.
That said, given the Necker cubes, the ambiguity is in the vertices. Pick any vertex $v$ from the figure. There are two point of views here - $v$ can either point upwards, or point downwards. If $v$ points upwards, then so does all the other vertices, and similar for the other choice. So two point of views, two choices - hence, the ambiguity group is $\Bbb Z/2\Bbb Z$. More formally, the symmetry is obtained from reflecting the whole figure (imagined as sitting in $\Bbb R^3$) along an appropriate hyperplane. Obviously, the symmetry group then is the cyclic group of order 2.
To answer $\sf \color{Fuchsia}{(D)}$. Yes, this is indeed a variant of Čech cohomology.
First, let me explain what cohomology really is. If $X$ is a sufficiently nice topological space, you can always set up a homeomorphism $X \cong T$ of $X$ with a simplicial complex $T$. Such a homeomorphism is called a triangulation of $X$. Note that there need not be a unique such homeomorphism, so there may be lots of ways to triangulate a given nice topological space.
Declare $\Delta_n(T)$ to be the free abelian group generated by $n$-simplices of $T$. Let $\Delta^n(T)$ be $\hom(\Delta_n(T), \Bbb Z)$, the dual of $\Delta_n(T)$. Elements of $\Delta^n(T)$ are called $n$-cochains. There is a natural map $\partial : \Delta^{n-1}(T) \to \Delta^n(T)$ taking an $n$-cochain $\psi : \Delta_{n-1}(T) \to \Bbb Z$ to the $n$-cochain $\psi \circ \text{bd} : \Delta^n(T) \to \Delta^{n-1}(T) \to \Bbb Z$ where $\text{bd}$ sends an $n$-simplex in $T$ to sum of signed faces of that $n$-simplex (signs chosen appropriately to take care of orientation). $\partial$ is called the coboundary map. One can check that the map $\partial \partial : \Delta^{n-1}(T) \to \Delta^n(T) \to \Delta^{n+1}(T)$ is the zero map. The sequence of maps
$$\cdots \to \Delta^{n-1}(T) \to \Delta^n(T) \to \Delta^{n+1}(T) \to \cdots$$
is then a cochain complex, that is, going two steps in the sequences lands you into $0$. As a consequence of $\partial \partial = 0$, we have that $\text{im} \partial \subset \ker \partial$. Elements of $\text{im} \partial$ are called coboundaries, and elements of $\ker \partial$ are called cocycles. As everything is abelian, $\text{im} \partial$ is also normal in $\ker \partial$. We set $H^n(X; \Bbb Z) := \ker \partial/\text{im} \partial$. This is called the simplicial cohomology of $X$, and is indeed independent of the triangulation $T$ we chose. It's a topological invariant of $X$. We'll see something similar happening in Penrose's paper.
If $X$ is the given ambiguous figure, let $\{U_i\}$ be a good cover of $X$. That is, $X = \bigcup U_i$ and each of $U_i$, $U_i \cap U_j$, $U_i \cap U_j \cap U_k$ and further intersections, are homeomorphic to balls $B^n$ (a more general condition would be that they are all contractible). Let each intersection $U_{i_1} \cap \cdots \cap U_{i_n}$ be denoted as $U_{i_1 \cdots i_n}$. Treat these sets as symbols (i.e., formal intersection), and distinguish $U_{ij}$ from $U_{ji}$ even though they are the same set.
$G$ be the ambiguity group of $X$. Define an $n$-dimensional cochain to be a function $\varphi^n : \{U_{i_1 \cdots i_n}\} \to G$ such that $\varphi^n(U_{i_1 \cdots i_k \cdots i_l \cdots i_n}) = \varphi^n(U_{i_1 \cdots i_l \cdots i_k \cdots i_n})^{-1}$ holds for any $k, l$, where $\{U_{i_1\cdots i_n}\}$ is the collection of all possible intersection of $n+1$ many sets from the cover. When $n=0$, it's just an assignment of a an element of $G$ to each $U_i$.
Let $C_n(X)$ denote the group of all $n$-dimensional cochains, where we multiply two cochains by multiplying their values in $G$. Define the coboundary map $\partial : C_{n-1}(X) \to C_{n}(X)$ given by sending $\varphi^{n-1}$ to the $n$-cochain $\varphi^n$ assigning to each $U_{i_i \cdots i_n}$ the number $\prod_j \varphi^{n-1}(U_{i_1 \cdots \hat{i_j} \cdots i_n})^{\varepsilon_j}$ where $\varepsilon_j = (-1)^n$ if $j$ is even and $(-1)^{n+1}$ if $j$ is odd, and $\hat{i_j}$ means that term is missing from the indices. So for example, in the tribar example, assigning to each $Q_i$ the number $q_i$ is a $0$-cochain and applying the coboundary map one gets an assignment of each (oriented) intersection $Q_{ij} = Q_i \cap Q_j$ to the number $q_i/q_j$, a $1$-cochain in our language.
The final thing to check is that $\partial \partial : C_{n-1}(X) \to C_{n}(X) \to C_{n+1}(X)$ is the zero map. This is a rather tedious and technical thing to check, but if the readers try verifying it, they will understand the motivation behind the rather bizarre definition of $\partial$ above. To sum up, we have that
$$\cdots \to C_{n-1}(X) \stackrel{\partial}{\to} C_n(X) \stackrel{\partial}{\to} C_{n+1}(X) \to \cdots$$
is a cochain complex. We can thus take it's cohomology $H^n(X; G) := \ker \partial/\text{im}\partial$ (the $G$ is there to indicate that the cochains are $G$-valued. Indeed, we could have put any group to be the coefficient group there, but it seems Penrose finds it natural to use the ambiguity group of $X$). For $n = 1$, this is precisely the $1$st cohomology group of Penrose.
Let us compute the cohomology group of the tribar. $H^1(X;\Bbb R^+)$ is the group of $1$-cocycles modulo $1$-coboundaries. As there are no nontrivial $2$-cochains, every $1$-cochain is a cocycle. That is to say, we're computing $1$st cohomology of the two-term cochain $$0 \to C_0(X) \stackrel{\partial}{\to} C_1(X) \to 0$$ which is just $C_1(X)/\text{im} \partial$. Note that this has a slight similarity with your interpretation (the definition of $H$), but is not quite the same. In any case, I can find a $1$-cocycle in $C_1(X)$ which is not a coboundary using the clever technique devised by Penrose: the $1$-cochain $\psi^1$ sending $Q_{ij} = Q_i \cap Q_j$ to $d_{ij} = A_i/A_j$ is independent of the choice of $A_k$'s, and I claim that it is a nontrivial cochain (i.e., is not a boundary). Well, if it was a boundary, then the equality $d_{ij} = \psi^1(Q_{ij}) = \psi^0(Q_i)/\psi^0(Q_j)$ would hold for some $0$-cochain $\psi^0$. But then we'd have $\tau(\psi^1) = d_{12}d_{23}d_{31} = 1$.
However, if we choose the placement of tribar appropriately so that $A_{12}$ and $A_{21}$ has the same distance from origin (i.e., $d_{12} = 1$) and $A_{23}$ is farther away from origin than $A_{32}$ (so that $d_{23} > 1$, then $A_{13}$ would automatically be closer than $A_{31}$ (that is, $d_{31} = d_{13}^{-1} > 1$), which implies $\tau(\psi^1)$ is strictly greater than $1$, forcing $\psi^1$ to be nontrivial. Once we have this nontrivial cochain, we can get $|\Bbb R^+|$-many nontrivial cochains by multiplying $\psi^1(Q_{12})$ by $1$, $\psi^1(Q_{23})$ by $\lambda$, and $\psi^1(Q_{31})$ by $1/\lambda$ for each $\lambda > 0$. This gives a natural identification with $G$. I think with some work you can show that these are all the nontrivial $1$-cocycles there are, and conclude $H^1(X; \Bbb R^+) \cong \Bbb R^+$. In general, it need not be true that $H^1(Q; G)$ is the same as the ambiguity group $G$, but the $1$-st cohomology group $H^1(Q; G)$ would be a module over the ambiguity group $G$.
I picked the open cover $\{U_i\}$ of $X$ to be good, i.e., all sets and all intersections are contractible. The reason is that by the nerve theorem, $X$ would have the homotopy type of the nerve $\mathcal{N}$ of $\{U_i\}$. Čech cohomology just computes cohomology of the nerve $\mathcal{N}$, but since this is homotopy equivalent to $X$, that is the same as taking cohomology of $X$, so it's independent of whatever good cover we chose. I think something similar must be at work for these ambiguous/impossible figures. This probably answers $\sf \color{Fuchsia}{(B)}$, but only Penrose knows what he had in mind.
One more thing I haven't addressed yet.
Ultimately, I'd like to be able to look at an impossible figure and systematically derive its cohomology group, just like I can derive a figure's symmetry group. Or alternately, create impossible figures with given cohomology group. But perhaps the analogy isn't tenable, as cohomology groups aren't really symmetry groups at all.
The first part of this answers gives a rough overview of how you could recognize the ambiguity group from a given ambiguous picture, so I presmume I have answered that bit. I do not know what a rigorous definition of the ambiguity group might be, but I presume it's some sort of inherent symmetry group of the chosen cover $\{U_i\}$. Also, certainly the analogy is tenable: if you read my answer to $\sf \color{Fuchsia}{(A)}$, you'd see that this ambiguity group is indeed some sort of symmetry groups which captures symmetries "induced from ambiguity". However, I do not know if you can construct an ambiguous figure such that the ambiguity group $G$ is isomorphic to a given group. Here's some food for thought: Assume $G$ is a finitely presented abelian group. Pick generators $x_1, \cdots, x_n$ of $G$. $F$ be the free abelian group generated by these. If $Y$ is some ambiguous figure with infinite cycle ambiguity group, then disjoint union of $n$ copies of $Y$ gives you a figure with ambiguity group isomorphic to $F$. Now given relators $r_i = 1$ for $G$, try identifying the copies of $Y$ appropriately to kill the symmetries corresponding to $r_i$. Can this be done?
PS: I wanted to add an explicit computation of the Penrose cohomology $H^1(X; \Bbb Z/2)$ of the Necker cube as a summary, but I'd need some pictures for that and I'm too ashamed of my drawing skills to add it. Maybe if OP (and others reading this) are really interested.