New twist on a Putnam problem
Solution 1:
The case of the abelian group $Z_2$ came up before on this forum. Is there a non-constant function $f:\mathbb{R}^2 \to \mathbb{Z}/2\mathbb{Z}$ that sums to 0 on corners of squares?
I gave an answer to that, which after some simplification, and using some of the sets of the OP, goes as follows: Let $f$ be the function from the plane into $Z_2$, and use the relations:
[a] $f(0,0)+f(1,0)+f(1,1)+f(0,1)=0,$
[b] $f(0,0)+f(-1,0)+f(-1,-1)+f(0,-1)=0,$
[c] $f(1,0)+f(0,1)+f(-1,0)+f(0,-1)=0.$
Adding these in $Z_2$, everything cancels exept $f(1,1)+f(-1,-1).$ This means (since we're in $Z_2$) that the values of $f$ at diagonally opposite points of any square are the same, from which by rescaling the square and moving it, we can see the map $f$ is constant.
EDIT: Extension to the case of the group $Z_4$. Suppose $f$ maps the plane to $Z_4$ and the sum of $f$ on corners of every square is $0$. By reducing mod 2 and using the above result that maps into $Z_2$ with this corner-sum property are constant, we can deduce that in the case of this map $f$ to $Z_4$ there are only two possible cases left: The "1,3" case, in which the value of $f$ at each point is in $\{1,3\}$ and the "2,4" case in which the value of $f$ at each point is in $\{2,4\}$.
Now in either of these cases, the values at the vertices of any square are either all the same or else there are two of each value. For example in the 1,3 case we cannot have three 1's and one 3 at the corners of any square, since $1+1+1+3=2 \mod 4.$ The same happens with three 3's and one 1, or with three 2's and a 4, or with three 4's and a 2.
To simplify the rest we'll use the symbols $a,b$ to mean the two distinct elements in either the 1,3 case or the 2,4 case. So by what has been said, every square must have its corners labelled either all $a$, or all $b$, or else two $a$ and two $b$ on the corners.
Our claim now is that there is no way to label the nine points of the "double square" in this way, in such a way that the diagonally opposite corners of the "big square" end up with different labels from $a,b$. By the "double square" we mean the collection of nine points $(x,y)$ with each of $x,y$ taken from the set $\{-1,0,1\}$ (as described in the OP and in the above proof). There are then in an obvious way a "big square", four "small squares" we can call lower right, lower left, etc., and a "diamond" (also a square) formed by the midpoints of the sides of the big square.
Now suppose it is possible to put different values at diagonally opposite points of the big square. Without loss, assume the lower right is given the value $a$ and the upper left the value $b$. Then the other two points of the big square are also $a,b$ (not both $a$ or both $b$); again without loss suppose the lower left of the big square is $a$ while the upper right of the big square is $b$.
Now the center of the square must be labelled either $a$ or $b$, and we may suppose it is labelled $a$ (if it were $b$ we could rotate the square 180 degrees and interchange the letters $a,b$).
Now there are two possiblilties for the midpoint of the right side of the big square.
case 1: we put the value $a$ at this right side midpoint. Then we already have three $a$ labels on the lower right square, so are forced to place another $a$ at the fourth point of this square, i.e. at the midpoint of the bottom side of the big square. Then easily the bottom left square ends up labelled all with $a$. Now looking at the top two smaller squares, each now has three of its vertices chosen, with two $a$ and one $b$ for the choices so far. This means that the midpoint of the top side of the large square must be given the value $b$, and finally our contradiction, since now the diamond square has three points labelled $a$ and one point (top of diamond) labelled $b$. As noted there cannot be three $a$ and one $b$ at the corners of any square.
case 2: we put the value $b$ at the right side midpoint. This time we are forced to place a $b$ at the midpoint of the bottom side of the big square, then another $b$ at the midpoint of the left side of the big square, and now the two smaller upper squares require there to be an $a$ at the midpoint of the top side of the large square. Again we reach a contradiction, this time because the diamond square is now labelled with three $b$ and one $a$ corner.
To complete the argument for the group $Z_4$: From what we've shown, every square must have the same value mod 4 at any two of its diametrically opposite vertices. Then by expanding and moving such squares we can position one with its opposite corners at any two points $P,Q$ in the plane, making the map constant in $Z_4$.
The question regarding other abelian groups of exponent dividing 4 may be approachable in this way; if so I don't see it yet.
ADDED: Thanks to Gerry Myerson for pointing out that Mercio's solution to the $Z_2$ problem shows how to extend to any finite abelian group $G$. A good answer to look at for anyone interested! (same reference, "Is there a nonconstant...")
Solution 2:
As noted in several other answers, the purely discrete version of the problem (with ${\mathbb Z}^2$ in place of ${\mathbb R}^2$) has a few solutions, and the possibility of rescaling in $\mathbb R$ forces any solution in this case. Here is some more information on the discrete problem.
By considering first smaller families of squares or triangles, one can avoid the torsion problem and obtain nice characterizations of the solutions. Then, a final small adjustment is needed to obtain a full charcaterization to the solution of the initial problem.
Theorem Let $G$ be an arbitrary abelian group, and let $V$ be the module of all maps$f: {\mathbb Z}^2 \to G$ whose sums are zero on all the squares of the form $(x,y)+[0,1]^2$ and $\lbrace (x,y), (x+1,y+1), (x,y+2), (x-1,y+1)\rbrace$. Then the ${\mathbb Z}$-linear map $$ \phi : V \to G^4,\ \ f \mapsto (f(1,1),f(0,0),f(1,0),f(2,0)) $$ is an isomorphism. Indeed, for $(a,b,c,d)\in G^4$ the unique $f$ such that $\phi(f)=(a,b,c,d)$, is given by the following rule : for integers $x,y,i,j$,
$$ f(x,y)= \begin{cases} -ya & -(i+j-1) b& &+(i-j)d & \text{if }\ x=2i,& y=2j \\ -ya & +(i-j-1) b& -c& -(i+j)d & \text{if}\ x=2i,& y=2j+1\\ ya & -(i-j)b& c & +(i+j)d & \text{if}\ x=2i+1,& y=2j \\ ya & +(i+j)b& & -(i-j)d & \text{if}\ x=2i+1,& y=2j+1 \\ \end{cases}\tag{*} $$
Corollary Let $G$ be an arbitrary abelian group, and let $f: {\mathbb Z}^2 \to G$ be a map. Then the sum of $f$ is zero over all the rectangles of ${\mathbb Z}^2$ iff $f$ satisfies (*) above for some four tuple $(a,b,c,d)$ with $4a=4b=4c=4d=0$ in $G$.
Proof of the theorem :