How to tranform a square-ish tile into an arbitrary quadrilateral
Solution 1:
You recently wrote a comment to an answer of mine. That answer describes how to compute a projective transformation given four points and their images. Your comment asked whether that approach could work for complex coordinates.
It does for the complex projective plane, but that is hard to visualize. You'd get homogeneous coordinate vectors with three complex coordinates, and most of them would not have a counterpart in the Euclidean plane. However, the approach also works for the complex projective line, one dimension lower. Without the "projective", the "complex line" means single complex number. Which you can separate into real and imaginary part and thus represent as a point in the plane. Making this a projective line means you use homogeneous coordinates, as a second complex number and essentially look at the ratio of these. This is the same as the normal complex line with a single point at infinity added to it. That structure is also called the Riemann sphere.
Projective transformations of the complex projective line are Möbius transformations. Following the approach from my answer, they are uniquely defined given three points and their images.
So if you use the corners of a pair of times to define such a Möbius transformation, and the transformation defined in this way happens to reliably map the fourth corner of one time into the fourth corner of the other, then this approach would make sense for your use cases. That's a fairly strong condition, and lacking an authoritative formula describing your shape, I'm not going too check it for you.
Note that Möbius transformations map lines and circles to lines and circles. So if your reference square gets deformed into some straight-edged polygon, I'd expect it's images in the tiling to be bounded by a chain of circular arcs. The fact that your tiling is depicted with straight edges everywhere suggests that perhaps assuming a Möbius transformation between any two tiles is not in the spirit of that picture, even if the corner positions were to allow such a transformation of the corner points. That's for you to decide, though.
You actually might have yet another constraint: you probably want the transformation of adjacent tiles to be compatible. So that if you transform your deformed square to one, and to the other, then the images of the shared boundary are the same for both of these. That's a very strong condition. It would be possible to guarantee this if all your tiles are images of a single tile under repeated application of the same transformation. But neither a projective transformation of the real plane nor the complex line would have the properties you depict. So you are looking at different transformations from one tile to the next, and still need the tiles to fit together.
You could of course do some approximations. Transform the boundary according to each of the adjacent tiles then take some form of average between the two results. But at that point, the problem is mostly aesthetic, with very little mathematical motivation to rely on.
Solution 2:
I have attempted to reproduce more or less faithfuly your figure:
The coordinates of the points being in two arrays $x$ and $y$ (see Matlab program below). But it's not clear what your objective is. What would be the role of the squares ?
The transform you need should be
- either a conformal transform (i.e., given by a complex function $Z=f(z)$ giving in particular a preservation of angles, like in the second figure here (with $Z=exp(z/2)$)
- or a projective transform acting on each separate tile.
Edit: the way to define a projective transform that maps any quadrilateral $(x_k,y_k)$ on any target quadrilateral $(x'_k,y'_k)$.
In terms of unknowns, you are looking for $8$ coefficients $a,b,c,d,e,f,g,h$ such that
$$\begin{cases}x'_k&=&\dfrac{ax_k+by_k+c}{gx_k+hy_k+1}\\y'_k&=&\dfrac{dx_k+ey_k+f}{gx_k+hy_k+1}\end{cases}$$
providing you a linear system of 8 equations/constraints (4 for the abscissas, 4 for the ordinates).
Another explanation here.
Matlab program
a=4;b=0.3;s=0.0085;
k=1;
for t=-1.2:s:0.7;
r=(1+tanh(a*t));
ang=t*58;
x(k)=(r+b).*cos(ang);
y(k)=(r+b).*sin(ang);
k=k+1;
end;
L=length(x);
plot(x,y,LS,'on');
n=13;
for k=1:n
I=k:n:L;
plot(x(I),y(I),'r',LS,'on')
end;