Change of Variable formula for a non-differentiable mapping.

Let $\Omega \subset \Bbb R^n$. For a diffeomorphism (or merely a differentiable bijection) $\varphi:\Omega \to \varphi(\Omega)$, we have the formula $$ \int_{\Omega} f\circ\varphi^{-1}(x)\, dx = \int_{\varphi^{-1}(\Omega)} f(y)|D\varphi(y)| \,dy. $$

How much can we generalize the class in which $\varphi$ is allowed to lie in? Is it enough that we have, says, a bijection $\varphi\in W_{\text{loc}}^{1,\infty}(\Omega ;\Bbb R^n)$ or even $\varphi\in W_{\text{loc}}^{1,1}(\Omega ;\Bbb R^n)$?

How much does the result depends on the domain $\Omega$? Is there a big difference between a compact and an open domain? Does the regularity of the boundary $\partial \Omega$ play any role?

I'd also appreciate if you have a good reference to this kind of result so that I can read further into this interesting issue. Happy New Year to all of you.

Perhaps one of the most general classes of maps, defined on a set $\Omega \subset \Bbb R^n$, for which the change of variables formula $$ \int\limits_{\Omega} f\circ\varphi^{-1}(x)\, \mathrm{d}x = \int\limits_{\varphi^{-1}(\Omega)} f(y)|D\varphi(y)| \,\mathrm{d}y \label{1}\tag{1} $$ (or a proper generalization) holds is the one considered by Piotr Hajłasz in [1]. To describe his results, it is useful to preliminarily recall some concepts.

• A function $u:\Omega \to \Bbb R$ is approximately totally differentiable at $x_0\in\Omega$ if there exists a real vector $\mathsf{D}u|_{x_0}=(\mathsf{D}u_1,\ldots,\mathsf{D}u_n)$ such that, for every $\varepsilon$, $x_0$ is a point of density for the set $$ A_\varepsilon=\left\{ x\in\Omega\,\left|\;\frac{|u(x)-u(x_0)-\langle\mathsf{D}u|_{x_0},x-x_0\rangle|}{|x-x_0|}<\varepsilon\right.\right\} $$ Saying that $u$ is approximately totally differentiable or is approximately totally differentiable a.e. should have an obvious meaning.

• The class of approximately totally differentiable a.e. functions was characterized by Hassler Whitney in [2], pp. 144-147 (the statement of Whitney is slightly different though equivalent to the one reported in [1] pp. 93-94), by the following theorem 1: let $u: E \to \Bbb R$ be measurable, $E \subseteq \Bbb R^n$. Then the following conditions are equivalent:
  (a) $u$ is approximately totally differentiable a.e. in $E$.
  (b) $u$ is approximately derivable with respect to each variable a.e. in $E$.
  (c) Denoting by $|\cdot|$ the Lebesgue measure, for each $\varepsilon > 0$ there exists a closed set $F\subseteq E$ and a function $v\in C^1(\Bbb R^n)$ such that $$ |E\setminus F|<\varepsilon \text{ and }u|_F = v|_F. $$

• An approximately totally a.e. differentiable map $\varphi:\Omega \to \varphi(\Omega)$ is a map whose each component $\varphi_i$, $i=1,\ldots, n$ is approximately totally differentiable a.e. on its domain of definition $\Omega$.

• Let $\varphi:\Omega \to \Bbb R^n$. We say that $\varphi$ satisfies the condition N (Lusin’s condition) if for any $E\subseteq\Omega$, $$ |E|=0 \implies |f(E)|=0. $$

• Let $\varphi:\Omega \to \Bbb R^n$, and $E\subseteq\Omega$. The Banach indicatrix is the function $N_\varphi(\cdot ,E):\Bbb R^n\to \Bbb N\cup\{\infty\}$ defined by $$ N_\varphi(y, E) = \sharp(\varphi^{−1}(y) \cap E). $$ where $\sharp$ denotes cardinality measure of the given set.

After those preliminaries we can try to answer the OP questions:

How much can we generalize the class in which $\varphi$ is allowed to lie in?

It is a main result of [1] (Theorem 2, §2 pp. 94-96) ,that a generalization of formula \eqref{1} holds for the class of approximately totally a.e. differentiable maps.
Precisely, theorem 2 of [1] states that if $\varphi:\Omega \to \Bbb R^n$ is any mapping, where $\Omega \subseteq \Bbb R^n$ is an arbitrary open subset, satisfying one of the conditions (a), (b), (c), of theorem 1, then we can redefine it on a subset of measure zero in such a way that the new $\varphi$ satisfies the Lusin condition $N$.
If $\varphi$ satisfies one of the conditions (a), (b), (c) and the condition $N$, then for every measurable function $f : \Bbb R^n \to \Bbb R$ and every measurable subset $E$ of $\Bbb R^n$ the following statements are true:

1. The functions $f(y)|D\varphi(y)|$ and $(f\circ\varphi^{-1}(x))N_\varphi(x, E)$ are measurable.

2. If moreover $f \ge 0$ then $$ \int\limits_E f(y)|D\varphi(y)|\mathrm{d}y = \int\limits_{\Bbb R^n} f\circ\varphi^{-1}(x)N_\varphi(x, E)\mathrm{d}x. \label{2}\tag{2} $$

3. If one of the functions $f(y)|D\varphi(y)|$ and $(f\circ\varphi^{-1}(x))N_\varphi(x, E)$ is integrable then so is the other (integrability of $f |D\varphi|$ concerns the set $E$) and the formula of \eqref{2} holds.

Note that

• Formula \eqref{2} is proved first for non-negative functions $f\ge 0$: the general case follows by the decomposition $f= f^+ − f^−$ ([1], §2 p. 96).

• I have modified the notation of [1] in order to show how formula \eqref{2} is a generalization of formula \eqref{1}, since this last one, proposed by the OP, has a non standard structure (even if it is perfectly equivalent to the standard one).

Is it enough that we have, says, a bijection $\varphi\in W_{\text{loc}}^{1,\infty}(\Omega ;\Bbb R^n)$ or even $\varphi\in W_{\text{loc}}^{1,1}(\Omega ;\Bbb R^n)$?

As recalled by Hajłasz ([1], example p. 94, and §3 p. 96), since the partial derivatives of $\varphi\in W_{\text{loc}}^{1,1}(\Omega ;\Bbb R^n)$ maps are defined a.e., these satisfy conditions (b) and (c) of theorem 1, implying that theorem 2 (and formula \eqref{2}) holds for them, so $\varphi\in W_{\text{loc}}^{1,1}(\Omega ;\Bbb R^n)$ is sufficient for the validity of formula \eqref{2}. Moreover Hajłasz ([1], §3 p. 96-98) is able to strengthen the theorem for these maps: however, this requires the same modification mechanism used in the general case, since there are continuous $W_{\text{loc}}^{1,1}(\Omega ;\Bbb R^n)$ maps which do not satisfy the Lusin's condition $N$.

How much does the result depends on the domain $\Omega$? Is there a big difference between a compact and an open domain? Does the regularity of the boundary $\partial \Omega$ play any role?

As you can see in the hypotheses of theorem 2, the domain $\Omega$ is only assumed to be an arbitrary open subset of $\Bbb R^n$ and it seems that its proof does not depend on the boundary structure (regularity) of the domain nor on its compactness (provided $\Omega$ has a non void interior, i.e. is compact in the sense that it has a compact closure). Thanks to Behnam Esmayli for clarifying this point in his comment.

Bibliography

[1] Piotr Hajłasz (1993), "Change of variables formula under minimal assumptions", Colloquium Mathematicum, 64, n. 1, pp. 93-101, ISSN 0010-1354; 1730-6302/e, DOI 10.4064/cm-64-1-93-101, MR1201446, Zbl 0840.26009.

[2] Hassler Whitney (1951), "On totally differentiable and smooth functions", Pacific Journal of Mathematics, Vol. 1 (1951), No. 1, 143–159, ISSN 0030-8730, DOI: 10.2140/pjm.1951.1.143, MR0043878, Zbl 0043.05803.