how to place a rope with a given length within an orthogonal triangle (see picture)

I would like to know what is the optimal way of placing the red rope of a given length $p$, where $\sqrt{2}<p<2$, in the orthogonal triangle ABCA, so that the green area is minimized (see attached figure). A condition is that the rope along with the segment AB must formulate a convex set.

So, is there a unique optimal way of placing the rope? Are there only two possible optimal (symmetrical) ways? Are there more? Are there infinite? Why? Most importantly, what is the value of the minimized area, for a given length $p$?

Just by intuition, I would say that I would place the rope for some of its portion alongside segment AC (just for a small portion) and then go for a straight shot to point B (or do the symmetrical trajectory by going for a straight shot from point A to some point on CB and then go down to point B along side CB). Portions of the rope are allowed to lie on the the perimeter of the triangle ABCA, but not allowed to exceed the perimeter of the triangle. The rope can also be placed in such ways that it exhibits corners, as long as it formulates a convex set with segment AB.

The endpoints of the rope are nailed down to points A and B.

Thank you for your feedback.

Let's first solve the problem for polylines with a fixed (possibly high) number of nodes $P_0=A, P_1, \ldots, P_n=B$. The conditions that the polyline has the desired length ans is convex and within the given triangle are closed, so that the set of tuples $(P_1,\ldots, P_n)$ is compact. As the area in question is a continuous function of the $P_i$, we conclude that the minimum is actually attained. Let $S_n(p)$ be the minimal area obtainable with a polyline of $n$ nodes.

Consider what happens if we move one node $P_i$ (the top node in the following illustration) of a polyline. By the length condition, $P_i$ is restricted to an elliptical arc with foci $P_{i-1}$ and $P_{i+1}$ (red ellipsis). The ellipsis may degenerate to a line segment, in which case $Pi$ must be on $P_{i-1}P_{i+1}$ and is redundant. Assume this is not the case. By the convexity condition, $P_i$ is limited in its position by the prolongations of $P_{i-2}P_{i-1}$ and of $P_{i+2}P_{i+1}$ (red line segments). Similarly, in the case of $i=1$ or $i=n-1$ we have the triangle sides $AC$ or $BC$ as red lines.

If moving $P_i$ under these constraints can make the green triangle smaller, we also make the overall enclosed area smaller. For a minimizing polyline, this is not possible. We conclude that in a minimizing polyline, $P_i$ is at one of the two intersections of the red lines with the ellipsis, i.e., either $P_{i-1}$ is on the line segment $P_{i-2}P_i$ or $P_{i+1}$ is on the line segment $P_iP_{i+2}$ (or, in case of $i=1$: $P_1$ is on $AC$; or, in case $i=n-1$: $P_{n-1}$ is on $BC$). Hence, if $1<i<n-1$, one of the points $P_{i-1},P_i,P_{i+1}$ is redundant.

We conclude that the optimal result for polylines with $n\gg1$ nodes is the same as the optimum for $n=3$, i.e., $$ S_n(p)=S_3(p)\qquad \text{for }n\ge 3.$$ And for the latter, we need only consider the cases where $P_1\in AC$ and $P_2\in BC$.

Hence the optimum over all polylines looks somewhat like this:

Let $\theta=\angle P_2P_1C$. Note that $\angle CP_2P_1=90^\circ -\theta$. Infinitesimally moving $P_1$ so that $u$ changes by an infinitesimal amount $\mathrm du$, will change $v$ by $-\cos\theta\, \mathrm du$ and the green area by $\frac 12 v \sin\theta\, \mathrm du$. Likewise, changing $w$ by an infinitesimal $\mathrm dw$ will change $v$ by $-\sin\theta\, \mathrm dw$ and the green area by $\frac 12 v \cos\theta\, \mathrm dw$. In order to keep $p$ constant, we must have $(1-\cos\theta)\mathrm du+(1-\sin\theta)\mathrm d w=0$. This makes the change in area $$ \begin{align}\mathrm dA &= \frac 12 v \sin\theta\, \mathrm du+\frac 12 v \cos\theta\, \mathrm dw\\ &=\frac v2\left(\sin\theta\,\mathrm du+\cos\theta\,\mathrm dw\right)\\ &=\frac v2\left(\sin\theta\,\mathrm du-\frac{1-\cos\theta}{1-\sin\theta}\cos\theta\,\mathrm du\right)\\ &=\left(\sin\theta(1-\sin\theta)-\cos\theta(1-\cos\theta)\right)\frac{v\,\mathrm du}{2(1-\sin\theta)}\\ &=(\sin\theta-\cos\theta)(1-\sin\theta-\cos\theta)\frac{v\,\mathrm du}{2(1-\sin\theta)}\\\end{align}$$ Note that $1-\sin\theta-\cos\theta<0$ for $0^\circ <\theta<90^\circ$. Therefore, $\frac{\mathrm dA}{\mathrm du}$ is negative if $\sin\theta<\cos\theta$ and positive if $\sin\theta>\cos\theta$. We conclude that the minimum is attained when and only when $\sin\theta=\cos\theta$, i.e., when $P_1P_2\|AB$. Let $h$ be the height of the trapezoid $ABP_2P_1$. Then its bottom line is $\sqrt 2$ and its top is $v=\sqrt 2-2h$. Also, $u=w=h\sqrt 2$, so that $$h=\frac{p-\sqrt 2}{2\sqrt 2-2} $$ and ultimately $$ S_3(p)=h\cdot\frac{v+\sqrt 2}{2}=\frac{-p^2+4p+2-4\sqrt 2}{12-8\sqrt 2}.$$ One verifies that this quadratic is an increasing function of $p\in[\sqrt 2, 2]$.

Finally, let us note that the restriction to polylines poses no problems: Assume an arbitrary curve of length $p$ produces an area $\tilde S <S_3(p)$. Then this curve can be approximated by a circumscribed and slightly longer polyline of length $p'>p$, with an area $\tilde S'$ exceeding that for the curve by an arbitrarily small amount (if only we take a large enough number of nodes). In particular, we still have $S_3(p')\le \tilde S'<S_3(p)$, contradicting the fact that $S_3$ is strictly increasing.

Thus ultimately

$$ {S_{\text{opt}}(p)=\frac{-p^2+4p+2-4\sqrt 2}{12-8\sqrt 2}}$$

and the optimum is attained for the trapezoid.