How to find Maximum of the following non-linear function of $x,y$ without using calculus

From the diagram we see, it is sufficient to solve for $x\ge 0$, $y\ge 0$ and $4x+3y\le 75$. We use AM-GM inequality.

We write $$z=\frac{3}{2}y \left(4x+\frac{2}{3}y \right)=\frac{9}{14}\cdot \frac{7}{3}y\left( 4x+\frac{2}{3}y\right)$$

Then by AM-GM, $$\left( \frac{7}{3}y\left( 4x+\frac{2}{3}y\right) \right) ^{1/2} \le \frac{\tfrac{7}{3}y + (4x+\tfrac{2}{3}y)}{2} \le \frac{75}{2}$$ $$\Rightarrow z \le \frac{9}{14}\cdot \left( \frac{75}{2} \right)^2 =\frac{15^4}{56}$$ The equality is achieved for $\tfrac{7}{3}y = (4x+\tfrac{2}{3}y)$ which gives the point $(x,y)=(\tfrac{375}{56},\tfrac{225}{14})$.

To find the critical points of your multivariate function $z$, you need to find where the partial derivatives are zero (or undefined):

$$\frac{\partial z}{\partial x}=6y=0\Rightarrow y=0$$ $$\frac{\partial z}{\partial y}=6x+2y=0\Rightarrow x=0$$

So $(0,0)$ is a critical point of $z$ on the entire plane. Normally, you would then check whether to discard this point based on whether it's in the region of interest, but it really doesn't matter since it's on the boundary of the region of interest.

The next thing to do is look for critical points on the boundary curves. To do that, we can parametrize the boundaries as follows:

$$(x,y)=(0,t_1),\quad t_1\in[0,25]$$ $$(x,y)=(t_2,0),\quad t_2\in[0,75/4]$$ $$(x,y)=((75-3t_3)/4,t_3),\quad t_3\in[0,25]$$

Taking these boundaries one at a time, we can find the critical points:

$$z(0,t_1)=t_1^2$$ $$\frac{{\rm d}z}{{\rm d}t_1}=2t_1=0\Rightarrow t_1=0$$

This value of $t_1$ is on the boundary of our curve, so again it doesn't really matter whether we mark down $(0,0)$ as a critical point. Next:

$$z(t_2,0)=0$$ $$\frac{{\rm d}z}{{\rm d}t_2}=0$$

Here we have a rather uncommon situation--every point on this curve is equal at zero. More importantly, they are all critical points. It's possible that they are all at the maximum, so we'll keep this family of points in mind when we decide which is maximal. Finally:

$$z((75-3t_3)/4,t_3)=t_3^2+t_3(75-3t_3)\frac{3}{2}=-\frac{7t_3^2}{2}+\frac{225t_3}{2}$$ $$\frac{{\rm d}z}{{\rm d}t_2}=\frac{225}{2}-7t_3=0\Rightarrow t_3=\frac{225}{14}$$

This value of $t_3=225/14\approx 16.07$ is inside our limits for the parametrization, so $(375/56,225/14)$ is a critical point we must consider.

Finally, we must consider the endpoints of the boundary curves: $(0,0),(0,25),(75/4,0)$. You can now see why it doesn matter if we consider the $(0,0)$ from earlier--we have to consider it now anyways.

$$z(x,0)=0$$ $$z(0,25)=25^2=625$$ $$z(375/56,225/14)=\frac{225}{14}\left(6\frac{375}{56}+\frac{225}{14}\right)=\frac{50625}{56}\approx904.01$$

Thus $\frac{50625}{56}$ is the maximum value of the function on the given region, and it occurs at $(x,y)=(\frac{375}{56},\frac{225}{14})$.

Note: I've typed this up on mobile without my computer, so it likely that some of the numerical calulations may be off. For that reason, I reccomend following my steps with your own calculations.

First, as you may have observed from your diagram, the constraint $3x + 4y \leq 100$ is redundant. Indeed, if we multiply both hands of the constraint $4x + 3y \leq 75$ by $4/3$, we get $$\frac{16}{3}x + 4y \leq 100.$$ Furthermore, because of the constraint $x \geq 0$, then we have $3x + 4y \leq \frac{16}{3}x + 4y$. Hence, $3x + 4y \leq 100$ too.

Therefore, we will ignore the constraint $3x + 4y \leq 100$ from now onwards.

Let an optimal solution for $(x, y)$ be $(x_*, y_*)$. Then we know $x_* \geq 0$, $y_* \geq 0$, and $4x_* + 3y_* \leq 75$.

Claim. We have either $y_* = 0$ or $4x_* + 3y_* = 75$.

Proof. Suppose that $4x_* + 3y_* = 75 - r$ for some real number $r \geq 0$. Then $(x_* + \frac{1}{4}r, y_*)$ is in the feasible region, because it satisfies all three constraints:

$(x_* + \frac{1}{4}r) \geq x_* \geq 0$,
$y_* \geq 0$,
$4(x_* + \frac{1}{4}r) + 3y_* = 4x_* + 3y_* + r = 75$.

Hence, the value at $(x_* + \frac{1}{4}r, y_*)$ cannot exceed the optimal value. We know that the value at $(x_* + \frac{1}{4}r, y_*)$ is $$y_*\left(6\left(x_* + \frac{1}{4}r\right) + y_*\right) = 6x_*y_* + y_*^2 + \frac{3}{2}ry_*,$$ while the optimal value is $$y_*(6x_* + y_*) = 6x_*y_* + y_*^2.$$ As a consequence, we must have $$\frac{3}{2}ry_* \leq 0.$$ Recall that $r$ and $y_*$ are non-negative. Hence, we must have either $y_* = 0$ or $r = 0$. The latter implies $4x_* + 3y_* = 75$. $\square$

So, to find the optimal solution, we now know that we only need to check two cases:

Case 1: Suppose $y_* = 0$. In this case, the value is $0 \cdot (6x_* + 0) = 0$.
Case 2: Suppose $4x_* + 3y_* = 75$. Then $x_* = (75 - 3y_*)/4$, so the optimal value becomes \begin{align*} y_*(6x_* + y_*) &= 6x_*y_* + y_*^2 \\[0.5em] &= 6 \cdot \left(\frac{75 - 3y_*}{4}\right) \cdot y_* + y_*^2 \\[0.5em] &= \left(-\frac{9}{2}y_*^2 + \frac{225}{2}y_*\right) + y_*^2 \\[0.5em] &= -\frac{7}{2}y_*^2 + \frac{225}{2}y_*. \end{align*} The optimal solution for $y_*$ to achieve the maximum value of this quadratic polynomial can be easily found (for instance, by completing the square or taking derivative) to be $y_* = 225/14$. Substituting back, we get $x_* = 375/56$ and hence $$y_*(6x_* + y_*) = \frac{50625}{56}.$$