Not so easy optimization of variables?

What is the maximum value of $x^2+y^2$, where $(x,y)$ are solutions to $2x^2+5xy+3y^2=2$ and $6x^2+8xy+4y^2=3$. (calculus is not allowed). I tried everything I could but whenever I got for example $or$ $x^2+y^2=f(y)$ or $f(x)$ the function $f$ would always be a concave up parabola, so I could not find a maximum for either variable. However, I also don't see how you could solve it if you leave both variables on one side. And by the way I know that you can solve for $x$ and $y$ using the quadratic formula and get $4$ different solutions but I am looking for a much more efficient way than that.

This question came from a math competition from the Math Honor Society, Mu Alpha Theta.

The two constraints are conic sections. These can intersect in at most 4 points! So you only have to check these. Multiply the first constraint by 3, the second by 2 and subtract. The result is $ 6x^2 + xy - y^2 = (3x-y)(2x+y) = 0 $. So $ y=3x $ or $ y = -2x $. Substituting this back into either of the constraints will give you x and y. Check which gives the larger $ x^2 + y^2 $.

Edit: my suggestion does not work because a solution to the third eqn need not satisfy the original two eqns.

Multiply the first by $8$ to get: $16x^2+40xy+24y^2=16$

Multiply the second by $5$ to get: $30x^2+40xy+20y^2=15$

Subtract the two to get: $14x^2 - 4y^2 = -1$

Thus, you have to maximize $-\frac{1}{14}+\frac{4y^2}{14}+y^2$

Now it should be doable...

Since this question popped up to the top of the stack recently, I thought I'd add another answer, since there is a way to solve the problem without needing to determine the intersection points themselves (although they can be obtained with a small amount of additional calculation).

The hyperbola is $ \ 2x^2 + 5xy + 3y^2 = 2 \ $ , the ellipse is $ \ 6x^2 + 8xy + 4y^2 = 3 \ $ .

The conic sections described by the given equations, although rotated with respect to the coordinate axes, are centered on the origin, and thus retain their symmetry about the origin ["if the point $ \ (x,y) \ $ lies on the curve, so does $ \ (-x , -y) $ " ] . The function $ \ x^2 + y^2 \ $ , the "distance-from-the-origin-squared" function, is also symmetric about the origin. This means that its extrema on such curves will lie at points so symmetrically placed, and thus will lie on lines passing through the origin, $ \ y = mx \ $ (or, in certain situations, on the $ \ y-$ axis). I suspect this is the key observation that the poser of the problem may have intended, since this greatly reduces the amount of calculation, as we shall see. (It is understandable that this might be missed, as the effect of symmetries in optimization seems to be rather under-emphasized in teaching the topic; I found this through my own experience with such problems.)

Applying this additional condition, the function to be optimized becomes $ \ x^2 + (mx)^2 \ = \ (m^2 + 1) \ x^2 \ . $ While it still appears that we will need to solve for two variables, this condition permits us to eliminate one of them. The equations of the conic sections become

$$ 2x^2 \ + \ 5mx^2 \ + \ 3m^2x^2 \ = \ 2 \ \ \text{and} \ \ 6x^2 \ + \ 8mx^2 \ + \ 4m^2x^2 \ = \ 3 \ \ . $$

Dividing both equations through by $ \ x^2 \ , $ multiplying each equation by the appropriate constant, and equating the two, we obtain

$$ 6 \ + \ 15m \ + \ 9m^2 \ = \ \frac{6}{x^2} \ = \ \ 12 \ + \ 16m \ + \ 8m^2 $$

$$ \Rightarrow \ \ m^2 \ - \ m \ - \ 6 \ = \ ( m + 2 ) \ ( m - 3 ) \ = \ 0 \ \ \Rightarrow \ \ m \ = -2 \ , \ 3 \ \ , $$

thereby confirming the factorization and the lines through the origin shown by Bill Kleinhans (and André Nicolas in the other posting of this problem). We next return to either of the conic section equations to write

$$ 3m^2 \ + \ 5m \ + \ 2 \ = \ \frac{2}{x^2} \ \ \Rightarrow \ \ x^2 \ = \ \frac{2}{3m^2 + 5m + 2} \ \ \text{or} $$

$$ 4m^2 \ + \ 8m \ + \ 6 \ = \ \frac{3}{x^2} \ \ \Rightarrow \ \ x^2 \ = \ \frac{3}{4m^2 + 8m + 6} , $$

allowing us to write our function to be extremized as

$$ \frac{2 \ (m^2 + 1) }{3m^2 + 5m + 2} \ \ \text{or} \ \ \frac{3 \ (m^2 + 1) }{4m^2 + 8m + 6} \ \ . $$

Inserting our two slope values into either of these expressions yields

$$ m \ = \ -2 \ \ \rightarrow \ \ \frac{5}{2} \ \ , \ \ m \ = \ 3 \ \ \rightarrow \ \ \frac{5}{11} \ \ . $$

Hence, the minimum value for $ \ x^2 + y^2 \ , \ \frac{5}{11} \ , $ occurs at the intersection points of the two conic sections lying on the line $ \ y = 3x \ , $ and the maximum value, $ \ \frac{5}{2} \ , $ occurs for those intersections on the line $ \ y = -2x \ . $ The graph below illustrates the situation.

Although they are not needed to answer the question, we can determine the intersection points themselves from the foregoing:

$$ m \ = \ -2 \ \ \rightarrow \ \ x^2 \ = \ \frac{1}{2} \ \ \Rightarrow \ \ ( \ \pm \frac{\sqrt{2}}{2} \ , \ \mp \sqrt{2} \ ) \ \ ; $$

$$ m \ = \ 3 \ \ \rightarrow \ \ x^2 \ = \ \frac{1}{22} \ \ \Rightarrow \ \ ( \ \pm \frac{1}{\sqrt{22}} \ , \ \pm \frac{3}{\sqrt{22}} \ ) \ \ . $$

These do in fact agree with the extremal values for the "distance-squared" function.

$$ \ \ $$

The approach described by response will also work (it was the first method I looked at, as well), although it still needs to be taken together with the rest of the equations. The intersection points do lie on the "vertical" hyperbola $ \ 4y^2 - 14x^2 = 1 \ $ (shown in magenta in the graph above), and the symmetrical arrangement of the intersections can again be used to find the extrema of our function and locate the points:

$$ 4m^2x^2 \ - \ 14x^2 \ = \ 1 $$

$$ \Rightarrow \ \ 3m^2 \ + \ 5m \ + \ 2 \ = \ \frac{2}{x^2} \ = \ 8m^2 \ - \ 28 \ \ \Rightarrow \ \ 5m^2 \ - \ 5m \ - \ 30 \ = \ 0 \ \ \text{or} $$

$$ \Rightarrow \ \ 4m^2 \ + \ 8m \ + \ 6 \ = \ \frac{3}{x^2} \ = \ 12m^2 \ - \ 42 \ \ \Rightarrow \ \ 8m^2 \ - \ 8m \ - \ 48 \ = \ 0 \ \ , $$

leading us once more to the slope values we obtained above, to be used in the function $ \ \frac{m^2 + 1}{2 \ (2m^2 - 7 )} \ . $