how to solve binary form $ax^2+bxy+cy^2=m$, for integer and rational $ (x,y)$

EDIT, March 2016: I wrote a program that finds this stuff quickly, also identifies the "fundamental" solutions from which all others may be derived. Good if the question is to solve for one specific target value. Oh, I told it to use only $x,y \geq 0.$ Going backwards by the automorphism matrix does give solutions with smaller absolute values, for a while. This method includes an effective bound on the entries that allows proving that all such "fundamental" solutions have been found.

jagy@phobeusjunior:~$ ./Pell_Target_Fundamental_A
8  15
9  17

  25^2 - 69 3^2 = 4

3 x^2 + 3 x y -5 y^2 = 55

Thu Mar 31 11:12:43 PDT 2016

x:  4  y:  1 ratio: 0.25  fundamental 
x:  5  y:  4 ratio: 0.8  fundamental 
x:  12  y:  13 ratio: 1.08333  fundamental 
x:  25  y:  28 ratio: 1.12  fundamental 
x:  47  y:  53 ratio: 1.12766
x:  100  y:  113 ratio: 1.13
x:  291  y:  329 ratio: 1.13058
x:  620  y:  701 ratio: 1.13065
x:  1171  y:  1324 ratio: 1.13066
x:  2495  y:  2821 ratio: 1.13066
x:  7263  y:  8212 ratio: 1.13066
x:  15475  y:  17497 ratio: 1.13066
x:  29228  y:  33047 ratio: 1.13066
x:  62275  y:  70412 ratio: 1.13066
x:  181284  y:  204971 ratio: 1.13066
x:  386255  y:  436724 ratio: 1.13066
x:  729529  y:  824851 ratio: 1.13066
x:  1554380  y:  1757479 ratio: 1.13066
x:  4524837  y:  5116063 ratio: 1.13066

Thu Mar 31 11:13:03 PDT 2016
8  15
9  17
Inverse of given automorphism of quadratic form:
17  -15
-9  8
jagy@phobeusjunior:~$ 

Given a solution $(x,y),$ you get another solution from $$ (8x+15y,9x+17y). $$ Repeat forever. Going backwards is $$ (17x-15y,-9x+8y). $$

For $y=1,$ it should have been $3x^2 + 3 x - 60 = 0,$ so $x^2 + x - 20 = 0.$ Also you got the xy order wrong at the end, start with $$ (4,1)(-5,1)(-4,-1)(5,-1). $$ These are just $$ \pm (4,1); \pm (5,-1). $$

EEDDIITT: unfortunate, you missed two infinite strings. I carefully drew Conway's topograph for one cycle. The four solutions with small entries, in distinct orbits, are $(4,1), (5,-1),(5,4), (9,-4).$

The four strings, continued in both directions, become $$ \ldots (32972,-17497);(1321,-701);(53,-28); (4,1); (47,53) ; (1171,1324); (29228,33047); \ldots $$ $$ \ldots (62275,-33047);(2495,-1324);(100,-53);(5,-1); (25,28) ; (620,701); (15475,17497); \ldots $$ $$ \ldots (15475,-8212);(620,-329);(25,-13);(5,4); (100,113) ; (2495,2821); (62275,70412); \ldots $$ $$ \ldots (132687,-70412);(5316,-2821);(213,-113);(9,-4); (12,13) ; (291,329); (7263,8212); \ldots $$

I decided to make a very careful tree outline so that I could fill in the relevant bits of the Conway topograph and have it clearly visible. In brief, one diagram shows behavior along the "river." However, specific representations of a number with larger absolute value happen a bit away from the river, in trees... I have carefully shown how the four representations of $55$ with small entries, arise in the diagram.

Here is a jpeg of the relevant portion of Conway's topograph for this problem. This is from The First Lecture in The Sensual Quadratic Form. Note that the 2 by 2 matrix

$$ A \; = \; \left( \begin{array}{rr} 8 & 15 \\ 9 & 17 \end{array} \right) $$ is visible towards the right of the diagram, the value $3$ with coordinate vector $$ \left( \begin{array}{r} 8 \\ 9 \end{array} \right), $$ and on the lower right of that we find the value $-5$ with coordinate vector $$ \left( \begin{array}{r} 15 \\ 17 \end{array} \right). $$ That 2 by two matrix, which is where I got the formula $ (8x+15y,9x+17y), $ is the generator of the automorphism group of the quadratic form, also called the orthogonal group, the rotation group, etc. The traditional name for it was "automorph," not so many use that any more except me, i guess. The automorph also comes, quickly, from the Lagrange cycle method, illustrated in the computer output below that. The best book for that is Duncan A. Buell, Binary Quadratic Forms.

jagy@phobeusjunior:~/old drive/home/jagy/Cplusplus$ ./indefCycle 3 3 -5

  0  form              3           3          -5


           1           0
           0           1

To Return  
           1           0
           0           1

0  form   3 3 -5   delta  -1
1  form   -5 7 1   delta  7
2  form   1 7 -5   delta  -1
3  form   -5 3 3   delta  1
4  form   3 3 -5


  form   3 x^2  + 3 x y  -5 y^2 

minimum was   1rep   x = -1   y = -1 disc   69 dSqrt 8.3066238629  M_Ratio  7.666667
Automorph, written on right of Gram matrix:  
8  15
9  17
=========================================
jagy@phobeusjunior:~/old drive/home/jagy/Cplusplus$ 

Places I have talked about this or posted diagrams:

Generate solutions of Quadratic Diophantine Equation

Another quadratic Diophantine equation: How do I proceed?

How to find solutions of $x^2-3y^2=-2$?

Generate solutions of Quadratic Diophantine Equation

Finding all solutions of the Pell-type equation $x^2-5y^2 = -4$

Find all integer solutions for the equation $|5x^2 - y^2| = 4$

Maps of primitive vectors and Conway's river, has anyone built this in SAGE?

Infinitely many systems of $23$ consecutive integers

Finding integers of the form $3x^2 + xy - 5y^2$ where $x$ and $y$ are integers, using diagram via arithmetic progression

Small integral representation as $x^2-2y^2$ in Pell's equation

Solving the equation $ x^2-7y^2=-3 $ over integers