Use MySQL spatial extensions to select points inside circle

There are no geospatial extension functions in MySQL supporting latitude / longitude distance computations. There is as of MySQL 5.7.

You're asking for proximity circles on the surface of the earth. You mention in your question that you have lat/long values for each row in your flags table, and also universal transverse Mercator (UTM) projected values in one of several different UTM zones. If I remember my UK Ordnance Survey maps correctly, UTM is useful for locating items on those maps.

It's a simple matter to compute the distance between two points in the same zone in UTM: the Cartesian distance does the trick. But, when points are in different zones, that computation doesn't work.

Accordingly, for the application described in your question, it's necessary to use the Great Circle Distance, which is computed using the haversine or another suitable formula.

MySQL, augmented with geospatial extensions, supports a way to represent various planar shapes (points, polylines, polygons, and so forth) as geometrical primitives. MySQL 5.6 implements an undocumented distance function st_distance(p1, p2). However, this function returns Cartesian distances. So it's entirely unsuitable for latitude and longitude based computations. At temperate latitudes a degree of latitude subtends almost twice as much surface distance (north-south) as a degree of longitude(east-west), because the latitude lines grow closer together nearer the poles.

So, a circular proximity formula needs to use genuine latitude and longitude.

In your application, you can find all the flags points within ten statute miles of a given latpoint,longpoint with a query like this:

 SELECT id, coordinates, name, r,
        units * DEGREES(ACOS(LEAST(1.0, COS(RADIANS(latpoint))
                  * COS(RADIANS(latitude))
                  * COS(RADIANS(longpoint) - RADIANS(longitude))
                  + SIN(RADIANS(latpoint))
                  * SIN(RADIANS(latitude))))) AS distance
   FROM flags
   JOIN (
        SELECT 42.81  AS latpoint,  -70.81 AS longpoint, 
               10.0 AS r, 69.0 AS units
        ) AS p ON (1=1)
  WHERE MbrContains(GeomFromText (
        CONCAT('LINESTRING(',
              latpoint-(r/units),' ',
              longpoint-(r /(units* COS(RADIANS(latpoint)))),
              ',', 
              latpoint+(r/units) ,' ',
              longpoint+(r /(units * COS(RADIANS(latpoint)))),
              ')')),  coordinates)

If you want to search for points within 20 km, change this line of the query

               20.0 AS r, 69.0 AS units

to this, for example

               20.0 AS r, 111.045 AS units

r is the radius in which you want to search. units are the distance units (miles, km, furlongs, whatever you want) per degree of latitude on the surface of the earth.

This query uses a bounding lat/long along with MbrContains to exclude points that are definitely too far from your starting point, then uses the great circle distance formula to generate the distances for the remaining points. An explanation of all this can be found here. If your table uses the MyISAM access method and has a spatial index, MbrContains will exploit that index to get you fast searching.

Finally, the query above selects all the points within the rectangle. To narrow that down to only the points in the circle, and order them by proximity, wrap the query up like this:

 SELECT id, coordinates, name
   FROM (
         /* the query above, paste it in here */
        ) AS d
  WHERE d.distance <= d.r
  ORDER BY d.distance ASC 

UPDATE

Use ST_Distance_Sphere() to calculate distances using a lat/long

http://dev.mysql.com/doc/refman/5.7/en/spatial-convenience-functions.html#function_st-distance-sphere

This assumes the coordinates in the table are stored as a POINT() datatype in a column labeled 'point'. The function X(point) and Y(point) extract the latitude and longitude values from the point value respectively.

SET @lat = the latitude of the point
SET @lon = the longitude of the point
SET @rad = radius in Kilometers to search from the point
SET @table = name of your table

SELECT
    X(point),Y(point),*, (
      6373 * acos (
      cos ( radians( @lat ) )
      * cos( radians( X(point) ) )
      * cos( radians( Y(point) ) - radians( @lon ) )
      + sin ( radians( @lat ) )
      * sin( radians( X(point) ) )
    )
) AS distance
FROM @table
HAVING distance < @rad

If you want to do it in miles, replace the constant 6373 with 3959

For those wanting to reduce the query syntax, here's a common implementation of a user defined MySQL function for implementing a distance function based on the Haversine formulae.

CREATE FUNCTION HAVERSINE ( coord1 POINT, coord2 POINT )
RETURNS DOUBLE
DETERMINISTIC
BEGIN
    DECLARE dist DOUBLE;
    SET rlat1 = radians( X( coord1 ) );
    SET rlat2 = radians( X( coord2 ) );
    SET rlon1 = radians( Y( coord1 ) );
    SET rlon2 = radians( Y( coord2 ) );
    SET dist  = ACOS( COS( rlat1 ) * COS( rlon1 ) * COS( rlat2 ) * COS( rlon2 ) + COS( rlat1 ) * SIN( rlon1 ) * COS( rlat2 ) * SIN( rlon2 ) + SIN( rlat1 ) * SIN( rlat2 ) ) * 6372.8;
    RETURN dist;
END

Buffers won't help you much in MySQL < 5.6, since buffer is a polygon, and polygon operations in MySQL < 5.6 are implemented as "Minimal Bounding Rectangles" (MBR), which are pretty useless.

Since MySQL 5.6, the full non-MBR st_* operations were implemented. But the best solution for you, in case of circle, is to use undocumented function st_distance:

select *
from waypoints
where st_distance(point(@center_lon, @center_lat), coordinates) <= radius;

It was hard to find, since it's undocumented :-) But it's mentioned on this blog, whose author also filled the mentioned bugreport. There are caveats though (citing the blog):

The bad news is:

1) All functions still only use the planar system coordinates. Different SRIDs are not supported.

2) Spatial indexes (RTREE) are only supported for MyISAM tables. One can use the functions for InnoDB tables, but it will not use spatial keys.

Point 1) means that the unit of distance will be the same as the unit of coordinates (degrees in case of WGS84). If you need distance in meters, you have to use projected coordination system (e.g. UTM or similar) that has units corresponding to meters.

So, in case you don't want to go with these caveats, or in case of MySQL < 5.6, you will have to write your own custom distance function.

for the sake of completeness, as of MySQL 5.7.6. you can use the ST_Distance_Sphere function which achieves the same result:

SET @pt1 = ST_GeomFromText('POINT(12.3456 34.5678)');

SELECT * from 
(SELECT * ,(ST_Distance_Sphere(@pt1, location, 6373)) AS distance FROM mydb.Event ORDER BY distance) x WHERE x.distance <= 30;

In this case, we provide the approximate radius of the Earth in kilometers (6373) and a point (@pt1). This code will calculate the distance (in kilometers) between that point (long 12.3456, lat 34.5678) and all the points contained in the database where the distance is 30km or less.