You can use the following : First we get the last saved image by checking which was the last modified image. Then check if last modified time is in the last few seconds. You may also have to check the exact location of where camera stores the image.

private boolean deleteLastFromDCIM() {

        boolean success = false;
        try {
            File[] images = new File(Environment.getExternalStorageDirectory()
                    + File.separator + "DCIM/Camera").listFiles();
            File latestSavedImage = images[0];
            for (int i = 1; i < images.length; ++i) {
                if (images[i].lastModified() > latestSavedImage.lastModified()) {
                    latestSavedImage = images[i];
                }
            }

                    // OR JUST Use  success = latestSavedImage.delete();
            success = new File(Environment.getExternalStorageDirectory()
                    + File.separator + "DCIM/Camera/"
                    + latestSavedImage.getAbsoluteFile()).delete();
            return success;
        } catch (Exception e) {
            e.printStackTrace();
            return success;
        }

    }

Unfortunately, some smart phones save images in another folder such as DCIM/100MEDIA. So can't rely to these solution. I prefer use this way:

String[] projection = new String[] {
     MediaStore.Images.ImageColumns._ID,
     MediaStore.Images.ImageColumns.DATA,
     MediaStore.Images.ImageColumns.BUCKET_DISPLAY_NAME,
     MediaStore.Images.ImageColumns.DATE_TAKEN,
     MediaStore.Images.ImageColumns.MIME_TYPE};     

final Cursor cursor = managedQuery(
     MediaStore.Images.Media.EXTERNAL_CONTENT_URI,projection, null, null,
     MediaStore.Images.ImageColumns.DATE_TAKEN + " DESC"); 

if(cursor != null){
     cursor.moveToFirst();
     // you will find the last taken picture here and can delete that
}

I tried to find out if a second copy exists and delete the copy. I used the above code to find the last taken picture.

Notice: Don't use cursor.close(); after using managedQuery, Leave the cursor for the Android system to manage and don't call that. You can see managedQuery()

Notice2: The managedQuery method is deprecated and it should be avoided, implement CursorLoaders instead.