How to programmatically remove a file from "share with me" in Google drive

The problem is that the user in question doesn't own the file. After a lot of digging I realised that what you want to do is to remove the permissions for the user on the file in question.

The first thing you need to do is run an about.get on the current user:

return service.About.Get().Execute();

This will give you the permission id of that user

"permissionId": "060305882255734372",

Once that is done you can then do a permissions.get on the file for that user:

var response = service.Permissions.Get(fileId, permissionId).Execute();

Response

{
 "kind": "drive#permission",
 "id": "06030588225573437",
 "type": "user",
 "role": "writer"
}

Which will give you the permission id on the file for the user in question.

Then you can delete the permission on the file for the user using permission.delete

var response = service.Permissions.Delete(fileId, permissionId).Execute();