OpenCV template matching and transparency

What's the way OpenCV handles transparency in image during template matching?

The problem is that the template image needs to have transparent parts, because in the original image there could be anything at those places.

I tried all of the methods, and none of them yielded positive results (e.g. position of template in original image wasn't detected correctly).

It doesn't seem like OpenCV handles alpha the way you want it to.

You have two options:

1. Write your own cross-correlation method that will use the alpha channel
2. Transform your images so your alpha channel becomes irrelevant

Since the first option is straightforward, I will explore the second option here. I'm going to re-use the sample code I provided to a similar question earlier. If you apply cross-correlation directly to your images, the background interferes with the template matching (in particular, light background parts). If you play around with color channels, you will find that matching in the blue channel gives the correct result. This depends on the image content and isn't a consistent way to solve the problem.

Another option is to perform edge detection (e.g. Sobel) on the image and template, and perform cross-correlation then. Here are the edge detected images (I used the Sobel edge detector on the Luma channel in GIMP, and then some intensity stretching).

As you can see, the alpha channel here has become irrelevant, as most of the terrain has become zero intensity and will not contribute to the cross-correlation calculation. So now cross-correlation can be directly applied, giving the desired result:

misha@misha-desktop:~/Desktop/stackoverflow$ python cross-correlation.py map-blue.png building-maskz-blue.png 
(163, 244)

Finally, here's another related question.

PS. What game is this?

I have a slightly more brain-dead solution to this issue which actually seems to work reasonably well: Replace the template image's alpha channel with noise, which more or less makes the transparent regions statistically insignificant during the matching process.

For example, my use case involved searching for emoji characters in screen captures from iOS. The iOS keyboard background changes colors depending on context, which makes the matching process problematic if you commit to a particular background color in your template image.

Here's the raw template image on alpha:

Here's the processed template with noise filling in for the alpha channel:

I sent the processed template image through the Template Matching sample code provided in the OpenCV documentation. On either dark or light background, the match is found with reasonable confidence.

Searching on dark backgrounds:

Searching on light backgrounds:

In comparison, leaving the template's alpha channel transparent — or committing to a dark or light background — did not return acceptable matches.

OpenCV 3.0 offers native support for template matching with masked templates. Refer to the new documentation:

Parameters:

image ...

templ ...

result ...

method ...

mask Mask of searched template. It must have the same datatype and size with templ. It is not set by default.

[Slight Digression]

Note that template matching with masked reference images (the larger image) is not possible though. And that makes sense, given OpenCV uses FFT based template matching.

Therefore, if you need to perform template matching only at specific regions of your reference images, you will need to implement your own method for that or mask the output of cv::matchTemplate.

Implementing it from scratch should compensate for cases where you only want to search for the template at very specific regions (i.e.: around harris corners).