C++ conversion from NumPy array to Mat (OpenCV)

I am writing a thin wrapper around ArUco augmented reality library (which is based on OpenCV). An interface I am trying to build is very simple:

• Python passes image to C++ code;
• C++ code detects markers and returns their locations and other info to Python as tuple of dicts.

However, I couldn't figure out how to represent an image in Python to pass it to C++. For GUI and camera management I am going to use PyQt, so initially it is going to be QImage, but I can't simply pass it to OpenCV (or I can?). At first, I tried to use nested tuples to represent row, column and color of each pixel, so I ended up with this sample code:

using namespace cv;
namespace py = boost::python;

void display(py::tuple pix)
{
    /*
        Receive image from Python and display it.
    */
    Mat img(py::len(pix), py::len(pix[0]), CV_8UC3, Scalar(0, 0, 255));
    for (int y = 0; y < py::len(pix); y++)
        for (int x = 0; x < py::len(pix[y]); x++)
        {
            Vec3b rgb;
            for (int i = 0; i < 3; i++)
                rgb[i] = py::extract<int>(pix[y][x][i]);
            img.at<Vec3b>(Point(x, y)) = rgb;
        }
    imshow("Image", img);
    waitKey(0);
}

BOOST_PYTHON_MODULE(aruco)
{
    py::def("display", display);
}

It turned out to be painfully slow (a few seconds for a single frame), so I went googling and found solution that should be much faster: use NumPy arrays, so the code would look something like that:

void display(py::object array)
{
    Mat img;
    // ... some magic here to convert NumPy array to Mat ...
    imshow("Image", img);
    waitKey(0);
}

However, I have no idea how to convert NumPy Array (which in C++ level is just a Python Object) to OpenCV Mat. I would appreciate any help here.

Alternatively, maybe NumPy is not really needed, so I could just pass QImage Python object directly to C++ layer? Or maybe there is a different approach to this problem? Any advice is appreciated!

The best solution in your situation is using custom boost::python converter for cv::Mat object. OpenCV has Python wrapper and when you are using this wrapper you are operating on Numpy arrays - you don't even need to know that those arrays are converted to cv::Mat objects while "crossing the c++ <-> python border". Writing such converter for simple type is quite easy, however creating converter for cv::Mat isn't simple. Fortunetely someone else already did this - here is version for OpenCV 2.x and here for 3.x. If you are not familiar with boost::python converters, this article should help you.
Hope it helps, if you wil have any problems, let us know.

I wrote this example for who didn't know there is Boost Numpy module. You can see how to convert Mat to NDArray and vice versa. it will gives you idea the way of convert ndarray.

#define BOOST_PYTHON_STATIC_LIB
#define BOOST_LIB_NAME "boost_numpy35"
//#include <boost/config/auto_link.hpp>
#include <boost/python.hpp>
#include <boost/python/numpy.hpp>
#include <iostream>
#include <opencv2/opencv.hpp>

namespace py = boost::python;
namespace np = boost::python::numpy;

void Init() {
    // set your python location.
    wchar_t str[] = L"D:\\Anaconda3\\envs\\tensorflow_vision";

    Py_SetPythonHome(str);

    Py_Initialize();
    np::initialize();
}

np::ndarray ConvertMatToNDArray(const cv::Mat& mat) {
    py::tuple shape = py::make_tuple(mat.rows, mat.cols, mat.channels());
    py::tuple stride = py::make_tuple(mat.channels() * mat.cols * sizeof(uchar), mat.channels() * sizeof(uchar), sizeof(uchar));
    np::dtype dt = np::dtype::get_builtin<uchar>();
    np::ndarray ndImg = np::from_data(mat.data, dt, shape, stride, py::object());

    return ndImg;
}

cv::Mat ConvertNDArrayToMat(const np::ndarray& ndarr) {
    //int length = ndarr.get_nd(); // get_nd() returns num of dimensions. this is used as a length, but we don't need to use in this case. because we know that image has 3 dimensions.
    const Py_intptr_t* shape = ndarr.get_shape(); // get_shape() returns Py_intptr_t* which we can get the size of n-th dimension of the ndarray.
    char* dtype_str = py::extract<char *>(py::str(ndarr.get_dtype()));

    // variables for creating Mat object
    int rows = shape[0];
    int cols = shape[1];
    int channel = shape[2];
    int depth;

    // you should find proper type for c++. in this case we use 'CV_8UC3' image, so we need to create 'uchar' type Mat.
    if (!strcmp(dtype_str, "uint8")) {
        depth = CV_8U;
    }
    else {
        std::cout << "wrong dtype error" << std::endl;
        return cv::Mat();
    }

    int type = CV_MAKETYPE(depth, channel); // CV_8UC3

    cv::Mat mat = cv::Mat(rows, cols, type);
    memcpy(mat.data, ndarr.get_data(), sizeof(uchar) * rows * cols * channel);

    return mat;
}

int main()
{
    using namespace std;

    try
    {
        // initialize boost python and numpy
        Init();

        // import module
        py::object main_module = py::import("__main__");
        py::object print = main_module.attr("__builtins__").attr("print"); // this is for printing python object

        // get image
        cv::Mat img;
        img = cv::imread("Lenna.jpg", cv::IMREAD_COLOR);
        if (img.empty())
        {
            std::cout << "can't getting image" << std::endl;
            return -1;
        }

        // convert Mat to NDArray
        cv::Mat cloneImg = img.clone(); // converting functions will access to same data between Mat and NDArray. so we should clone Mat object. This may important in your case.
        np::ndarray ndImg = ConvertMatToNDArray(cloneImg);

        // You can check if it's properly converted.
        //print(ndImg);

        // convert NDArray to Mat
        cv::Mat matImg = ConvertNDArrayToMat(ndImg); // also you can convert ndarray to mat.

        // add 10 brightness to converted image
        for (int i = 0; i < matImg.rows; i++) {
            for (int j = 0; j < matImg.cols; j++) {
                for (int c = 0; c < matImg.channels(); c++) {
                    matImg.at<cv::Vec3b>(i, j)[c] += 10;
                }
            }
        }

        // show image
        cv::imshow("original image", img);
        cv::imshow("converted image", matImg);
        cv::waitKey(0);
        cv::destroyAllWindows();
    }
    catch (py::error_already_set&)
    {
        PyErr_Print();
        system("pause");
    }

    system("pause");
    return 0;
}