Converting YUV into BGR or RGB in OpenCV

In newer version of OPENCV there is a built in function can be used to do YUV to RGB conversion

cvtColor(src,dst,CV_YUV2BGR_YUY2);

specify the YUV format after the underscore, like this CV_YUYV2BGR_xxxx

It looks to me like you're decoding a YUV422 stream as YUV444. Try this modification to the code you provided:

for(int i = 0, j=0; i < 1280 * 720 * 3; i+=6, j+=4)
{
    m_RGB->imageData[i] = pData[j] + pData[j+3]*((1 - 0.299)/0.615);
    m_RGB->imageData[i+1] = pData[j] - pData[j+1]*((0.114*(1-0.114))/(0.436*0.587)) - pData[j+3]*((0.299*(1 - 0.299))/(0.615*0.587));
    m_RGB->imageData[i+2] = pData[j] + pData[j+1]*((1 - 0.114)/0.436);
    m_RGB->imageData[i+3] = pData[j+2] + pData[j+3]*((1 - 0.299)/0.615);
    m_RGB->imageData[i+4] = pData[j+2] - pData[j+1]*((0.114*(1-0.114))/(0.436*0.587)) - pData[j+3]*((0.299*(1 - 0.299))/(0.615*0.587));
    m_RGB->imageData[i+5] = pData[j+2] + pData[j+1]*((1 - 0.114)/0.436);
}

I'm not sure you've got your constants correct, but at worst your colors will be off - the image should be recognizable.