Playing a custom avi data stream using QtMultimedia

I need to play back a custom AVI file that contains a classic video stream, an audio stream but also a custom data stream.

The custom stream contains data that is visualized by some custom widgets; those widgets only need that each custom frame is written in a buffer at the proper time.

Our application is based on Qt, and already uses QMediaPlayer/QVideoWidget to play traditional videos, but the additional custom stream makes things more complicated, because AFAIK QMediaPlayer only plays the video/audio and ignores everything else.

I would like to avoid to reinvent the entire qt-multimedia, but I am not sure how to make the best out of the available Qt classes.

My ideas so far are:

1. Write a custom media player class that demuxes and decodes the video using ffmpeg, implements the timing, uses QAudioOutput for playing the audio, produces a stream of QVideoFrames to be played on the video and write the custom data to some buffer for visualization.

   The problem: In order to avoid writing the code to rescale/convert the video frames, I would like to reuse QVideoWidget, but it seems to work only with the "real" QMediaPlayer.

2. Demux the input file and feed QMediaPlayer with the AV streams. Demux the input with ffmpeg (possibly leaving the decoding to the Qt backend), have one QIODevice to retrieve only the video/audio streams from the input file and another one to retrieve the data stream. Play the video/audio with QMediaPlayer.

                 +-------+                          
                 | QFile |                          
                 +---^---+                          
                     |                              
                  inherits                          
                     |                              
           +--------------------+
           |    MyAviDemuxer    |
           |                    |
           |  holds a queue of  |
           |  demuxed packets   |
           +--------------------+
           |                    |
     readDataPacket      readVideoPacket
           |                    |
   +-------v--------+  +--------v-----------+            +-----------+
   | MyCustomReader |  | MyVideoAudioStream +--inherits--> QIODevice |
   +----------------+  +--------+-----------+            +-----------+
                                |       
                             setMedia                  
                                |                  
                        +-------v-------+           
                        | QMediaPlayer  |           
                        +---------------+           
   

   The problem: synchronize the timing of the data stream with QMediaPlayer, handle headers and metadata correctly.

I am slightly inclined to option 1, just because it gives me more control, but I am wondering if I missed an easier solution (even Windows-only).

I understand you have quite the customized class structure but maybe you could use some advice from a coding newbie. I think you should be using some more basic existing data types together with your custom classes.

Solution for: synchronizing the timing of the data stream with QMediaPlayer:
Try using some timer threads (combination of Thread and timer). Make one that uses whatever the stream index is of MyVideoAudioStream (using time as the variable in the index) and "Mycustomreader" (using an array of packets with time as the variable in the index) as it's body. Add into the body some logic that cycles through the position( @param:time) in QMediaPlayer. From this,you could parse through the execution code of both at the same time. As time increases,the position in QMediaPlayer and the index of your stream would increase.

If you don't have an index or position in your custom stream,I highly suggest you create one.

It looks like Qt actually already supports the concept of data streams to some degree - http://doc.qt.io/qt-5/qmediastreamscontrol.html#details shows that it's among the selectable types of streams for a qmediastreamscontrol.

Other docs including http://doc.qt.io/qt-5/qmediaserviceproviderplugin.html suggest that you could create a QMediaServiceProviderPlugin that implements video and audio QMediaControl interfaces (possibly by subclassing an existing media service provider), an also create your own QMediaControl interface subclass to create a Control to handle your raw data.

Hopefully implementing in this way would allow you to use existing facilities for splitting apart the streams, handling headers, and similar functionality.

Unfortunately the specifics of building a QMediaService seems to be "outside of the scope of this documentation and support on the relevant mailing lists or IRC channels should be sought." (http://doc.qt.io/qt-5/qmediaservice.html#details). The source (http://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/multimedia) could be of some use doing this, however, in addition to, perhaps, the source at http://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/plugins, which includes the directshow / gstreamer / coreaudio plugins.

In any case I would try to subclass and re-implement as little as possible