Adding other video codecs / DVD support to JavaFX 2.2

Believe me, I feel and know your frustration. I have pondered this for a while, but I had to use un-straight means of solving my issues.

There are many ways around this, each with limitations but depends on what works for you:

1. Docs say WebView works with HTML5, which plays videos supported on the platform (Though sadly not flash). If using a webview to play video works for you, you can try this out. You can even draw over it with other nodes.

2. Portable VLC Player! If maybe you're developing some sort of projector/director app and you want fullscreen video, you can have portable VLC player play the video in fullscreen in one screen with it's controls in the other. Used this solution and it works quite well for mac and windows. :) Only thing is you can't draw nodes on the video as it's an external app, with just the illusion of fullscreen video of your app.

3. If you ever need to utilize the power of flash within your javafx 2.0 application, then use a swt-based browser(or something Like the DJ Project if you're a Swinger) as they support all features of your native browser.

I've now managed to compile MKV support into JavaFX successfully, and it does take some, but not a great deal of effort on the native layer also. See here for the discussion surrounding it, and here for the result submitted as a patch / JIRA ticket.

I've written a much more comprehensive guide on the process here which may be of interest to anyone else looking to go down this route.

What follows is my brief investigation before I actually seriously looked at compiling other media support in, though I'll leave it here for reference.

Now that JFX8 has been released and is completely open source, I've spent a bit of time looking at how this could be done, and whether it could be done without patching the JFX source. Unfortunately the answer to that latter point is an almost definite no, at least not without horrible bytecode manipulation hacks. I may look into this more practically at a later date, but I'll document what I've worked out so far from the source available.

The magic starts from the Media constructor, which is ultimately where the MediaException pops out from (with the MEDIA_UNSUPPORTED flag if you try to play an unsupported format.) From there it creates the Locator, whose constructor ensures that the URL is one that's supported. It's init() method is then called in a separate thread, which performs some sanity checking on the URL string, reads the file, then proceeds to try to work out what the format is.

The relevant code for this part of the method is thus:

if (scheme.equals("file") || scheme.equals("jar")) {
    InputStream stream = getInputStream(uri);
    stream.close();
    isConnected = true;
    contentType = MediaUtils.filenameToContentType(uriString); // We need to provide at least something
}

if (isConnected) {
    // Check whether content may be played.
    // For WAV use file signature, since it can detect audio format
    // and we can fail sooner, then doing it at runtime.
    // This is important for AudioClip.
    if (MediaUtils.CONTENT_TYPE_WAV.equals(contentType)) {
        contentType = getContentTypeFromFileSignature(uri);
        if (!MediaManager.canPlayContentType(contentType)) {
            isMediaSupported = false;
        }
    } else {
        if (contentType == null || !MediaManager.canPlayContentType(contentType)) {
            // Try content based on file name.
            contentType = MediaUtils.filenameToContentType(uriString);

            if (Locator.DEFAULT_CONTENT_TYPE.equals(contentType)) {
                // Try content based on file signature.
                contentType = getContentTypeFromFileSignature(uri);
            }

            if (!MediaManager.canPlayContentType(contentType)) {
                isMediaSupported = false;
            }
        }
    }

    // Break as connection has been made and media type checked.
    break;
}

From this we can see a first "dumb" attempt is made to grab the file content based on its name (this is what MediaUtils.filenameToContentType() does.) There's then some special cases for checking for different types of wav file, but if that fails then we fall back on a cleverer check which looks at the actual file signature. Both these checks are in MediaUtils. This latter check is much more extensive, and looks at the first few bytes of the file to see if it can work out a format that way. If it can't, then it bails out and throws the exception that then pops out as our dreaded MEDIA_UNSUPPORTED flag.

If the type is identified correctly though, there's still another hurdle to go through - it has to be supported by the current platform. Some platforms are loaded dynamically depending on the environment, however the GSTPlatform always exists, thus we would need to put any additional (universal) formats here. This is relatively simple, a CONTENT_TYPES array exists which just holds the array of supported formats.

Unfortunately cloning the JavaFX repo seems to be failing for me at the moment, otherwise I'd attempt to put some of this in practice. But in lieu of the above, what actually needs to happen to add support for further formats? It actually doesn't seem hugely difficult.

1. In MediaUtils, support needs to be added to the filenameToContentType() method to handle the new file extension. This is trivial.

2. In the same class, support needs to be added to the fileSignatureToContentType() method to work out the file type based on its signature. This is a tad more complex, but still not too bad. This may even be optional, since the current code only seems to use this as a fallback if the format isn't identified correctly (or at all) from the file extension. A comprehensive list of file signatures for different formats can be found here which should help with this task.

3. In GSTPlatform, the new content type needs to be added to the list of supported content types.

On the Java side of things, this appears to be all that's necessary to get it to accept the content type and at least attempt to pass it down to the native Gstreamer layer.

However, I'm no expert in GStreamer, so while I'm aware there's many more formats that it can handle and play that JavaFX currently refuses, I'm unsure as to how exactly they've removed this capacity. They've definitely done it in the Java layer above, but they may have also done it on the native GStreamer level - at this point I'm unsure.

I assume they've made some changes to GStreamer for JFX8 - but at the present time they're not listed on the relevant project page, so it's quite hard to work out exactly what they've changed for this version.

The next step would be to grab the JFX8 source, build with the above proposed changes for a new content type, and then see what errors (if any) occur on the native level, then take it from there.

And now, Javafx2.1 finally supports mp4 H.264 so you should now be good to go without the above posted stunts. :)