How do I convert an OpenGL GLKView to a MTLKit Metal based View?

A few years ago, Apple started warning anyone using GLKit in their app that OpenGL was going away:

warning: OpenGLES is deprecated. Consider migrating to Metal instead
warning: GLKit is deprecated. Consider migrating to MetalKit instead

My app uses a complex OpenGL class, and I don't know either OpenGL or Metal. Apple has a few WWDC sessions on this, but they are targeted at OpenGL experts. Since Apple is going to remove OpenGL someday, I want to start this now before I only have a few months to do it. What should I do?

tldr;

Once I started seeing the build error messages in iOS12:

warning: OpenGLES is deprecated. Consider migrating to Metal instead
warning: GLKit is deprecated. Consider migrating to MetalKit instead

I knew I had to do something. Who knows exactly when Apple will remove OpenGL and GLKit? Trust me, you don't want to wait until you have just a few months to convert to Metal, as the process is by no means straight forward.

What follows is the process I used to convert an Objective-C/OpenGL view into Metal. It was a long arduous process and several times I put my head on my desk and cried in frustration.

The fundamental steps I took were ones I would suggest others adopt too:

1. Remove all business logic and anything not directly related to OpenGL from the view, and restructure the primary app as necessary.
2. Create a test harness app that you will use for the conversion, and absolutely put it under version control.
3. Add the OpenGL view to the test harness.
4. Once the ViewController can drive the view, and you can see it, you are ready to start the transition.

In my case, I had three hurtles to jump: convert the view to Swift, recreate the functionality in Metal, then replace all GLK vector and matrix values and operations to simd.

My suggestion for proceeding:

1. Convert any ObjectiveC to Swift (I used Swiftify, free for limited translation, however I had a subscription)
2. Add a MTKView to the test harness, and put code switches in the ViewController so as to alternately look at either view (comparing both was a big help to me).
3. Since I didn't know either OpenGL or Metal, I spent a lot of time downloading open source Metal projects and tutorials.
4. Create the Metal boilerplate (based on examples/tutorials) along with a shader.
5. Put a pad on your desk so when you bang your head in frustration trying to get anything to show in the Metal view you don't seriously hurt yourself.
6. Once you are over the hill, convert the GLK values/operations to simd, making use of the translation functions shown later.

I cannot stress this enough—commit every time you change a few operations and test them! You will surely break things and that way you can reference earlier working code.

The test harness will prove useful, as you will likely find that timing changes result in undesired behavior. In my case I created two harnesses, a second that had more of the app code in it so I could better debug the actual usage.

Project

I forked an open source project Panorama. The master branch contains the Metal/simd code, and the Swift-OpenGL branch contains the original ObjectiveC code along with the Swift conversion. This lets a reader compare the two side-by-side. However, you don't need to referernce that to gleen much of how to convert OpenGL code in ObjectiveC to Swift, or to convert GLKit vectors and matrices to simd, as follows.

ObjectiveC to Swift

The OpenGL code makes much use of pointers, and those are a bit more burdensome in Swift. For instance:

GLfloat *m_TexCoordsData;  // treated as an array of pairs of floats
glTexCoordPointer(2, GL_FLOAT, 0, m_TexCoordsData);

became

struct Pointer2 {
    private var array: [SIMD2<Float>]

    init(size: Int) {
        let n: SIMD2<Float> = [Float.nan, Float.nan]
        array = Array<SIMD2<Float>>(repeating: n, count: size)
    }

    subscript(index: Int) -> SIMD2<Float>{
        get { return array[index] }
        set(newValue) { array[index] = newValue }
    }

    mutating func usingRawPointer(block: WithRawPtr) {
        array.withUnsafeBytes { (bufPtr) -> Void in
            block(bufPtr.baseAddress!)
        }
    }
}
    private var tPtr: Pointer2   // m_TexCoordsData
    tPtr.usingRawPointer(block: { (ptr) in
        glTexCoordPointer(2, UInt32(GL_FLOAT), 0, ptr)
    })

This all went away in the final Metal code. Doing the conversion turned up latent bugs too!

In addition, I had to cast (convert) many of the values to stricter Swift types:

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);

became

glTexParameteri(UInt32(GL_TEXTURE_2D), UInt32(GL_TEXTURE_WRAP_S), GLint(GL_REPEAT))

and this process was just tedious. However, the Swift code was thinner and easier to read IMHO.

A couple of functions translated easily:

GLKQuaternion GLKQuaternionFromTwoVectors(GLKVector3 u, GLKVector3 v) {
    GLKVector3 w = GLKVector3CrossProduct(u, v);
    GLKQuaternion q = GLKQuaternionMake(w.x, w.y, w.z, GLKVector3DotProduct(u, v));
    q.w += GLKQuaternionLength(q);
    return GLKQuaternionNormalize(q);
}

became

func GLKQuaternionFromTwoVectors(_ u: GLKVector3, _ v: GLKVector3) -> GLKQuaternion {
    let w = GLKVector3CrossProduct(u, v)
    var q = GLKQuaternionMake(w.x, w.y, w.z, GLKVector3DotProduct(u, v))
    q.w += GLKQuaternionLength(q)
    return GLKQuaternionNormalize(q)
}

Later you will see that the translation to simd was not so easy.

OpenGL to Metal

Unfortunately, there is no magic wand here. Apple has some WWDC sessions on this, but they didn't really enlighten me. Metal uses two types of kernels: compute and shader, with compute the easier. However in my case I had to use a shader, which was more difficult for me to grasp.

Metal Resources

A good place to start if you know nothing of Metal is this Metal Tutorial on the Ray Wenderlich site. A second article there is even more helpful: Moving From OpenGL to Metal on the Ray Wenderlich site. Both have copious references to more Metal material.

Two other excellent articles I found helpful: Donald Pinckney's Blog (Older). Another helpful author: Alex Barbulescu

The guy who literally wrote the book on Metal is Warren Moore. His book and articles are invaluable!

Things to keep in mind

OpenGL uses a clip space of -1 to 1 (z values). You need to account for this in your shader. Warren Moore personally suggested I insure my shader was not returning negative z values by using this code:

v.z = v.z * 0.5 + v.w * 0.5;

This obviates the need to completely redo your OpenGL code that might have used negative z values.

The backgroundColor of a MTLView is not set by using that property, but setting the clearColor.

The communication from App space to shader space is done using structures, which must be separately defined in each respectively. For instance, in my app this struct:

private struct Uniforms {
  let projectionMatrix: simd_float4x4
  let attitudeMatrix: simd_float4x4
}

is defined in the shader as:

struct Uniforms {
  float4x4 projectionMatrix;
  float4x4 attitudeMatrix;
};

These structs are application defined.

Textures

If you are using images to create textures, it's a bit different in Metal. This

NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithBool:YES], GLKTextureLoaderOriginBottomLeft, nil];
GLKTextureInfo *info=[GLKTextureLoader textureWithContentsOfFile:path options:options error:&error];
glBindTexture(GL_TEXTURE_2D, info.name);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, IMAGE_SCALING);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, IMAGE_SCALING);

became

let loader: MTKTextureLoader = MTKTextureLoader(device: mtlDevice)
do {
    let texture = try loader.newTexture(cgImage: cgImage, options: [
        MTKTextureLoader.Option.origin: MTKTextureLoader.Origin.bottomLeft,
        MTKTextureLoader.Option.SRGB: false // yes and image washed out
    ])
    return texture
} catch {

Note that you need to set the origin to bottomLeft.

Final Comments

Unless you plan on really learning Metal deeply, you will be doing a lot of experimentation and asking questions. Having a test harness app will prove invaluable as you spend hours trying to get your code to do what you want.

Translate GLK to simd

Any Apple code is surely full of GLKVectors, GLKMatrices, and related functions. Unfortunately, there are no tools to convert them—you must do it by hand, line by line, and sometimes there is not simd equivalent. Sometime I used Xcode's search and replace, but not often.

GLK -> SIMD

First, to get the simd macros, add this to your source files: import simd

• GLfloat -> Float
• GLint -> Int
• GLKMatrix4 -> simd_float4 (typealias for SIMD4)
• GLKMatrix4Identity -> matrix_identity_float4x4 (not easy to find)
• GLKMatrix4Invert -> simd_inverse(simd_float4x4)
• GLKMatrix4Make -> simd_float4x4(simd_float4, simd_float4, simd_float4, simd_float4)
• GLKMatrix4MakeFrustum -> no replacement, function provided below
• GLKMatrix4MakeLookAt -> no replacement, function provided below
• GLKMatrix4MakeWithQuaternion -> simd_matrix4x4(simd_quatf)
• GLKMatrix4Multiply -> simd_float4x4 * simd_float4x4
• GLKMatrix4MultiplyVector3 -> no replacement, function provided below
• GLKMatrix4MultiplyVector4 ->simd_float4x4 * simd_float4
• GLKQuaternion -> simd_quatf
• GLKQuaternionLength -> simd_quatf.length
• GLKQuaternionMake -> simd_quaternion(_ x: Float, _y: Float, _ z: Float, _ w: Float)
• GLKQuaternionNormalize -> simd_quatf.normalized
• GLKTextureInfo -> MTLTexture
• GLKVector3 -> simd_float3
• GLKVector3CrossProduct -> simd_cross(simd_float3, simd_float3)
• GLKVector3DotProduct -> simd_dot(simd_float3, simd_float3)
• GLKVector3Make -> simd_make_float3(_ x: Float, _y: Float, _ z: Float)
• GLKVector3Normalize -> simd_normalize(simd_float3)
• GLKVector4 -> simd_float4
• GLKVector4Make -> simd_make_float4(_ x: Float, _y: Float, _ z: Float, _ w: Float)

I should note that Dash was a tremendous help in digging through the simd functions.

The two functions referenced above:

func simd_make_look_at_float4x4(
                                eyeX: Float,
                                eyeY: Float,
                                eyeZ: Float,
                                centerX: Float,
                                centerY: Float,
                                centerZ: Float,
                                upX: Float,
                                upY: Float,
                                upZ: Float
) -> simd_float4x4 {
    // https://stackoverflow.com/questions/9053377/ios-questions-about-camera-information-within-glkmatrix4makelookat-result
    let ev = simd_float3(eyeX, eyeY, eyeZ)
    let cv = simd_float3(centerX, centerY, centerZ)
    let uv = simd_float3(upX, upY, upZ)

    let subbed = ev - cv
    let n = simd_normalize(subbed)

    let cross_p = simd_cross(uv, n)
    let u = simd_normalize(cross_p)

    let v = simd_cross(n, u)

    let c0: simd_float4 = [u[0], v[0], n[0], 0]
    let c1: simd_float4 = [u[1], v[1], n[1], 0]
    let c2: simd_float4 = [u[2], v[2], n[2], 0]

    let v0 = simd_dot(-1*u, ev)
    let v1 = simd_dot(-1*v, ev)
    let v2 = simd_dot(-1*n, ev)
    let c3: simd_float4 = [v0, v1, v2, 1]

    let m: simd_float4x4 = simd_float4x4(columns: (c0, c1, c2, c3))
    return m
}

func simd_make_frustum_float4x4(frustum: Float, aspectRatio: Float) -> simd_float4x4 {
    // https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/glFrustum.xml
    let left    = -frustum
    let right   = frustum
    let bottom  = -frustum/aspectRatio
    let top     = frustum/aspectRatio
    let near     = PanoramaView.Z_NEAR
    let far      = PanoramaView.Z_FAR

    let m00 = (2.0 * near) / (right - left)

    let m11 = (2.0 * near) / (top - bottom)

    let m20 = (right + left) / (right - left)
    let m21 = (top + bottom) / (top - bottom)
    let m22 = -1 * (far + near) / (far - near)
    let m23 = Float(-1)

    let m32 = -1 * (2 * far * near) / (far - near)

    let c0: simd_float4 = [m00, 0, 0, 0]
    let c1: simd_float4 = [0, m11, 0, 0]
    let c2: simd_float4 = [m20, m21, m22, m23]
    let c3: simd_float4 = [0, 0, m32, 0]
    let m = simd_float4x4(columns: (c0, c1, c2, c3))

    return m
}

// Translated from the original Panorama code
func simd_make_quaternion_from_two_vectors(_ u: simd_float3, _ v: simd_float3) -> simd_quatf {
    let w: simd_float3 = simd_cross(u, v)
    var q: simd_quatf = simd_quaternion(w.x, w.y, w.z, simd_dot(u, v))
    q.real += q.length
    return q.normalized
}

Translate back and forth between GLK and simd

These functions are found in the Panorama repository mentioned earlier, in a file GLK-Metal-Tools.swift. If as recommended you translate back and forth after your controller is solely simd, you can put these in your view to as you slowly remove the GLK code.

func glkV3_to_simd(_ v3: GLKVector3) -> simd_float3 {
    let v: simd_float3 = simd_make_float3(v3.x, v3.y, v3.z)
    return v
}
func simd3_to_glk(_ v3: simd_float3) -> GLKVector3 {
    let v = GLKVector3Make(v3[0], v3[1], v3[2])
    return v
}

func glkV4_to_simd(_ v3: GLKVector4) -> simd_float4 {
    let v: simd_float4 = simd_make_float4(v3.x, v3.y, v3.z, v3.w)
    return v
}

func simd4x4_to_glk(_ m: simd_float4x4) -> GLKMatrix4 {
    var array: [GLKVector4] = []
    for i in 0..<4 {
        let fv: simd_float4 = m[i]
        let v: GLKVector4 = GLKVector4Make(fv[0], fv[1], fv[2], fv[3]);
        array.append(v)
    }
    let mg: GLKMatrix4 = GLKMatrix4MakeWithColumns(array[0], array[1], array[2], array[3]);
    return mg;
}
func glkm4_to_simd(_ m: GLKMatrix4) -> simd_float4x4 {
    var array: [simd_float4] = []
    for i in 0..<4 {
        let fv: GLKVector4 = GLKMatrix4GetColumn(m, Int32(i))
        let v: simd_float4 = simd_make_float4(fv[0], fv[1], fv[2], fv[3]);
        array.append(v)
    }
    let ms: simd_float4x4 = simd_matrix(array[0], array[1], array[2], array[3]);
    return ms;
}

I used these print routines to check various values during development, you might find them of use too:

func print4x4SIMD(
    msg: String,
    m: simd_float4x4
) {
    var s = ""
    s += "---COL: \(msg)\n"

    let (c0, c1, c2, c3) = m.columns

    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 0, c0[0], c0[1], c0[2], c0[3])
    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 1, c1[0], c1[1], c1[2], c1[3])
    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 2, c2[0], c2[1], c2[2], c2[3])
    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 3, c3[0], c3[1], c3[2], c3[3])

    print("\n\(s)\n")
}
func print4x4GLK(
    msg: String,
    m: GLKMatrix4
) {
    var s = ""
    s += "---COL: \(msg)\n"

    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 0, m.m00, m.m01, m.m02, m.m03)
    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 1, m.m10, m.m11, m.m12, m.m13)
    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 2, m.m20, m.m21, m.m22, m.m23)
    s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 3, m.m30, m.m31, m.m32, m.m33)

    print("\n\(s)\n")
}

simd_float4x4 rotation

While I didn't use this yet, I may need it someday (it's untested):

func matrix_from_rotation(radians: Float, v _v: simd_float3) -> simd_float4x4 {
    // https://www.haroldserrano.com/blog/rotating-a-2d-object-using-metal
    let v: simd_float3 = simd_normalize(_v)
    let cos: Float = cos(radians)
    let cosp: Float = 1 - cos
    let sin: Float = sin(radians)

    let col0 = simd_float4(
        cos + cosp * v.x * v.x,
        cosp * v.x * v.y + v.z * sin,
        cosp * v.x * v.z - v.y * sin,
        0
    )

    let col1 = simd_float4(
        cosp * v.x * v.y - v.z * sin,
        cos + cosp * v.y * v.y,
        cosp * v.y * v.z + v.x * sin,
        0.0
    )

    let col2 = simd_float4(
        cosp * v.x * v.z + v.y * sin,
        cosp * v.y * v.z - v.x * sin,
        cos + cosp * v.z * v.z,
        0.0
    )

    let col3 = simd_float4(0, 0, 0, 1)

    let m: simd_float4x4 = simd_float4x4(columns: (col0, col1, col2, col3))
    return m
}

Conclusion

This project took me about half a year working one weekend day per week, but it spanned a period of 18 months. The reason: I spent so many days making no progress, getting bizarre corrupted output, or no output, that when I finally got the primary view to show in Metal as it did in, I put the project away. I was just too burned out to go on.

That said, the end of in iOS is going to end, and as the years roll by that end is getting nearer.

I was originally going to stop when I got Metal working with the GLK vectors and matrices, but was urged to convert to simd now by Warren Moore.

It was a moment of pure ecstasy when I finally built my company's app and there wasn't a single compiler warning related to GLKit!