How to apply a Vignette CIFilter to a live camera feed in iOS?
While trying to apply a simple vignette filter to the raw camera feed of an iPhone6, with the help of Metal and Core Image, I see a lot of lag between the frames being processed and rendered in an MTKView
The approach which I have followed is (MetalViewController.swift):
- Get raw camera output using
AVCaptureVideoDataOutputSampleBufferDelegate
- Convert
CMSampleBuffer
>CVPixelBuffer
>CGImage
- Create an
MTLTexture
with thisCGImage
.
Point no. 2 and 3 are inside the method named: fillMTLTextureToStoreTheImageData
- Apply a
CIFilter
to theCIImage
fetched from theMTLTexture
in theMTKViewDelegate
func draw(in view: MTKView) {
if let currentDrawable = view.currentDrawable {
let commandBuffer = self.commandQueue.makeCommandBuffer()
if let myTexture = self.sourceTexture{
let inputImage = CIImage(mtlTexture: myTexture, options: nil)
self.vignetteEffect.setValue(inputImage, forKey: kCIInputImageKey)
self.coreImageContext.render(self.vignetteEffect.outputImage!, to: currentDrawable.texture, commandBuffer: commandBuffer, bounds: inputImage!.extent, colorSpace: self.colorSpace)
commandBuffer?.present(currentDrawable)
commandBuffer?.commit()
}
}
}
The performance is not at all what Apple mentioned in this doc: https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_tasks/ci_tasks.html#//apple_ref/doc/uid/TP30001185-CH3-TPXREF101
Am I missing something?
Solution 1:
Your step 2 is way too slow to support real-time rendering... and it looks like you're missing a couple of steps. For your purpose, you would typically:
Setup:
- create a pool of
CVPixelBuffer
- usingCVPixelBufferPoolCreate
- create a pool of metal textures using
CVMetalTextureCacheCreate
For each frame:
- convert
CMSampleBuffer
>CVPixelBuffer
>CIImage
- Pass that
CIImage
through your filter pipeline - render the output image into a
CVPixelBuffer
from the pool created in step 1 - use
CVMetalTextureCacheCreateTextureFromImage
to create a metal texture with your filtered CVPixelBuffer
If setup correctly, all these steps will make sure your image data stays on the GPU, as opposed to travelling from GPU to CPU and back to GPU for display.
The good news is all this is demoed in the AVCamPhotoFilter sample code from Apple https://developer.apple.com/library/archive/samplecode/AVCamPhotoFilter/Introduction/Intro.html#//apple_ref/doc/uid/TP40017556. In particular see the RosyCIRenderer
class and its superclass FilterRenderer
.