The one killer feature I had in mind with ColorMyWorld was to have it do real-time color analysis: point the video camera at an object, grab a sample swatch from the video frame, find its average color, and display the closest matching color. I never implemented this because the app seemed good enough without it, but the absence of this killer feature gnawed at my soul.

Armed with a few more months of iOS/Obj-C experience, I’ve finally circled back to ColorMyWorld to give it the attention it deserves. It’s been given a big code refactoring, but more importantly, I’ve taken a swing at implementing real-time analysis. That means working with the AVFoundation framework.

It was encouraging to see how quickly I was able to finish the first iteration of the feature. It took me a few hours to get the closest color to print to the debug console, and another few hours to get the camera to display on the screen and to allow the user to lock and unlock the sampling mode by holding their finger to the screen.

At this point, everything was looking pretty darn functional, but there was a problem I expected: rotating the device really messed things up. I expected the camera to rotate when I rotate the device. It didn’t.

To explain what happened, I’ll have to explain what I know about AVFoundation.

With AVFoundation, you set up a session to capture input from the device’s camera or the microphone (AVCaptureSession, AVCaptureDevice, and AVCaptureDeviceInput). You then set up the session with one or more ways to output the captured data (AVCaptureOutput, AVCaptureVideoDataOutput, etc.). The session then uses connections (AVCaptureConnection) to send data from the inputs to its outputs.

To display what the camera sees, you have to use a preview layer. A preview layer is a CALayer subclass that displays the video that the camera input is capturing. You basically create a preview layer, add it as a sublayer to a view, and set it to receive data from the capture session. Pretty basic stuff, only a few lines of code.

The problem is that rotating the view in which the preview layer exists does not rotate the preview layer. There is a deprecated way of rotating the preview layer, but the Apple-approved method is to grab the connection between the session and the preview layer and set its videoOrientation property. Okay, so in your view controller’s viewDidLayoutSubviews method, have it access the preview layer’s connection to its session and set this to the UIDevice’s orientation.

But I discovered something else that was weird: the image output from the camera was oriented differently than it was in the preview layer. My mind was boggled. Why is this happening? I already set the connection to the correct orientation!

Well, there’s a very good explanation for why this is: your AVCaptureSession captures raw data from the camera, but it has a few different ways of outputting that data. One way is to render that data in the preview layer, and another way is to send the data to a buffer that you can read and transform into UIImage instances.

These two things use different connections! That’s why setting the preview layer’s connection to orient the image did not change the image output’s orientation. To do that, you have to access the video data output, get its connection to the session, and change its videoOrientation as well.