Microsoft's new camera app brings AI to your iPhone

Microsoft sure loves it when research projects beget actual products, and it just released another for the masses to play with. Pix is a replacement camera app (what?) available for iPhones and iPads (what?), and in short, it promises better photos of the people around you without any extra work on your part. It'll run on just about every iOS device from the iPhone 5s on, and an Android version is in the works too. (Microsoft didn't have a firm answer when I asked if these features would make their way into the Windows 10 Mobile camera.) And you know what? In some ways, I wish this was the camera app that Apple built in the first place.

First, the basics. The most important thing you need to know about Pix is that it's been tuned to make your pictures of people look better.

"There are things the Apple camera does that we don't do and might not ever do," said Josh Weisberg, GM of computational photography at Microsoft. "The goal was around people photos -- can we make better people photos than the stock camera? And we succeeded."

From the moment you start Pix, it's capturing what your camera is pointed at -– you can never tell when something's about to happen, after all. Once you press the shutter button, Pix snaps 10 frames, and Weisberg says that's where the magic really kicks in. Algorithms evaluate those ten frames for obvious things like sharpness or exposure, but also underlying characteristics like whether a person in the shot seems happy or sad. When that near-instantaneous process is done, you'll be given up to three "Best Images." The image data from the leftover photos is used to enhance those winners before being deleted. All of this happens on the fly and without any extra fiddling, so you don't need to be a photo buff to snap some great shots.

If the app detects a bunch of similar pictures, it'll stitch them into a Live Image, but only when it thinks what's going on in the photos is interesting. Oh, and the Hyperlapse feature that Microsoft has been working on for years is here too. This time, though, you can turn existing photos into time-lapses, or just use it to stabilize video you just shot.

Using Pix is very much a learning process, and I don't just mean for you, the user. According to Weisberg, the app sends anonymized bits of "telemetry" -- settings data and what Best Images people fave'd or deleted -- back to the mothership, where human judges will examine them and adjust the image processing algorithms accordingly. Basically, the more you use Pix, the more insight it gains into what makes a photo good. All told, Weinberg was right: The app really is helpful for improving your photos of people. Usually, anyway.

In no time at all, I was snapping photos using Pix that came out punchier and with a greater emphasis on the people in the shot. When the testing period inevitably overlapped with post-work drinks at a local dive, Pix shined even brighter. I mean that literally too. Smartphone camera sensors often flounder in dim, dank conditions, leaving software to do the heavy lifting required to make a passable photo. Microsoft's photo processing was both super-fast and mostly great at brightening up pictures of my colleagues and removing grain without making things look unnatural. I was utterly impressed... until I wasn't.

My biggest issue with Pix in its current form is that it's inconsistent. Sometimes my shots were clear improvements over what Apple's camera app was capable of. Other times, though, Apple's softwaer had a clear edge. Take landscape photos, for instance: Even before Microsoft's instantaneous image processing did its thing, the app had trouble exposing shots with bright backgrounds. Pix's outdoor shots tended to be a little blown out, while Apple's camera software was generally better at balancing exposure levels.

And for all the work that went into teaching Pix to enhance photos of people, it still struggles at times. A "Best Image" it suggested of a colleague in the dimly lit dive mentioned earlier was noticeably less crisp than the image the camera actually captured. In the app's zeal to brighten up her face, it smoothed out her features a little too much. Long story short, the version of Pix I played with was still fairly hit or miss.

But that doesn't mean you shouldn't use it. In fact, I'd strongly recommend giving this a download, even if you're not the sort of person who already juggles multiple camera apps. The benefits of better image processing can be seen from the get-go, but the weightier, far more fascinating goal is to see how much Microsoft's system can learn about good photographs. In a way, it's almost as though we're collectively training it to better understand art. The very nature of Microsoft's algorithmic processing means these early issues will probably get ironed out over time, and I'm fascinated to see how long it takes before Pix becomes great in every situation.

Sources: Engadget