As part of its Scale conference today in San Jose, Facebook’s computational photography team is announcing the completion of an internal research project dedicated to helping you clean up your poorly taken 360 photos. The team used a deep neural network to identify crooked 360 photos and reorient them to maintain realism.
You’ve probably used leveling tools before on your smartphone to salvage pictures taken at awkward angles. But Matt Uyttendaele, one of the research scientists working on the project, explained to me in an interview that traditional computer vision researchers would approach this problem by looking to identify straight lines in a photo converging at a vanishing point (when two parallel lines appear to intersect).
But this approach isn’t super generalizable because a lot of photos simply don’t have enough parallel lines to act as reference points. So instead, Uyttendaele and his team trained up a neural net, specifically AlexNet, on rotated images labeled with tilt and roll values. It turned out that having enough of this data was actually one of the biggest challenges of the entire effort.
Once the team cobbled together 500,000 non-rotated images, they artificially rotated them. This yielded a nice data set to build a model for 360 image correction. The feature hasn’t been deployed yet, but it’s expected that this should happen in the coming months once product decisions have been made as to how users will opt-in to the corrective capability and testing is complete.