IST research group at the forefront of digital photography revolution

UNIVERSITY PARK, Pa. — The rising popularity of digital photography in recent years has opened up a whole new world for both amateur and professional photographers. It is now easy for anyone with a camera to share their photos with wide audiences through social networking sites, blogs, email and other outlets.

While photographic images can arouse a wide variety of emotions in people, there is no universal standard for measuring aesthetic value. However, the research group of James Wang, a professor in Penn State’s College of Information Sciences and Technology, was recently granted two patents, 8,755,596 and 8,781,175, by the U.S. Patent and Trademark Office (USPTO) for content analysis methods that are intended to help digital photographers refine their skills by providing instant feedback on visual features generally believed to make photographs more pleasing to the eye.

“People are taking a lot of pictures on mobile phones,” Wang said. “We hope they can leverage technological advancements to improve their picture-taking.”

The patent, “Studying aesthetics in photographic images using a computational approach,” describes a method to automatically infer the aesthetic quality of a picture by comparing visual features with those of manually-rated photos. A second patent, “On-site composition and aesthetics feedback through exemplars for photographers,” extends further by describing a comprehensive system to enhance the aesthetic quality of the photographs captured by mobile consumers that provides on-site composition and aesthetics feedback through retrieved examples. Overall aesthetics feedback predicts the aesthetic ratings for both color and monochromatic images. An algorithm is used to provide ratings for color images, while new features and a new model are developed to treat monochromatic images. The system was designed keeping the next generation photography needs in mind and is the first of its kind.

Wang, along with Ritendra Datta and Dhiraj Joshi, then doctoral candidates in the Department of Computer Science and Engineering (CSE), and Jia Li, a professor of the Department of Statistics, described the theories and statistical methods they used to develop the technology in a publication back in 2006. Content analysis in photographic images has been studied by the multimedia and vision research community in the past decade, according to the authors. Culturally significant pictures are being archived in digital libraries, and online photo sharing communities are becoming increasingly common. While there are no firm rules for judging aesthetics, “certain features in photographic images are believed, by many, to please humans more than certain others.” The researchers tackled the problem computationally and experimentally through a statistical learning approach.

“In this age of digital picture explosion, it is critical to continuously develop intelligent systems for automatic image content analysis,” they wrote in the paper.

In the content analysis system developed by Wang and his associates, when a photographer takes a digital picture, it is processed by a computer program. The system compares the photo’s attributes with a data set of rated photos, then generates a rating based on factors such as color, saturation, depth of field (the range of distance that is acceptably sharp in the photograph) and shape convexity. People learn to rate the aesthetics of pictures from experiences gathered by seeing other pictures, the researchers wrote in their paper, and “our opinions are often governed by what we have seen in the past.”

“We can use a familiarity measure to evaluate aesthetics,” Wang said.

The main source of data for the researchers’ computational aesthetics work was obtained from a large online photo sharing community, The website, which was started in 1997, attracts a large number of photographers. The researchers chose this online community, they stated, because it provides photos that are rated by a relatively diverse group. Many amateur and professional photographers visit the site frequently, share photos, and rate and comment on photos taken by peers.

“Of interest to us was the fact that many of these photographs were peer-rated in terms of two qualities, namely aesthetics and originality,” the researchers wrote.

Through their observations of the gathered data from, the researchers discovered a strong correlation between the aesthetics and originality ratings for a given photo. The more original a photograph was perceived to be, the higher its aesthetic rating. Typically, they wrote, “a very original concept leads to good aesthetic value, while beauty can often be characterized by originality in view angle, color, lighting or composition.”

“The more novel (the photo), generally the higher the perceived aesthetics,” Wang said.

The possible benefits of building a computational aesthetics model, according to Wang et. al., include the ability of photographers to get a rough estimate of their shot composition quality, leading to adjustment in camera parameters or shot positioning for improved aesthetics. Camera manufacturers can also incorporate a “suggested composition” feature into their products. In addition, Wang said, managers of image hosting websites such as Flickr can use the aesthetics scores to rank the photos on their site.

Looking towards the future, Wang said, he and his associates will seek avenues for further developing and marketing the content analysis system. They are interested in potentially licensing the technology, particularly to smartphone manufacturers or companies that develop search engines. As far as improving the model itself, Wang said that he and his fellow researchers would like to be able to “provide as much useful feedback to photographers as possible.” He envisions a smartphone that would provide immediate feedback to photographers that would resemble a dialogue more than a score generator, and he would like for the system to provide suggestions on how to maximize the emotional impact of photos on viewers.

“The digital photography revolution is likely just at its early stage,” Wang said. “Through harnessing the vast user-generated data on the Internet, future cameras can be equipped with some professional-level knowledge about photography and be able to provide intuitive on-site guidance to photographers.”

Author: Stephanie Koons