Using deep learning to “see” inside homes across the world

How much does someone's living room tell about how they live? Peeking into another person's life might be just part of natural human curiosity, but the answer to this question may provide insights in a wide range of aspects of human behavior. A new study published in EPJ Data Science uses the power of machine learning to explore patterns of home decors—and what they could tell about their owners—in popular accommodation website Airbnb.

The Internet has provided the world with more images than can be viewed in a lifetime. Some sites, like Craigslist, Zillow, and Airbnb, specifically let us see the interiors of peoples’ homes, nests of revealing human creativity, design, style and culture. When we geolocate these images—that is, put them on a map—we can find spatial and cultural/geographic trends in how people decorate across the world.

What we did

In 2016, our team conducted a pilot study with geolocated Airbnb photos by measuring how ornate or austere interior room photos were in 10 different cities. In 2017 and 2018, we expanded our analysis to include only living rooms in 107 global cities and performed a deep dive into six U.S. cities. We used computer vision techniques to detect the presence of plants, books, wall art, decor and vibrant colors in over 50,000 homes on six continents.

What we found out

On a global scale, we found that, regionally, Europe and North America lead in ornamental objects, followed by South East Asia. The Middle East had the greatest variation. Bright, vibrant colors and many decor elements were found in India and Morocco. Vibrant colors were also found frequently in Japan. There were many books in Europe, the U.S. in South America, and many plants in China and Scandinavian countries. Italian, Indian and Brazilian cities had many pieces of wall art. We found that different regions used these elements differently.

On a national scale, using Chicago, Houston, Los Angeles, New York City, Philadelphia, Washington, DC, we found that the use of decorative elements was clustered into neighborhoods, but not to a great extent. We also found that decoration patterns did not correlate with socio-economic status of the neighborhood, meaning that wealthier and more deprived neighborhoods, and neighborhoods with different ethnic or racial backgrounds used these elements at similar rates. Looking into specific neighborhoods, we found places where rentals had many pieces of art in Brooklyn, and places where rentals often had books on display, such as the Fishtown neighborhood of Philadelphia.

What’s next

What’s new about this research (from the 20,000 foot view) is that it uses cutting-edge technology & big data to examine a cultural phenomenon in human geography and interior design. Applying deep learning methods to art and design questions is an exciting research frontier.

There are new things we can do as we come through millions of images to learn something meaningful about the world. Our next steps may be to train machine learning models to detect specific decoration styles, to and detect specific symbolic decorative objects like national flags, sports team decorations, cultural crafts, or even Kippan, a favorite IKEA sofa, to explore spatial trends in globalization of goods or local identity.

View the latest posts on the On Society homepage