news image

Facebook is now using AI to help blind users “see” and experience the social network

07 Apr 2016

Facebook’s first blind engineer, Matt King, introduces image recognition technology which automatically generates captions for photos in the News Feed of vision-impaired users

There are 39 million blind people around the globe. But just because they are blind that doesn’t mean they don’t use the Internet. They rely on tools called screen readers to narrate websites and apps or translate them into braille - one line of text or link at a time. And because screen readers work by looking at a website’s code, not just what is visible on a display, sites can be almost incomprehensible to non-developers. Also there is the problem of image perception, since websites and apps have become more and more visual in nature.


Social media has been a big part of that shift. According to Facebook people share more than 2 billion photos per day across its various products and apps alone. And over one billion people check their Facebook feeds every day, mindlessly scrolling through texts and photos to keep up with friends and family. The bad news is that pictures often come with no description whatsoever, making it impossible for blind users to experience social networking.


Despite all obstacles Facebook’s first blind engineer, Matt King, is now revolutionizing social media as we know it. He has been developing a tool which automatically generates captions for photos in the News Feed of vision-impaired users. The tool is called Automatic Alternative Text and it uses image recognition technology to identify objects in photos and describe them to users through voice over. It works when a user has Apple's built-in screen reader turned on and selects a picture in the app. Then Facebook uses artificial intelligence algorithms to detect basic features in the image and create a new alt text that the screen reader will share with the user. The solution can pick out particular characteristics of the people in a photo like smiles, beards and eyeglasses. Also it can analyze an image in a more general sense, determining that a photo depicts sun or ocean waves or snow. For example, if a picture shows a couple overlooking the ocean while wearing sunglasses, the new alt text will be something like this: “This image may contain: two people, smiling, sunglasses, sky, outdoor, water.” Eventually King hopes to achieve a true storyteller, but at least for now this solution gives blind users a basic idea what’s in images otherwise they couldn’t see.


The new AI-powered feature is now available in English for iOS users in the US, UK, Canada, Australia and New Zealand. Facebook also plans to introduce the Automatic Alternative Text to more platforms, languages and countries, however, there is still a lot of work to be done on the prototype. Meanwhile, other companies are hard at work to create more inclusive technology. Just last week, both Microsoft and Twitter announced tools similar to the one King has been working on at Facebook for months.


Ultimately, CEO Mark Zuckerberg recognizes that the photo description tool is “an important step towards making sure everyone has equal access to information and is included in the conversation.” We agree that this is not just a matter of making life easier for blind people - it's a matter of how they're actually going to get along in the 21st century of technology reign.