Mobile apps give the blind and visually impaired a new sense of freedom

Ask anyone who was around before smartphones , and they'll tell you life was different. But for Nutsiri "Earth" Kidkul, who went blind eight years ago, life-changing doesn't begin to describe the impact of mobile apps .

Before specialized apps came along, Kidkul would have to plan and prepare much more for basic tasks like grocery shopping, traveling or reading documents. What's more, Kidkul often needed assistance from other people to go about her day.

"I was always at the mercy of their time or schedule," Kidkul says.

Kidkul uses, for example, Microsoft Seeing AI, a talking camera app, to help her read her own mail and documents. She finds it convenient and accurate enough that she doesn't have to rely on others.

"It gives me a sense of privacy to be able to sort and read my own mail without assistance from friends or family," Kidkul says.

Over the last decade, mobile apps have evolved from silly fart programs and mindless games to services that make a massive difference in people's lives. Accessibility jumped to the forefront roughly two and a half years ago, when giants such as Microsoft and Apple made it a priority. These apps don't just offer superficial user interface improvements but incorporate sophisticated tech such as AI and voice recognition to provide tangible benefits like having your mail read to you.

The apps show a commitment to considering a wider, more diverse audience. But there's a business reason too -- more accessible tech means tapping new users and avoiding the backlash that may come from ignoring an underserved community. The World Health Organization estimated 1.3 billion people living with some form of vision impairment in the world as of 2018. Kikdul is among the 36 million who are blind.

During its I/O developer conference last week, Google devoted a segment of its keynote to a slew of new ways the company is tackling accessibility. Google CEO Sundar Pichai introduced features like Live Caption, which helps users transcribe audio and video, and Project Euphonia, which uses AI to help people with speech impairments.

A world of new possibilities

After Kidkul went blind, she began attending the Braille Institute in Los Angeles as a student. She started volunteering more and later participated in the Institute's technology internship program. Kidkul is now the lead tech instructor, a position she's held for four years.

Kidkul leans on a number of apps that she says make life more convenient.

InstaCart simplifies grocery shopping, and Money Reader helps Kidkul identify different monetary denominations without assistance. Most items she needs to buy can be delivered to her door through Amazon .

She says she uses Smart Ride to navigate public transportation, but otherwise, her go-to apps are Uber or Lyft .

"When I'm ready for a ride, I just call an Uber [or] Lyft and the car just shows up wherever I'm standing," Kidkul says. "I also don't have to pay the driver (since payment happens through the app). Much easier."

AI in apps

The accessibility development isn't just about adding a new button or feature. A number of apps have incorporated the technology to make everyday tasks easier for the blind and visually impaired.

The Be My Eyes app connects seeing volunteers and users who need assistance with a video call. The live call can help its users with checking expiration dates on food, where to point their camera, distinguishing colors, reading instructions and more.  

The Seeing AI app, which Kidkul uses daily, employs artificial intelligence to read product barcodes, product names, instructions and currency. Additionally, the app can help distinguish colors and the difference between objects. The app can also help users recognize friends and describe people around them, including registering facial emotions.

Earlier this year, Chinese electronics giant Huawei released the Facing Emotions app, which can identify seven different emotions with AI facial recognition software. In addition to AI, Facing Emotions uses the camera of the Huawei Mate 20 Pro to turn emotions into conveyable sound.

Instagram alsohas taken strides in making its app more accessible to the blind and visually impaired. The app now includes alternative text, and custom alternative text. Automatic alternative text pairs with a screen reader to describe photos to the user.

Voice command in apps

Incorporating voice commands into apps make actions like driving safer and simple tasks like getting weather info easier. And it makes a tremendous difference in accessibility for the blind and visually impaired.

Last October, Google's Accessibility Team launched the Voice Access app. The program lets users operate their Android devices completely hands-free. Voice Access operates through voice commands and assigns numbers to functions and apps. Users can text, scroll, navigate apps and more by saying "OK Google."

Though the app was designed for people with limited mobility, blind and visually impaired people might also find it helpful.

Other apps have added or expanded their voice command features in the last year like YouTube for Android and Pandora.

At CES 2019, Google announced that its Assistant device would respond and function even if your Android device is locked.  

Braille in cars

Even though Kidkul uses ride-share apps and public transportation, the future could hold more possibilities in cars for blind and low vision people.

Last May, introduced a Braille device for car windows that could help the blind and visually impaired see the world. Feel the View uses the vibrations in the vehicle's window to "show" what's outside.

The prototype technology converts an image from the built-in external facing camera and converts the image to grayscale. The shades of gray are translated to vibrations that vary in intensity. If a person touches the window, it can be felt like Braille.

Feel the View also incorporates a voice assistant and AI to help put the image into context.

We've still got a long way to go

Even as apps keep advancing, there's still a problem with the phone itself, according to Kristen Shinohara, an assistant professor at the Rochester Institute of Technology, where she helps direct the Center for Accessibility and Inclusion Research Lab. She focuses on the design of technologies that are usable for people with disabilities.   
"It would be fair to say that there is, fortunately, an increasing awareness of the need to make smartphone apps accessible to visually impaired users, but that there is yet a lack of knowledge in how to design and implement accessible features in current smartphone technology," Shinohara said in an email.

Kidkul, however, sees the benefit. She teaches the use of smartphones, desktop computing and assistive devices, like the Victor Reader and Braille display. Kidkul says smartphones have come a long way in making people with blindness and low vision feel more connected and independent, in areas including transportation, staying in touch with friends and family, shopping and identifying objects.

"I use many of these apps myself and as a technology instructor here at Braille Institute, I can tell you that technology has been a game changer for the blind and visually impaired," Kidkul says. "We can do virtually anything someone sighted can do using our smartphone."

But Kidkul shares Shinohara's belief that there's room for improvement. Kidkul says devices themselves need to be more accessible and developers need more education in building accessibility elements into apps. For example, often buttons or images aren't labeled, so VoiceOver or TalkBack screen readers can't identify them. Kidkul also says maps are an area for improvement because graphics are so heavily involved.

Shinohara said what's needed is a fundamental shift in the way technology design is thought about. Instead of designing apps only for the blind and visually impaired, accessibility should inherently be folded into the development process.

"Nearly any app is, technically, usable by a blind or visually impaired user if care has been taken to create the correct elements," Kidkul says.