Audio signifiers

Research by design

TL; DR Skimming a website is difficult for visually impaired users because they do not have access to visual cues. This concept works with screen readers to enable the eyes-free skimming of websites.

Duration: 12 months
Image of the manage signfiers window that allows users to customize audio signifiers and a set of tutorials for new users to learn using audio signifiers.

Location: Capstone project funded by Mozilla Research Grant 2018
Role: Interaction Design, UX Research, Independent project

Methods: Research by design, Moderated remote usability tests,
Tools: Sketch, Dreamweaver, Javascript, Web Audio APIs, Trello

Current screen readers require visiting each item sequentially.

Thus, visually impaired users need a lot of time to get a high-level overview of the web content. However, sighted users rely on visual cues to quickly skim through web content. This gap in web experience hinders the ability to explore many websites in a short time, which can be frustrating.

Skimming is easy for sighted users because they have access to composition, colors, and typography. But it is difficult for visually impaired users. Hence, visually impaired users utilize voice speed, search, and filters.
How might we enable web content skimming when using a screen reader?
Key insights from interviews, contextual inquiry, and user evaluation.
Icon representing missing valuable content
Users are unaware of valuable content

Visually impaired users miss out on the usefulness of the website because valuable content is difficult to reach quickly.

Icon representing changes on the website.
Clueless about changes on a website

Many changes on the websites are conveyed visually, making changes difficult to identify and unclear.

Icon representing errors
Difficulty to find error source

Many users find it difficult to trace the source of validation errors while filling forms.

Icon representing earmarking.
Need for earmarking or personalization

The ability to add custom markers or flags allows users to retrace the web content enabling selective focus.

Audio signifiers facilitate eyes-free skimming via screen readers.

This concept complements screen reader output without being obtrusive. Further, the sounds are inspired by real-world counterparts to add materiality to the web experience.

Audio signifiers are used to convey current status of the website. To mark important or new content. To convey events, hierarchy, or placement. To grab user attention when errors occur.
How it works?
To convey the status of the website and user
To notify users and mark important content
To convey action and movement
To grab users attention
Let's look at the design process that led to this design!
Discovering pain points through user interviews and contextual inquiry.

Current screen readers do not provide a quick overview of the website. Hence, visually impaired users have to sequentially tab through several UI elements before they can decide whether the website is useful. This requires a lot of time.

Image of a visually impaired interviewed while using the computer.

“Sometimes, I do not even pay attention to the screen reader. I quickly go to the part which I need by tabbing.”

- Software Developer
Image of a female user who works as a Principal Accessibility Consultant.

"I do online shopping, read the news, or connect with my friends on social networking sites. But, I miss out on so much content until someone says something."

- Principal Accessibility Consultant
Image of a social worker working on a phone.

“Often, I am unaware of what changed on the website if I click on something. For example, am I already logged in?”

- Social Worker
Image of a undergraduate student using a personal computer.

"Learning screen readers was not easy. It took me a while to get used to it. So I played a lot of games!"

- Undergraduate Student
Icon with a bulb to represent key findings

Key finding: Visually impaired users miss out on a lot of information and changes on the websites.

Looking from the users' perspective through archetypes or persona.

I realized that any design change should make learning screen readers easier for new users. But, it should not hinder experienced users who have been using it for more than 20 years. Creating persona inspired by user stories helped to portray user intentions.

Jennifer is a blogger and motivational speaker. She often travels to give talks or conduct workshops. She likes to stay updated about the latest news and writes several emails and book reviews. She values Connectivity because she likes to stay connected to her network. For her productivity is important as she wishes to meet deadlines and prompt responses. She has Has to collaborate with several people and dislikes losing time when things change.

"Using JAWS is second nature to me. It has been there since I was 15, I know it is not perfect, but it works."

Dillon is an undergraduate student. He is new to the extensive use of screen readers. He dislikes working on group projects and assignments. He is an avid gamer and loves writing software code. He values effortless learning as he likes to tinker with new technology. He loves multi-tasking and likes to get involved in many things. Prefers personalization and loves to modifies things to suit his preferences. He is trying to be self-reliance and wants to become autonomous.

"Getting used to a screen reader has been a challenge. I had to practice a lot to get this good!"

ing key learning.

Key learning: Experienced users rely on their workarounds to overcome the pain points while using screen readers. New users need assistance to get used to screen readers.

Brainstorming broad concepts to kickstart ideation.

I began by speculating a future where web experience is conveyed by audio instead of sight. Then I mapped several discernable properties of sound to replace visual cues. This led to acoustic experience ideas beneficial towards six kinds of websites.

Concept 1 is a grid wall-based acoustic experience for online shopping sites. Concept 2 is ripples based acoustic experience for social networking sites. Concept 3 is an ellipse acoustic experience for online streaming sites. Concept 4 is a fluid acoustic experience for storytelling or portfolios. Concept 5 is a layered acoustic experience for news, magazines, and galleries. Concept 6 is a concave acoustic experience for directories or contact pages.
to represent key learnings

Key learning: Several properties of sound such as pitch, frequency, position, distance, stereo image, space (wet/dry reverb percentage, type of reflections), envelope (Attack, Decay, Sustain, Release) have functional qualities. Moreover, cohesiveness and compositional properties have aesthetic qualities.

Testing the prototypes with participants

After building the prototypes, moderated remote evaluations were done with 10 participants. First, the participants were asked to familiarize themselves with sounds and their purpose. Later they interacted with the prototypes, which imitated the problematic situations. Finally, participants were asked to describe their experience by filling the google form. Let's look at the key findings.

View prototypes now
Image from usability study over zoom video conference.
70% of participants confirmed that audio signifiers enable the skimming of web content. Average time of 18 min to learn sounds and the associated events they represent. Average five number of attempts to learn sounds and the associated events. Less than 1-minute average time to learn sounds when participants chose custom sounds. Participants could remember 4 out of 7 sounds consistently at any given time. 6 out of 10 participants felt audio signifiers aid learning screen readers.
Tutorials and customizing sounds promote user acceptance.

User evaluations demonstrated the usefulness of audio signifiers in specific scenarios. Additionally, it unveiled that users needed tutorials to learn various ways to use audio signifiers. Moreover, they expressed the need for customization. Hence, I created a workflow to enable customization and learning through tutorials.

Image of the workflow for audio signifier settings where users can turn on/off sounds, play tutorials, and add new audio signifiers.

I felt the pressure of achieving deadlines as the project was funded by the Mozilla research grant. My ability to handle pressure and doing whatever it takes enhanced greatly!

Recruiting participants was tricky. Hence, I had to think out of the box and publish podcast-style audio to get people interested.

The next steps in this project are conducting remote moderated evaluation studies for Phase 3 and writing a deadline ready research paper.

Caricature of a sighted user who relies on the vision for interacting with the web. Plus, a visually impaired user who relies on hearing.