The HTML5 specification introduces the audio and video media elements, and with them the opportunity to dramatically change the way we integrate media on the web. The current API provides ways to play and get limited information about audio and video, but gives no way to programatically access or create such media. We present a new API for these media elements, as well as a working Firefox prototype, which allows web developers to read and write raw audio data. Through numerous working demonstrations, we show how access to this data allows for improved web accessibility, opens new possibilities for real-time media manipulation, and extends the web in ways that have traditionally required proprietary plugins. For example, we look at how an audio data API can be used to create such things as in-browser synthesizers, audio based animated visualizations, digital signal processing, text to speech, etc. We will also explore the code necessary for web developers to work with audio data, calculate FFT, and other audio algorithms in JavaScript.

Granting direct access to the audio stream in the browser, opens up the web as a powerful tool for accessibility, innovation and creativity. There are a myriad of readers for the web such as SuperNova for the visually impaired, but having worked with people who have absolutely no vision, there is an obvious gap in usability. Someone who is partially sighted or color-blind, can change the size of the screen and switch color palettes. But people who have very-little to no-sight-at-all have to rely on text-to-speech. The problem with this method is simple to explain, but difficult to solve. Have you ever tried to navigate a drop-down menu with your eyes closed using text-to-speech? This alone renders the web unusable in seconds. Imagine having absolutely no visual cues to help you know what to click on. Imagine having no idea if you are in the navigation or in the content. What if the tab indexes have been neglected on a site you need to pay your bills, or change personal information with a company? You would have to reply on other people to help you on the web, and you would miss out on all the benefits that the web can bring to your daily life.