w3cx-4of5-html5.2x
Coursera - Front-End Web Development - Apps and Games

W3Cx Front-End Web Developer

W3Cx-4of5-HTML5.2x - Apps and Games

Table of contents

About W3C and the Web

- About W3C and the Web

- Why accessibility is important

- Why internationalization is important

Course information

* Welcome to "HTML5.2x - Apps and Games"

* Course outline, grading and due dates

* Course practical information

* Course tools

Module 1: Advanced HTML5 Multimedia

1.1. Video introduction - Module 1

1.2. The Timed Text Track API

1.3. Advanced features for audio and video players

1.4. Creating tracks on the fly, syncing HTML content with a video

1.5. The Web Audio API

Module 2: Game programming with HTML5

2.1. Video introduction - Module 2

2.2. Basic concepts of HTML5 game development

2.3. A simple game framework: graphics, animations and interactions

2.4. Time-based animation

2.5. Animating multiple objects, collision detection

2.6. Sprite-based animation

2.7. Game states

Module 3: HTML5 file upload and download

3.1. Video introduction

3.2. File API and Ajax / XHR2 requests

3.3. Drag and drop: the basics

3.4. Drag and drop: working with files

3.5. Forms and files

3.6. IndexedDB

3.7. Conclusion on client-side persistence

Module 4: Web components and other HTML5 APIs

4.2.Web Components

4.3.Web Workers

4.4.The Orientation and Device Motion APIs

4.5.Where to from here?

W3Cx-4of5-HTML5.2x – Apps and Games HTML5.2x Apps and Games - git

Course outline

During this course, you will discover advanced HTML5 techniques to help you develop innovative projects and applications. Please try to work on one of the many proposed optional projects, to be found at the end of each section. Remember that if you are not comfortable with JavaScript, no worries. Just start creating from one of the provided examples or follow our JavaScript Introduction course. And most of all, have fun!

Play with audio and video in "Module 1: Advanced HTML5 multimedia".
Start programming an HTML5-based game of your own in Module 2! Learn the core techniques for writing 2D video games that run at 60 frames/s. Discover basic game development concepts, set up a single game framework, understand time-based animation, animate multiple objects, use sprite sheets, detect collisions, deal with gamepad events, and many more.
Module 3 is about drag and drop, upload/download files with Ajax/XHR2 and IndexedDB. The Indexed Database API is a recommended standard interface for a local database of records holding simple values and hierarchical objects. Note that IndexedDB is also useful for storing your HTML5 game scores!
This is the last set of lectures! Module 4 gives a lot of space to Web components as they help build Web pages using ready to use standardized building blocks. Web components comprise Custom Elements, Shadow DOM and HTML Imports and Templates.
These specifications are under continuous development at W3C.

This module also gives you a flavor of other HTML5 APIs such as the Orientation API which is useful for monitoring and controlling games and other activities; and Web Workers which introduce the power of parallel processing to Web apps.

This HTML5 Apps and Games course is part of the Front-End Web Developer" (FEWD) Professional Certificate program. To get this FEWD certificate, you will need to successfully pass all 5 courses that compose that program. Find out more on w3cx.org!

If you already have a verified certificate in one or more of these courses, you do NOT need to re-take that course.

Additional information:

If you are new to the edX platform, we encourage you to (check out DemoX], a quick walk-through of an edX experience. It will help answer basic “how to's” on taking an edX course.
As an alternative, do not hesitate to browse the courses' FAQs in edX Help Center.

Welcome!

This course is one of the courses composing the "Front-End Web Developer”

Professional Certificate program.

You will dive into advanced techniques, combining HTML5, CSS and JavaScript, to create your own HTML5 app and/or game.

During this course, you will notably learn:

Advanced multimedia features with the Track and Audio APIs
HTML5 games techniques
More APIs, including Web Workers and Service Workers
Web components
Persistence techniques for data storage including IndexedDB, File System API and Drag and Drop

Web Browsers and Editors

Web browsers

Not surprisingly, it would be helpful to have a browser (short for "Web Browser") installed so that you can see the end result of your source code. Most common browsers are Edge (and IE), Firefox, Chrome, Safari, etc.

Look for the history of Web browsers (on Wikipedia). An interesting resource is the [market and platform market share] (https://www.w3counter.com/globalstats.php) (updated regularly).

Web editors

While any text editor, like NotePad or TextEdit, can be used to create Web pages, they don't necessarily offer a lot of help towards that end. Some others offer more facilities for error checking, syntax coloring and saving some typing by filling things out for you. Check the following sample:

Sublime Text is quite popular with developers, though there can be a bit of a learning curve to use its many features.
Notepad - on Windows PC's.
Visual Studio - on Windows PC's, many developers are already familiar with it.
TextEdit - This is available on Macs, but be sure you're saving as plain text, not as a .rtf or .doc file.
BlueGriffon is a WYSIWYG ("What You See Is What You Get") content editor for the World Wide Web. Powered by Gecko, the rendering engine of Firefox, it's a modern and robust solution to edit Web pages in conformance to the latest Web Standards.
XCode - Mac developers may be familiar with XCode.
Atom is another cross platform editor, created by GitHub.
Vim or Emacs are great editors on which the Web was built, but if you're not already familiar with these, this isn't the time to try.

To help you practice during the whole duration of the course, you will use the following online editor tools. Pretty much all the course's examples will actually use these.

JS Bin

JS Bin is an open source collaborative Web development debugging tool. This tool is really simple, just open the link to the provided examples, look at the code, look at the result, etc. And you can modify the examples as you like, you can also modify / clone / save / share them.

Tutorials can be found on the Web (such as this one) or on YouTube. Keep in mind that it's always better to be logged in (it's free) if you do not want to lose your contributions/personal work.

CodePen

CodePen is an HTML, CSS, and JavaScript code editor that previews/showcases your code bits in your browser. It helps with cross-device testing, real-time remote pair programming and teaching.

This is a great service to get you started quickly as it doesn't require you to download anything and you can access it, along with your saved projects from any Web browser. Here's an article which will be of-interest if you use CodePen: Things you can do with CodePen [Brent Miller, February 6, 2019].

There are many other handy tools such as JSFiddle, and Dabblet (Lea Verou's tool that we will use extensively in a future CSS course). Please share your favorite tool on the discussion forum, and explain why! Share also your own code contributions, such as a nice canvas animation, a great looking HTML5 form, etc. Sharing them using JS Bin, or similar tools, would be really appreciated.

W3C Validators

For over 20 years, the W3C has been developing and hosting [free and open source tools] used every day by millions of Web developers and Web designers. All the tools listed below are Web-based, and are available as downloadable sources or as free services on the W3C Developers tools site.

W3C Validator

The W3C validator checks the markup validity of various Web document formats, such as HTML.

CSS Validator

The CSS validator checks Cascading Style Sheets (CSS) and (X)HTML documents that use CSS stylesheets.

Unicorn

Unicorn is W3C's unified validator, which helps people improve the quality of their Web pages by performing a variety of checks. Unicorn gathers the results of the popular HTML and CSS validators, as well as other useful services, such as RSS/Atom feeds and http headers.

Link Checker

The W3C Link Checker looks for issues in links, anchors and referenced objects in a Web page, CSS style sheet, or recursively on a whole Web site. For best results, it is recommended to first ensure that the documents checked use valid HTML Markup.

Internationalization Checker

The W3C Internationalization Checker provides information about various internationalization-related aspects of your page, including the HTTP headers that affect it. It also reports a number of issues and offers advice about how to resolve them.

W3C CheatSheet

The W3C cheatsheet provides quick access to useful information from a variety of specifications published by W3C. It aims at giving in a very compact and mobile-friendly format a compilation of useful knowledge extracted from W3C specifications, completed by summaries of guidelines developed at W3C, in particular Web accessibility guidelines, the Mobile Web Best Practices, and a number of internationalization tips.

Its main feature is a lookup search box, where one can start typing a keyword and get a list of matching properties/elements/attributes/functions in the above-mentioned specifications, and further details on those when selecting the one of interest.

The W3C cheatsheet is only available as a pure Web application.

Browser Compatibility

The term browser compatibility refers to the ability of a given Web site to appear fully functional on the browsers available in the market.

The most powerful aspect of the Web is what makes it so challenging to build for: its universality. When you create a Web site, you’re writing code that needs to be understood by many different browsers on different devices and operating systems!

To make the Web evolve in a sane and sustainable way for both users and developers, browser vendors work together to standardize new features, whether it’s a new HTML element, CSS property, or JavaScript API. But different vendors have different priorities, resources, and release cycles — so it’s very unlikely that a new feature will land on all the major browsers at once. As a Web developer, this is something you must consider if you’re relying on a feature to build your site.

We are then providing references to the browser support of HTML5 features presented in this course using 2 resources: Can I Use and Mozilla Developer Network (MDN) Web Docs.

Can I use

Can I Use provides up-to-date tables for support of front-end Web technologies on desktop and mobile Web browsers. Below is a snapshot of what information is given by CanIUse when searching for "CSS3 colors".

MDN Web Docs

To help developers make these decisions consciously rather than accidentally,

MDN Web Docs provides browser compatibility tables in its documentation pages, so that when looking up a feature you’re considering for your project, you know exactly which browsers will support it.

External resources:

Help Build the Web Platform

Most of the technologies you use when developing Web applications and Web sites are designed and standardized in W3C in a completely open and transparent process.

In fact, all W3C specifications are developed in public GitHub repositories, so if you are familiar with GitHub, you already know how to contribute to W3C specifications! This is all about raising issues (with feedback and suggestions) and/or bringing pull requests to fix identified issues.

Contribute

Contributing to this standardization process might be a bit scary or hard to approach at first, but understanding at a deeper level how these technologies are built is a great way to build your expertise.

If you're looking to an easy way to dive into this standardization processes, check out which issues in the W3C GitHub repositories have been marked as "good first issue" and see if you find anything where you think you would be ready to help.

Shape the future

Another approach is to go and bring feedback on ideas for future technologies: the W3C Web Platform Community Incubator Group was built as an easy place to get started to provide feedback on new proposals or bring brand-new proposals for consideration.

Happy Web building!

What is W3C?

As steward of global Web standards, W3C's mission is to safeguard the openness, accessibility, and freedom of the World Wide Web from a technical perspective.

W3C's primary activity is to develop protocols and guidelines that ensure long-term growth for the Web. The widely adopted Web standards define key parts of what actually makes the World Wide Web work.

A few history bits

In March 1989, while at CERN, Sir Tim Berners-Lee wrote “Information Management: A Proposal” outlining the World Wide Web. Tim’s memo was about to revolutionize communication around the globe. He then created the first Web browser, server, and Web page. He wrote the first specifications for URLs, HTTP, and HTML.

_{Tim Berners-Lee at his desk in CERN, 1994}

In October 1994, Tim Berners-Lee founded the World Wide Web Consortium (W3C) at the Massachusetts Institute of Technology, Laboratory for Computer Science [MIT/LCS] in collaboration with CERN, where the Web originated (see information on the original CERN Server), with support from DARPA and the European Commission.

In April 1995, Inria became the first European W3C host, followed by Keio University of Japan (Shonan Fujisawa Campus) in Asia in 1996. In 2003, ERCIM took over the role of European W3C Host from Inria. In 2013, W3C announced Beihang University as the fourth Host.

A few figures

As of August 2020, W3C:

Is a member-driven organization composed of approx 430 companies, universities, start-ups, etc. from all over the world.
Holds 46 technical groups, including Working Groups and Interest Groups where technical specifications are discussed and developed.
Published over 7,254 published technical reports, including 434 Web standards (or W3C Recommendations) - since January 1st,1995.
Runs a translation program to foster the translation of its specifications: see the translation matrix currently listing 309 available translations of W3C recommendations.
Hosts 338 Community and Business Groups, where developers, designers, and anyone passionate about the Web have a place to hold discussions and publish ideas.
Gathers over 13,129 active participants constituting the W3C community.
Has a technical staff composed of 64 people, spread on all five continents

W3C's core values

Committed to core values of an open Web that promotes innovation, neutrality, and interoperability, W3C and its community are setting the vision and standards for the Web, ensuring the building blocks of the Web are open, accessible, secure, international and have been developed via the collaboration of global technical experts.

Essential Steps in Web i18n

You find below three examples (and checks!) to help you to ensure that your Web page works for people around the world, and to make it work differently for different cultures, where needed. Let's meet the words 'charset' and 'lang', soon to become your favorite markup ;)

Example #1: character encoding declaration

A character encoding declaration is vital to ensure that the text in your page is recognized by browsers around the world, and not garbled. You will learn more about what this is, and how to use it as you work through the course. For now, just ensure that it's always there.

Check #1: There is a character encoding declaration near the start of your source code, and its value is UTF-8.

<head>
<meta charset="utf-8"/>
...
</head>

Example #2: primary language declaration

For a wide variety of reasons, it's important for a browser to know what language your page is written in, including font selection, text-to-speech conversion, spell-checking, hyphenation and automated line breaking, text transforms, automated translation, and more. You should always indicate the primary language of your page in the <html> tag. Again you will learn how to do this during the course. You will also learn how to change the language, where necessary, for parts of your document that are in a different language.

Check #2: The HTML tag has a lang attribute which correctly indicates the language of your content.

This example below indicates that the page is in French.

<!doctype html>
<html lang="fr">
<head>
...

Example #3: cultural bias

People around the world don't always understand cultural references that you are familiar with, for example the concept of a 'home run' in baseball, or a particular type of food. You should be careful when using examples to illustrate ideas. Also, people in other cultures don't necessarily identify with pictures that you would recognize, for example, hand gestures can have quite unexpected meanings in other parts of the world, and photos of people in a group may not be representative of populations elsewhere. When creating forms for capturing personal details, you will quickly find that your assumptions about how personal names and addresses work are very different from those of people from other cultures.

Check #3: If your content will be seen by people from diverse cultures, check that your cultural references will be recognized and that there is no inappropriate cultural bias.

Don't worry!

The following 7 quick tips summarize some important concepts of international Web design. They will become more meaningful as you work through the course, so come back and review this page at the end.

Encoding: use the UTF-8 (Unicode) character encoding for content, databases, etc. Always declare the encoding.
Language: declare the language of documents and indicate internal language changes.
Navigation: on each page include clearly visible navigation to localized pages or sites, using the target language.
Escapes: use characters rather than escapes (e.g. á á or á) whenever you can.
Forms: use UTF-8 on both form and server. Support local formats of names/addresses, times/dates, etc.
Localizable styling: use CSS styling for the presentational aspects of your page. So that it's easy to adapt content to suit the typographic needs of the audience, keep a clear separation between styling and semantic content, and don't use 'presentational' markup.
Images, animations & examples: if your content will be seen by people from diverse cultures, check for translatability and inappropriate cultural bias.

You will find more quick tips on the Internationalization quick tips page. Remember that these tips do not constitute complete guidelines.

Internationalization checker

When you start creating Web pages, you can also run them through the W3C's Internationalization Checker. If there are internationalization problems with your page, this checker explains what they are and what to do about it.

Module 1 Advanced HTML5 multimedia

In the [W3Cx HTML5 Coding Essentials and Best Practices](https://www.edx.org/course/html5-coding-essentials-and-best-practices) course, we saw that <video> and <audio> elements can have <track> elements. A <track> can have a label, a kind (subtitles, captions, chapters, metadata, etc.), a language (srclang attribute), a source URL (src attribute), etc.

Here is a small example of a video with 3 different tracks ("......" masks the real URL here, as it is too long to fit in this page width!):

Video code extract!

1.  <video id="myVideo" preload="metadata" controls crossOrigin="anonymous">
2.    <source src="https://...../elephants-dream-medium.mp4" type="video/mp4">
3.    <source src="https://...../elephants-dream-medium.webm" type="video/webm">
4.      <track label="English    subtitles" kind="subtitles" srclang="en"
5.        src="https://...../elephants-dream-subtitles-en.vtt">
6.      <track label="Deutsch subtitles" kind="subtitles" srclang="de"
7.        src="https://...../elephants-dream-subtitles-de.vtt" default>
8.      <track label="English chapters" kind="chapters" srclang="en"
9.        src="https://...../elephants-dream-chapters-en.vtt">
10. </video>

And here is how it renders in your current browser (please play the video and try to show/hide the subtitles/captions):

Notice that the support for multiple tracks may differ significantly from one browser to another, in particular if you are using old versions.

Here is a quick summary (as of May 2020).

Safari provides a menu you can use to choose which subtitle/caption track to display. If one of the defined text tracks has the default attribute set, then it is loaded by default. Otherwise, the default is off.

Chrome and Opera both provide a subtitle menu and load the text track set that matches the browser language. If none of the available text tracks match the browser’s language, then it loads the track with the default attribute, if there is one. Otherwise, it loads none. Let's say that support is very incomplete (!).
Firefox provides also a subtitle menu but will show the first defined text track only if it has default set. It will load all tracks in memory as soon as the page is loaded.

Also, there is a Timed Text Track API in the HTML5/HTML5.1 specification that enables us to manipulate <track> contents from JavaScript. Do you recall that text tracks are associated with WebVTT files? As a quick reminder, let's look at a WebVTT file:

WEBVTT

WEBTT code extract!

1. 
2.  1
3.  00:00:15.000 --> 00:00:18.000 align:start
4.  <v Proog>On the left we can see...</v>
5. 
6.  2
7.  00:00:18.167 --> 00:00:20.083 align:middle
8.  <v Proog>On the right we can see the...</v>
9. 
10. 3
11. 00:00:20.083 --> 00:00:22.000
12. <v Proog>...the <c.highlight>head-snarlers</c></v>
13. 
14. 4
15. 00:00:22.000 --> 00:00:24.417 align:end
16. <v Proog>Everything is safe. Perfectly safe.</v>

The different time segments are called "cues" and each cue has an id (1, 2, 3 and 4 in the above example), a startTime and an endTime, and a text content that can contain HTML tags for styling (<b>, etc...) or be associated with a "voice" as in the above example. In this case, the text content is wrapped inside <v name_of_speaker>... elements.

It's now time to look at the JavaScript API for manipulating tracks, cues, and events associated with their life cycle. In the following lessons, we will look at different examples which use this API to implement missing features such as:

how to build a menu for choosing the subtitle track language to display,
how to display a synchronized description of a video (useful for disabled people, for example),
how to display a clickable transcript aside the video (similar to what the edX video player does),
how to show chapters,
how to use JSON encoded cue contents (useful for showing external resources in the HTML document while a video is playing),
etc.

Module 1

Hi! Welcome to Part 2 of the W3C HTML5 course. How about we start by looking at advanced HTML5 multimedia features? First, we will look at the Web Audio API that helps processing and synthesizing audio in Web applications. You will be able to load sound samples into memory and play them, loop them, process them through a chain of sound effects such as reverberation, delay, graphic equalizer, compressor, distortion, etc. You can also write nice real time visualizations like dancing frequency graphs, animated waveforms that dance with the music, or generate music programmatically.

The Web Audio API is particularly suited for games or for music applications.

A second nice multimedia feature is the Track API.

With it, you will be able to synchronize a video with elements in your document.

For example: display a Google Map, an HTML description or a Wikipedia page aside the video, while it’s playing.

As always, do not hesitate to practice coding looking at the interactive examples, and then please share your own creations in the discussion forum!

We hope you will enjoy this first week and we wish you the best!

1.2.1 Intro to the Timed Text Track API

Hi, today I've prepared for you a small example of a video that is associated with three different tracks.

Two for subtitles, in English and in German, and one track for chapters.

First, before going further, let's look at how this is rendered in different browsers. With Google Chrome, we've got a CC button here, that will enable or disable subtitles. What we can see is that by default the subtitles that are displayed are in German (in this example).

And I can just switch them on and off. It loaded the first track that has the default attribute. And I have no menu for choosing what track, what language I want to be displayed here …

If we look at FireFox, it's even worse!

We don't have any menu at all, no CC button. I cannot switch on the subtitles, because as of December 2015, FireFox will load only the first track, if it has the default attribute.

This is not the case, so we don't have any subtitles and we cannot display them.

With Safari, on my Mac, it's better because I've got a subtitle menu and I can choose between the different tracks. I can switch to English subtitles, and english subtitles will be displayed.

I'm on a MacIntosh so I cannot show you... but with other browsers like Internet Explorer or Microsoft Edge, there are situations similar to Safari.

What can we do to increase the features of the default player? We can use what we call the Track API, for asking which tracks are available, activating them, and so on. Now, I would like to remind you the structure of a track. I'm just going to display the content of one of these tracks.

The tracks are made of cues and what we call a cue is a kind of time segment that is defined with a starting time and an ending time. And the cue can have an ID, in that case it is a numeric ID (1, 2, 3), and a content that can be HTML with bold, italic elements and it can also be a voice so when you see "v" followed by the name of the character that is speaking, it’s a voice.

We are going to look at what we can do with such tracks during the course and we will see how to handle a chapter menu, how to display a nice transcript on the side of the video, that you can click to jump at the exact time the video tells the words that are on the transcript. And we will see also how to choose the subtitle or caption track language for the video.

This is finished for this small introduction video, I will just conclude by this thing here: explaining this crossOrigin="anonymous". We saw that during the HTML5 Part 1 course and many people asked questions about this. This is because of security constraints.

In browsers, when you've got the HTML page that is on a different location than the video file and the tracks files, you will have security constraints errors. And if your server is configured for accepting different origins, then you can add this attribute crossOrigin="anonymous" in your HTML document and it is going to work.

The server here: mainline.i3s.unice.fr has been configured for allowing external HTML pages to include the videos it hosts and the subtitles it hosts, this is the reason. You can use the DropBox public directory here because Dropbox also enables cross origin requests.

This is all for this first video, I’ll see you in the next one!

The Timed Text Track JavaScript API

In the W3Cx HTML5 Coding Essentials and Best Practices course, we saw that <video> and <audio> elements can have <track> elements. A <track> can have a label, a kind (subtitles, captions, chapters, metadata, etc.), a language (srclang attribute), a source URL (src attribute), etc.

Here is a small example of a video with 3 different tracks ("......" masks the real URL here, as it is too long to fit in this page width!):

Code extract!

<video id="myVideo" preload="metadata" controls crossOrigin="anonymous">
  <source src="https://...../elephants-dream-medium.mp4" type="video/mp4">
  <source src="https://...../elephants-dream-medium.webm" type="video/webm">
  <track label="English subtitles" kind="subtitles" srclang="en"
    src="https://...../elephants-dream-subtitles-en.vtt">
  <track label="Deutsch subtitles" kind="subtitles" srclang="de"
    src="https://...../elephants-dream-subtitles-de.vtt" default>
  <track label="English chapters" kind="chapters" srclang="en"
    src="https://...../elephants-dream-chapters-en.vtt">
</video>

And here is how it renders in your current browser (please play the video and try to show/hide the subtitles/captions):

Notice that the support for multiple tracks may differs significantly from one browser to another, in particular if you are using old versions.

You can read this article by Ian Devlin: "HTML5 Video Captions – Current Browser Status", written in April 2015, for further details.

Here is a quick summary:

Safari provides a menu you can use to choose which subtitle/caption track to display. If one of the defined text tracks has the default attribute set, then it is loaded by default. Otherwise, the default is off.

Chrome and Opera both provide a subtitle menu and load the text track set that matches the browser language. If none of the available text tracks match the browser’s language, then it loads the track with the default attribute, if there is one. Otherwise, it loads none. Let's say that support is very incomplete (!).
Firefox provides a subtitle menu but will show the first defined text track only if it has default set. It will load all tracks in memory as soon as the page is loaded.

There is a Timed Text Track API in the HTML5/HTML5.1 specification that enables us to manipulate <track> contents from JavaScript. Do you recall that text tracks are associated with WebVTT files? As a quick reminder, let's look at a WebVTT file:

WEBVTT code extract!

1.  WEBVTT
2.   
3.  1
4.  00:00:15.000 --> 00:00:18.000 align:start
5.  <v Proog>On the left we can see...</v>
6.   
7.  2
8.  00:00:18.167 --> 00:00:20.083 align:middle
9.  <v Proog>On the right we can see the...</v>
10.  
11. 3
12. 00:00:20.083 --> 00:00:22.000
13. <v Proog>...the <c.highlight>head-snarlers</c></v>
14.  
15. 4
16. 00:00:22.000 --> 00:00:24.417 align:end
17. <v Proog>Everything is safe. Perfectly safe.</v>

The different time segments are called "cues" and each cue has an id (1, 2, 3 and 4 in the above example), a startTime and an endTime, and a text content that can contain HTML tags for styling (<b>, etc...) or be associated with a "voice" as in the above example. In this case, the text content is wrapped inside <v name_of_speaker>... elements.

It's now time to look at the JavaScript API for manipulating tracks, cues, and events associated with their life cycle. In the following lessons, we will look at different examples which use this API to implement missing features such as:

how to build a menu for choosing the subtitle track language to display,
how to display a synchronized description of a video (useful for disabled people, for example),
how to display a clickable transcript aside the video (similar to what the edX video player does),
how to show chapters,
how to use JSON encoded cue contents (useful for showing external resources in the HTML document while a video is playing),
etc.

1.2.2 The HTML Track Element, getting the status of a track

Hi, in this video I will show you how we can work with the track elements from JavaScript, just to know which track has been loaded and which track is active.

For that, we will manipulate different properties of the HTML track element from JavaScript.

The first thing I am going to do is to add a small div at the end of the document for displaying the different track statuses. I added a div here called trackStatusesDiv with a heading... I am going to add some CSS to visualize this area.

Like that, we will have the description of the track here. I added a border and some margins and so on. From JavaScript, we can not do anything before the page has been loaded, so I am adding a window.onload listener, and all the treatments will be in this function. The first thing I am going to do is get these track elements here... and I am going to get them in a variable called htmlTracks. How can I get them?

I'm going to stop the automatic refresh on JSBin for the moment.

So, querySelectorAll is a function that will return a collection with all the tracks, an array with all the tracks, all the HTML elements. I am going to call a function called displayTrackStatuses that I write here. I will first iterate on these tracks and display in the console the different values. We are doing a loop.

I will first display something in the console.

I am going to add a current track, it will be easier.

I can write currentTrack.label for example, that will display the value of the different attributes.

This is just for checking that my code is OK.

Open the console, I have got one error...

If I click here "Run with JS" I can see that it's working.

What I can display is the label, I can also display the kind... you remember the kind: subtitles, subtitles, chapters for the different track subtitles, subtitles and chapters and I can also display the language with srclang. So English, Deutsch (for German).

I can also display what is called the status. It is readyState, this is a property you can use only from JavaScript. It says that for track number 1, the value is 0, for track number 2 is 2, for track number 3, it's 0. 2 means that the track is loaded and 0 means that the track is not available.

We've got the first track, the English subtitles are not available: status readyState 0 and we've got the German subtitles that have been loaded because it has the default attribute, and we showed that in Google Chrome the track with the default attribute is loaded when the page is loaded.

This is how we can consult the different statuses of a track from JavaScript.

I am going to copy and paste some code for just displaying this in a nicer way here.

This is how we can display the different statuses and I can add a button that will call this just to refresh the different statuses.

Let's add a button, we call it refresh, and when we click on it, we will call displayTrackStatuses. If I click here... just checking there is no error…it will refresh this thing. So now I am going to try this code with Safari; because you remember Safari has a menu for changing the different tracks.

I prepared that already, so here I can start playing the same video that has Deutsch subtitles loaded by default, you can see that here. If I choose English subtitles now, and I refresh track statuses: you can see that the English subtitles are loaded now and the Deutsch subtitles are also available. We are going to use these different attributes for forcing some tracks to load programmatically from JavaScript and this will enable us to make a sort of menu for choosing the different tracks.

I will explain that in a next video.

The HTML track element

Let's go back to our example. Below is the HTML code:

Code extract!

1. <video id="myVideo" preload="metadata" controls crossOrigin="anonymous">
2.   <source src="https://...../elephants-dream-medium.mp4" type="video/mp4">
3.   <source src="https://...../elephants-dream-medium.webm" type="video/webm">
4.     <track label="English subtitles" kind="subtitles" srclang="en"
         src="https://...../elephants-dream-subtitles-en.vtt" >
5.     <track label="Deutsch subtitles" kind="subtitles" srclang="de"
         src="https://</b>...../elephants-dream-subtitles-de.vtt" default>
6.     <track label="English chapters" kind="chapters" srclang="en"
7.       src="https://</b>...../elephants-dream-chapters-en.vtt">
8. </video>
9.  
10.<div id="trackStatusesDiv">
11.  <h3>HTML track descriptions</h3>
12.</div>

This example defines three <track> elements. From JavaScript, we can manipulate these elements as "HTML elements" - we will call them the "HTML views" of tracks.

Getting the status of an HTML track

Example at JSBin that displays the different elements we can get from an HTML track:

Here is the JavaScript source code:

Function displayTrackStatuses code extract!

1.  var video, htmlTracks;
2.  var trackStatusesDiv;
3.  window.onload = function() {
4.  // called when the page has been loaded
5.  video = document.querySelector("#myVideo");
6.  trackStatusesDiv = document.querySelector("#trackStatusesDiv");
7.  // Get the tracks as HTML elements
8.  htmlTracks = document.querySelectorAll("track");
9.  // displays their statuses in a div under the video
10. displayTrackStatuses(htmlTracks);
11. };
12. function displayTrackStatuses(htmlTracks) {
13. // displays track info
14. for(var i = 0; i < htmlTracks.length; i++) {
15. var currentHtmlTrack = htmlTracks[i];
16. var label = "<li>label = " + currentHtmlTrack.label + "</li>";
17. var kind = "<li>kind = "   + currentHtmlTrack.kind + "</li>";
18. var lang = "<li>lang = "   + currentHtmlTrack.srclang + "</li>";
19. var readyState = "<li>readyState = " + currentHtmlTrack.readyState + "</li>"
20. trackStatusesDiv.innerHTML += "<li>Track:" + i + ":</b></li>"
21.    + "<ul>" + label + kind + lang + readyState + "</ul>";
22. }
23. }

The code is rather straightforward:

We cannot access any HTML element before the page has been loaded. This is why we do all the work in the window.onload listener,
Line 6 we get a pointer to the div with id=trackStatusesDiv, that will be used to display track statuses,
Line 8: we get all the track elements in the document. They are HTML track elements,
Line 12: we call a function that will build some HTML to display the track status in the div we got from line 7.
Lines 14-19: we iterate on the HTML tracks, and for each track we get the label, the kind and the srclang attribute values. Notice, at line 19, the use of the readyState attribute, only used from JavaScript, that will give the current HTML track state.

You can see on the screenshot (or from the JSBin example) that the German subtitle file has been loaded, and that none of the other tracks have been loaded.

Possible values for the readyState attribute of HTML tracks:

0 = NONE ; the text track's cues have not been obtained
1 = LOADING ; the text track is loading with no errors yet. Further cues can still be added to the track by the parser
2 = LOADED ; the text track has been loaded with no errors
3 = ERROR ; the text track was enabled, but when the user agent attempted to obtain it, something failed. Some or all of the cues are likely missing and will not be obtained

Now, it's time to look at the twin brother of an HTML track: the corresponding TextTrack object!

1.2.3 The TextTrack Object

Hi! Now we are preparing ourselves for reading the content of the different text tracks and display them.

But before that, I must introduce what we call the TextTrack object that is a JavaScript object that is associated to the HTML elements. A track has two different views... maybe it is simpler to say that it has got a HTML view that means you can do a getElementById and manipulate the HTML element, this track element here, from JavaScript. Or we can also work with its twin brother that is a text track, and this new view is the one we are going to use for forcing a track to be loaded, and for reading its content. And for forcing subtitles or caption track to be displayed. We just slightly modify the previous example by displaying the mode

The mode is a property from the TextTrack, not from the HTML track. And this mode can be "disabled", "showing" or "hidden". And when it is disabled, reading the video will not fire any event related to the track. We will talk about events later but a disabled track is the same if we have no track at all.

A track that is "showing" is displayed in the video, if the implementation of the video player supports that. And a track that is "hidden" is just not displayed.

How did we manipulate and access this mode property?

The displayTrackStatuses function, that we wrote earlier, displayed the different properties of the HTML track, like the label, the kind or the language. This time, we accessed his twin brother, the TextTrack by using the track property.

Every HTML track element has a track property that is a TextTrack.

Here, from the current HTML track, I am getting the TextTrack (currentTextTrack).

This is the object we use to access the mode and display it here. Another interesting thing is that if we set the mode, if we modify the value of the mode, from "disabled" to "showing" or to "hidden", it will force the track to be loaded asynchronously in the background by the browser. We added in this example two buttons, "force load track 0" and "force load track 2" because by default, the track 0, the English subtitles, is not loaded. And the chapters, in track number 2, are not loaded either. We are going to force the track 0 to be loaded.

If I click here "force load track 0", you see that the status changes - the mode changes to "hidden" and the track now is loaded. What happened in the background?

Let's have a look at the code we wrote. I am going to zoom a little bit...

The button we clicked is this one: "force load track 0" here, called a function named forceLoadTrack(0) that I prepared.

What does this function do?

It will call another function called getTrack that will check if the track is already loaded.

If it is already loaded, then the second parameter here, is a callback function,

It will be called because the track is ready to be read.

In the case the track has not been loaded, we will set the mode to "hidden" and then we will trigger the browser so that it will load asynchronously, in the background, the track.

And when the track is ready, then, and only then, we will call readContent.

Let's have a look at this getTrack function that we wrote. It says getTrack, please load me the TextTracks corresponding to the HTML track number n.

So here is the function. The first thing we do is that from the HTML track, we get the text track.

Then we check on the HTML track if it is already loaded.

If it is the case, then we will call the function that has been passed as the second parameter: it's the readContent. And the readContent is just here, for the moment it will not read the content really, but it will just update the status.

If I click on « force load track 2 » for example, it will load the track and when the track is arrived, it will call the displayStatus() that will show the updated status of the track.

In the case the track is not here, the readyState is not equal to 2, then we will force the track to be loaded. By doing this we set the mode to "hidden".

This may will take some time: you understand that the browser is loading on the Web the track. It may take 2 seconds for example. We need to have a listener that will listen to the load event.

So htmlTrack.addEventListenner('load'...) will trigger only when the track has been loaded, and only in that case we will call the callback function: the readContent that has been passed in the second parameter, in order to read the track.

If I look at the console, and if I start again the application. Only the second track has been loaded, I click "force load track 0", it says "forcing the track to be loaded", it loads the track and it calls the callback "reading content of loaded track".

If I click again the same button, it says "the text track is already loaded" and I am going to read it now. We cannot load a track several times, if it is already loaded, we must just use it. In the next video, we will show how we can effectively read the content of the track and do something with it.

The object that contains the cues (subtitles or captions or chapter description from the WebVTT file) is not the HTML track itself. It is another object that is associated with it: a TextTrack object!

The TextTrack JavaScript object has different methods and properties for manipulating track content, and is associated with different events. But before going into detail, let's see how to obtain a TextTrack object.

Obtaining a TextTrack object that corresponds to an HTML track

First method: get a TextTrack from its associated HTML track.

The HTML track element has a track property which returns the associated TextTrack object.

Example source code:

Code extract!

// HTML tracks
var htmlTracks = document.querySelectorAll("track");

// The TextTrack object associated with the first HTML track
var textTrack = htmlTracks[0].track;
var kind = textTrack.kind;
var label = textTrack.label;
var lang = textTrack.language;

// etc.

Note that once we get a TextTrack object, we can manipulate the kind, label, language attributes (be careful, it's not srclang, like the equivalent attribute name for HTML tracks). Other attributes and methods are described later in this lesson.

Second method: get TextTrack from the HTML video element.

The <video> element (and <audio> element too) has a TextTrack property accessible from JavaScript:

var videoElement = document.querySelector("#myVideo");
var textTracks = videoElement.textTracks; // one TextTrack for each HTML track element
var textTrack = textTracks[0]; // corresponds to the first track element
var kind = textTrack.kind // e.g. "subtitles"
var mode = textTrack.mode // e.g. "disabled", "hidden" or "showing"

The mode property of TextTrack objects TextTrack objects have a mode property, that is set to one of:

"showing": the track is either already loaded, or is being loaded by the browser. As soon as it is completely loaded, subtitles or captions will be displayed in the video. Other kinds of track will be loaded but will not necessarily show anything visible in the document. All tracks that have mode="showing" will fire events while the video is being played.
"hidden": the track is either already loaded, or is being loaded by the browser. All tracks that have mode="hidden" will fire events while the video is being played. Nothing will be visible in the standard video player GUI.
"disabled": this is the mode where tracks are not being loaded. If a loaded track has its mode set to "disabled", it will stop firing events, and if it was in mode="showing" the subtitles or captions will stop being displayed in the video player.

TextTrack content can only be accessed if a track has been loaded! Use the mode property to force a track to be loaded!

BE CAREFUL: you cannot access a TextTrack content if the corresponding HTML track has not been loaded by the browser! It is possible to force a track to be loaded by setting the mode property of the TextTrack object to "showing" or "hidden".
Tracks that are not loaded have their mode property of "disabled".

Here is an example that will test if a track has been loaded, and if it hasn't, will force it to be loaded by setting its mode to "hidden". We could have used "showing"; in this case, if the file is a subtitle or a caption file, then the subtitles or captions will be displayed on the video as soon as the track has finished loading.

Try the example at JSBin

Here is what we added to the HTML code:

Code extract!

1.  <button id="buttonLoadFirstTrack"
2.          onclick="forceLoadTrack(0);"
3.          disabled>
4.     Force load track 0
5.  </button>
6.  <button id="buttonLoadThirdTrack"
7.          onclick="forceLoadTrack(2);"
8.          disabled>
9.     Force load track 2
10. </button>

The buttons will call a function named forceLoadTrack(trackNumber) that takes as a parameter the number of the track to get (and force load if necessary).

Here are the additions we made to the JavaScript code from the previous example:

Functions readContent, getTrack, & forceLoadTrack code extract!

1.  function readContent(track) {
2.    console.log("reading content of loaded track...");
3.    displayTrackStatuses(htmlTracks); // update document with new
      track statuses
4.  }
5.   
6.  function getTrack(htmlTrack, callback) {
7.    // TextTrack associated to the htmlTrack
8.    var textTrack = htmlTrack.track;
9.  
10.   if(htmlTrack.readyState === 2) {
11.     console.log("text track already loaded");
12.     // call the callback function, the track is available
13.     callback(textTrack);
14.   } else {
15.     console.log("Forcing the text track to be loaded");
16.  
17.     // this will force the track to be loaded
18.     textTrack.mode = "hidden";  
        // loading a track is asynchronous, we must use an event listener
19.     htmlTrack.addEventListener('load', function(e) {
20.     // the track is arrived, call the callback function
21.     callback(textTrack);
22.       });
23.     }
24. }
25. 
26. function forceLoadTrack(n) {
27.   // first parameter = track number,
28.   // second = a callback function called when the track is loaded,
29.   // that takes the loaded TextTrack as parameter
30.   getTrack(htmlTracks[n], readContent);
31. }

Explanations:

Lines 26-31: the function called when a button has been clicked. This function in turn calls the getTrack(trackNumber, callback) function. It passes the readContent callback function as a parameter. This is typical JavaScript asynchronous programming: the getTrack() function may force the browser to load the track and this can take some time (a few seconds), then when the track has downloaded, we ask the getTrack function to call the function we passed (the readContent function, which is known as a callback function), with the loaded track as a parameter.
Line 6: the getTrack function. It first checks if the HTML track is already loaded (line 10). If it is, it calls the callback function passed by the caller, with the loaded TextTrack as a parameter. If the TextTrack is not loaded, then it sets its mode to "hidden". This will instruct the browser to load the track. Because that may take some time, we must use a load event listener on the HTML track before calling the callback function. This allows us to be sure that the track is really completely loaded.
Lines 1-4: the readContent function is only called with a loaded TextTrack. Here we do nothing special for the moment except that we refresh the different track statuses in the HTML document.

1.2.4 Working With Cues

Hi! We will continue the last example from the previous video, and this time when we will click on the "force load track 0" or "force load track 2" buttons.

You remember the track number 0 (the English subtitles) was not loaded, readyState=0 here says the track is not loaded. Track 2 also was not loaded: it contains the English chapters of the video. This time, I will explain how we can read the content of the file. If I click on "force load track 0", I see here the content of the WebVTT file. I didn't read it as pure text, I used the track API for accessing individually each cue, each one of these elements here is a cue, and I access the id, the start time, the end time and the content, that we call the text content of each cue. If I click on "force load track 2", I see the chapters definitions here, so chapter 1 of the video goes from 0 to 26 seconds, and it corresponds to the introduction part of the video.

How did we do that? We just completed the readContent function that previously just showed the statuses of the different tracks. Remember that when we clicked on a button, we forced the text track corresponding to the HTML track to be loaded in memory, and then we can read it. A TextTrack object has different properties and the most important one is called cues. The cues is the list of every cue inside the VTT file, and each cue corresponds to a time segment, has an id and a text content.

If you do track.cues, you've got the list of the cues and you can iterate on them.

For each cue, we are going to get its id: cue.id here. It corresponds to the id of the cue number i. In my example, I have got an index in the loop, I get the current cue,

I get the id of this cue. I can also get the start time, the end time and the text. So, cue.text corresponds exactly at this sentence highlighted here. This is the only thing I wanted to show you, because the next time we are going to do something really interesting with this content here, we are going to display on the side of the video a clickable transcript. And when we will click on it, the video will jump to the corresponding position. This is exactly what the edX video player does, the one you are watching at right now.

A TextTrack object has different properties and methods;

kind: equivalent to the kind attribute of HTML track elements. Its value is either "subtitles", "caption", "descriptions", "chapters", or "metadata". We will see examples of chapters, descriptions and metadata tracks in subsequent lessons.
label: the label of the track, equivalent of the label attribute of HTML track elements.
language: the language of the text track, equivalent to the srclang attribute of HTML track elements (be careful: it's not the same spelling!)
mode: explained earlier. Can have values equal to: "disabled"|"hidden"|"showing". Can force a track to be loaded (by setting the mode to "hidden" or "showing").
cues: get a list of cues as a TextTrackCueList object. This is the complete content of the WebVTT file!
activeCues: used in event listeners while the video is playing. Corresponds to the cues located in the current time segment. The start and end times of cues can overlap. In reality this may rarely happen, but this property exists in case it does, returning a TextTrackCueList object that contains all active tracks at a given time.
addCue(cue): add a cue to the list of cues.
removeCue(cue): remove a cue from the list of cues.
getCueById(id): returns the cue with a given id (not implemented by all browsers - a polyfill is given in the examples from the next lessons).

A TextTrackCueList is a collection of cues, each of which has different properties and methods;

id: the cue id as written in the line that starts cues in the WebVTT file.
startTime and endTime: define the time segment for the cue, in seconds, as a floating point value. It is not the formatted String we have in the WebVTT file (see screenshot below),
text: the cue content.
getCueAsHTML(): a method that returns an HTML version of the cue content, not as plain text.
Others such as align, line, position, size, snapToLines, etc., that correspond to the position of the cue, as specified in the WebVTT file. See the HTML5 course Part 1 about cue positioning.

Example that displays the content of a track

Here is an example at JSBin that displays the content of a track:

We just changed the content of the readContent(track) method from the example in the previous lesson:

Function readContent extract!

1.  function readContent(track) {
2.     console.log("reading content of loaded track...");
3.     //displayTrackStatuses(htmlTracks);
4.     // instead of displaying the track statuses, we display
5.     // in the same div, the track content//
6.     // first, empty the div
7.     trackStatusesDiv.innerHTML = "";
8.  
9.     // get the list of cues for that track   
10.    var cues = track.cues;
11.    // iterate on them
12.    for(var i=0; i < cues.length; i++) {
13.       // current cue
14.       var cue = cues[i];
15.       var id = cue.id + "<br>";
16.       var timeSegment = cue.startTime + " => " + cue.endTime + "<br>";
17.       var text = cue.text + "<P>"
18.       trackStatusesDiv.innerHTML += id + timeSegment + text;
19.    }
20. }

As you can see, the code is simple: you first get the cues for the given TextTrack (it must be loaded; this is the case since we took care of it earlier), then iterate on the list of cues, and use the id, startTime, endTime and text properties of each cue.

This technique will be used in one of the next lessons, and we will show you how to make a clickable transcript on the side of the video - something quite similar to what the edX video player does.

1.2.5 Listening to Events

Ok. This time we will talk about track events and cue events.

First, let's start by a small demonstration. If I play this video, the video is going on and I can listen to events like 'cuenter' and 'cueexit'.

Each time a new cue is entered, we will display it here, and each time it is exited, we will display it here.

We saw that we can display them in sync with the video now. Each time a cue is reached, it means that the current time entered a new time segment defined by the starting and ending time of a cue.

Let's have a look again at one of the VTT files.

Each cue holds a start time and a end time so when the time enters the 15th second, we've got a cueenter event and we can get this content and show it on the HTML page.

When we go out of this time period, when we go further than 18 seconds, we exit this cue and we enter this cue.

How are these events handled in the JavaScript code? Everything is done in the readContent method that we saw earlier.

This time instead of iterating on different cues of the TextTrack, we will just, for each cue individualy, we will iterate on the cues and add a listener on that cue, an exist listener and an enter listener.

So what do we do? We iterate in the cues called addCueListenners for the current cue.

This method, addCueListeners, it will define two listeners: the cue enter listener and the cue exit event listener.

On the cue enter listener we just create a string "entered cue id=" and "text=" that will correspond to the text displayed when a cue is just reached.

We display the id and we display the text.

The same thing is done when we exit.

When we exit, we just display the id "exited cue id=". This is how we can have individual enter and exit listeners for each cue.

It will enable us to highlight the current cue in a transcript while the video is playing.

The only problem is that as of December 2015, FireFox still does not recognize these sorts of listeners.

The implementation is not done yet, so you can use a fallback. You can use a listener on the track that will listen to the 'cuechange' event.

If I just comment the function addCueListeners and I uncomment this piece of code that has a cueChange listener on the track itself, then instead of knowing that we entered of exited and individual cue, we can get, for every new time segment, the list of the cues that are triggered, that should be activated and displayed for this time segment. As the different cues can overlap, the time segments can overlap -it is not often the case but it may occur- what the callback function from this listener gives is a list of active cues. Most of the time you have got only one.

Anyway, you can just work with the list of the cues. Here we just take the first active cue because we are assuming that the cues are not overlapping.

I added a small test here because sometimes we've got some strange ghost cues that are active and not defined. I do not exactly what was the problem when I test it but I added this test here to avoid some error messages...

We have got the first active cue and we just display it. We get the id, we get the text and this is all. In that case, if I run the application again, instead of having enter and exit, I will just have cue change events.

It starts at 15 seconds.

I had a small bug here, it is not this.id but cue.id.... we can start again.

This is a fallback for FireFox if you want to display the cues in sync with the video.

The next video will show how to display a transcript here with the current cue highlighted and you can click on them in order to jump to the right place in the video.

After that I think we will have seen the most useful properties, methods, events you can use with tracks and cues...

Instead of reading the whole content of a track at once, like in the previous example, it might be interesting to process the track content cue by cue, while the video is being played.

For example, you choose which track you want - say, German subtitles - and you want to display the subtitles in sync with the video, below the video, with your own style and animations...

Or you display the entire set of subtitles to the side of the video and you want to highlight the current one...

For this, you can listen for different sorts of events.

The two types of cue event are:

enter and exit events fired for cues.
cuechange events fired for TextTrack objects (good support).

Example of cuechange listener on TextTrack

// track is a loaded TextTrack
track.addEventListener("cuechange", function(e) {
   var cue = this.activeCues[0];
   console.log("cue change");
   // do something with the current cue
});

In the above example, let's assume that we have no overlapping cues for the current time segment.

The above code listens for cue change events: when the video is being played, the time counter increases. And when this time counter value reaches time segments defined by one or more cues, the callback is called.

The list of cues that are in the current time segments are in this.activeCues; where this represents the track that fired the event.

In the following lessons, we show how to deal with overlapping cues (cases where we have more than one active cue).

Example of enter and exit event listeners on a track's cues.

Function addCueListeners code!

1.   // iterate on all cues of the current track
2.   var cues = track.cues;
3.   for(var i=0, len = cues.length; i < len; i++) {
4.     // current cue, also add enter and exit listeners to it
5.     var cue = cues[i];
6.     addCueListeners(cue);
7.  
8.     ...
9.   }
10.  
11. function addCueListeners(cue) {
12.   cue.onenter = function(){
13.     console.log('enter cue id=' + this.id);
14.     // do something
15.   };
16. 
17.   cue.onexit = function(){
18.     console.log('exit cue id=' + cue.id);
19.      // do something else
20.   };
21. } // end of addCueListeners...

Showing real examples of event listeners

Here is an example at JSBin that shows how to listen for cuechange events:

Source code extract:

Function readContent code!

1.  function readContent(track) {
2.     console.log("adding cue change listener to loaded track...");
3.     trackStatusesDiv.innerHTML = "";
4.  
5.     // add a cue change listener to the TextTrack
6.     track.addEventListener("cuechange", function(e) {
7.        var cue = this.activeCues[0];
8.        if(cue !== undefined)
9.           trackStatusesDiv.innerHTML += "cue change: text = " + cue.text + "<br>";
10.    });
11. 
12.    video.play();
13. }

And here is another modified version of this example at JSBin, that shows how to use enter and exit events on cues:

Source code extract:

Functions readContent & addCueListeners code extract!

1.  function readContent(track) {
2.     console.log("adding enter and exit listeners to all cues of this > track");
3.  
4.     trackStatusesDiv.innerHTML = "";
5.  
6.     // get the list of cues for that track
7.     var cues = track.cues;
8.     // iterate on them
9.     for(var i=0; i < cues.length; i++) {
10.       // current cue
11.       var cue = cues[i];   
12.       addCueListeners(cue);</b>
13.  }
14. 
15. video.play();
16. }
17.  
18. function addCueListeners(cue) {
19.    cue.onenter = function(){
20.       trackStatusesDiv.innerHTML += 'entered cue id=' + this.id + " > "
21.                                  + this.text + "<br>";
22.    };
23.    cue.onexit = function(){
24.       trackStatusesDiv.innerHTML += 'exited cue > id=' + this.id + "<br>";
25.    };
26. } // end of addCueListeners...

1.3 Advanced Features for <audio> and <video> Players

1.3.1 With a Clickable Transcript on the Side

Hi! This time, we will just go a little bit further than in the previous examples.

When we click on a button, we will force the loading of the track, we will read the content, add cue listeners in order to trigger events when the video is played and we will display them on the side this time, and there are hyperlinks you can click and if I click somewhere, the video starts at the corresponding location and you can see that the cues are highlighted in black as the video advances.

There are not a lot of subtitles at this location but... you can see that the transcript is highlighted as the video is playing.

What have we added to the previous example?

The first thing is that we just defined a rectangular area here: it is just a div with an id="transcript", and we added some CSS here, for locating the video and the transcript on the same horizontal position, so that the video can grow and the transcript too.

We use floating positions, I put ‘float:left;’ for the transcript that is on the right because if I put 'right' it will grow but it will be aligned on the right... and I prefer it on the left.

We can give a look at the CSS, there is nothing complicated and this can scroll because of the overflow:auto; rule we added to the div.

When we click on the buttons here, we call a function called loadTranscript(), instead of forceLoadTrack(0) and forceLoadTrack(1), this time we just ask for a particular language and, it's implicit, but we are also looking for track files that are not chapters.

Let's have a look at this loadTranscript() function here. So the loadTranscript() function has a parameter that is the language.

The first thing we do is that we clear the div, we are just setting the content to null and then we disable all the tracks: we set the mode of the all the tracks ‘disabled’ because when we click here and we can change the language of the transcript, we need to disable all track and enable just the one we are interested in.

How can we locate the right track with the language?

We just iterate on the tracks... this is the text tracks object... and we get the current track as an HTML element and as a TextTrack and using the TextTrack, we just check the language and the kind.

And if the language is equal to the one we are looking for, and if the kind is different than chapters, then this is the track we would like.

By forcing the mode to "showing", in case the track has not already been loaded, it will trigger (it will ask the browser) to load the file.

This is where we test if the file is already been loaded, it is exactly the same test we did in the previous example.

If the track is already loaded, just display the cues in sync with the video and if the track has not been loaded, display the cues after the track has been loaded.

This is the same function as the getContent we had, except that we renamed it.

This takes the track as an HTML element and the TextTrack as parameters.

Let's have a look at how it is done.

"displayCuesAfterTrackLoaded" just waits for the load event to be triggered and then it called display the cues function that will display the cues in sync.

Either we call it directly if the track is loaded, or we know that the loading has been triggered if necessary, and we just wait in the load event listener.

Let's have a look at what displayCues function does.

The displayCues function is exactly the same as the readContent we had earlier.

It gets the cue list for the given track, add listeners to the track.

And instead of just displaying the plain text below the video as we did earlier, we will just make a nice format and we will add the id of the cue in the element we are creating.

Let's have a look... I'm calling the inspector... let's have a look at one of the list items here.

You can see that in the list item we use the CSS class called cues, just for the formatting, for putting them in blue and adding an underline when the mouse is over, and we use the id of the cue in the list item.

So the id is 10, and we also created an onclick listener that calls the function we will detail later, that is call jumpTo.

And here is the starting time of the cue.

What we did is that we created a list item with a given id and if we click on it, it has a click listener that will call the jumpTo with the time as the parameter.

This is the trick, this onclick listener that will make the video jump to the right position.

How did we create that? We use classical techniques.

We created a string called clickableTransText that is just an HTML list item with the id, the onclick listener that is built with the start time of the cue and we just add this list item to the div.

This function here, addToTranscriptDiv, just adds in the DOM the HTML fragment.

We can give a look at addToTranscriptDiv.

It just does transcriptDiv.innerHTML += this text.

The jumpTo method that makes the video jump... in order to jump to a particular time in the video we are just setting the currentTime property of the video element and we want it to start playing as soon as the jump is done.

So video = document.querySelector("#myVideo") is just the video HTML element.

I recommend to look slowly at the code, it is a bit longer because we added some formatting for the voices and so on, but it is not complicated.

Take your time and look at how it's done....

Foreword about the set of five examples presented in this section: the code of the examples is larger than usual, but each example integrates blocks of code already presented and detailed in the previous lessons.

Example #1: create an accessible player with a clickable transcript of the video presentation.

It might be interesting to read the content of a track before playing the video.

This is what the edX video player does: it reads a single subtitle file and displays it as a transcript on the right.

In the transcript, you can click on a sentence to make the video jump to the corresponding location.

We will learn how to do this using the track API.

Read the WebVTT file at once using the track API and make a clickable transcript.

Here we decided to code something similar, except that we will offer a choice of track/subtitle language.

Our example offers English or German subtitles, and also another track that contains the chapter descriptions (more on that later).

Using a button to select a language (track), the appropriate transcript is displayed on the right. Like the edX player, we can click on any sentence in order to force the video to jump to the corresponding location.

While the video is playing, the current text is highlighted.

Some important things here:

Browsers do not load all the tracks at the same time, and the way they decide when and which track to load differs from one browser to another. So, when we click on a button to choose the track to display, we need to enforce the loading of the track, if it has not been loaded yet.
When a track file is loaded, then we iterate on the different cues and generate the transcript as a set of <li>...</li> elements. One <li> per cue/sentence.
We define the id attribute of the <li> to be the same as the cue.id value. In this way, when we click on a <li> we can get its id and find the corresponding cue start time, and make the video jump to that time location.
We add an enter and an exit listener to each cue. These will be useful for highlighting the current cue. Note that these listeners are not yet supported by FireFox (you can use a cuechange event listener on a TextTrack instead - the source code for FireFox is commented in the example).

Try this example at JSBin:

HTML code!

1.  <section id="all">
2.  <button disabled id="buttonEnglish"
3.           onclick="loadTranscript('en');">
4.      Display English transcript
5.  </button>
6.  <button disabled id="buttonDeutsch"
7.           onclick="loadTranscript('de');">
8.      Display Deutsch transcript
9.  </button>
10. </p>
11. <video id="myVideo" preload="metadata" controls crossOrigin="anonymous">
12.      <source src="https://...../elephants-dream-medium.mp4"
13.              type="video/mp4">
14.      <source src="https://...../elephants-dream-medium.webm"
15.              type="video/webm">
16.      <track label="English subtitles"
17.             kind="subtitles"
18.             srclang="en"
19.             src="https://...../elephants-dream-subtitles-en.vtt" >
20.      <track label="Deutsch subtitles"
21.             kind="subtitles"
22.             srclang="de"
23.             src="https://...../elephants-dream-subtitles-de.vtt"
24.             default</b>>
25.      <track label="English chapters"
26.             kind="chapters"
27.             srclang="en"
28.             src="https://...../elephants-dream-chapters-en.vtt">
29. </video>
30. <div id="transcript"></div>
31. </section>

CSS code:

CSS code extract!

1.  #all {
2.     background-color: lightgrey;
3.     border-radius:10px;
4.     padding: 20px;
5.     border:1px solid;
6.     display:inline-block;
7.     margin:30px;
8.     width:90%;
9.  }
10.  
11. .cues {
12.    color:blue;
13. }
14.  
15. .cues:hover {
16.    text-decoration: underline;
17. }
18.  
19. .cues.current {
20.    color:black;
21.    font-weight: bold;
22. }
23.  
24. #myVideo {
25.    display: block;
26.    float : left;
27.    margin-right: 3%;
28.    width: 66%;
29.    background-color: black;
30.    position: relative;
31. }
32.  
33. #transcript {
34.    padding: 10px;
35.    border:1px solid;
36.    float: left;
37.    max-height: 225px;
38.    overflow: auto;
39.    width: 25%;
40.    margin: 0;
41.    font-size: 14px;
42.    list-style: none;
43. }

JavaScript code:

JavaScript code extract!

1.  var video, transcriptDiv;
2.  // TextTracks, html tracks, urls of tracks
3.  var tracks, trackElems, tracksURLs = []; 
4.  var buttonEnglish, buttonDeutsch;
5.   
6.  window.onload = function() {
7.     console.log("init");
8.     // when the page is loaded, get the different DOM nodes
9.     // we're going to work with
10.    video = document.querySelector("#myVideo");
11.    transcriptDiv = document.querySelector("#transcript");
12. 
13.    // The tracks as HTML elements
14.    trackElems = document.querySelectorAll("track");
15. 
16.    // Get the URLs of the vtt files
17.    for(var i = 0; i < trackElems.length; i++) {
18.       var currentTrackElem = trackElems[i];
19.       tracksURLs[i] = currentTrackElem.src;
20.    }
21. 
22.    buttonEnglish = document.querySelector("#buttonEnglish");
23.    buttonDeutsch = document.querySelector("#buttonDeutsch");
24. 
25.    // we enable the buttons only in this load callback,
26.    // we cannot click before the video is in the DOM
27.    buttonEnglish.disabled = false;
28.    buttonDeutsch.disabled = false;
29. 
30.    // The tracks as TextTrack JS objects
31.    tracks = video.textTracks;
32. };
33.  
34. function loadTranscript(lang) {
35.   // Called when a button is clicked
36. 
37.   // clear current transcript
38.   clearTranscriptDiv();
39. 
40.   // set all track modes to disabled. We will only activate the
41.   // one whose content will be displayed as transcript
42.   disableAllTracks();
43. 
44.   // Locate the track with language = lang
45.   for(var i = 0; i < tracks.length; i++) {
46.     // current track
47.     var track = tracks[i];
48.     var trackAsHtmlElem = trackElems[i];
49. 
50.     // Only subtitles/captions are ok for this example...
51.     if((track.language === lang) && (track.kind !== "chapters")) {
52.        track.mode="showing";
53.  
54.        if(trackAsHtmlElem.readyState === 2) {
55.           // the track has already been loaded
56.           displayCues(track);
57.        } else {
58.           displayCuesAfterTrackLoaded(trackAsHtmlElem, track);
59.        }
60. 
61.        /<i> Fallback for FireFox that still does not implement cue enter and exit events
62.          track.addEventListener("cuechange", function(e) {
63.              var cue = this.activeCues[0];
64.              console.log("cue change");
65.              var transcriptText = document.getElementById(cue.id);
66.              transcriptText.classList.add("current");
67.          });
68.       </i>/
69.     }
70.   }
71. }
72. 
73. function displayCuesAfterTrackLoaded(trackElem, track) {
74.   // Create a listener that will only be called once the track has
75.   // been loaded. We cannot display the transcript before
76.   // the track is loaded
77.    trackElem.addEventListener('load', function(e) {
78.       console.log("track loaded");
79.       displayCues(track);
80.    });
81. }
82. 
83. function disableAllTracks() {
84.   for(var i = 0; i < tracks.length; i++)
85.      // the track mode is important: disabled tracks do not fire events
86.      tracks[i].mode = "disabled"; 
87. }
88.  
89. function displayCues(track) { 
90.    // displays the transcript of a TextTrack
91.    var cues = track.cues;
92. 
93.    // iterate on all cues of the current track
94.    for(var i=0, len = cues.length; i < len; i++) {
95.       // current cue, also add enter and exit listeners to it
96.       var cue = cues[i];
97.       addCueListeners(cue);
98.  
99.       // Test if the cue content is a voice <v speaker>....</v>
100.       var voices = getVoices(cue.text);
101.       var transText="";
102.       if (voices.length > 0) {
103.          for (var j = 0; j < voices.length; j++) { // how many voices?
104.             transText += voices[j].voice + ': ' + removeHTML(voices[j].text);
105.          }
106.       } else
107.          transText = cue.text; // not a voice text
108. 
109.       var clickableTransText = "<li class='cues' id=" + cue.id
110.                                + " onclick='jumpTo("
111.                                + cue.startTime + ");'" + ">"
112.                                + transText + "</li>";
113.  
114.       addToTranscriptDiv(clickableTransText);
115.    }
116. }
117.  
118. function getVoices(speech) { 
119.    // takes a text content and check if there are voices
120.    var voices = []; // inside
121.    var pos = speech.indexOf('<v'); // voices are like <v Michel> ....
122.    while (pos != -1) {
123.       endVoice = speech.indexOf('>');
124.       var voice = speech.substring(pos + 2, endVoice).trim();
125.       var endSpeech = speech.indexOf('</v>');
126.       var text = speech.substring(endVoice + 1, endSpeech);
127.       voices.push({
128.           'voice': voice,
129.           'text': text
130.       });
131.       speech = speech.substring(endSpeech + 4);
132.       pos = speech.indexOf('<v');
133.   }
134.   return voices;
135. }
136.  
137. function removeHTML(text) {
138.   var div = document.createElement('div');
139.   div.innerHTML = text;
140.   return div.textContent || div.innerText || '';
141. }
142. 
143. function jumpTo(time) {
144.   // Make the video jump at the time position + force play
145.   // if it was not playing
146.   video.currentTime = time;
147.   video.play();
148. }
149.  
150. function clearTranscriptDiv() {
151.   transcriptDiv.innerHTML = "";
152. }
153.  
154. function addToTranscriptDiv(htmlText) {
155.   transcriptDiv.innerHTML += htmlText;
156. }
157.  
158. function addCueListeners(cue) {
159.   cue.onenter = function(){
160.      // Highlight current cue transcript by adding the
161.      // cue.current CSS class
162.      console.log('enter id=' + this.id);
163.      var transcriptText = document.getElementById(this.id);
164.      transcriptText.classList.add("current");
165. };
166.  
167. cue.onexit = function(){
168.    console.log('exit id=' + cue.id);
169.    var transcriptText = document.getElementById(this.id);
170.    transcriptText.classList.remove("current");
171. };
172. 
173. } // end of addCueListeners...

Load a WebVTT file using Ajax/XHR2 and parse it manually.

This is an old example written in 2012 at a time when the track API was not supported by browsers. It downloads WebVTT files using Ajax and parses it "by hand". Notice the complexity of the code, compared to the previous example that uses the track API instead. We give this example as is. Sometimes, bypassing all APIs can be a valuable solution, especially when support for the track API is sporadic, as was the case in 2012...

Here is an example at JSBin that displays the values of the cues in the different tracks:

This example, adapted from an example from (now offline) dev.opera.com, uses some JavaScript code that takes a WebVTT subtitle (or caption) file as an argument, parses it, and displays the text on screen, in an element with an id of transcript.

Extract from HTML code:

HTML video code extract!

1.  ...
2.  <video preload="metadata" controls >
3.   <source src="https://..../elephants-dream-medium.mp4" type="video/mp4">
4.   <source src="https://..../elephants-dream-medium.webm" type="video/webm">
5.      <track label="English subtitles" kind="subtitles" srclang="en"
6.           src="https://..../elephants-dream-subtitles-en.vtt" default>
7.      <track label="Deutsch subtitles" kind="subtitles" srclang="de"
8.             src="https://..../elephants-dream-subtitles-de.vtt">
9.      <track label="English chapters" kind="chapters" srclang="en"
10.            src="https://..../elephants-dream-chapters-en.vtt">
11. </video>
12.  ...
13.    <h3>Video Transcript</h3>
14.    <button onclick="loadTranscript('en');">English</button>
15.    <button onclick="loadTranscript('de');">Deutsch</button>
16.     </div>
17.     <div id="transcript"></div>
18. ...

JavaScript code:

Functions loadTranscript, loadTranscriptFile, getVoices, removeHTML extract!

1.  // Transcript.js, by dev.opera.com
2.  function loadTranscript(lang) {
3.     var url = "https://mainline.i3s.unice.fr/mooc/" +
4.         'elephants-dream-subtitles-' + lang + '.vtt';
5.  
6.     // Will download using Ajax + extract subtitles/captions   
7.     loadTranscriptFile(url); 
8.  }
9.   
10. function loadTranscriptFile(webvttFileUrl) {
11.    // Using Ajax/XHR2 (explained in detail in Module 3)
12.    var reqTrans = new XMLHttpRequest();
13. 
14.    reqTrans.open('GET', webvttFileUrl);
15. 
16.    // callback, called only once the response is ready
17.    reqTrans.onload = function(e) { 
18. 
19.        var pattern = /^([0-9]+)$/;
20.      var patternTimecode = /^([0-9]{2}:[0-9]{2}:[0-9]{2}[,.]{1}[0-9]{3})    --> ([0-9]
21.                             {2}:[0-9]{2}:[0-9]{2}[,.]{1}[0-9]{3})(.</i>)$/;
22. 
23.        var content = this.response; // content of the webVTT file
24. 
25.        var lines = content.split(/r?n/); // Get an array of text lines
26.        var transcript = '';
27.        for (i = 0; i < lines.length; i++) {
28.          var identifier = pattern.exec(lines[i]);
29. 
30.          // is there an id for this line, if it is, go to next line
31.          if (identifier) { 
32.            i++;
33.            var timecode = patternTimecode.exec(lines[i]); 
34.            // is the current line a timecode?
35.            if (timecode && i < lines.length) {      
36.               // if it is go to next line     
37.               i++;
38.               // it can only be a text line now
39.               var text = lines[i];  
40.                         
41.               // is the text multiline?
42.               while (lines[i] !== '' && i < lines.length) {   
43.                  text = text + 'n' + lines[i];
44.                  i++;
45.               }
46. 
47.               var transText = '';
48.               var voices = getVoices(text);
49.               // is the extracted text multi voices ? 
50.               if (voices.length > 0) {
51.                  // how many voices ?
52.                  for (var j = 0; j < voices.length; j++) { 
53.                  transText += voices[j].voice + ': '
54.                            + removeHTML(voices[j].text)
55.                            + '<br />';
56.               }
57.           } else 
58.              // not a voice text
59.              transText = removeHTML(text) + '<br />'; 
60. 
61.          transcript += transText;
62.        }
63.      }
64. 
65.      var oTrans = document.getElementById('transcript');
66.      oTrans.innerHTML = transcript;
67.    }
68. };
69.  reqTrans.send(); // send the Ajax request
70. }
71.  
72. function getVoices(speech) {  // takes a text content and check if there are voices 
73.   var voices = [];            // inside
74.   var pos = speech.indexOf('<v'); // voices are like <v Michel> ....
75. 
76.   while (pos != -1) {
77.     endVoice = speech.indexOf('>');
78.     var voice = speech.substring(pos + 2, endVoice).trim();
79.     var endSpeech = speech.indexOf('</v>');
80.     var text = speech.substring(endVoice + 1, endSpeech);
81.     voices.push({
82.        'voice': voice,
83.        'text': text
84.     });
85.     speech = speech.substring(endSpeech + 4);
86.     pos = speech.indexOf('<v');
87.   }
88.   return voices;
89. }
90.  
91. function removeHTML(text) {
92.   var div = document.createElement('div');
93.   div.innerHTML = text;
94.   return div.textContent || div.innerText || '';
95. }

1.3.2 Captions, Descriptions, Chapters and Metadata

Example #2: showing video description while playing, listening to events, changing the mode of a track.

Each track has a mode property (and a mode attribute) that can be: "disabled", "hidden" or "showing". More than one track at a time can be in any of these states. The difference between "hidden" and "disabled" is that hidden tracks can fire events (more on that at the end of the first example) whereas disabled tracks do not fire events.

Here is an example at JSBin that shows the use of the mode property, and how to listen for cue events in order to capture the current subtitle/caption from JavaScript. You can change the mode of each track in the video element by clicking on its button. This will toggle the mode of that track. All tracks with mode="showing" or mode="hidden" will have the content of their cues displayed in real time in a small area below the video.

In the screen-capture below, we have a WebVTT file displaying a scene's captions and descriptions.

Extract from HTML code:

Video code extract!

1.  <html lang="en">
2.  ...
3.  <body onload="init();">
4.    ...
5.    <p>
6.    <video id="myVideo" preload="metadata"
7.      poster ="https://...../sintel.jpg"
8.      crossorigin="anonymous"
9.      controls="controls"
10.     width="640" height="272">
11. 
12.     <source src="https://...../sintel.mp4"
13.       type="video/mp4" />
14.     <source src="https://...../sintel.webm"
15.       type="video/webm" />
16.     <track src="https://...../sintel-captions.vtt"
17.       kind="captions"
18.       label="English Captions"
19.       default</b>/>
20.     <track src="https://...../sintel-descriptions.vtt"
21.       kind="descriptions"
22.       label="Audio Descriptions" />
23.     <track src="https://...../sintel-chapters.vtt"
24.       kind="chapters"
25.       label="Chapter Markers" />
26.     <track src="https://...../sintel-thumbs.vtt"
27.       kind="metadata"
28.       label="Preview Thumbs" />
29.   </video>
30.   </p>
31. 
32.   <p>
33.   <div id="currentTrackStatuses"></div>
34.   <p>
35.   <p>
36.   <div id="subtitlesCaptions"></div>
37.   </p>
38. 
39.   <p>
40.     <button onclick="clearSubtitlesCaptions();">
41.       Clear subtitles/captions log
42.     </button>
43.   </p>
44. 
45.   <p>Click one of these buttons to toggle the mode of each track:</p>
46.   <button onclick="toggleTrack(0);">
47.     Toggle english caption track mode
48.   </button>
49.   <br>
50.   <button onclick="toggleTrack(1);">
51.     Toggle audio description track mode
52.   </button>
53.   <br>
54.   <button onclick="toggleTrack(2);">
55.     Toggle chapter track mode
56.   </button>
57.   <br>
58.   <button onclick="toggleTrack(3);">
59.     Toggle preview thumbnail track modes
60.   </button>
61. 
62. </body>
63. </html>

JavaScript code:

Functions init, displayTrackStatus, appendToScrollableDiv, clearDiv, clearSubtitlesCaptions, toggleTrack, logCue extract!

var tracks, video, statusDiv, subtitlesCaptionsDiv;
 
function init() {
   video = document.querySelector("#myVideo");
   statusDiv = document.querySelector("#currentTrackStatuses");
      subtitlesCaptionsDiv = document.querySelector("#subtitlesCaptions");
   tracks = document.querySelectorAll("track");

   video.addEventListener('loadedmetadata', function() {
      console.log("metadata loaded");

      // defines cue listeners for the active track; we can do this only after the 
      // video metadata have been loaded
      for(var i=0; i<tracks.length; i++) {
         var t = tracks[i].track;
         if(t.mode === "showing") {
            t.addEventListener('cuechange', logCue, false);
         }
       }
       // display in a div the list of tracks and their status/mode value
       displayTrackStatus();  
    });
}
 
function displayTrackStatus() {
    // display the status / mode value of each track.
    // In red if disabled, in green if showing
    for(var i=0; i<tracks.length; i++) {
       var t = tracks[i].track;
       var mode = t.mode;

       if(mode === "disabled") {
          mode = "<span style='color:red'>" + t.mode + "</span>";
       } else if(mode === "showing") {
          mode = "<span style='color:green'>" + t.mode + "</span>";
       }
       appendToScrollableDiv(statusDiv, "track " + i + ":" + t.label
                                        + " " + t.kind+" in "
                                        + mode + " mode");
    }
}
function appendToScrollableDiv(div, text) {
   // we've got two scrollable divs. This function
   // appends text to the div passed as a parameter
   // The div is scrollable (thanks to CSS overflow:auto)
   var inner = div.innerHTML;
   div.innerHTML = inner + text + "<br/>";
   // Make it display the last line appended
   div.scrollTop = div.scrollHeight;
}
 
function clearDiv(div) {
   div.innerHTML = '';
}
 
function clearSubtitlesCaptions() {
   clearDiv(subtitlesCaptionsDiv);
}
 
function toggleTrack(i) {
   // toggles the mode of track i, removes the cue listener
   // if its mode becomes "disabled"
   // adds a cue listener if its mode was "disabled"
   // and becomes "hidden"
   var t = tracks[i].track;
   switch (t.mode) {
      case "disabled":
         t.addEventListener('cuechange', logCue, false);
         t.mode = "hidden";
         break;
      case "hidden":
         t.mode = "showing";
         break;
      case "showing":
         t.removeEventListener('cuechange', logCue, false);
         t.mode = "disabled";
         break;
    }
    // updates the status
    clearDiv(statusDiv);
    displayTrackStatus();
    appendToScrollableDiv(statusDiv,"<br>" + t.label+" are now " +t.mode);
}
 
function logCue() {
   // callback for the cue event
   if(this.activeCues && this.activeCues.length) {
      var t = this.activeCues[0].text; // text of current cue
      appendToScrollableDiv(subtitlesCaptionsDiv, "Active "
                                          + this.kind + " changed to: " + t);
   }
}

1.3.3 With Buttons for Choosing the Subtitle Language

Example #3: adding buttons for choosing the subtitle/caption track

You might have noticed that with some browsers, before 2018, the standard implementation of the video element did not let the user choose the subtitle language. Now, recent browsers offers a menu to choose the track to display.

However, before it was available, it was easy to implement this feature using the Track API.

Here is a simple example at JSBin: we added two buttons below the video to enable/disable subtitles/captions and let you choose which track you prefer.

HTML code:

HTML code extract!

1.  ...
2.  <body onload="init()">
3.   ...
4.  <video id="myVideo" preload="metadata" controls crossOrigin="anonymous" >
5.       <source src="https://...../elephants-dream-medium.mp4"
6.               type="video/mp4">
7.       <source src="https://...../elephants-dream-medium.webm"
8.               type="video/webm">
9.       <track  label="English subtitles"
10.              kind="subtitles"
11.              srclang="en"
12.              src="https://...../elephants-dream-subtitles-en.vtt"
13.              default</b>>
14.      <track  label="Deutsch subtitles"
15.              kind="subtitles"
16.              srclang="de"
17.              src="https://...../elephants-dream-subtitles-de.vtt">
18.      <track  label="English chapters"
19.              kind="chapters"
20.              srclang="en"
21.              src="https://...../elephants-dream-chapters-en.vtt">
22. </video>
23. <h3>Current track: <span id="currentLang"></span></h3>
24. <div id="langButtonDiv"></div>
25. </section>
26. ...

JavaScript code:

JavaScript code extract!

1.  var langButtonDiv, currentLangSpan, video;
2.   
3.  function init() {
4.     langButtonDiv = document.querySelector("#langButtonDiv");
5.     currentLangSpan = document.querySelector("#currentLang");
6.     video = document.querySelector("#myVideo");
7.  
8.     console.log("Number of tracks = "
9.                 + video.textTracks.length);
10.    // Updates the display of the current track activated
11.    currentLangSpan.innerHTML = activeTrack();
12.    // Build the buttons for choosing a track
13.    buildButtons();
14. }
15.  
16. function activeTrack() {
17.    for (var i = 0; i < video.textTracks.length; i++) {
18.       if(video.textTracks[i].mode === 'showing') {
19.          return video.textTracks[i].label + " ("
20.                 + video.textTracks[i].language + ")";
21.       }
22.    }
23.    return "no subtitles/caption selected";
24. }
25.  
26. function buildButtons() {
27.    if (video.textTracks) { // if the video contains track elements
28.       // For each track, create a button
29.       for (var i = 0; i < video.textTracks.length; i++) {
30.          // We create buttons only for the caption and subtitle tracks
31.          var track = video.textTracks[i];
32.         if((track.kind !== "subtitles") && (track.kind !== "captions"))
33.             continue;
34. 
35.          // create a button for track number i         
36.          createButton(video.textTracks[i]); 
37.       }
38.    }
39. }
40. 
41. function createButton(track) {
42.    // Create a button
43.    var b = document.createElement("button");
44.    b.value=track.label;
45.    // use the lang attribute of the button to keep trace of the
46.    // associated track language. Will be useful in the click listener
47.    b.setAttribute("lang", track.language); 
48.    b.addEventListener('click', function(e) {
49.      // Check which track is the track with the language we're looking for
50.      // Get the value of the lang attribute of the clicked button
51.      var lang = this.getAttribute('lang'); 
52. 
53.      for (var i = 0; i < video.textTracks.length; i++) {
54.        if (video.textTracks[i].language == lang) {
55.           video.textTracks[i].mode = 'showing';
56.        } else {
57.           video.textTracks[i].mode = 'hidden';
58.        }
59.      }
60.      // Updates the span so that it displays the new active track
61.     currentLangSpan.innerHTML = activeTrack();
62.   });
63.   // Creates a label inside the button
64.   b.appendChild(document.createTextNode(track.label));
65.   // Add the button to a div at the end of the HTML document
66.   langButtonDiv.appendChild(b);
67. }
68.

External resources

If you are interested in building a complete custom video player, MDN offers an online tutorial with further information about styling and integrating a "CC" button.
The MDN documentation on Web Video Text Tracks Format (WebVTT).

1.3.4 With a Simple Chapter Navigation Menu

Example #4: making a simple chapter navigation menu

We can use WebVTT files to define chapters. The syntax is exactly the same as for subtitles/caption .vtt files. The only difference is in the declaration of the track. Here is how we declared a chapter track in one of the previous examples (in bold in the example below):

HTML code:

HTML code extract!

<video id="myVideo" preload="metadata" controls crossOrigin="anonymous">
  <source src=<https://...../elephants-dream-medium.mp4>
    type="video/mp4">
  <source src=<https://...../elephants-dream-medium.webm>
    type="video/webm">
  <track label="English subtitles"
    kind="subtitles"
    srclang="en"
    src="https://...../elephants-dream-subtitles-en.vtt" >
  <track label="Deutsch subtitles"
    kind="subtitles"
    srclang="de"
    src=<https://...../elephants-dream-subtitles-de.vtt>
    default>
  <track label="English chapters"
    kind="chapters"
    srclang="en"
    src="https://...../elephants-dream-chapters-en.vtt">
</video>

If we try this code in an HTML document, nothing special happens. No magic menu, no extra button!

Currently, no browser takes chapter tracks into account. You could use one of the enhanced video players presented during the HTML5 Part 1 course, but as you will see in this lesson: making your own chapter navigation menu is not complicated.

Let's start by examining the sample .vtt file:

WEBVTT code elephant-dream-chapters-en.vtt extract!

1.  WEBVTT
2.  
3.  chapter-1
4.  00:00:00.000 --> 00:00:26.000
5.  Introduction
6.  
7.  chapter-2
8.  00:00:28.206 --> 00:01:02.000
9.  Watch out!
10. 
11. chapter-3
12. 00:01:02.034 --> 00:03:10.000
13. Let's go
14. 
15. chapter-4
16. 00:03:10.014 --> 00:05:40.000
17. The machine
18. 
19. chapter-5
20. 00:05:41.208 --> 00:07:26.000
21. Close your eyes
22. 
23. chapter-6
24. 00:07:27.125 --> 00:08:12.000
25. There's nothing there
26. 
27. chapter-7
28. 00:08:13.000 --> 00:09:07.500
29. The Colossus of Rhodes

There are 7 cues (one for each chapter). Each cue id is the word "chapter-" followed by the chapter number, then we have the start and end time of the cue/chapter, and the cue content. In this case: the description of the chapter ("Introduction", "Watch out!", "Let's go", etc...).

Hmm... let's try to open this chapter track with the example we wrote in a previous lesson - the one that displayed the clickable transcript for subtitles/captions on the right of the video. We need to modify it a little bit:

We add a "show English chapters" button with a click event listener similar to this:

<button disabled id="buttonEnglishChapters" onclick="loadTranscript('en', 'chapters');"</b>>
  Display English chapter markers
</button>

We modify the loadTranscript function from the previous example, so that it matches both the srclang and the kind attribute of the track.

Here is a new version:

function loadTranscript(lang, kind) {
  ...
  // Locate the track with lang and kind that match the parameters
  for(var i = 0; i < tracks.length; i++) {
    ...
    if((track.language === lang) && (track.kind === kind)) {
    // display it contents...
    }
  }
}

Simple approach: chapters as clickable text on the right of the video.

Try it on JSBin; this version includes the modifications we presented earlier - nothing more. Notice that we kept the existing buttons to display a clickable transcript:

Look at the JavaScript and HTML tab of the JSBin example to see the source code. It's the same as in the clickable transcript example, except for the small changes we explained earlier.

Chapter navigation, illustrated in the video player below, is fairly popular.

In addition to the clickable chapter list, this one displays an enhanced progress bar created using a canvas. The small squares are drawn corresponding to the chapter cues' start and end times. You could modify the code provided, in order to add such an enhanced progress indicator.

However, we will see how we can do better by using JSON objects as cue contents. This will be the topic of the next two lessons!

1.3.5 With Thumbnails, Using JSON Cues

Example #5: create a chapter menu with image thumbnails.

Instead of using text (optionally using HTML for styling, multi lines, etc.), it is also possible to use JSON objects as cue values that can be manipulated from JavaScript. JSON means "JavaScript Object Notation". It's an open standard for describing JavaScript objects as plain text.

Here is an example cue from a WebVTT file encoded as JSON instead of plain text. JSON is useful for describing "structured data", and processing such data from JavaScript is easier than parsing plain text.

WebVTT code extract!

1.  WEBVTT
2.  Wikipedia
3.  00:01:15.200 --> 00:02:18.800
4.  {
5.  "title": "State of Wikipedia",
6.  "description": "Jimmy Wales talking ...",
7.  "src": "https://upload.wikimedia.org/...../120px-Wikipedia-logo-
8.  v2.svg.webp",
9.  "href": "https://en.wikipedia.org/wiki/Wikipedia"
10. }

This JSON object is a JavaScript object encoded as a text string. If we listen for cue events or if we read a WebVTT file as done in previous examples, we can extract this text content using the cue.text property. For example:

JS code extract!

1.  var videoElement = document.querySelector("#myvideo");
2.  var textTracks = videoElement.textTracks; // one for each track element
3.  var textTrack = textTracks[0]; // corresponds to the first track element
4.  
5.  var cues = textTrack.cues;
6.  var cue = cues[0]; // first cue
7.  
8.  // cue.text is in JSON format, with JSON.parse we turn it back
9.  // to a real JavaScript object
10. var obj = JSON.parse(cue.text); 
11. 
12. var title = obj.title; // "State of Wikipedia"
13. var description = obj.description; // Jimmy Wales talking...
14. etc...

This is a powerful way of embedding metadata, especially when used in conjunction with listening for cue and track events.

Improved approach: make a nicer chapter menu by embedding a richer description of chapter markers

Earlier we saw an example that could display chapter markers as clickable text on the right of a video.

This example used only standard plain text content for the cues:

WebVTT code extract!

1.  WEBVTT
2.  
3.  chapter-1
4.  00:00:00.000 --> 00:00:26.000
5.  Introduction
6.  
7.  chapter-2
8.  00:00:28.206 --> 00:01:02.000
9.  Watch out!
10. ...

We used this example to manually capture the images from the video that correspond to each of the seven chapters:

We clicked on each chapter link on the right, then paused the video,
then we used a screen capture tool to grab each image that corresponds to the beginning of chapter,
Finally, we resized the images with Photoshop to approximately 200x400 pixels.

(For advanced users: it's possible to semi-automatize this process using the ffmepg command line tool, see for example; this and that).

Here are the images which correspond to the seven chapters of the video from the previous example:

To associate these images with its chapter description, we will use JSON objects as cue contents:

elephants-dream-chapters-en-JSON.vtt:

WebVTT code extract!

1.  WEBVTT
2.   
3.  chapter-1
4.  00:00:00.000 --> 00:00:26.000
5.  {
6.  "description": "Introduction",
7.  "image": "introduction.jpg"
8.  }
9.   
10.  
11. chapter-2
12. 00:00:28.206 --> 00:01:02.000
13. {
14. "description": "Watch out!",
15. "image": "watchOut.jpg"
16. }
17. ...

Before explaining the code, we propose that you try this example at JSBin that uses this new .vtt file:

HTML code:

HTML code extract!

1.  ...
2.  <video id="myVideo" preload="metadata" controls crossOrigin="anonymous">
3.    <source src="https://...../elephants-dream-medium.mp4"
4.      type="video/mp4">
5.    <source src="https://...../elephants-dream-medium.webm"
6.      type="video/webm">
7.    <track label="English subtitles"
8.      kind="subtitles"
9.    srclang="en" src="https://...../elephants-dream-subtitles-en.vtt" >
10.   <track label="Deutsch subtitles"
11.     kind="subtitles"
12.   srclang="de" src="https://...../elephants-dream-subtitles-de.vtt" default>
13.   <track label="English chapters"
14.     kind="chapters"
15.   srclang="en" src="https://...../elephants-dream-chapters-en-JSON.vtt">
16. </video>
17. <h2>Chapter menu</h2>
18. <div id="chapterMenu"></div>
19.  ...

It's the same code we had in the first example, except that this time we use a new WebVTT file that uses JSON cues to describe each chapter. For the sake of simplicity, we also removed the buttons and all the code for displaying a clickable transcript of the subtitles/captions on the right of the video.

JavaScript code:

JavaScript code extract!

1.  var video, chapterMenuDiv;
2.  var tracks, trackElems, tracksURLs = [];
3.   
4.  window.onload = function() {
5.     console.log("init");
6.     // When the page is loaded
7.     video = document.querySelector("#myVideo");
8.     chapterMenuDiv = document.querySelector("#chapterMenu");
9.  
10.    // Get the tracks as HTML elements
11.    trackElems = document.querySelectorAll("track");
12.    for(var i = 0; i < trackElems.length; i++) {
13.       var currentTrackElem = trackElems[i];
14.       tracksURLs[i] = currentTrackElem.src;
15.    }
16. 
17.    // Get the tracks as JS TextTrack objects
18.    tracks = video.textTracks;
19. 
20.    // Build the chapter navigation menu for the given lang and kind</b>
21.    buildChapterMenu('en', 'chapters');</b>
22. };
23.  
24. function buildChapterMenu(lang, kind) {
25.   // Locate the track with language = lang and kind="chapters"
26.   for(var i = 0; i < tracks.length; i++) {
27.      // current track
28.      var track = tracks[i];
29.      var trackAsHtmlElem = trackElems[i];
30. 
31.      if((track.language === lang) && (track.kind === kind)) {
32.         // the track must be active, otherwise it will not load
33.         track.mode="showing"; // "hidden" would work too
34.  
35.         if(trackAsHtmlElem.readyState === 2) {
36.            // the track has already been loaded
37.            displayChapterMarkers(track);
38.         } else {
39.          displayChapterMarkersAfterTrackLoaded(trackAsHtmlElem, track);
40.         }
41.      }
42.    }
43. }
44.  
45. function displayChapterMarkers(track) {
46.    var cues = track.cues;
47. 
48.    // We must not see the cues on the video
49.    track.mode = "hidden";
50. 
51.    // Iterate on cues
52.    for(var i=0, len = cues.length; i < len; i++) {
53.       var cue = cues[i];
54.  
55.       var cueObject = JSON.parse(cue.text);</b>
56.       var description = cueObject.description;</b>
57.       var imageFileName = cueObject.image;</b>
58.    var imageURL = "https://mainline.i3s.unice.fr/mooc/" + imageFileName;
59. 
60.       // Build the marker. It's a figure with an img and a figcaption inside.
61.       // The img has an onclick listener that will make the video jump
62.       // to the start time of the current cue/chapter
63.       var figure = document.createElement('figure');
64.       figure.classList.add("img");
65. 
66.       figure.innerHTML = "<img onclick='jumpTo("
67.                          + cue.startTime + ");' class='thumb' src='"
68.                          + imageURL + "'><figcaption class='desc'>"
69.                          + description + "</figcaption></figure>";
70.       // Add the figure to the chapterMenuDiv
71.       chapterMenuDiv.insertBefore(figure, null);
72.     }
73. }
74.  
75. function displayChapterMarkersAfterTrackLoaded(trackElem, track) {
76.   // Create a listener that will only be called when the track has
77.   // been loaded
78.   trackElem.addEventListener('load', function(e) {
79.      console.log("chapter track loaded");
80.      displayChapterMarkers(track);
81.   });
82. }
83.  
84. function jumpTo(time) {
85.   video.currentTime = time;
86.   video.play();
87. }
88.

Explanations:

Lines 4-18: when the page is loaded, we assemble all of the track HTML elements and their corresponding TextTrack objects.
Line 19: using that we can build the chapter navigation menu. All is done in the window.onload callback, so nothing happens until the DOM is ready.
Lines 24-43: the buildChapterMenu function first locates the chapter track for the given language, then checks if this track has been loaded by the browser. Once it has been confirmed that the track is loaded, the function displayChapters is called.
Lines 45-65: the displayChapters(track) function will iterate over all of the cues within the chapter track passed as its parameter. For each cue, the JSON content is re-formatted back into a JavaScript object (line 55) and the image filename and description of the chapter/cue are extracted (lines 56-57). Then an HTML description for the chapter is built and added to the div element with id=chapterMenu.

Here is the HTML code for one menu marker:

<figure class="img">
    >  <img onclick="jumpTo(0);" class="thumb" src="https://...../introduction.jpg">
   <figcaption class="desc">
      Introduction
   </figcaption>
</figure>

Notice that we add a click listener to each thumbnail image. Clicking a chapter thumbnail will cause the video to jump to the chapter time location (the example above is for the first chapter with start time = 0).

We also added CSS classes "img", "thumb" and "desc", which make it easy to style and position the thumbnails using CSS.

CSS source code extract:

CSS code extract!

1.  #chapterMenuSection {
2.     background-color: lightgrey;
3.     border-radius:10px;
4.     padding: 20px;
5.     border:1px solid;
6.     display:inline-block;
7.     margin:0px 30px 30px 30px;
8.     width:90%;
9.  }
10.  
11. figure.img {
12.   margin: 2px;
13.   float: left;
14. }
15.  
16. figcaption.desc {
17.   text-align: center;
18.   font-weight: normal;
19.   margin: 2px;
20. }
21.  
22. .thumb {
23.   height: 75px;
24.   border: 1px solid #000;
25.   margin: 10px 5px 0 0;
26.   box-shadow: 5px 5px 5px grey;
27.   transition: all 0.5s;
28. }
29.  
30. .thumb:hover {
31.   box-shadow: 5px 5px 5px black;
32. }

A sample menu marker is shown below (it's also animated - hover your mouse over the thumbnail to see its animated shadow):

Introduction

Combining techniques: a clickable transcript and a chapter menu

This example is the same as the previous one except that we have kept the features that we saw previously: the buttons for displaying a clickable transcript. The code is longer, but it's just a combination of the "clickable transcript" example from the previous lesson, and the code from earlier in this lesson.

Try it at JSBin

1.4.1 Creating Tracks on the Fly

In this lesson, we are going to show:

The addTextTrack method for adding a TextTrack to an html <track> element,
The VTTCue constructor, for creating cues programmatically, and
the addCue method for adding cues on the fly to a TextTrack etc.

These methods will allow us to create TextTrack objects and cues on the fly, programatically.

The presented example shows how we can create "sound sprites": small sounds that are parts of a mp3 file, and that can be played separately. Each sound will be defined as a cue in a track associated with the <audio> element.

Let's create on the fly a WebVTT file with many cues, in order to cut a big sound file into segments and play them on demand

This JsBin demonstration, adapted from an original demo by Sam Dutton, uses a single mp3 file that contains recorded animal sounds.

Below is the sound file. You can try to play it:

Explanations:

The demo uses a JavaScript array for defining the different animal sounds in this audio file:

JavaScript code extract!

var sounds = [
    {
       id: "purr",
       startTime: 0.200,
       endTime: 1.800
    },
    {
       id: "meow",
       startTime: 2.300,
       endTime: 3.300
    },
    {
       id: "bark",
       startTime: 3.900,
       endTime: 4.300
    },
    {
       id: "baa",
       startTime: 5.000,
       endTime: 5.800
    }
    ...
];

The idea is to create a track on the fly, then add cues within this track. Each cue will be created with the id, the start and end time taken from the above JavaScript object. In the end, we will have a track with individual cues located at the time location where an animal sound is in the mp3 file.

Then we generate buttons in the HTML document, and when the user clicks on a button, the getCueById method is called, then the start and end time properties of the cue are accessed and the sound is played (using the currentTime property of the audio element).

Polyfill for getCueById: Note that this method is not available on all browsers yet. A simple polyfill is used in the examples presented. If the getCueById method is not implemented (this is the case in some browsers), it's easy to use this small polyfill:

Code extract!

1.   // for browsers that do not implement the getCueById() method
2.  
3.   // let's assume we're adding the getCueById function to a TextTrack object
4.   //named "track"
5.  if (typeof track.getCueById !== "function") {
6.     track.getCueById = function(id) {
7.       var cues = track.cues;
8.       for (var i = 0; i != track.cues.length; ++i) {
9.         if (cues[i].id === id) {
10.          return cues[i];
11.        }
12.      }
13.   };
14.  }

Techniques

To add a TextTrack to a track element, use the addTextTrack method (of the audio or video element). The function's signature is addTextTrack(kind[,label[,language]]) where kind is our familiar choice between subtitles, captions, chapters, etc. The optional label is any text you'd like to use describing the track; and the optional language is from our usual list of BCP-47 abbreviations, eg 'de', 'en', 'es', 'fr' (etc).

The VTTCue constructor enables us to create our own cue class-instances programmatically. We create a cue instance by using the new keyword. The constructor function expects three familiar arguments, thus: new VTTCue(startTime, endTime, id) - more detail is available from the MDN and the W3C's two applicable groups.

To add cue-instances to a TextTrack on-the-fly, use the track object's addCue method, eg track.addCue(cue). The argument is a cue instance - as above. Note that the track must be a TextTrack object because addCue does not work with HTMLTrackElement Objects.

HTML source code extract:

Code extract!

...
<h1>Playing audio sprites with the track element</h1>
 <p>A demo by Sam Dutton, adapted for JsBin by M.Buffa</p>
 
<div id="soundButtons" class="isSupported"></div>
...
 window.onload = function() {
    // Create an audio element programmatically
 var audio = newAudio("https://mainline.i3s.unice.fr/mooc/animalSounds.mp3");
 
    audio.addEventListener("loadedmetadata", function() {
      // When the audio file has its metadata loaded, we can add
      // a new track to it, with mode = hidden. It will fire events
      // even if it is hidden
      var track = audio.addTextTrack("metadata", "sprite track", "en");</b>
      track.mode = "hidden";</b>
 
      // for browsers that do not implement the getCueById() method
      if (typeof track.getCueById !== "function") {
         track.getCueById = function(id) {
            var cues = track.cues;
            for (var i = 0; i != track.cues.length; ++i) {
               if (cues[i].id === id) {
               return cues[i];
            }
         }
      };
   }
 
   var sounds = [
      {
        id: "purr",
        startTime: 0.200,
        endTime: 1.800
      },
      {
        id: "meow",
        startTime: 2.300,
        endTime: 3.300
      },
      ...
   ];
 
   for (var i = 0; i !== sounds.length; ++i) {
      // for each animal sound, create a cue with id, start and end time
      var sound = sounds[i];
   var cue = new VTTCue(sound.startTime, sound.endTime, sound.id); </b>
      cue.id = sound.id;
      // add it to the track
      track.addCue(cue);
      // create a button and add it to the HTML document
      document.querySelector("#soundButtons").innerHTML += 
                       "<button class='playSound' id=" 
                       + sound.id + ">" +sound.id 
                       + "</button>";  }
 
   var endTime;
   audio.addEventListener("timeupdate", function(event) {
      // When we play a sound, we set the endtime var.
      // We need to listen when the audio file is being played, 
      // in order to pause it when endTime is reached.
      if (event.target.currentTime > endTime)
         event.target.pause();
   });
 
   function playSound(id) {
     // Plays the sound corresponding to the cue with id equal
     // to the one passed as a parameter. We set the endTime var
     // and position the audio currentTime at the start time
     // of the sound
     var cue = track.getCueById(id);
     audio.currentTime = cue.startTime;
     endTime = cue.endTime;
     audio.play();
  };
  // create listeners for all buttons
  var buttons = document.querySelectorAll("button.playSound");
  for(var i=; i < buttons.length; i++) {    
     buttons[i].addEventListener("click", function(e) {
        playSound(this.id);
     });
  }
 });
};

1.4.2 Update the Document in Sync with a Media Playing

Mixing JSON cue content with track and cue events, makes the synchronization of elements in the HTML document (while the video is playing) much easier.

Example of track event listeners that use JSON cue contents

Here is a small code extract that shows how we can capture the JSON content of a cue when the video reaches its start time. We do this within a cuechange listener attached to a TextTrack:

Code extract!

textTrack.oncuechange = function (){
    // "this" is the textTrack that fired the event.
    // Let's get the first active cue for this time segment
    var cue = this.activeCues[0]; 
    var obj = JSON.parse(cue.text);
    // do something
}

Here is a very impressive demo by Sam Dutton that uses JSON cues containing the latitude and longitude of the camera used for filming the video, to synchronize two map views: every time the active cue changes, the Google map and equivalent Google street view are updated.

WARNING: as this Google service is no longer free of charge, you might see "for development purpose only" messages during the execution of this demo. You'll need a valid Google API key in order to remove these messages.

Example of a cue content from this demonstration:

{"lat":37.4219276, "lng":-122.088218, "t":1331363000}

Cue events and cue content:

We can acquire a cue DOM object using the techniques we have seen previously, or by using the new HTML5 TextTrack getCueById() method.

var videoElement = document.querySelector("#myvideo");
var textTracks = videoElement.textTracks; // one for each track element
var textTrack = textTracks[0]; // corresponds to the first track element

// Get a cue with ID="wikipedia"
var cue = textTrack.getCueById("Wikipedia");

And once we have a cue object, it is possible to add event listeners to it:

cue.onenter = function(){
   // display something, play a sound, update any DOM element...
};

cue.onexit = function(){
   // do something else
};

If the getCueById method is not implemented (this is the case in some browsers), we use the @@polyfill presented in the previous section:

// for browsers that do not implement the getCueById() method

// let's assume we're adding the getCueById function to a TextTrack object
//named "track"
if (typeof track.getCueById !== "function") {
  track.getCueById = function(id) {
    var cues = track.cues;
    for (var i = 0; i != track.cues.length; ++i) {
      if (cues[i].id === id) {
        return cues[i];
      }
    }
 };
}

Example that displays a wikipedia page and a google map while a video is playing

Try the example at JSBin

HTML code extract:

HTML source code!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.  <meta charset="utf-8">
5.  <title>Example syncing element of the document with video metadata in webVTT file</title>
6.  </head>
7.  <body >
8.  <main>
9.  <video id="myVideo" controls crossorigin="anonymous" >
10.       <source src="https://mainline.i3s.unice.fr/mooc/samuraiPizzacat.mp4"
11.            type="video/mp4">
12.    ...
13.    </source>
14. <track label="urls track"
15.         src="https://...../SamuraiPizzaCat-metadata.vtt"
16.         kind="metadata" >
17.  </track>
18. </video>
19.    <div id="map"></div>
20. </main>
21.  
22. <aside>
23.        <iframe sandbox="allow-same-origin" id="myIframe" > </iframe>
24. </aside>
25. <h3>Wikipedia URL: <span id="currentURL"> Non défini </span></h3>
26.  
27. <script src="https://maps.google.com/maps/api/js?sensor=false"></script>
28. ...

JavaScript code:

JavaScript source code extract!

1.  window.onload = function() {
2.      var videoElement = document.querySelector("#myVideo");
3.      var myIFrame = document.querySelector("#myIframe");
4.      var currentURLSpan = document.querySelector("#currentURL");
5.  
6.      var textTracks = videoElement.textTracks; // one for each track element
7.      var textTrack = textTracks[0]; // corresponds to the first track element
8.     
9.      // change mode so we can use the track
10.     textTrack.mode = "hidden";
11.     // Default position on the google map
12.     var centerpos = new google.maps.LatLng(48.579400,7.7519);
13.  
14.     // default options for the google map
15.     var optionsGmaps = {
16.        center:centerpos,
17.        navigationControlOptions: {style:
18.                  google.maps.NavigationControlStyle.SMALL},
19.        mapTypeId: google.maps.MapTypeId.ROADMAP,
20.        zoom: 15
21.     };
22.  
23.     // Init map object
24.     var map = new google.maps.Map(document.getElementById("map"),
25.                                   optionsGmaps);
26.  
27.     // cue change listener, this is where the synchronization between
28.     // the HTML document and the video is done
29.     textTrack.oncuechange = function (){
30.        // we assume that we have no overlapping cues
31.        var cue = this.activeCues[0];
32.        if(cue === undefined) return;
33. 
34.        // get cue content as a JavaScript object
35.        var cueContentJSON = JSON.parse(cue.text);
36. 
37.        // do different things depending on the type of sync (wikipedia, gmap)
38.        switch(cueContentJSON.type) {
39.          case'WikipediaPage':
40.             var myURL = cueContentJSON.url;
41.             var myLink = "<a href="" + myURL + "">" + myURL + "</a>";
42.             currentURLSpan.innerHTML = myLink;
43. 
44.             myIFrame.src = myURL; // assign url to src property
45.             break;
46.          case 'LongLat':
47.             drawPosition(cueContentJSON.long, cueContentJSON.lat);
48.             break;
49.        }
50.    };
51.  
52.    function drawPosition(long, lat) {
53.       // Make new object LatLng for Google Maps
54.       var latlng = new google.maps.LatLng(lat, long);
55.  
56.       // Add a marker at position
57.       var marker = new google.maps.Marker({
58.           position: latlng,
59.           map: map,
60.           title:"You are here"
61.       });
62. 
63.       // center map on longitude and latitude
64.       map.panTo(latlng);
65.    }
66. };

All the critical work is done by the cuechange event listener, lines 27-50. We have only the one track, so we set its mode to "hidden" (line 10) in order to be sure that it will be loaded, and that playing the video will fire cuechange events on it. The rest is just Google map code and classic DOM manipulation for updating HTML content (a span that will display the current URL, line 42).

1.5.1 Introduction

Welcome to the WebAudio API lesson! I personnally love this API, playing with it is a lot of fun as you will discover! I hope you will like it as much as I do!

The audio and video elements are used for playing streamed content, but we do not have a real control on the audio. They come with a powerful API as we saw during the previous course and the previous lessons of this course: we can build a custom user interface, make our own play, stop, pause buttons.

We can control the video from JavaScript, listen to events and manage playlists, etc. However, we have no real control on the audio signal: fancy visualizations are impossible to do. The ones that dance with the music, and sound effects such as reverberation, delay, make an equalizer, control the stereo, put the signal on the left or on the right is impossible. Furthermore, playing multiple sounds in sync is nearly impossible due to the streamed nature of the signal. For video games, we will need to play very fast different sounds, and you cannot wait for the stream to arrive before starting playing it.

Web Audio is the solution to all these needs and with Web Audio you will be able to get the output signal from the audio and video elements, and process it with multiple effects. You will be able to work with samples loaded in memory. This will enable perfect syncing, accurate loops, you will be able to mix sounds, etc. You can also generate music programmatically for creating synthetic sounds or virtual instruments. This part will not be covered by this course even if I give links to interesting libraries and demos that do that.

Let’s have a look at some applications. The first thing I wanted to show you is just want we can do with the standard audio element.

This is the standard audio element [music] that just plays a guitar riff that is coming from a server, but we can get control on the audio stream and do such things like that [music]. As you can see I control the stereo balancing here, and we have a real time waveform and volume meters visualization.

Another thing we can do is that we can load samples in memory.

This is an application I wrote for playing multitracks songs. So we are loading MP3s and decoding them in memory so that we can click anywhere on the song, I can make loops like this [music].

As you can see, we can isolate the tracks, we can mix them in real time.

Another application that works with samples in memory is this small example you will learn how to write it in the course: we loaded two different short sounds [sounds] in memory and we can play them repeatedly [sounds] or we can add effects like changing the pitch, changing the volume with some random values and play them with random intervals [sounds].

We can see that the application to video games is straightforward.

Another thing you can do is use synthetics sounds, we will not cover the techniques, but you can use some libraries. This is a library that works with synthetic sounds, you do not have to load a file for having these sounds [sounds].

This is a library for making 8 bits sounds like the very first computers and video games in the 80's used to produce. You can also make very complex application, like a vocoder [sounds], or a synthesizer music instrument [sounds]. Ok you have got the idea.

This is all the interesting things you can do, and you can also learn how to debug such applications. I will make a video especially for that, but using FireFox, you can activate, in the setting of the dev tools, the Web Audio debug tab.

I clicked here on Web Audio, and this added a new tab here Web Audio, and if I reload the page, I can see the graph corresponding to the route of the signal.

Here we have got a source -this is called the audio graph- so we've got the source, and we've got a destination. The source is the original sound. In that case it is a mediaElementAudioSource node that corresponds to the audio element here.

The signal goes to another node that is provided by the Web Audio API and implemented natively in your browser, it is a StereoPanner for separating the sound between left and right.

Then it goes to an analyzer here that will draw the blue waveform and finally to the destination, and the destination is the speakers. I also routed the signal to another part of the graph just for displaying two different analyzers corresponding to the left and right channels. This is for the volume meters here [music].

And if you click on a node, you can see that some node have parameters.

On the stereoPanner, that enables me to balance the sound to the left or to the right, you can see if I change that and click again, I can debug the different properties of each node. You will learn how to build this graph, how to assemble the different nodes, what are the most useful nodes for adding effects, controlling the volume, controlling the stereo, making a equalizer, creating fancy visualizations, and so on.

Welcome to the Web Audio world and during a few lessons, you will learn step by step how to do such an application.

Shortcomings of the standard APIs that we have discussed so far...

In Module 2 of the HTML5 Coding Essentials course, you learned how to add an audio or video player to an HTML document, using the <audio> and <video> elements.

For example:

<audio src="https://mainline.i3s.unice.fr/mooc/LaSueur.mp3" controls>

... render like this in your document:

Under the hood, this HTML code:

initiates a network request to stream the content,
deals with decoding/streaming/buffering the incoming data,
renders audio controls,
updates the progress indicator, time, etc.

You also learned that it's possible to write a custom player: to make your own controls and use the JavaScript API of the <audio> and <video> elements; to call play() and pause(); to read/write properties such as currentTime; to listen to events (ended, error, timeupdate, etc.); and to manage a playlist, etc.

However, there are many things we still cannot do, including:

Play multiple sounds or music in perfect sync,
Play non-streamed sounds (this is a requirement for games: sounds must be loaded in memory),
Output directly to the speakers; adding special effects, equalizer, stereo balancing, reverb, etc.
Any fancy visualizations that dance with the music (e.g. waveforms and frequencies).

The Web Audio API fulfills such missing parts, and much more.

In this course, we do not cover the whole Web Audio API specification. Instead, we focus on the parts of the API that can be useful for writing enhanced multimedia players (that work with streamed audio or video), and on parts that are useful for games (i.e. parts that work with small sound samples loaded in memory). There is the API that specializes in music synthesis and scheduling notes, that we will not study oin this course.

Here's a screenshot from one example we will study: an audio player with animated waveform and volume meters that 'dance' with the music:

Web Audio concepts

The audio context

The canvas used a graphic context for drawing shapes and handling properties such as colors and line widths.

The Web Audio API takes a similar approach, using an AudioContext for all its operations.

Using this context, the first thing we do when using this API is to build an "audio routing graph" made of "audio nodes" which are linked together (most of the time in the course, we are going to call it the "audio graph"). Some node types are for "audio sources", another built-in node is for the speakers, and many other types exist, that correspond to audio effects (delay, reverb, filter, stereo panner, etc.), audio analysis (useful for creating fancy visualizations of the real time signal). Others, which are specialized for music synthesis, are not studied in this course.

The AudioContext also exposes various properties, such as sampleRate, currentTime (in seconds, from the start of AudioContext creation), destination, and the methods for creating each of the various audio nodes.

The easiest way to understand this principle is to look at a first example at JSBin.

This example is detailed in the next lesson. For the moment, all you need to know is that it routes the signal from an <audio> element using a special node that bridges the "streamed audio" world to the Web Audio World, called a MediaElementSourceNode, then this node is connected to a GainNode which enables volume control. This node is then connected to the speakers. We can look at the audio graph of this application using a recent version of FireFox. This browser is the only one (@@as at November 2015) to provide a view of the audio graph, which is very useful for debugging:

Use the Audio Chrome extension to see the WebAudio graph in devtools

For a long time, FireFox had a very good WebAudio debugger built in its devtools, but it has been discontinued in 2019. Meanwhile you can use a Google Chrome extension named "WebAudio Inspector" (or "Audion"). You can install it from the Chrome Web Store].

Once installed, open a Web page that contains some WebAudio code (this one for example), open the Developer Tools (function key F12, then the gear-wheel), and locate the “Web Audio” (Editor) option. Once enabled, return to Developer Tools and open the Web Audio tab. Then reload the target webpage so that all Web audio activity can be monitored by the tool. You can click on the WebAudio graph nodes to see their properties' values.

Note that JSBin examples should be opened in standalone mode (not in editor mode).

Audio nodes are linked via their inputs and outputs, forming a chain that starts with one or more sources, goes through one or more nodes, then ends up at a destination (although you don't have to provide a destination if you just want to visualize some audio data, for example).

The AudioDestination node above corresponds to the speakers. In this example, the signal goes from left to right: from the MediaElementSourceNode (we will see in the code that it's the audio stream from an <audio> element), to a Gain node (and by adjusting the gain property we can set the volume of the sound that outputs from this node), then to the speakers.

Typical code to build an audio routing graph (the one used in the above example)

HTML code extract:

HTML source code extract!

1.  <audio src="https://mainline.i3s.unice.fr/mooc/drums.mp3"
2.          id="gainExample"
3.          controls loop
4.          crossorigin="anonymous">
5.  </audio>
6.  <br>
7.  <label for="gainSlider">Gain</label>
8.  <input type="range" min="0" max="1" step="0.01" value="1" id="gainSlider" />

JavaScript source code:

JavaScript source code!

1.  // This line is a trick to initialize the AudioContext
2.  // that will work on all recent browsers
3.  var ctx = window.AudioContext || window.webkitAudioContext;
4.  var audioContext;
5.  
6.  var gainExemple, gainSlider, gainNode;
7.   
8.  window.onload = function() {
9.  
10.   // get the AudioContext
11.   audioContext = new ctx();
12.  
13.   // the audio element
14.   player = document.querySelector('#gainExample');
15.   player.onplay = () => {
16.      audioContext.resume();
17.   }
18.   gainSlider = document.querySelector('#gainSlider');
19.  
20.   buildAudioGraph();
21. 
22.   // input listener on the gain slider
23.   gainSlider.oninput = function(evt){
24.     gainNode.gain.value = evt.target.value;
25.   };
26. };
27.  
28. function buildAudioGraph() {
29.   // create source and gain node
30.   var gainMediaElementSource = audioContext.createMediaElementSource(player);
31.   gainNode = audioContext.createGain();
32. 
33.   // connect nodes together
34.  gainMediaElementSource.connect(gainNode);
35.  gainNode.connect(audioContext.destination);
36. }
37.

Explanations:

Here we applied a commonly used technique:

As soon as the page is loaded: initialize the audio context (line 11). Here we use a trick so that the code works on all browsers: Chrome, FF, Opera, Safari, Edge. The trick at line 3 is required for Safari, as it still needs the WebKit prefixed version of the AudioContext constructor.
Then we build a graph (line 20).
The build graph function first builds the nodes, then connects them to build the audio graph. Notice the use of audioContext.destination for the speakers (line 35). This is a built-in node. Also, the MediaElementSource node "gainexample" which is the HTML's audio element.

Example of bigger graphs

Web Audio nodes are implemented natively in the browser. The Web Audio framework has been designed to handle a very large number of nodes. It's common to encounter applications with several dozens of nodes: some, such as this Vocoder application, use hundreds of nodes (the picture below has been taken while the WebAudio debugger was still included in FireFox, you should get similar results with the Chrome WebAudio Inspector extension).

1.5.2 Working with Streamed Content

Hello! Using Web Audio with streamed content is really easy. I’m going to show you the most simple example we can do, directly on JSBin.

I first create an audio element, using the standard way.

Here we are, ok, [music] so this is just the standard audio element. I will add the “crossOrigin=anonymous” attribute because when we stream audio content and we control it with Web Audio, we need to follow the “same origin policy” constraint. That means that the HTML page and the JavaScript code that manipulate the content of the stream, should be normally located on the same server as the audio file. This is not the case here because the HTML I’m typing is hosted on jsbin.com. By adding the “crossOrigin=anonymous” attribute, this will send different HTTP headers to the mainline.i3s.unice.fr server.

And this server has been configured for accepting external requests. So this will make the example work.

From JavaScript, if I want to get the audio stream, I must first wait until the page is loaded.

I’m writing a window.onload event listener, and the first thing I do is to get a handle on the audio player. I need to add an id attribute here… Ok, like that!

Like the canvas, that is using a graphic context, here, with Web Audio, we use an audio context.

I created the context. Now, I can create a special source node using context.createMediaElementSource, that takes as a single parameter a video or an audio element. I’m using the player variable here, that corresponds to the audio element.

Now, I connect this source to the destination. A destination is a special node that corresponds to the speakers.

Each node has a connect and a disconnect method. I’m using the connect method here, and I’m using ctx.destination as the destination node.

If I play it [music], the stream is going directly to the speakers. And if I comment this line, the string is disconnected and nothing is outputed. Once you got an handle on the audio stream, using createMediaElementSource, the behavior of the audio element is changed; all the signal, the audio signal, is routed to your own audio graph, not to the default route that goes to the speakers.

And I can visualize this graph. With JsBin, I need to be in standalone mode, like that.

I open the devtools, I check that Web Audio is activated, here, and now I can go to the Web Audio tab, reload the page, and I can see my graph here.

Ok, that was all for this very first lesson. You learnt how to make the simplest audio graph possible, and how to visualize the graph. In the next lesson, we will add in the middle, here, different nodes for processing the sound like having control over the stereo, making filters... like an equalizer, and things like that...

The MediaSourceElement node In the previous lesson, we encountered the MediaElementSource node that is used for routing the sound from a <video> or <audio> element stream. The above video shows how to make a simple example step by step, and how to setup FireFox for debugging Web Audio applications and visualize the audio graph.

Typical use:

Example at JSBin

HTML:

<audio id="player" controls crossorigin="anonymous" loop>
   <source src="https://mainline.i3s.unice.fr/mooc/guitarRiff1.mp3">
   Your browser does not support the audio tag.
</audio>

JavaScript:

var ctx = window.AudioContext || window.webkitAudioContext;
var context = new ctx();

var mediaElement = document.querySelector('#player');
var sourceNode = context.createMediaElementSource(mediaElement);
sourceNode.connect(context.destination); // connect to the speakers

The MediaElementSource node is built using context.createMediaElementSource(elem), where elem is an <audio> or a <video> element.

Then we connect this source Node to other nodes. If we connect it directly to context.destination, the sound goes to the speakers with no additional processing.

In the following lessons, we will see the different nodes that are useful with streamed audio and with the MediaElementSource node. Adding them in the audio graph will enable us to change the sound in many different ways.

1.5.3 Most Useful Filter Nodes

All definitions come from the [Mozilla Developer Network (MDN) pages giving details about the Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API).

Let's study the most useful filter nodes: gain, stereo panner, filter, convolver (reverb).

Gain node

Useful for setting volume... see the Gain node's documentation.

Definition: "The GainNode interface represents a change in volume. It is an AudioNode audio-processing module that causes a given gain to be applied to the input data before its propagation to the output. A GainNode always has exactly one input and one output, both with the same number of channels."

Example at JSBin, or try it in your browser:

Gain Node

Source code extract:

Source code extract!

1.  /* Gain Node */
2.   
3.  var gainExample = document.querySelector('#gainExample');
4.  var gainSlider = document.querySelector('#gainSlider');
5.   
6.  var gainMediaElementSource = audioContext.createMediaElementSource(gainExample);
7.  var gainNode = audioContext.createGain();
8.   
9.  gainMediaElementSource.connect(gainNode);
10. gainNode.connect(audioContext.destination);
11.  
12. gainSlider.oninput = function(evt){
13. gainNode.gain.value = evt.target.value;
14. };

The gain property (line 13 in the above code) corresponds to the multiplication we apply to the input signal volume. A value of 1 will keep the volume unchanged. A value < 1 will lower the volume (0 will mute the signal), and a value > 1 will increase the global volume, with a risk of clipping. With gain values > 1, we usually add a compressor node to the signal chain to prevent clipping. You will see an example of this when we discuss the compressor node.

Stereo panner

See the Stereo Panner node's documentation.

Definition: "The StereoPannerNode interface of the Web Audio API represents a simple stereo panner node that can be used to pan an audio stream left or right. The pan property takes a value between -1 (full left pan) and 1 (full right pan).

Example at JSBin, or try it in your browser:

Balance

Source code extract:

Source code extract!

1.  // the audio element
2.  playerPanner = document.querySelector('#pannerPlayer');
3.  pannerSlider = document.querySelector('#pannerSlider');
4.   
5.  // create nodes
6.  var source = audioContext.createMediaElementSource(playerPanner);
7.  pannerNode = audioContext.createStereoPanner();
8.  
9.  // connect nodes together
10. source.connect(pannerNode);
11. pannerNode.connect(audioContext.destination);
12.  
13. // input listener on the gain slider
14. pannerSlider.oninput = function(evt){
15.   pannerNode.pan.value = evt.target.value;
16. };

Biquad filter

Definition: "The BiquadFilterNode interface represents a simple low-order filter, and is created using the AudioContext.createBiquadFilter() method. It is an AudioNode that can represent different kinds of filters, tone control devices, and graphic equalizers. A BiquadFilterNode always has exactly one input and one output."

See also the Biquad Filter node's documentation.

Example at JSBin, or try it in your browser, with a lowpass filter, only the frequency slider will have a noticeable effect:

The most useful slider is the frequency slider (that changes the frequency value property of the node). The meaning of the different properties (frequency, detune and Q) differs depending on the type of the filter you use (click on the dropdown menu to see the different types available). Look at this documentation for details on the different filters and how the frequency, detune and Q properties are used with each of these filter types.

Here is a nice graphic application that shows the frequency responses to the various filters, you can choose the type of filters and play with the different property values using sliders:

Multiple filters are often used together. We will make a multi band equalizer in a next lesson, and use six filters with type=peaking.

Source code extract:

Source code extract!

1.  var ctx = window.AudioContext || window.webkitAudioContext;
2.  var audioContext = new ctx();
3.   
4.  /* BiquadFilterNode */
5.   
6.  var biquadExample = document.querySelector('#biquadExample');
7.  var biquadFilterFrequencySlider =
8.        document.querySelector('#biquadFilterFrequencySlider');
9.  var biquadFilterDetuneSlider =
10.       document.querySelector('#biquadFilterDetuneSlider');
11. var biquadFilterQSlider =
12.       document.querySelector('#biquadFilterQSlider');
13. var biquadFilterTypeSelector =
14.       document.querySelector('#biquadFilterTypeSelector');
15.  
16. var biquadExampleMediaElementSource =
17.       audioContext.createMediaElementSource(biquadExample);
18.  
19. var filterNode = audioContext.createBiquadFilter();
20.  
21. biquadExampleMediaElementSource.connect(filterNode);
22.  
23. filterNode.connect(audioContext.destination);
24.  
25. biquadFilterFrequencySlider.oninput = function(evt){
26.    filterNode.frequency.value = parseFloat(evt.target.value);
27. };
28.  
29. biquadFilterDetuneSlider.oninput = function(evt){
30.    filterNode.detune.value = parseFloat(evt.target.value);
31. };
32.  
33. biquadFilterQSlider.oninput = function(evt){
34.    filterNode.Q.value = parseFloat(evt.target.value);
35. };
36.  
37.  
38. biquadFilterTypeSelector.onchange = function(evt){
39.     filterNode.type = evt.target.value;
40. };

Convolver node: useful for convolution effects such as reverberation

Definition: "The ConvolverNode interface is an AudioNode that performs a Linear Convolution on a given AudioBuffer, often used to achieve a reverb effect. A ConvolverNode always has exactly one input and one output."

See the Convolver node's documentation.

Example at JSBin, THIS EXAMPLE DOES NOT WORK IN YOUR BROWSER as the edX platforms disables Ajax loading in its HTML pages. Try it at JSBin!

Reverb (Dry/Wet)

From Wikipedia: a convolution is a mathematical process which can be applied to an audio signal to achieve many interesting high-quality linear effects. Very often, the effect is used to simulate an acoustic space such as a concert hall, cathedral, or outdoor amphitheater. It can also be used for complex filter effects, like a muffled sound coming from inside a closet, sound underwater, sound coming through a telephone, or playing through a vintage speaker cabinet. This technique is very commonly used in major motion picture and music production and is considered to be extremely versatile and of high quality.

Each unique effect is defined by an impulse response. An impulse response can be represented as an audio file and can be recorded from a real acoustic space such as a cave, or can be synthetically generated through a wide variety of techniques. We can find many high quality impulses on the Web (for example @@TJS OK? here). The impulse used in the example is the one recorded at the opera: La Scala Opera of Milan, in Italy. It's a .wav file.

Try this demo to see the difference between different impulse files!

So before building the audio graph, we need to download the impulse. For this, we use an Ajax request (this will be detailed during Module 3), but for the moment, just take this function as it is... The Web Audio API requires that impulse files should be decoded in memory before use. Accordingly, once the requested file has downloaded, we call the decodeAudioData method. Once the impulse is decoded, we can build the graph. So typical use is as follows:

Source code extract!

var impulseURL = "https://mainline.i3s.unice.fr/mooc/Scala-Milan-Opera-Hall.wav";
var decodedImpulse;
...
loadImpulse(impulseURL, function() {
   // we only get here once the impulse has finished
   // loading and is decoded
   buildAudioGraphConvolver();
});
 
...
function loadImpulse(url, callback) {

   ajaxRequest = new XMLHttpRequest();
   ajaxRequest.open('GET', url, true);
   ajaxRequest.responseType = 'arraybuffer'; // for binary transfer
 
   ajaxRequest.onload = function() {
      // The impulse has been loaded
      var impulseData = ajaxRequest.response;
      // let's decode it.
      audioContext.decodeAudioData(impulseData, function(buffer) {
         // The impulse has been decoded
         decodedImpulse = buffer;
         // Let's call the callback function, we're done!
         callback();
     });
   };

   ajaxRequest.onerror = function(e) {
      console.log("Error with loading audio data" + e.err);
   };

   ajaxRequest.send();
}

Now let's consider the function which builds the graph. In order to set the quantity of reverb we would like to apply, we need two separate routes for the signal:

One "dry" route where we directly connect the audio source to the destination,
One "wet" route where we connect the audio source to the convolver node (that will add a reverb effect), then to the destination,
At the end of both routes, before the destination, we add a gain node, so that we can specify the quantity of dry and wet signal we're going to send to the speakers.

The audio graph will look like this (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

And here is the function which builds the graph:

Source code extract!

1.  function buildAudioGraphConvolver() {
2.    // create the nodes
3.    var source = audioContext.createMediaElementSource(playerConvolver);
4.    convolverNode = audioContext.createConvolver();
5.    // Set the buffer property of the convolver node with the decoded impulse
6.    convolverNode.buffer = decodedImpulse;
7.  
8.    convolverGain = audioContext.createGain();
9.    convolverGain.gain.value = 0;
10. 
11.   directGain = audioContext.createGain();
12.   directGain.gain.value = 1;
13. 
14. 
15.   // direct/dry route source -> directGain -> destination
16.   source.connect(directGain);
17.   directGain.connect(audioContext.destination);
18. 
19.   // wet route with convolver: source -> convolver
20.   // -> convolverGain -> destination
21.   source.connect(convolverNode);
22.   convolverNode.connect(convolverGain);
23.   convolverGain.connect(audioContext.destination);
24. }

Note that at line 6 we use the decoded impulse. We could not have done this before the impulse was loaded and decoded.

The Dynamics Compressor node

Definition: "The DynamicsCompressorNode interface provides a compression effect, which lowers the volume of the loudest parts of the signal in order to help prevent clipping and distortion that can occur when multiple sounds are played and multiplexed together at once. This is often used in musical production and game audio."

It's usually a good idea to insert a compressor in your audio graph to give a louder, richer and fuller sound, and to prevent clipping. See the Dynamics Compressor node's documentation.

Example you can try on JSBin or try it here in your browser:

Gain Turn compressor On

In this example we set the gain to a very high value that will make a saturated sound. To prevent clipping, it is sufficient to add a compressor right at the end of the graph! Here we use the compressor with all default settings.

NB: This course does not go into detail about the different properties of the compressor node, as they are largely for musicians with the purpose of enabling the user to set subtle effects such as release, attack, etc.

Audio graph with the compressor activated (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

Extract of the HTML code:

Source code extract!

<audio src="https://mainline.i3s.unice.fr/mooc/guitarRiff1.mp3"
        id="compressorExample" controls loop
        crossorigin="anonymous"></audio>
<br>
<label for="gainSlider1">Gain</label>
<input type="range" min="0" max="10" step="0.01"
        value="8" id="gainSlider1" />
<button id="compressorButton">Turn compressor On</button>

JavaScript source code:

JavaScript source code!

1.  // This line is a trick to initialize the AudioContext
2.  // that will work on all recent browsers
3.  var ctx = window.AudioContext || window.webkitAudioContext;
4.  var audioContext;
5.  var compressorExemple, gainSlider1, gainNode1, compressorNode;
6.  var compressorButton;
7.  var compressorOn = false;
8.   
9.  window.onload = function() {
10. 
11.   // get the AudioContext
12.   audioContext = new ctx();
13.  
14.   // the audio element
15.   compressorExemple = document.querySelector('#compressorExample');
16.   gainSlider1 = document.querySelector('#gainSlider1');
17.   // button for turning on/off the compressor
18.   compressorButton = document.querySelector('#compressorButton');
19. 
20.   buildAudioGraph();
21. 
22.   // input listener on the gain slider
23.   gainSlider1.oninput = function(evt) {
24.     gainNode1.gain.value = evt.target.value;
25.   };
26. 
27.   compressorButton.onclick = function(evt) {
28.      if(compressorOn) {
29.         // disconnect the compressor and make a
30.         // direct route from gain to destination
31.         compressorNode.disconnect(audioContext.destination);
32.         gainNode1.disconnect(compressorNode);
33.         gainNode1.connect(audioContext.destination);
34.         compressorButton.innerHTML="Turn compressor: On";
35.      } else {
36.         // compressor was off, we connect the gain to the compressor
37.         // and the compressor to the destination
38.         gainNode1.disconnect(audioContext.destination);
39.         gainNode1.connect(compressorNode);
40.         compressorNode.connect(audioContext.destination);
41.         compressorButton.innerHTML="Turn compressor: Off";
42.      }
43.      compressorOn = !compressorOn;
44.    };
45. };
46.  
47. function buildAudioGraph() {
48.    // create source and gain node
49.    var gainMediaElementSource =
50.       audioContext.createMediaElementSource(compressorExemple);
51.    gainNode1 = audioContext.createGain();
52.    gainNode1.gain.value = parseFloat(gainSlider1.value);
53. 
54.    // do not connect it yet
55.    compressorNode = audioContext.createDynamicsCompressor(); //  connect nodes together
56.    gainMediaElementSource.connect(gainNode1);
57.    gainNode1.connect(audioContext.destination);
58. }

Explanations:

There is nothing special here compared to the other examples in this section, except that we have used a new method disconnect (line 32 and line 38), which is available on all types of nodes (except ctx.destination) to modify the graph on the fly. When the button is clicked, we remove or add a compressor in the audio graph (lines 28-42) and to achieve this, we disconnect and reconnect some of the nodes.

1.5.4 Writing an Equalizer

Example #1: an audio equalizer with an <audio> element

Example at JSBin, here is a screenshot:

This example uses six BiquadFilter nodes with type="peaking".

If you read the description of this filter type: "Frequencies inside the range get a boost or an attenuation; frequencies outside it are unchanged." This is exactly what we need to write a multi band equalizer! We're going to use several sliders, each of which boosts one range of frequency values.

The definition says that:

the frequency property value of a filter will indicate the middle of the frequency range getting a boost or an attenuation, each slider corresponds to a filter whose frequency will be set to 60Hz, 170Hz, 350Hz, 1000Hz, 3500Hz, or 10000Hz.
the gain property value of a filter corresponds to the boost, in dB, to be applied; if negative, it will be an attenuation. We will code the sliders' event listeners to change the gain value of the corresponding filter.
the Q property values control the width of the frequency band. The greater the Q value, the smaller the frequency band. We'll ignore it for the purposes of this example.

HTML code extract:

HTML source code extract!

1.  <h2>Equalizer made with the Web Audio API</h2>
2.   
3.  <div class="eq">
4.    <audio id="player" controls crossorigin="anonymous" loop>
5.       <source src="https://mainline.i3s.unice.fr/mooc/drums.mp3">
6.       Your browser does not support the audio tag.
7.    </audio>
8.  
9.    <div class="controls">
10.     <label>60Hz</label>
11.     <input type="range"
12.            value="0" step="1" min="-30" max="30"
13.            oninput="changeGain(this.value, 0);">
14.     </input>
15.     <output id="gain0">0 dB</output>
16.   </div>
17. 
18.   <div class="controls">
19.    <label>170Hz</label>
20.    <input type="range"
21.           value="0" step="1" min="-30" max="30"
22.           oninput="changeGain(this.value, 1);">
23.    </input>
24.    <output id="gain1">0 dB</output>
25.   </div>
26. 
27.   <div class="controls">
28.     <label>350Hz</label>
29.     <input type="range"
30.            value="0" step="1" min="-30" max="30"
31.            oninput="changeGain(this.value, 2);">
32.     </input>
33.     <output id="gain2">0 dB</output>
34.   </div>
35. ...
36. </div>

JavaScript code:

JavaScript source code!

1.  //Builds an equalizer with multiple biquad filters
2.   
3.  var ctx = window.AudioContext || window.webkitAudioContext;
4.  var context = new ctx();
5.   
6.  var mediaElement = document.getElementById('player');
7.  var sourceNode = context.createMediaElementSource(mediaElement);
8.   
9.  // Creates the equalizer, comprised of a set of biquad filters
10.  
11. var filters = [];
12.  
13. // Set filters
14. [60, 170, 350, 1000, 3500, 10000].forEach(function(freq, i) {
15.    var eq = context.createBiquadFilter();
16.    eq.frequency.value = freq;
17.    eq.type = "peaking";
18.    eq.gain.value = 0;
19.    filters.push(eq);
20. });
21.  
22. // Connects filters in sequence
23. sourceNode.connect(filters[0]);
24. 
25. for(var i = 0; i < filters.length - 1; i++) {
26.    filters[i].connect(filters[i+1]);
27. }
28.  
29. // Connects the last filter to the speakers
30. filters[filters.length - 1].connect(context.destination);
31.  
32. // Event listener called by the sliders
33. function changeGain(sliderVal,nbFilter) {
34.    var value = parseFloat(sliderVal);
35.   filters[nbFilter].gain.value = value;
36. 
37.   // Updates output labels
38.   var output = document.querySelector("#gain"+nbFilter);
39.   output.value = value + " dB";
40. }

Here is the final audio graph (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

Example #2: equalizer with a <video> element

We cloned the previous example and simply changed the <audio>... part of the HTML code by:

<video id="player" width="320" height="240" controls crossOrigin="anonymous">
  <source src="https://mainline.i3s.unice.fr/mooc/elephants-dream-medium.mp4" >
</video>

And the example works in the same way, but this time with a video. Try moving the sliders to change the sound!

Example at JSBin:

1.5.5 Waveforms

Hi, Today I will show you how to write a waveform that will danse with the music. I prepared a small skeleton that is just composed of an audio element with a mp3 file that will be streamed when I press the play button.

The sound is captured in an audio graph using a media element source like we explained in a previous lesson.

Let's start from the beginning and look at this example. When the page is loaded, we go to the onload listener, we create an audio context because we are going to work with the Web Audio API, and then we are just using the standard way for using a canvas.

We get the canvas, and we get the canvas context, so we are ready to draw things in the canvas here. Then we build an audio graph and then we start an animation... and for the moment the animation just draws an horizontal line 60 times per seconds. We will look at it later.

Let's have a look at the audio graph. The audio graph is made of a source node that is the media element source. It corresponds to the audio element.

Then we create an analyser, this is a special node that will provide on demand the time domain and frequency domain analysis data, and this data will be useful for drawing a waveform or for drawing frequencies that will dance with the music, or for drawing volume meters, or for doing beat detection. We will see examples in the next lessons.

When you create an analyser node, you specify the size of the Fast Fourier Transform (FFT), do not worry if you do not know exactly what this technique is. It is just the size that will have an influence on the number of data you will have to draw.

Here for waveform, classical values are 1024 or 2048. They must be powers of 2.

The number of data we will get depends on this size: it is exactly the FFT size divided by 2. In this example I set the FFT size to 1024 so I will get 512 data...

Here is how we can declare a buffer that will get the data. It is called dataArray here and we will use it in the animation loop.

I've got a source node that corresponds to the audio stream, an analyser node that will analyse the stream, and then I connect the source to the analyser, and the analyser to the destination, and the destination is the speakers.

Let's have a look at the animation loop! We clear the canvas, and in the end we call again the animation loop, so that the animation will be done 60 times per seconds... and the way we will draw the waveform is just a set of connected line that we call "a path".

The "path" is a way of drawing that we presented during the HTML5 Part 1 course.

So here, for drawing an horizontal, flat waveform, we just do a loop on the number of data. Here, we are not using the real data but we will do a loop 512 times and we compute an increment in x, so that depending on the width of the canvas and on the number of data we have got to draw, we will add this increment to the x coordinate.

And the y coordinate here is just faked because we use height/2.

I can add a random element here if you like, and we will see that I can just fake some data to be drawn. So, instead of drawing this, I will use the real values. How can we get the data? In the animation loop, 60 times per seconds, we call the analyser.getByteTimeDomainData and we pass the dataArray of the correct size.

And just after this call, the dataArray will contain the data we want to draw. And what is interesting here, is the value of each data... and we will compute the y coordinate depending on that. I'm just declaring a variable that will get the value. This value is between 0 and 255 because we are working with bytes, and bytes are 8 bits encoded data and the value is between 0 and 255. I will normalize it. So now it's between 0 and 1. Then, in order to compute the y value, I just have to scale it to the height of the canvas, like this...

And now if I play the sound, I've got the waveform that is animated.

These 3 lines are very straightforward, just in order to transform a value between

0 and 255 and scale it to the height of the canvas...

Do try on and study the code in the JSBin example created during the Live Coding Video

WebAudio offers an Analyser node that provides real-time frequency and time-domain analysis information. It leaves the audio stream unchanged from the input to the output, but allows us to acquire data about the sound signal being played. This data is easy for us to process since complex computations such as Fast Fourier Transforms are being executed, behind the scenes.

Example #1: audio player with waveform visualization

Example at JSBin

Do things in order!

First, select the audio context and the canvas context, then build the audio graph, and finally run the animation loop.

Typical operations to perform once the HTML page is loaded:

Source code!

1.  window.onload = function() {
2.     // get the audio context
3.     audioContext= ...;
4.  
5.     // get the canvas, its graphic context...
6.     canvas = document.querySelector("#myCanvas");
7.     width = canvas.width;
8.     height = canvas.height;
9.     canvasContext = canvas.getContext('2d');
10. 
11.    // Build the audio graph with an analyser node at the end
12.    buildAudioGraph();
13. 
14.    // starts the animation at 60 frames/s
15.    requestAnimationFrame(visualize);
16. };

Step #1: build the audio graph with an analyser node at the end

If we want to visualize the sound that is coming out of the speakers, we have to put an analyser node at almost the end of the sound graph. Example #1 shows a typical use: an <audio> element, a MediaElementElementSource node connected to an Analyser node, and the analyser node connected to the speakers (audioContext.destination). The visualization is a graphic animation that uses the requestAnimationFrame API presented in teh W3C HTML5 Coding Essentials and Best Practices course (Module 4).

Typical code for building the audio graph:

HTML code:

1.  <audio src="https://mainline.i3s.unice.fr/mooc/guitarRiff1.mp3"
2.      id="player" controls loop crossorigin="anonymous">
3.  </audio>
4.  <canvas id="myCanvas" width=300 height=100></canvas>

JavaScript code:

JavaScript source code!

1.  function buildAudioGraph() {
2.     var mediaElement = document.getElementById('player');
3.     var sourceNode = audioContext.createMediaElementSource(mediaElement);
4.  
5.     // Create an analyser node
6.     analyser = audioContext.createAnalyser();
7.  
8.     // set visualizer options, for lower precision change 1024 to 512,
9.     // 256, 128, 64 etc. bufferLength will be equal to fftSize/2
10.    analyser.fftSize = 1024;
11.    bufferLength = analyser.frequencyBinCount;
12.    dataArray = new Uint8Array(bufferLength);
13. 
14.    sourceNode.connect(analyser);
15.    analyser.connect(audioContext.destination);
16. }

With the exception of lines 8-12, where we set the analyser options (explained later), we build the following graph (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

Step #2: write the animation loop

The visualization itself depends on the options which we set for the analyser node. In this case we set the FFT size to 1024 (FFT is a kind of accuracy setting: the bigger the value, the more accurate the analysis will be. 1024 is common for visualizing waveforms, while lower values are preferred for visualizing frequencies). Here is what we set in this example:

analyser.fftSize = 1024;
bufferLength = analyser.frequencyBinCount;
dataArray = new Uint8Array(bufferLength);

Line 2: we set the size of the FFT,
Line 3: this is the byte array that will contain the data we want to visualize. Its length is equal to fftSize/2.

When we build the graph, these parameters are set - effectively as constants, to control the analysis during play-back.

Here is the code that is run 60 times per second to draw the waveform:

Source code!

1.  function visualize() {
2.    // 1 - clear the canvas
3.    // like this: canvasContext.clearRect(0, 0, width, height);
4.  
5.    // Or use rgba fill to give a slight blur effect
6.    canvasContext.fillStyle = 'rgba(0, 0, 0, 0.5)';
7.    canvasContext.fillRect(0, 0, width, height);
8.  
9.    // 2 - Get the analyser data - for waveforms we need time domain data
10.   analyser.getByteTimeDomainData(dataArray);
11.  
12.   // 3 - draws the waveform
13.   canvasContext.lineWidth = 2;
14.   canvasContext.strokeStyle = 'lightBlue';
15.  
16.   // the waveform is in one single path, first let's
17.   // clear any previous path that could be in the buffer
18.   canvasContext.beginPath();
19. 
20.   var sliceWidth = width / bufferLength;
21.   var x = 0;
22.  
23.   for(var i = 0; i < bufferLength; i++) {
24.     // dataArray values are between 0 and 255,
25.     // normalize v, now between 0 and 1
26.     var v = dataArray[i] / 255;
27.     // y will be in [0, canvas height], in pixels
28.     var y = v * height;
29.  
30.     if(i === 0) {
31.       canvasContext.moveTo(x, y);
32.     } else {
33.       canvasContext.lineTo(x, y);
34.     }
35.  
36.     x += sliceWidth;
37.   }
38.  
39.   canvasContext.lineTo(canvas.width, canvas.height/2);
40. 
41.   // draw the path at once
42.   canvasContext.stroke();
43. 
44.   // once again call the visualize function at 60 frames/s
45.   requestAnimationFrame(visualize)Explanations:

Lines 9-10: we ask for the time domain analysis data. The call to getByteTimeDomainData(dataArray) will fill the array with values corresponding to the waveform to draw. The returned values are between 0 and 255. See the specification for details about what they represent exactly in terms of audio processing.

Below are other examples that draw waveforms.

Example #2: video player with waveform visualization

Using a <video> element is very similar to using an <audio> element. We have made no changes to the JavaScript code here; we Just changed "audio" to "video" in the HTML code.

Example at JSBin:

Example #3: both previous examples, this time with the graphic equalizer

Adding the graphic equalizer to the graph changes nothing, we visualize the sound that goes to the speakers. Try lowering the slider values - you should see the waveform changing.

Example at JSBin

Example at JSBin:

1.5.6 Frequencies

First typical example

Example at JSBin:

This time, instead of a waveform we want to visualize an animated bar chart. Each bar will correspond to a frequency range and 'dance' in concert with the music being played.

The frequency range depends upon the sample rate of the signal (the audio source) and on the FFT size. While the sound is being played, the values change and the bar chart is animated.
The number of bars is equal to the FFT size / 2 (left screenshot with size = 512, right screenshot with size = 64).
In the example above, the Nth bar (from left to right) corresponds to the frequency range N * (samplerate/fftSize). If we have a sample rate equal to 44100 Hz and a FFT size equal to 512, then the first bar represents frequencies between 0 and 44100/512 = 86.12Hz. etc. As the amount of data returned by the analyser node is half the fft size, we will only be able to plot the frequency-range to half the sample rate. You will see that this is generally enough as frequencies in the second half of the sample rate are not relevant.
The height of each bar shows the strength of that specific frequency bucket. It's just a representation of how much of each frequency is present in the signal (i.e. how "loud" the frequency is).

You do not have to master the signal processing 'plumbing' summarised above - just plot the reported values!

Enough said! Let's study some extracts from the source code.

This code is very similar to the first example given at the top of this page. We've set the FFT size to a lower value, and rewritten the animation loop to plot frequency bars instead of a waveform:

Code extract!

1.  function buildAudioGraph() {
2.    ...
3.    // Create an analyser node
4.    analyser = audioContext.createAnalyser();
5.  
6.    // Try changing to lower values: 512, 256, 128, 64...
7.    // Lower values are good for frequency visualizations,
8.    // try 128, 64 etc.? 
9.    analyser.fftSize = 256;
10.   ...
11. }

This time, when building the audio graph, we have used a smaller FFT size. Values between 64 and 512 are very common here. Try them in the JSBin example! Apart from the lines in bold, this function is exactly the same as in the first example.

The new visualization code:

Code extract!

1.  function visualize() {
2.    // clear the canvas
3.    canvasContext.clearRect(0, 0, width, height);
4.  
5.    // Get the analyser data
6.    analyser.getByteFrequencyData(dataArray);
7.   
8.    var barWidth = width / bufferLength;
9.    var barHeight;
10.   var x = 0;
11. 
12.   // values go from 0 to 255 and the canvas heigt is 100. Let's    rescale
13.   // before drawing. This is the scale factor
14.   heightScale = height/128;
15. 
16.   for(var i = 0; i < bufferLength; i++) {
17.     // between 0 and 255
18.     barHeight = dataArray[i];
19.  
20.     // The color is red but lighter or darker depending on the value
21.     canvasContext.fillStyle = 'rgb(' + (barHeight+100) + ',50,50)';
22.     // scale from [0, 255] to the canvas height [0, height]    pixels
23.     barHeight *= heightScale;
24.     // draw the bar
25.        canvasContext.fillRect(x, height-barHeight/2, barWidth, barHeight/2);
26.  
27.     // 1 is the number of pixels between bars - you can change it
28.     x += barWidth + 1;
29.   }
30. 
31.   // once again call the visualize function at 60 frames/s
32.   requestAnimationFrame(visualize);
33. }

Explanations:

Line 6: this is different to code which draws a waveform! We ask for byteFrequencyData (vs byteTimeDomainData earlier) and it returns an array of fftSize/2 values between 0 and 255.
Lines 16-29: we iterate on the value. The x position of each bar is incremented at each iteration (line 28) adding a small interval of 1 pixel between bars (you can try different values here). The width of each bar is computed at line 8.
Line 14: we compute a scale factor to be able to display the values (ranging from 0 to 255) in direct proportion to the height of the canvas. This scale factor is used in line 23, when we compute the height of the bars we are going to draw.

Other examples: achieving more impressive frequency visualization

Example at JSBin with a different look for the visualization: please read the source code and try to understand how the drawing of the frequency is done.

Last example at JSBin with this time the graphic equalizer, a master volume (gain) and a stereo panner node just before the visualizer node:

And here is the audio graph for this example (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

Source code from this example's the buildAudioGraph function:

Code extract!

1.  function buildAudioGraph() {
2.     var mediaElement = document.getElementById('player');
3.        var sourceNode = audioContext.createMediaElementSource(mediaElement);
4.  
5.     // Create an analyser node
6.    analyser = audioContext.createAnalyser();
7.  
8.    // Try changing for lower values: 512, 256, 128, 64...
9.    analyser.fftSize = 1024;
10.   bufferLength = analyser.frequencyBinCount;
11.   dataArray = new Uint8Array(bufferLength);
12. 
13.   // Create the equalizer, which comprises a set of biquad filters
14.   // Set filters
15.   [60, 170, 350, 1000, 3500, 10000].forEach(function(freq, i) {
16.      var eq = audioContext.createBiquadFilter();
17.      eq.frequency.value = freq;
18.      eq.type = "peaking";
19.      eq.gain.value = 0;
20.      filters.push(eq);
21.    });
22.  
23.    // Connect filters in sequence
24.    sourceNode.connect(filters[0]);
25.    for(var i = 0; i < filters.length - 1; i++) {
26.      filters[i].connect(filters[i+1]);
27.    }
28. 
29.    // Master volume is a gain node
30.    masterGain = audioContext.createGain();
31.    masterGain.value = 1;
32. 
33.    // Connect the last filter to the speakers
34.    filters[filters.length - 1].connect(masterGain);
35. 
36.    // for stereo balancing, split the signal
37.    stereoPanner = audioContext.createStereoPanner();
38.    // connect master volume output to the stereo panner
39.    masterGain.connect(stereoPanner);
40. 
41.    // Connect the stereo panner to analyser and analyser to    destination
42.    stereoPanner.connect(analyser);
43.    analyser.connect(audioContext.destination);
44. }

1.5.7 Volume Meters

Important note: the volume meter implementations below use rough approximations and cannot be taken as the most accurate way to compute an exact volume. See at the end of the page for some extra explanations, as well as links to better (and more complex) implementations.

Example #1: add a single volume meter to the audio player

Try it at JSBin:

In order to have a "volume meter" which traces upward/downward with the intensity of the music, we will compute the average intensity of our frequency ranges, and draw this value using a nice gradient-filled rectangle.

Here are the two functions we will call from the animation loop:

Code extract!

1.  function drawVolumeMeter() {
2.    canvasContext.save();
3.  
4.    analyser.getByteFrequencyData(dataArray);
5.    var average = getAverageVolume(dataArray);
6.  
7.    // set the fill style to a nice gradient
8.    canvasContext.fillStyle=gradient;
9.  
10.   // draw the vertical meter
11.   canvasContext.fillRect(0,height-average,25,height);
12. 
13.   canvasContext.restore();
14. }
15.  
16. function getAverageVolume(array) {
17.   var values = 0;
18.   var average;
19. 
20.   var length = array.length;
21. 
22.   // get all the frequency amplitudes
23.   for (var i = 0; i < length; i++) {
24.     values += array[i];
25.   }
26. 
27.   average = values / length;
28.   return average;
29. }

Note that we are measuring intensity (line 4) and once the frequency analysis data is copied into the dataarray, we call the getAverageVolume function (line 5) to compute the average value which we will draw as the volume meter.

This is how we create the gradient:

Code extract!

1.  // create a vertical gradient of the height of the canvas
2.  gradient = canvasContext.createLinearGradient(0,0,0, height);
3.  gradient.addColorStop(1,'#000000');
4.  gradient.addColorStop(0.75,'#ff0000');
5.  gradient.addColorStop(0.25,'#ffff00');
6.  gradient.addColorStop(0,'#ffffff');

And here is what the new animation loop looks like (for the sake of clarity, we have moved the code that draws the signal waveform to a separate function):

Code extract!

1.  function visualize() {
2.  
3.    clearCanvas();
4.  
5.    drawVolumeMeter();
6.    drawWaveform();
7.   
8.     // call again the visualize function at 60 frames/s
9.    requestAnimationFrame(visualize);
10. }

Notice that we used the best practices seen in week 3 of the HTML5 part 1 course: we saved and restored the context in all functions that change something in the canvas context (see function drawVolumeMeter and drawWaveForm in the source code).

Example #2: draw two volume meters, one for each stereo channel

This time, let's split the audio signal and create a separate analyser for each output channel. We retain the analyser node that is being used to draw the waveform, as this works on the stereo signal (and is connected to the destination in order to hear full audio).

We added a stereoPanner node right after the source and a left/right balance slider to control its pan property. Use this slider to see how the left and right volume meter react.

In order to isolate the left and the right channel (for creating individual volume meters), we used a new node called a Channel Splitter node. From this node, we created two routes, each going to a separate analyser (lines 46 and 47 of the example below)

See the ChannelSplitterNode's documentation. Notice that there is also a ChannelMergerNode for merging multiple routes into a single stereo signal.
Use the connect method with extra parameters to connect the different outputs of the channel splitter node:
connect(node, 0, 0) to connect the left output channel to another node,
connect(node, 1, 0) to connect the right output channel to another node,

Example at JSBin:

This is the audio graph we've built (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

As you can see there are two routes: the one on top sends the output signal to the speakers and uses an analyser node to animate the waveform, meanwhile the one at the bottom splits the signal and send its left and right parts to separate analyser nodes which draw the two volume meters. Just before the split, we added a stereoPanner to enable adjustment of the left/right balance with a slider.

Source code extract:

Source code extract!

1.  function buildAudioGraph() {
2.    var mediaElement = document.getElementById('player');
3.    var sourceNode = audioContext.createMediaElementSource(mediaElement);
4.  
5.    // connect the source node to a stereo panner
6.    stereoPanner = audioContext.createStereoPanner();
7.    sourceNode.connect(stereoPanner);
8.  
9.    // Create an analyser node for the waveform
10.   analyser = audioContext.createAnalyser();
11. 
12.   // Use FFT value adapted to waveform drawing
13.   analyser.fftSize = 1024;
14.   bufferLength = analyser.frequencyBinCount;
15.   dataArray = new Uint8Array(bufferLength);
16. 
17.   // Connect the stereo panner to the analyser
18.   stereoPanner.connect(analyser);
19.    // and the analyser to the destination
20.   analyser.connect(audioContext.destination);
21. 
22.   // End of route 1.  We start another route from the
23.   // stereoPanner node, with two analysers for the meters
24. 
25.   // Two analysers for the stereo volume meters
26.   // Here we use a small FFT value as we're gonna work with
27.   // frequency analysis data
28.   analyserLeft = audioContext.createAnalyser();
29.   analyserLeft.fftSize = 256;
30.   bufferLengthLeft = analyserLeft.frequencyBinCount;
31.   dataArrayLeft = new Uint8Array(bufferLengthLeft);
32. 
33.   analyserRight = audioContext.createAnalyser();
34.   analyserRight.fftSize = 256;
35.   bufferLengthRight = analyserRight.frequencyBinCount;
36.   dataArrayRight = new Uint8Array(bufferLengthRight);
37.  
38.   // Split the signal
39.   splitter = audioContext.createChannelSplitter();
40. 
41.   // Connect the stereo panner to the splitter node
42.   stereoPanner.connect(splitter);
43. 
44.   // Connect each of the outputs from the splitter to
45.   // the analysers
46.   splitter.connect(analyserLeft,0,0);
47.   splitter.connect(analyserRight,1,0);
48. 
49.   // No need to connect these analysers to something, the sound
50.   // is already connected through the route that goes through
51.   // the analyser used for the waveform
52. }

And here is the new function for drawing the two volume meters:

Source code extract!

1.  function drawVolumeMeters() {
2.    canvasContext.save();
3.  
4.    // set the fill style to a nice gradient
5.    canvasContext.fillStyle=gradient;
6.  
7.    // left channel
8.    analyserLeft.getByteFrequencyData(dataArrayLeft);
9.    var averageLeft = getAverageVolume(dataArrayLeft);
10. 
11.   // draw the vertical meter for left channel
12.   canvasContext.fillRect(0,height-averageLeft,25,height);
13. 
14.   // right channel
15.   analyserRight.getByteFrequencyData(dataArrayRight);
16.   var averageRight = getAverageVolume(dataArrayRight);
17. 
18.   // draw the vertical meter for left channel
19.   canvasContext.fillRect(26,height-averageRight,25,height);
20. 
21.   canvasContext.restore();
22. }

The code is very similar to the previous one. We draw two rectangles side-by-side, corresponding to the two analyser nodes - instead of the single display in the previous example.

Extra explanations and resources

Indeed, the proposed examples are ok for making things "dancing in music" but rather inaccurate if you are looking for a real volume meter. Results may also change if you modify the size of the fft in the analyser node properties. There are accurate implementations of volume meters in WebAudio (see this volume meter example) but they use nodes that were out of the scope for this course. Also, a student from this course named "SoundSpinning" proposed also another approximation that gives more stable results. Read below:

SoundSpinning: "The only half close way I found for the meter levels is to use getFloatTimeDomainDatadata from the analyser, which seems to give a normalized array between -1 and 1. Then just plot the actual wave level values as we loop in the canvas rendering. This is still not great, since the canvas works at 60Hz while (most of the times) audio sampling is 44.1kHz, but it is closer. This also keeps the same levels no matter whatFFTsizeyou apply."
Here is a codepen with my proposed meters.

1.5.8 Sound Samples Loaded in Memory

For some applications, it may be necessary to load sound samples into memory and uncompress them before they can be used.

No streaming/decoding in real time means less CPU is used,

With all samples loaded in memory, it's possible to play them in sync with great precision,

It's possible to make loops, add effects, change the playback rate, etc.

And of course, if they are in memory and uncompressed, there is no wait time for them to start playing: they are ready to be used immediately!

These features are useful in video games: where a library of sounds may need to ready to be played. By changing the playback rate or the effects, many different sounds can be created, even with a limited number of samples (for instance, an explosion played at different speed, with different effects).

Let's try some demos!

Here is a first example at JSBin: click on the different buttons. Only two minimal sound samples are used in this example: shot1.mp3 and shot2.mp3. You can download many free sound samples like these from the freesound.org Web site.

Here is how the WebAudio graph looks like (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

Music applications such as Digital Audio Workstations (GarageBand-like apps) will need to play/record/loop music tracks in memory.

Try this impressive DAW that uses free sound samples from freesound.org! Each instrument is a small audio file that contains all the notes played on a real instrument. When you play a song (midi file) the app will play-along, selecting the same musical note from the corresponding instrument audio sample. This is all done with Web Audio and samples loaded in memory:

The author of this course wrote a multitrack audio player: it loads different mp3 files corresponding to different instruments and play/loop them in sync.

You can try it or get the sources on GitHub. The documentation is in the help menu.

Try also this small demonstration that uses the Howler.js library for loading sound samples in memory and playing them using WebAudio (we'll discuss this library later). Click on the main window and notice how fast the sound effects are played. Click as fast as you can!

Try the explosion demo at JSBin:

1.5.9 Load and play sound samples

Use an AudioBufferSourceNode as the source of the sound sample in the Web Audio graph.

There is a special node in Web Audio for handling sound samples, called an AudioBufferSourceNode.

This node has different properties:

buffer: the decoded sound sample.
loop: should the sample be played as an infinite loop - when the sample has played to its end, it is re-started from the beginning. (default is True), it also depends on the two next properties.
loopStart: a double value indicating, in seconds, in the buffer sample playing must restart. Its default value is 0.
loopEnd: a double value indicating, in seconds, at what point in the buffer sample playing must stop (and eventually loop again). Its default value is 0.
playbackRate: the speed factor at which the audio asset will be played. Since no pitch correction is applied on the output, this can be used to change the pitch of the sample.
detune: not relevant for this course.

Loading and decoding a sound sample

Before use, a sound sample must be loaded using Ajax, decoded, and set to the buffer property of an AudioBufferSourceNode.

Try the example at JSBin:

In this example, as soon as the page is loaded, we send an Ajax request to a remote server in order to get the file shoot2.mp3. When the file is loaded, we decode it. Then we enable the button (before the sample was not available, and thus could not be played). Now you can click on the button to make the noise.

Notice in the code that each time we click on the button, we rebuild the audio graph.

This is because AudioBufferSourceNodes can be used only once!
But don't worry, Web Audio is optimized for handling thousands of nodes...

HTML code extract:

<button id="playButton" disabled=true>Play sound</button>

JavaScript source code:

JavaScript code extract!

1.  var ctx;
2.   
3.  var soundURL =
4.  'https://mainline.i3s.unice.fr/mooc/shoot2.mp3';
5.  var decodedSound;
6.   
7.  window.onload = function init() {
8.     // The page has been loaded
9.  
10.    // To make it work even on browsers like Safari, that still
11.    // do not recognize the non prefixed version of AudioContext
12.  var audioContext = window.AudioContext || window.webkitAudioContext;
13.  
14.    ctx = new audioContext();
15.  
16.    loadSoundUsingAjax(soundURL);
17. 
18.    // By default the button is disabled, it will be
19.    // clickable only when the sound sample will be loaded
20.    playButton.onclick = function(evt) {
21.       playSound(decodedSound);
22.    };
23. };
24.  
25. function loadSoundUsingAjax(url) {
26.    var request = new XMLHttpRequest();
27. 
28.    request.open('GET', url, true);
29.    // Important: we're loading binary data
30.    request.responseType = 'arraybuffer';
31.  
32.    // Decode asynchronously
33.    request.onload = function() {
34.       console.log("Sound loaded");
35. 
36.       // Let's decode it. This is also asynchronous
37.       ctx.decodeAudioData(request.response,
38.           function(buffer) { // success
39.              console.log("Sound decoded");
40.              decodedSound = buffer;    
41.              // we enable the button
42.              playButton.disabled = false;
43.           },
44.           function(e) { // error
45.              console.log("error");
46.           }
47.       ); // end of decodeAudioData callback
48.     };   // end of the onload callback
49. 
50.     // Send the request. When the file will be loaded,
51.     // the request.onload callback will be called (above)
52.     request.send();
53. }
54.  
55. function playSound(buffer){
56.     // builds the audio graph, then start playing the source
57.     var bufferSource = ctx.createBufferSource();
58.     bufferSource.buffer = buffer;
59.     bufferSource.connect(ctx.destination);
60.     bufferSource.start(); // remember, you can start() a source only once!
61. }

Explanations:

When the page is loaded, we first call the loadSoundUsingAjax function for loading and decoding the sound sample (line 16), then we define a click listener for the play button. Loading and decoding the sound can take some time, so it's an asynchronous process. This means that the call to loadSoundUsingAjax will return while the downloading and decoding is still in progress. We can define a click listener on the button anyway, as it is disabled by default (see the HTML code). Once the sample has been loaded and decoded, only then will the button be enabled (line 42).
The loadSoundUsingAjax function will first create an XmlHttpRequest using the "new version of Ajax called XhR2" (described in detail during week 3). First we create the request (lines 26-30): notice the use of 'arrayBuffer' as a responseType for the request. This has been introduced by Xhr2 and is necessary for binary file transfer. Then the request is sent (line 52).
Ajax is an asynchronous process: once the browser receives the requested file, the request. onload callback will be called (it is defined at line 33), and we can decode the file (an mp3, the content of which must be uncompressed in memory). This is done by calling ctx.decodeAudioData(file, successCallback, errorCallback). When the file is decoded, the success callback is called (lines 38-43). We store the decoded buffer in the variable decodedSound, and we enable the button.
Now, when someone clicks on the button, the playSound function will be called (lines 55-61). This function builds a simple audio graph: it creates an AudioBufferSourceNode (line 57), sets its buffer property with the decoded sample, connects this source to the speakers (line 59) and plays the sound. Source nodes can only be used once (a "fire and forget" philosophy), so to play the sound again, we have to rebuild a source node and connect that to the destination. This seems strange when you learn Web Audio, but don't worry - it's a very fast operation, even with hundreds of nodes.

Loading and decoding multiple sounds: the BufferLoader utility

The problem: AJax requests are asynchronous

The asynchronous aspect of Ajax has always been problematic for beginners. For example, if our applications use multiple sound samples and we need to be sure that all of them are loaded and decoded, using the code we presented in the earlier example will not work as is. We cannot call:

loadSoundSample(urlOfSound1);
loadSoundSample(urlOfSound2);
loadSoundSample(urlOfSound3);
etc...

... because we will never know exactly when all the sounds have finished being loaded and decoded. All these calls will run operations in the background yet return instantly.

The BufferLoader utility object: useful for preloading sound and image assets

There are different approaches for dealing with this problem. During the HTML5 Coding Essentials and Best Practices course, we presented utility functions for loading multiple images. Here we use the same approach and have packaged the code into an object called the BufferedLoader.

Example at JSBin that uses the BufferLoader utility:

HTML code:

1.  <button id="shot1Normal" disabled=true>Shot 1</button>
2.  <button id="shot2Normal" disabled=true>Shot 2</button>

JavaScript code extract (does not contain the BufferLoader utility code):

JavaScript code extract!

1.  var listOfSoundSamplesURLs = [
2.  'https://mainline.i3s.unice.fr/mooc/shoot1.mp3',
3.  'https://mainline.i3s.unice.fr/mooc/shoot2.mp3'
4.  ];
5.   
6.  window.onload = function init() {
7.    // To make it work even on browsers like Safari, that still
8.    // do not recognize the non prefixed version of AudioContext
9.   var audioContext = window.AudioContext || window.webkitAudioContext;
10.  
11.   ctx = new audioContext();
12. 
13.   loadAllSoundSamples();
14. };
15.  
16. function playSampleNormal(buffer){
17.   // builds the audio graph and play
18.   var bufferSource = ctx.createBufferSource();
19.   bufferSource.buffer = buffer;
20.   bufferSource.connect(ctx.destination);
21.   bufferSource.start();
22. }
23.  
24.  
25. function onSamplesDecoded(buffers){
26.    console.log("all samples loaded and decoded");
27.    // enables the buttons
28.    shot1Normal.disabled=false;
29.    shot2Normal.disabled=false;
30. 
31.    // creates the click listeners on the buttons
32.    shot1Normal.onclick = function(evt) {
33.       playSampleNormal(buffers[0]);
34.    };
35. 
36.    shot2Normal.onclick = function(evt) {
37.       playSampleNormal(buffers[1]);
38.    };
39. }
40.  
41. function loadAllSoundSamples() {
42.   // onSamplesDecoded will be called when all samples
43.   // have been loaded and decoded, and the decoded sample will
44.   // be its only parameter (see function above)
45.  bufferLoader = new BufferLoader(ctx, listOfSoundSamplesURLs,onSamplesDecoded);
46. 
47.   // starts loading and decoding the files
48.   bufferLoader.load();
49. }

After the call to loadAllSoundSamples() (line 13), when all the sound sample files have been loaded and decoded, a callback will be initiated to onSamplesDecoded(decodedSamples), located at line 25. The array of decoded samples is the parameter of the onSamplesDecoded function.

The BufferLoader utility object is created at line 45 and takes as parameters: 1) the audio context, 2) an array listing the URLs of the different audio files to be loaded and decoded, and 3) the callback function which is to be called once all the files have been loaded and decoded. This callback function should accept an array as its parameter: the array of decoded sound files.

To study the source of the BufferLoaded object, look at the JavaScript tab in the example at JSBin.

Playing the two sound samples at various playback rates, repeatedly

This is a variant of the previous example (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension).

Example at JSBin:

In this example, we added a function (borrowed and adapted from this article on HTML5Rocks):

makeSource(buffer)

Here is the source code of this function:

Source code extract!

1.  function makeSource(buffer) {
2.     // build graph source -> gain -> compressor -> speakers
3.     // We use a compressor at the end to cut the part of the signal
4.     // that would make peaks
5.     // create the nodes
6.     var source = ctx.createBufferSource();
7.     var compressor = ctx.createDynamicsCompressor();
8.     var gain = ctx.createGain();
9.  
10.    // set their properties
11.    // Not all shots will have the same volume
12.    gain.gain.value = 0.2 + Math.random();
13. 
14.    source.buffer = buffer;
15. 
16.    // Build the graph
17.    source.connect(gain);
18.   gain.connect(compressor);
19.   compressor.connect(ctx.destination);
20. return source;
21. }

And this is the function that plays different sounds in a row, eventually creating random time intervals between them and random pitch variations:

Function playSampleRepeated code extract!

1.  function playSampleRepeated(buffer, rounds, interval, random, random2) {
2.     if (typeof random == 'undefined') {
3.        random = 0;
4.     }
5.     if (typeof random2 == 'undefined') {
6.        random2 = 0;
7.     }
8.  
9.     var time = ctx.currentTime;
10.    // Make multiple sources using the same buffer and play in quick succession.
11.    for (var i = 0; i < rounds; i++) {
12.       var source = makeSource(buffer);
13.       source.playbackRate.value = 1 + Math.random() * random2;
14.       source.start(time + i * interval + Math.random() * random);
15.   } 
16. }

Explanations:

Lines 11-15: we make a loop for building multiple routes in the graph. The number of routes corresponds to the number of times that we want the same buffer to be played. Note that the random2 parameter enables us to randomize the playback rate of the source node that corresponds to the pitch of the sound.
Line 14: this is where the sound is being played. Instead of calling source.start(), we call source.start(delay), this tells the Web Audio scheduler to play the sound after a certain time.
The makeSource function builds a graph from one decoded sample to the speakers. A gain is added that is also randomized in order to generate shot sounds with different volumes (between 0.2 and 1.2 in the example). A compressor node is added in order to limit the max intensity of the signal in case the gain makes it peak.

1.5.10 Sound samples and effects

Any of the effects that discussed during these lectures (gain, stereo panner, reverb, compressor, equalizer, analyser node for visualization, etc.) may be added to the audio graphs that we have built in our sound sample examples.

Below, we have mixed the code from two previous examples:

This one at JSBin:

And this one at JSBin (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

And here is the result (try it at JSBin):

Here is the audio graph of this example (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome WebAudio Inspector extension):

Look at the source code on JSBin, it's a quick merge of the two previous examples.

1.5.11 Useful third party libraries

It's best practice to know the Web Audio API itself. Many of the examples demonstrated during this course may be hard to write using high-level libraries. However, if you don't have too many custom needs, such libraries can make your life simpler! Also, some libraries use sound synthesis that we did not cover in the course and are fun to use - for example, adding 8-bit sounds to your HTML5 game!

Many JavaScript libraries have been built on top of WebAudio. We recommend the following:

HowlerJS: useful for loading and playing sound sample in video games. Can handle audio sprites (multiple sounds in a single audio file), loops, spatialization. Very simple to use. Try this very simple example we prepared for you at JsBin that uses HowlerJS!
Webaudiox, and in particular a helper built with this library, jsfx for adding 8-bit procedural sounds to video games, without the need to load audio files. Try the demo! There is also a sound generator you can try. When you find a sound you like, just copy and paste the parameter values into your code.
For writing musical applications, take a look at ToneJS !

Module 2

Hi! When I was 14, I was fan of the Ramones. I bought an electric guitar and I played in a rock band in my school. A few years later, I wanted to become a game creator. This is how I became a scientist in computer engineering. During this week, I went back in time, because

I'm going to teach you how to write HTML5 games.

During this week, you will greatly improve your knowledge of the HTML5 canvas API by learning the core techniques for writing 2D games that run at 60 frames/s.

We’ll teach you how to create a low level game framework that provides all the basic blocks you need for writing a video game: game loop with time based animation - this is an important technique that will enable your game to run at the same speed on different devices.

We will look at richer interactions with keyboard, mouse, gamepad, and also we will learn how to animate multiple objects on screen.

Not to forget learning about efficient collision detection, managing game states (start menu, game itself with different levels, game over at the end).

We will look also at the sprite based animation technique, that consists in extracting small images of a character posture and by drawing the different sub-images.

It gives the impression of the character moving. We will provide a sprite engine that will help you greatly use this technique. And finally we will look at how we can add music and sound effects with the Web Audio API seen during Week 1.

This is one of the most funny weeks of this course so we hope you will share your own creations and enjoy this funny part of the course.

2.2.1 History of JavaScript games

There is a widely-held belief that games running in Web browsers and without the help of plugins are a relatively new phenomenon. This is not true: back in 1998, Donkey Kong ran in a browser (screenshot below). It was a game by Scott Porter, written using only standard Web technologies (HTML, JavaScript, and CSS) .

Just a few years after the Web was born, JavaScript appeared - a simple script language with C-like syntax for interacting and changing the structure of documents - together with HTML, the HyperText Markup Language used for describing text documents. For the first time, particular elements could be moved across a browser's screen. This was noticed by Scott Porter who, in 1998, created the first JavaScript game library with the very original name, 'Game Lib'. At this time, Porter focused largely on creating ports of old NES and Atari games using animated gifs, but he also developed a Video Pool game in which he emulated the angle of a cue with a sprite of 150 different positions!

During the late 1990s and early 2000s, JavaScript increased in popularity, and the community coined the term 'DHTML' (Dynamic HTML), which was to be the first umbrella term describing a collection of technologies used together to create interactive and animated Web sites. Developers of the DHTML era hadn't forgotten about Porter's 'Game Lib', and within a couple of years, Brent Silby presented 'Game Lib 2'. It is still possible to play many games created with that library on his Web site.

The DHTML era was a time when JavaScript games were as good as those made in Flash. Developers made many DOM libraries that were useful for game development, such as Peter Nederlof's Beehive with its outstanding Rotatrix (which, personally, I think is one of the best HTML games EVER). The first very polished browser games were also developed; Jacob Sidelin, creator of 14KB Mario (screenshot on the right), created the very first page dedicated to JavaScript games.

And then came 2005: 'the year of AJAX'. Even though 'AJAX' just stands for 'Asynchronous JavaScript and XML', in practice it was another umbrella term describing methods, trends and technologies used to create a new kind of Web site - Web 2.0.

Popularization of new JavaScript patterns introduced the ability to create multiplayer connections or even true emulators of old computers. The best examples of this time were 'Freeciv' (screenshot on the left) by Andreas Rosdal - a port of Sid Meier's Civilization, and Sarien.net by Martin Kool, an emulator of old Sierra games.

And now we are entering a new era in the history of the Web: "HTML5"!

2.2.2 Elements and APIs useful for writing games

In the W3Cx [HTML5 Coding Essentials and Best Practices](https://www.edx.org/course/html5-coding-essentials-and-best-practices) course, we study the canvas, drawing, and animation elements. These are going to be revisited in more details in this section.

Here, we present some elements that are useful in writing games.

Drawing: the <canvas> element

The <canvas> is a new HTML element described as "a resolution-dependent bitmap canvas which can be used for rendering graphs, game graphics, or other visual images on the fly." It's a rectangle included in your page where you can draw using scripting with JavaScript. It can, for instance, be used to draw graphs, make photo compositions or do animations. This element comprises a drawable region defined in HTML code with height and width attributes.

You can have multiple canvas elements on one page, even stacked one on top of another, like transparent layers. Each will be visible in the DOM tree and has it's own state independent of the others. It behaves like a regular DOM element.

The canvas has a rich JavaScript API for drawing all kinds of shapes; we can draw wireframe or filled shapes and set several properties such as color, line width, patterns, gradients, etc. It also supports transparency and pixel level manipulations. It is supported by all browsers, on desktop or mobile phones, and on most devices it will take advantage of hardware acceleration.

It is undoubtedly the most important element in the HTML5 specification from a game developer's point of view, so we will discuss it in greater detail later in the course.

The W3C HTML Working Group published HTML Canvas 2D Context as W3C Recommendation (i.e., Web standard status).

Animating at 60 fps: the requestAnimationFrame API

The requestAnimationFrame API targets 60 frames per second animation in canvases. This API is quite simple and also comes with a high resolution timer. Animation at 60 fps is often easy to obtain with simple 2D games on major desktop computers. This is the preferred way to perform animation, as the browser will ensure that animation is not performed when the canvas is not visible, thus saving CPU resources.

Videos and animated textures: the <video> element

The HTML5 <video> element was introduced in the HTML5 specification for the purpose of playing streamed videos or movies, partially replacing the object element. The JavaScript API is nearly the same as the one of the <audio> element and enables full control from JavaScript.

By combining the capabilities of the <video> and <canvas> elements, it is possible to manipulate video data to incorporate a variety of visual effects in real time, and conversely, to use images from videos as "animated textures" over graphic objects.

Audio (streamed audio and real time sound effects): the <audio> element and the Web Audio API

The <audio> element

<audio> is an HTML element that was introduced to give a consistent API for playing streamed sounds in browsers. File format support varies between browsers, but MP3 works in nearly all browsers today. Unfortunately, the <audio> element is only for streaming compressed audio, so it consumes CPU resources, and is not adapted for sound effects where you would like to change the playing speed or add real time effects such as reverberation or doppler. For this, the Web Audio API is preferable.

The Web Audio API

This is a 100% JavaScript API designed for working in real time with uncompressed sound samples or for generating procedural music. Sound samples will need to be loaded into memory and decompressed prior to being used. Up to 12 sound effects are provided natively by browsers that support the API (all major browsers except IE, although Microsoft Edge supports it).

Interacting: dealing with keyboard and mouse events, the GamePad API

User inputs will rely on several APIs, some of which are well established, such as the DOM API that is used for keyboard, touch or mouse inputs. There is also a Gamepad API (in W3C Working Draft status) that is already implemented by some browsers, which we will also cover in this course. The Gamepad specification defines a low-level interface that represents gamepad devices.

Multi-participants feature: WebSockets

IMPORTANT INFORMATION: NOT COVERED IN THIS COURSE

Using the WebSockets technology (which is not part of HTML5 but comes from the W3C WebRTC specification - "Real-time Communication Between Browsers"), you can create two-way communication sessions between multiple browsers and a server. The WebSocket API, and useful libraries built on top of it such as socket.io, provide the means for sending messages to a server and receiving event-driven responses without having to poll the server for a reply.

2.2.3 The "game loop"

The "game loop" is the main component of any game. It separates the game logic and the visual layer from a user's input and actions.

Traditional applications respond to user input and do nothing without it - word processors format text as a user types. If the user doesn't type anything, the word processor is waiting for an action.

Games operate differently: a game must continue to operate regardless of a user's input!

The game loop allows this. The game loop is computing events in our game all the time. Even if the user doesn’t make any action, the game will move the enemies, resolve collisions, play sounds and draw graphics as fast as possible.

Different implementations of the 'Main Game Loop'

There are different ways to perform animation with JavaScript. A very detailed comparison of three different methods has already been presented in the W3Cx HTML5 Coding Essentials course. Below is a quick reminder of the methods, illustrated with new, short, online examples.

Performing animation using the JavaScript setInterval(...) function

Syntax: setInterval(function, ms);

The setInterval function calls a function or evaluates an expression at specified intervals of time (in milliseconds), and returns a unique id of the action. You can always stop this by calling the clearInterval(id) function with the interval identifier as an argument.

Try an example at JSBin : open the HTML, JavaScript and output tabs to see the code.

Source code extract:

1.  var addStarToTheBody = function(){
2.     document.body.innerHTML += "*";
3.  };
4.  
5.  //this will add one star to the document each 200ms (1/5s)
6.  setInterval(addStarToTheBody, 200);

WRONG:

1.  setInterval(‘addStarToTheBody()’, 200);
2.  setInterval(‘document.body.innerHTML += “*”;’, 200);

GOOD:

1.  setInterval(function(){
2.     document.body.innerHTML += “*”;
3.  }, 200);

or like we did in the example, with an external function.

Reminder from the HTML5 Coding Essentials course: with setInterval - if we set the number of milliseconds at, say, 200, it will call our game loop function EACH 200ms, even if the previous one is not yet finished. Because of this disadvantage, we might prefer to use another function, better suited to our goals.

Using setTimeout() instead of setInterval()

Syntax: setTimeout(function, ms);

The setTimeout function works like setInterval but with one little difference: it calls your function AFTER a given amount of time.

Try an example at JSBin: open the HTML, JavaScript and output tabs to see the code. This example does the same thing as the previous example by adding a "*" to the document every 200ms.

Source code extract:

Source code extract!

1.  var addStarToTheBody = function(){
2.      document.body.innerHTML += "*";
3.      // calls again itself AFTER 200ms
4.      setTimeout(addStarToTheBody, 200);
5.  };
6.  
7.  // calls the function AFTER 200ms
8.  setTimeout(addStarToTheBody, 200);

This example will work like the previous example. However, it is a definite improvement, because the timer waits for the function to finish everything inside before calling it again.

For several years, setTimeout was the best and most popular JavaScript implementation of game loops. This changed when Mozilla presented the requestAnimationFrame API, which became the reference W3C standard API for game animation.

Using the requestAnimationFrame API

Note: using requestAnimationFrame was covered in detail in the W3Cx HTML5 Coding Essentials course.

When we use timeouts or intervals in our animations, the browser doesn’t have any information about our intentions -- do we want to repaint the DOM structure or a canvas during every loop? Or maybe we just want to make some calculations or send requests a couple of times per second? For this reason, it is really hard for the browser’s engine to optimize the loop.

And since we want to repaint the game (move the characters, animate sprites, etc.) every frame, Mozilla and other contributors/developers introduced a new approach which they called requestAnimationFrame.

This approach helps the browser to optimize all the animations on the screen, no matter whether Canvas, DOM or WebGL. Also, if the animation loop is running in a browser tab that is not currently visible, the browser won't keep it running.

Basic usage, online example at JSBin.

Source code extract:

Source code extract!

1.  window.onload = function init() {
2.     // called after the page is entirely loaded
3.     requestAnimationFrame(mainloop);
4.  };
5.  
6.  function mainloop(timestamp) {
7.     document.body.innerHTML += "*";
8.  
9.     // call back itself every 60th of second
10.    requestAnimationFrame(mainloop);
11. }

Notice that calling requestAnimationFrame(mainloop) at line 10, asks the browser to call the mainloop function every 16.6 ms: this corresponds to 60 times per second (16.6 ms = 1/60 s).

This target may be hard to reach; the animation loop content may take longer than this, or the scheduler may be a bit early or late.

Many "real action games" perform what we call time-based animation. For this, we need an accurate timer that will tell us the elapsed time between each animation frame. Depending on this time, we can compute the distances each object on the screen must achieve in order to move at a given speed, independently of the CPU or GPU of the computer or mobile device that is running the game.

The timestamp parameter of the mainloop function is useful for exactly that: it gives a high resolution time.

We will cover this in more detail, later in the course.

2.3.1 A game framework skeleton

We are going to develop a game - not all at once, let's divide the whole job into a series of smaller tasks. The first step is to create a foundation or basic structure.

Let's start by building the skeleton of a small game framework, based on the Black Box Driven Development in JavaScript methodology. In other words: a game framework skeleton is a simple object-based model that uses encapsulation to expose only useful methods and properties.

We will evolve this framework throughout the lessons in this course, and cut it in different files once it becomes too large to fit within one single file.

Here is the starting point:

Source code extract!

1.  var GF = function(){
2.  
3.    var mainLoop = function(time){
4.      //Main function, called each frame
5.      requestAnimationFrame(mainLoop);
6.    };
7.  
8.    var start = function(){
9.      requestAnimationFrame(mainLoop);
10.   };
11. 
12.   // Our GameFramework returns a public API visible from outside its scope
13.   // Here we only expose the start method, under the "start" property name.
14.   return {
15.     start: start
16.   };
17. };

With this skeleton, it's very easy to create a new game instance:

1.  var game = new GF();
2.  
3.  // Launch the game, start the animation loop, etc.
4.  game.start();

Examples

Let's put something into the mainLoop function, and check if it works

Try this online example at JSBin, with a new mainloop: (check the JavaScript and output tabs). This page should display a different random number every 1/60 second. We don't have a real game yet, but we're improving our game engine :-)

Source code extract:

Source code extract!

1.  var mainLoop = function(time){
2.    // main function, called each frame
3.    document.body.innerHTML = Math.random();
4.  
5.    // call the animation loop every 1/60th of second
6.    requestAnimationFrame(mainLoop);
7.  };

Let's measure that animation's frame rate

Every game needs to have a function which measures the actual frame rate achieved by the code.

The principle is simple:

Count the time elapsed by adding deltas in the mainloop.
If the sum of the deltas is greater or equal to 1000, then 1s has elapsed since we started counting.
If at the same time, we count the number of frames that have been drawn, then we have the frame rate - measured in number of frames per second. Remember, it should be around 60 fps!

Quick glossary: the word delta is the name of a Greek letter (uppercase Δ, lowercase δ or 𝛿). The upper-case version is used in mathematics as an abbreviation for measuring the change in some object, over time - in our case, how quickly the mainloopis running. This dictates the maximum speed at which the game display will be updated. This maximum speed could be referred to as the rate of change. We call what is displayed at a single point-in-time, a frame. Thus the rate of change can be measured in frames per second (fps). Accordingly, our game's delta, determines the achievable frame rate - the shorter the delta (measured in mS), the faster the possible rate of change (in fps).

Here is a screenshot of an example and the code we added to our game engine, for measuring FPS (try it online at JSBin):

Source code extract:

Source code extract!

1.     // vars for counting frames/s, used by the measureFPS function
2.  var frameCount = 0;
3.  var lastTime;
4.  var fpsContainer;
5.  var fps;
6.  
7.  var measureFPS = function(newTime){
8.  
9.     // test for the very first invocation
10.    if(lastTime === undefined) {
11.      lastTime = newTime;
12.      return;
13.    }
14. 
15.    // calculate the delta between last & current frame
16.    var diffTime = newTime - lastTime;
17. 
18.    if (diffTime >= 1000) {
19.      fps = frameCount;
20.      frameCount = 0;
21.      lastTime = newTime;
22.    }
23. 
24.    // and display it in an element we appended to the
25.    // document in the start() function
26.    fpsContainer.innerHTML = 'FPS: ' + fps;
27.    frameCount++;
28. };

Now we can call the measureFPS function from inside the animation loop, passing it the current time, given by the high resolution timer that comes with the requestAnimationFrame API:

Source code extract!

1.  var mainLoop = function(time){
2.    // compute FPS, called each frame, uses the high resolution time parameter 
3.    // given by the browser that implements the requestAnimationFrame API
4.    measureFPS(time);
5.  
6.    // call the animation loop every 1/60th of second
7.    requestAnimationFrame(mainLoop);
8.  };

And the <div> element used to display FPS on the screen is created in this example by the start() function:

Source code extract!

1.  var start = function(){
2.    // adds a div for displaying the fps value
3.    fpsContainer = document.createElement(div);
4.    document.body.appendChild(fpsContainer);
5.  
6.    requestAnimationFrame(mainLoop);
7.  };

Hack: achieving more than 60 fPS? It's possible but to be avoided except in hackers' circles!

We also know methods of implementing loops in JavaScript which achieve even more than 60fps (this is the limit using requestAnimationFrame).

My favorite hack uses the onerror callback on an <img> element like this:

Source code extract!

1.  function mainloop(){
2.    var img = new Image;
3.  
4.    img.onerror = mainloop;
5.    img.src = 'data:image/png,' + Math.random();
6.  }

What we are doing here, is creating a new image on each frame and providing invalid data as a source of the image. The image cannot be displayed properly, so the browser calls the onerror event handler that is the mainloop function itself, and so on.

Funny right? Please try this and check the number of FPS displayed with this JSBin example.

Source code extract of this example:

Code extract!

1.  var mainLoop = function(){
2.    // main function, called each frame
3.  
4.    measureFPS(+(new Date()));
5.  
6.    // call the animation loop every LOTS of seconds using previous hack method
7.    var img = new Image();
8.    img.onerror = mainLoop;
9.    img.src = 'data:image/png,' + Math.random();
10. };

2.3.2 Introducing graphics

[Note: drawing within a canvas is studied in detail during the [W3C HTML5 Coding Essentials and Best Practices course](https://www.edx.org/course/html5-coding-essentials-and-best-practices), in module 3.]

Is this really a course about games? Where are the graphics?
Good news! We will add graphics to our game engine in this lesson! To-date we have talked of "basic concepts"; so without further ado, let's draw something, animate it, and move shapes around the screen :-)

Let's do this by including into our framework the same "monster" we used during the W3C HTML5 Coding Essentials and Best Practices course.

HTML5 canvas basic usage: drawing a monster

How to draw a monster in a canvas: you can try it online at JSBin.

HTML code (declaration of the canvas):

Source code extract!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.    <meta charset="utf-8">
5.    <title>Draw a monster in a canvas</title>
6.  </head>
7.  <body>
8.    <canvas id="myCanvas" width="200" height="200"></canvas>
9.  </body>
10. </html>

The canvas declaration is at line 8. Use attributes to give it a width and a height, but unless you add some CSS properties, you will not see it on the screen because it's transparent!

Let's use CSS to reveal the canvas, for example, add a 1px black border around it:

1.  canvas {
2.     border: 1px solid black;
3.  }

And here is a reminder of best practices when using the canvas, as described in the HTML5 Part 1 course:

Use a function that is called AFTER the page is fully loaded (and the DOM is ready), set a pointer to the canvas node in the DOM.
Then, get a 2D graphic context for this canvas (the context is an object we will use to draw on the canvas, to set global properties such as color, gradients, patterns and line width).
Only then can you can draw something,
Do not forget to use global variables for the canvas and context objects. I also recommend keeping the width and height of the canvas somewhere. These might be useful later.
For each function that will change the context (color, line width, coordinate system, etc.), start by saving the context, and end by restoring it.

Here is JavaScript code which implements those best practices:

Code extract!

1.  // useful to have them as global variables
2.  var canvas, ctx, w, h;
3.   
4.   
5.  window.onload = function init() {
6.    // Called AFTER the page has been loaded
7.    canvas = document.querySelector("#myCanvas");
8.  
9.    // Often useful
10.   w = canvas.width;
11.   h = canvas.height;
12. 
13.   // Important, we will draw with this object
14.   ctx = canvas.getContext('2d');
15. 
16.   // Ready to go!
17.   // Try to change the parameter values to move
18.   // the monster
19.   drawMyMonster(10, 10);
20. };
21.  
22. function drawMyMonster(x, y) {
23.   // Draw a big monster!
24.   // Head
25. 
26.   // BEST practice: save the context, use 2D transformations
27.   ctx.save();
28. 
29.   // Translate the coordinate system, draw relative to it
30.   ctx.translate(x, y);
31. 
32.   // (0, 0) is the top left corner of the monster.
33.   ctx.strokeRect(0, 0, 100, 100);
34. 
35.   // Eyes
36.   ctx.fillRect(20, 20, 10, 10);
37.   ctx.fillRect(65, 20, 10, 10);
38. 
39.   // Nose
40.   ctx.strokeRect(45, 40, 10, 40);
41. 
42.   // Mouth
43.   ctx.strokeRect(35, 84, 30, 10);
44. 
45.   // Teeth
46.   ctx.fillRect(38, 84, 10, 10);
47.   ctx.fillRect(52, 84, 10, 10);
48. 
49.   // BEST practice: restore the context
50.   ctx.restore();
51. }

In this small example, we used the context object to draw a monster using the default color (black) and wireframe and filled modes:

ctx.fillRect(x, y, width, height): draws a rectangle whose top left corner is at (x, y) and whose size is specified by the width and height parameters; and both outlined by, and filled with, the default color.
ctx.strokeRect(x, y, width, height): same but in wireframe mode.
Note that we use (line 30) ctx.translate(x, y) to make it easier to move the monster around. So, all the drawing instructions are coded as if the monster was in (0, 0), at the top left corner of the canvas (look at line 33). We draw the body outline with a rectangle starting from (0, 0). Calling context.translate "changes the coordinate system" by moving the "old (0, 0)" to (x, y) and keeping other coordinates in the same position relative to the origin.
Line 19: we call the drawMonster function with (10, 10) as parameters, which will cause the original coordinate system to be translated by (10, 10).
And if we change the coordinate system (this is what the call to ctx.translate(...) does) in a function, it is a best practice to always save the previous context at the beginning of the function and restore it at the end of the function (lines 27 and 50).

Animating the monster and including it in our game engine

Ok, now that we know how to move the monster, let's integrate it into our game engine:

add the canvas to the HTML page,
add the content of the init() function to the start() function of the game engine,
add a few global variables (canvas, ctx, etc.),
call the drawMonster(...) function from the mainLoop,
add a random displacement to the x, y position of the monster to see it moving,
in the main loop, do not forget to clear the canvas before drawing again; this is done using the ctx.clearRect(x, y, width, height) function.

You can try this version online at JSBin.

HTML code:

Code extract!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.  <meta charset="utf-8">
5.  <title>Trembling monster in the Game Framework</title>
6.  </head>
7.  <body>
8.  <canvas id="myCanvas" width="200" height="200"></canvas>
9.  </body>
10. </html>

JavaScript complete code:

Code extract!

1.  // Inits
2.  window.onload = function init() {
3.    var game = new GF();
4.    game.start();
5.  };
6.   
7.   
8.  // GAME FRAMEWORK STARTS HERE
9.  var GF = function(){
10.   // Vars relative to the canvas
11.   var canvas, ctx, w, h;
12.  
13.   ...
14. 
15.   var measureFPS = function(newTime){
16. 
17.     ...
18. };
19. 
20. // Clears the canvas content
21. function clearCanvas() {
22.    ctx.clearRect(0, 0, w, h);
23. }
24. 
25. // Functions for drawing the monster and perhaps other objects
26. function drawMyMonster(x, y) {
27.    ...
28. }
29. 
30. var mainLoop = function(time){
31.     // Main function, called each frame
32.     measureFPS(time);
33. 
34.     // Clear the canvas
35.     clearCanvas();
36. 
37.     // Draw the monster
38.     drawMyMonster(10+Math.random()*10, 10+Math.random()*10);
39. 
40.     // Call the animation loop every 1/60th of second
41.     requestAnimationFrame(mainLoop);
42. };
43.  
44. var start = function(){
45.     ...
46. 
47.     // Canvas, context etc.
48.     canvas = document.querySelector("#myCanvas");
49. 
50.     // often useful
51.     w = canvas.width;
52.     h = canvas.height;
53. 
54.     // important, we will draw with this object
55.     ctx = canvas.getContext('2d');
56.  
57.     // Start the animation
58.     requestAnimationFrame(mainLoop);
59. };
60.  
61. //our GameFramework returns a public API visible from outside its scope
62. return {
63.    start: start
64. };
65. };

Explanations:

Note that we now start the game engine in a window.onload callback (line 2), so only after the page has been loaded.
We also moved 99% of the init() method from the previous example into the start() method of the game engine, and added the canvas, ctx, w, h variables as global variables to the game framework object.
Finally, in the main loop we added a call to the drawMonster() function, injecting randomicity through the parameters: the monster is drawn with an x,y offset of between 0 and 10, in successive frames of the animation.
And we clear the previous canvas content before drawing the current frame (line 35).

If you try the example, you will see a trembling monster. The canvas is cleared and the monster drawn in random positions, at around 60 times per second!

Next, let's see how to interact with it using the mouse or the keyboard.

2.3.3 User interaction and event handling

Input & output: how do events work in Web apps & games?

HTML5 events

There is no input or output in JavaScript. We treat events caused by user actions as inputs, and we manipulate the DOM structure as output. Usually in games, we will maintain state variables representing moving objects like the position and speed of an alien ship, and the animation loop will refer to these variables in determining the movement of such objects.

In any case, the events are called DOM events, and we use the DOM APIs to create event handlers.

How to listen to events

There are three ways to manage events in the DOM structure. You could attach an event inline in your HTML code like this:

Method #1: declare an event handler in the HTML code

<div id="someDiv" onclick="alert('clicked!')"> content of the
div </div>

This method is very easy to use, but it is not the recommended way to handle events. Indeed, It works today but is deprecated (will probably be abandoned in the future). Mixing 'visual layer' (HTML) and 'logic layer' (JavaScript) in one place is really bad practice and causes a host of problems during development.

Method #2: attach an event handler to an HTML element in JavaScript

document.getElementById('someDiv').onclick = function() {
   alert('clicked!');
}

This method is fine, but you will not be able to attach multiple listener functions. If you need to do this, use the version shown below.

Method #3: register a callback to the event listener with the addEventListener method (preferred method)

document.getElementById('someDiv').addEventListener('click', function() {
   alert('clicked!');
}, false);

Note that the third parameter describes whether the callback has to be called during the captured phase. This is not important for now, just set it to false.

Details of the DOM event are passed to the event listener function

When you create an event listener and attach it to an element, the listener will create an event object to describe what happened. This object is provided as a parameter of the callback function:

element.addEventListener('click', function(event) {
   / now you can use event object inside the callback /
}, false);

Depending on the type of event you are listening to, you will consult different properties from the event object in order to obtain useful information such as: "which keys are pressed down?", "what is the location of the mouse cursor?", "which mouse button has been clicked?", etc.

In the following lessons, we will remind you now how to deal with the keyboard and the mouse (previously covered during the HTML5 Part 1 course) in the context of a game engine (in particular, how to manage multiple events at the same time), and also demonstrate how you can accept input from a game pad using the new Gamepad API.

Further reading

In the method #1 above, we mentioned that "Mixing 'visual layer' (HTML) and 'logic layer' (JavaScript) ... bad practice", and this is similarly reflected in many style features being deprecated in HTML5 and moved into CSS3. The management philosophy at play here is called "the separation of concerns" and applies in several ways to software development - at the code level, through to the management of staff. It's not part of the course, but professionals may find the following references useful:

Separation of concerns - Wikipedia, the free encyclopedia

2.3.4 Adding key listeners

A few reminders

This has been something of a nightmare for years, as different browsers had different ways of handling key events and key codes (read this if you are fond of JavaScript archaeology). Fortunately, it's much improved today and we can rely on methods that should work in any browser less than four years old.

After a keyboard-related event (eg keydown or keyup), the code of the key that fired the event will be passed to the listener function. It is possible to test which key has been pressed or released, like this:

Source code extract!

1.  window.addEventListener('keydown', function(event) {
2.     if (event.keyCode === 37) {
3.       // Left arrow was pressed
4.     }
5.  }, false);

At line 2, the key code of 37 corresponds to the left arrow key.

You can try key codes with this interactive example, and here is a list of keyCodes (from CSS Tricks

Game requirements: managing multiple keypress / keyrelease events

In a game, we often need to check which keys are being used, at a very high frequency - typically from inside the game loop that is looping at up to 60 times per second.

If a spaceship is moving left, chances are you are keeping the left arrow down, and if it's firing missiles at the same time you must also be pressing the space bar like a maniac, and maybe pressing the shift key to release smart bombs.

Sometimes these three keys might be down at the same time, and the game loop will have to take these three keys into account: move the ship left, release a new missile if the previous one is out of the screen or if it reached a target, launch a smart bomb if conditions are met, etc.

Keep the list of pertinent keys in a JavaScript object

The typical method used is: store the list of the keys (or mouse button or whatever game pad button...) that are up or down at a given time in a JavaScript object. For our small game engine we will call this object "inputStates".

We will update its content inside the different input event listeners, and later check its values inside the game loop to make the game react accordingly.

Add this to our game framework:

These are the changes to our small game engine prototype (which is far from finished yet):

We add an empty inputStates object as a global property of the game engine,
In the start() method, we add event listeners for each keydown and keyup event which controls the game.
In each listener, we test if an arrow key or the space bar has been pressed or released, and we set the properties of the inputStates object accordingly. For example, if the space bar is pressed, we set inputStates.space=true; but if it's released, we reset to inputStates.space=false.
In the main loop (to prove everything is working), we add tests to check which keys are down; and if a key is down, we print its name on the canvas.

Here is the online example you can try at JSBin

And here is the complete source code:

Code extract!

1.  // Inits
2.  window.onload = function init() {
3.    var game = new GF();
4.    game.start();
5.  };
6.  
7.  
8.  // GAME FRAMEWORK STARTS HERE
9.  var GF = function(){
10.    ... 
11. 
12.    // vars for handling inputs
13.    var inputStates = {};
14. 
15.    var measureFPS = function(newTime){
16.       ...
17.    };
18. 
19.    // Clears the canvas content
20.    function clearCanvas() {
21.      ctx.clearRect(0, 0, w, h);
22.    }
23. 
24.    // Functions for drawing the monster and perhaps other objects
25.    function drawMyMonster(x, y) {
26.       ...
27.    }
28. 
29.    var mainLoop = function(time){
30.      // Main function, called each frame
31.      measureFPS(time);
32. 
33.      // Clears the canvas
34.      clearCanvas();
35. 
36.      // Draws the monster
37.      drawMyMonster(10+Math.random()*10, 10+Math.random()*10);
38.  
39.     // check inputStates
40.     if (inputStates.left) {
41.       ctx.fillText("left", 150, 20);
42.     }
43.     if (inputStates.up) {
44.       ctx.fillText("up", 150, 50);
45.     }
46.    if (inputStates.right) {
47.       ctx.fillText("right", 150, 80);
48.    }
49.    if (inputStates.down) {
50.      ctx.fillText("down", 150, 120);
51.    }
52.    if (inputStates.space) {
53.      ctx.fillText("space bar", 140, 150);
54.    }
55. 
56.    // Calls the animation loop every 1/60th of second
57.    requestAnimationFrame(mainLoop);
58. };
59. 
60. var start = function(){
61.     ...
62.     // Important, we will draw with this object
63.     ctx = canvas.getContext('2d');
64.     // Default police for text
65.     ctx.font="20px Arial";
66. 
67.     // Add the listener to the main, window object, and update the states
68.     window.addEventListener('keydown', function(event){
69.       if (event.keyCode === 37) {
70.         inputStates.left = true;
71.       } else if (event.keyCode === 38) {
72.         inputStates.up = true;
73.       } else if (event.keyCode === 39) {
74.         inputStates.right = true;
75.       } else if (event.keyCode === 40) {
76.         inputStates.down = true;
77.       } else if (event.keyCode === 32) {
78.         inputStates.space = true;
79.       }
80.     }, false);
81. 
82.     // If the key is released, change the states object
83.     window.addEventListener('keyup', function(event){
84.       if (event.keyCode === 37) {
85.         inputStates.left = false;
86.       } else if (event.keyCode === 38) {
87.         inputStates.up = false;
88.       } else if (event.keyCode === 39) {
89.         inputStates.right = false;
90.       } else if (event.keyCode === 40) {
91.         inputStates.down = false;
92.       } else if (event.keyCode === 32) {
93.         inputStates.space = false;
94.       }
95.     }, false);
96. 
97. 
98.    // Starts the animation
99.    requestAnimationFrame(mainLoop);
100.  };
101. 
102. // our GameFramework returns a public API visible from outside its scope
103. return {
104.    start: start
105. };
106. };

You may notice that on some computers / operating systems, it is not possible to simultaneously press the up and down arrow keys, or left and right arrow keys, because they are mutually exclusive. However space + up + right should work in combination.

2.3.5 Adding mouse listeners

A few reminders

Working with mouse events requires detecting whether a mouse button is up or down, identifying that button, keeping track of mouse movement, getting the x and y coordinates of the cursor, etc.

Special care must be taken when acquiring mouse coordinates because the HTML5 canvas has default (or directed) CSS properties which could produce false coordinates. The trick to get the right x and y mouse cursor coordinates is to use this method from the canvas API:

// necessary to take into account CSS boundaries

var rect = canvas.getBoundingClientRect();

The width and height of the rect object must be taken into account. These dimensions correspond to the padding / margins / borders of the canvas. See how we deal with them in the getMousePos() function in the next example.

Here is an online example at JSBin that covers all cases correctly.

Move the mouse over the canvas and press or release mouse buttons. Notice that we keep the state of the mouse (position, buttons up or down) as part of the inputStates object, just as we do with the keyboard (per previous lesson).

Below is the JavaScript source code for this small example:

JavaScript code extract!

1.  var canvas, ctx;
2.  var inputStates = {};
3.  
4.  window.onload = function init() {
5.     canvas = document.getElementById('myCanvas');
6.     ctx = canvas.getContext('2d');
7.  
8.     canvas.addEventListener('mousemove', function (evt) {
9.        inputStates.mousePos = getMousePos(canvas, evt);
10.       var message = 'Mouse position:
    ' + inputStates.mousePos.x + ',' +                                 
      inputStates.mousePos.y;
11.       writeMessage(canvas, message);
12.    }, false);
13. 
14.    canvas.addEventListener('mousedown', function (evt) {
15.       inputStates.mousedown = true;
16.       inputStates.mouseButton = evt.button;
17.       var message = "Mouse button " + evt.button + " down at position: " + 
18.                     inputStates.mousePos.x + ',' + inputStates.mousePos.y;
19.       writeMessage(canvas, message);
20.    }, false);
21. 
22.    canvas.addEventListener('mouseup', function (evt) {
23.       inputStates.mousedown = false;
24.       var message = "Mouse up at position: " + inputStates.mousePos.x + ',' +     
25.                     inputStates.mousePos.y;
26.       writeMessage(canvas, message);
27.    }, false);
28. };
29. 
30. function writeMessage(canvas, message) {
31.   var ctx = canvas.getContext('2d');
32.   ctx.save();
33.   ctx.clearRect(0, 0, canvas.width, canvas.height);
34.   ctx.font = '18pt Calibri';
35.   ctx.fillStyle = 'black';
36.   ctx.fillText(message, 10, 25);
37.   ctx.restore();
38. }
39. 
40. function getMousePos(canvas, evt) {
41.   // necessary to take into account CSS boudaries
42.   var rect = canvas.getBoundingClientRect();
43.   return {
44.      x: evt.clientX - rect.left,
45.      y: evt.clientY - rect.top
46.   };
47. }

Making an object follow the mouse cursor

Try this example at JsBin.

Source code:

Code extract!

1.  var canvas, ctx, width, height;
2.  var rect = {x:40, y:40, rayon: 30, width:80, height:80, v:1};
3.  var mousepos = {x:0, y:0};
4.   
5.  function init() {
6.      canvas = document.querySelector("#myCanvas");
7.      ctx = canvas.getContext('2d');
8.      width = canvas.width;
9.      height = canvas.height;
10. 
11.     canvas.addEventListener('mousemove', function (evt) {
12.        mousepos = getMousePos(canvas, evt);
13.     }, false);
14. 
15.     mainloop();
16. }
17.  
18. function mainloop() {
19.    // 1) clear screen
20.    ctx.clearRect(0, 0, canvas.width, canvas.height);
21. 
22.    // 2) move object
23.     var dx = rect.x - mousepos.x;
24.     var dy = rect.y - mousepos.y;
25.     var angle = Math.atan2(dy, dx);
26. 
27.     rect.x -= rect.v*Math.cos(angle);
28.     rect.y -= rect.v*Math.sin(angle);
29. 
30.     // 3) draw object
31.     drawRectangle(angle);
32. 
33.     // request new frame
34.     window.requestAnimationFrame(mainloop);
35. }
36. 
37. function drawRectangle(angle) {
38.    ctx.save();
39. 
40.    // These two lines move the coordinate system
41.    ctx.translate(rect.x, rect.y);
42.    ctx.rotate(angle);
43.    // recenter the coordinate system in the middle
44.    // the rectangle. Like that it will rotate around
45.    // this point instead of top left corner
46.    ctx.translate(-rect.width/2, -rect.height/2);
47. 
48.    ctx.fillRect(0, 0, rect.width, rect.height);
49.    ctx.restore();
50. }
51.  
52. function getMousePos(canvas, evt) {
53.    // necessary to take into account CSS boudaries
54.    var rect = canvas.getBoundingClientRect();
55.    return {
56.       x: evt.clientX - rect.left,
57.       y: evt.clientY - rect.top
58.    };
59. }

Explanations:

Line 25 calculates the angle between mouse cursor and the rectangle,
Lines 27-28 move the rectangle v pixels along a line between the rectangle's current position and the mouse cursor,
Lines 41-46 translate the rectangle, rotate it, and recenter the rotational point to the center of the rectangle (in its new position).

Adding mouse listeners to the game framework

Now we will include these listeners into our game framework. Notice that we changed some parameters (no need to pass the canvas as a parameter of the getMousePos() function, for example).

The new online version of the game engine can be tried at JSBin:

Try pressing arrows and space keys, moving the mouse, and pressing the buttons, all at the same time. You'll see that the game framework handles all these events simultaneously because the global variable named inputStates is updated by keyboard and mouse events, and consulted to direct movements every 1/60th second.

JavaScript source code:

Code extract!

1.  // Inits
2.  window.onload = function init() {
3.    var game = new GF();
4.    game.start();
5.  };
6.  
7.  
8.  // GAME FRAMEWORK STARTS HERE
9.  var GF = function(){
10.   ...
11. 
12.   // Vars for handling inputs
13.   var inputStates = {};
14. 
15.   var measureFPS = function(newTime){
16.      ...
17.   };
18. 
19.   // Clears the canvas content
20.   function clearCanvas() {
21.     ctx.clearRect(0, 0, w, h);
22.   }
23. 
24.   // Functions for drawing the monster and perhaps other objects
25.   function drawMyMonster(x, y) {
26.     ...
27.   }
28. 
29.   var mainLoop = function(time){
30.    // Main function, called each frame
31.    measureFPS(time);
32. 
33.    // Clears the canvas
34.    clearCanvas();
35. 
36.    // Draws the monster
37.    drawMyMonster(10+Math.random()*10, 10+Math.random()*10);
38.    // Checks inputStates
39.    if (inputStates.left) {
40.      ctx.fillText("left", 150, 20);
41.    }
42.    if (inputStates.up) {
43.      ctx.fillText("up", 150, 40);
44.    }
45.    if (inputStates.right) {
46.      ctx.fillText("right", 150, 60);
47.    }
48.    if (inputStates.down) {
49.      ctx.fillText("down", 150, 80);
50.    }
51.    if (inputStates.space) {
52.      ctx.fillText("space bar", 140, 100);
53.    }
54.    if (inputStates.mousePos) {
55.      ctx.fillText("x = " + inputStates.mousePos.x + " y = " +
56.                            inputStates.mousePos.y, 5, 150);
57.    }
58.    if (inputStates.mousedown) {
59.      ctx.fillText("mousedown b" + inputStates.mouseButton, 5, 180);
60.    }
61. 
62.    // Calls the animation loop every 1/60th of second
63.    requestAnimationFrame(mainLoop);
64.   };
65. 
66. 
67.   function getMousePos(evt) {
68.     // Necessary to take into account CSS boudaries
69.     var rect = canvas.getBoundingClientRect();
70.     return {
71.       x: evt.clientX - rect.left,
72.       y: evt.clientY - rect.top
73.     };
74.   }
75. 
76.   var start = function(){
77.     ...
78.     // Adds the listener to the main window object, and updates the states
79.     window.addEventListener('keydown', function(event){
80.       if (event.keyCode === 37) {
81.         inputStates.left = true;
82.       } else if (event.keyCode === 38) {
83.         inputStates.up = true;
84.       } else if (event.keyCode === 39) {
85.         inputStates.right = true;
86.       } else if (event.keyCode === 40) {
87.         inputStates.down = true;
88.       } else if (event.keyCode === 32) {
89.         inputStates.space = true;
90.       }
91.     }, false);
92. 
93.     // If the key is released, changes the states object
94.    window.addEventListener('keyup', function(event){
95.       if (event.keyCode === 37) {
96.         inputStates.left = false;
97.       } else if (event.keyCode === 38) {
98.        inputStates.up = false;
99.       } else if (event.keyCode === 39) {
100.        inputStates.right = false;
101.       } else if (event.keyCode === 40) {
102.        inputStates.down = false;
103.       } else if (event.keyCode === 32) {
104.         inputStates.space = false;
105.       }
106.    }, false);
107. 
108.    // Mouse event listeners
109.    canvas.addEventListener('mousemove', function (evt) {
110.      inputStates.mousePos = getMousePos(evt);
111.    }, false);
112. 
113.    canvas.addEventListener('mousedown', function (evt) {
114.      inputStates.mousedown = true;
115.      inputStates.mouseButton = evt.button;
116.    }, false);
117. 
118.    canvas.addEventListener('mouseup', function (evt) {
119.      inputStates.mousedown = false;
120.    }, false);
121. 
122. 
123.    // Starts the animation
124.    requestAnimationFrame(mainLoop);
125.   };
126. 
127.   // Our GameFramework returns a public API visible from outside its scope
128.   return {
129.     start: start
130.   };
131. };

2.3.6 Gamepad events

Hi! In this lesson we will look at how we can manage such an input device! This is a Microsoft xbox 360 controller -a wired one- with an USB plug. And we will see how we can use the gamepad API that is available on modern browsers... except a few ones...

The first thing you can do is to add some even listeners for the gamepadconnect and gamepaddisconnected events.

If I plug in the game pad (I’m using Google Chrome for this demo), I plug it in...

Here we are! I need to press a button, for the gamepad to be detected. If I just plug it in: it won't be detected. On FireFox, I tried too... and it has been detected as soon as I plugged it in.

Once it's plugged, you can get a property of the event that is called gamepad... and you can get the number of buttons, and the number of axes.

Here, it says it's got 4 axes... the axes are for the joysticks... horizontal and vertical axes. We will see how to manage that in a minute. And it's got 17 buttons.

We can also detect when we disconnect it. So... I just unpluged it and it's also detected...

But, in order to scan... in order to know in real time the state of the different buttons...

and you've got some analogic buttons like these triggers... and you've got axes... they are the joysticks here... and buttons... you need to scan at a very fast frequency the state of the gamepad.

This is done in another example here, where I can press some buttons and you see that the buttons are detected.

And in case of an analogic button, I'm using a progress HTML element to draw/show the pressure... So, how do you manage these values?

You've got to have a mainloop that is very similar to the animation loop (or you can do this in the animation loop). And we call a method, a function called scanGamepads that will ask for the gamepad 60 times per second. Here we say "Navigator!

Hey browser! Give me all the gamepads you've got!" And you've got a gamepad array... if the gamepad is detected, it's non null and you can use it. In this example we use just one gamepad.

The first gamepad that is defined will be used for setting the "gamepad" global variable. This is the variable we check in the loop: "please, give me an updated status of the gamepad!", by calling the scanGamepad()... then we're going to check the buttons that are pressed... so how do we check the buttons?

We get the number of buttons: gamepad.buttons, we do an iteration on them, we get the current button and check if its pressed or not.

This is a boolean property: "pressed".

And in the case there is a "value" that is defined, it means it's for an analogic buttons, like the triggers here... and the value will be between 0 and 1. And this is what we draw here.

If you want to try another demo and look at the code for managing multiple gamepads, I added a link to this demo that has been done by people from Mozilla...

If you plug a second gamepad (I’ve got only one here), it will display another row for checking the state of the second gamepad.

Another thing that is interesting is to detect the joystick values here... you can see the progress bars moving. The joystick returns values between -1 and +1, 0 is the neutral position here.

The way you detect that is that instead of doing an iteration on the buttons: you do an iteration on the axes... the checkAxes function proposed in the course will just iterate on the axes array you get from the gamepad object.

gamepad.axes[i] here will returns the status... the value of the current axis.

axes[0] means horizontal here, axes[1] means vertical for the left joystick, axes[2] will mean left/right for the second joystick and axes[3] for the up/down.

This is how we manage that. Look at the code, it's very simple.

And in the course you will see how we can make the small monster move using the gamepad.

It's, I think, in the next lesson... I gave an example at the end, for moving the monster with the gamepad. We just reused the functions I've shown.

And here we can make the monster move using the gamepad as you can see... with the left joystick.

We just added scanGamepads() in the mailoop...updateGamepadStatus()...

and updateGamepadStatus() will scan the gamepads, check the buttons, and check the axes 60 times per second... the rest of the code is the same as I presented earlier. So I hope you enjoyed this part of the course and that you will use a gamepad in the small game you are going to develop during this week. Bye! Bye!

Some games, mainly arcade/action games, are designed to be used with a gamepad:

The Gamepad API is currently supported by all major browsers (including Microsoft Edge), except Internet Explorer - see the up to date version of this feature's compatbility table. Note that the API is still a draft and may change in the future. We recommend using a Wired Xbox 360 Controller or a PS2 controller, both of which should work out of the box on Windows XP, Windows Vista, Windows, and Linux desktop distributions. Wireless controllers are supposed to work too, but we haven't tested the API with them. You may find someone who has managed but they've probably needed to install an operating system driver to make it work.

Detecting gamepads

Events triggered when the gamepad is plugged in or unplugged

Let's start with a 'discovery' script to check that the GamePad is connected, and to see the range of facilities it can offer to JavaScript.

If the user interacts with a controller (presses a button, moves a stick) a gamepadconnected event will be sent to the page. NB the page must be visible! The event object passed to the gamepadconnected listener has a gamepad property which describes the connected device.

Example on JSBin

Code extract!

1.  window.addEventListener("gamepadconnected", function(e) {
2.     var gamepad = e.gamepad;
3.     var index = gamepad.index;
4.     var id = gamepad.id;
5.     var nbButtons = gamepad.buttons.length;
6.     var nbAxes = gamepad.axes.length;
7.  
8.     console.log("Gamepad No " + index +
9.                 ", with id " + id + " is connected. It has " +
10.                nbButtons + " buttons and " +
11.                nbAxes + " axes");
12. });

If a gamepad is disconnected (you unplug it), a gamepaddisconnected event is fired. Any references to the gamepad object will have their connected property set to false.

1.  window.addEventListener("gamepaddisconnected", function(e) {
2.     var gamepad = e.gamepad;
3.     var index = gamepad.index;
4.  
5.     console.log("Gamepad No " + index + " has been disconnected");
6.  });

Scanning for gamepads

If you reload the page, and if the gamepad has already been detected by the browser, it will not fire the gamepadconnected event again. This can be problematic if you use a global variable for managing the gamepad, or an array of gamepads in your code. As the event is not fired, these variables will stay undefined...

So, you need to regularly scan for gamepads available on the system. You should still use that event listener if you want to do something special when the system detects that a gamepad has been unplugged.

Here is the code to use to scan for a gamepad:

Code extract!

1.  var gamepad;
2.   
3.  function mainloop() {
4.     ...
5.     scangamepads();
6.   
7.     // test gamepad status: buttons, joysticks etc.
8.     ...
9.     requestAnimationFrame(mainloop);
10. }
11.  
12. function scangamepads() {
13.   // function called 60 times/s
14.   // the gamepad is a "snapshot", so we need to set it
15.   // 60 times / second in order to have an updated status
16.   var gamepads = navigator.getGamepads();
17. 
18.   for (var i = 0; i < gamepads.length; i++) {
19.     // current gamepad is not necessarily the first
20.     if(gamepads[i] !== undefined)
21.        gamepad = gamepads[i];
22.     }
23. }

In this code, we check every 1/60 second for newly or re-connected gamepads, and we update the gamepad global var with the first gamepad object returned by the browser. We need to do this so that we have an accurate "snapshot" of the gamepad state, with fixed values for the buttons, axes, etc. If we want to check the current button and joystick statuses, we must poll the browser at a high frequency and call for an updated snapshot.

From the specification: "getGamepads retrieves a snapshot of the data for the currently connected and interacted with gamepads."

This code will be integrated (as well as the event listeners presented earlier) in the next JSBin examples.

To keep things simple, the above code works with a single gamepad - here's a good example of managing multiple gamepads.

Detecting button status and axes values (joysticks)

Properties of the gamepad object

The gamepad object returned in the event listener has different properties:

id: a string indicating the id of the gamepad. Useful with

the mapping property below.
index: an integer used to distinguish multiple controllers (gamepad

1, gamepad 2, etc.).
connected: true if the controller is still connected, false if it

has been disconnected.
mapping: not implemented yet by most browsers. It will allow the

controllers of the gamepad to be remapped. A layout map is associated with the id of the gamepad. By default, and before they implement support for different mapping, all connected gamepads use a standard default layout.

axes: an array of floating point values containing the state of each

axis on the device. Usually these represent the analog sticks, with a pair of axes giving the position of the stick in its X and Y axes. Each axis is normalized to the range of -1.0...1.0, with -1.0 representing the up or left-most position of the axis, and 1.0 representing the down or right-most position of the axis.
buttons: an array

of GamepadButton objects containing the state of each button on the device. Each GamepadButton has a pressed and a value property.
The pressed property is a Boolean property indicating whether the

button is currently pressed (true) or unpressed (false).
The value property is a floating point value used to enable

representing analog buttons, such as the triggers, on many modern gamepads. The values are normalized to the range 0.0...1.0, with 0.0 representing a button that is not pressed, and 1.0 representing a button that is fully depressed.

Detecting whether a button is pressed

Digital, on/off buttons evaluate to either one or zero (respectively). Whereas analog buttons will return a floating-point value between zero and one.

Example on JSBin. You might also give a look at at this demo that does the same thing but with multiple gamepads.

Code for checking if a button is pressed:

Code extract!

1.  function checkButtons(gamepad) {
2.     for (var i = 0; i < gamepad.buttons.length; i++) {
3.        // do nothing is the gamepad is not ok
4.        if(gamepad === undefined) return;
5.        if(!gamepad.connected) return;
6.  
7.        var b = gamepad.buttons[i];
8.  
9.        if(b.pressed) {
10.          console.log("Button " + i + " is pressed.");
11.          if(b.value !== undefined)
12.              // analog trigger L2 or R2, value is a float in [0,
    1]
13.              console.log("Its value:" + b.val);
14.       }
15.    }
16. }

In line 11, notice how we detect whether the current button is an analog trigger (L2 or R2 on Xbox360 or PS2/PS3 gamepads).

Next, we'll integrate it into the mainloop code. Note that we also need to call the scangamepads function from the loop, to generate fresh "snapshots" of the gamepad with updated properties. Without this call, the gamepad.buttons will return the same states every time.

Code extract!

1.  function mainloop() {
2.     // clear, draw objects, etc...
3.     ...
4.     scangamepads();
5.     // Check gamepad button states
6.     checkButtons(gamepad);
7.  
8.     // animate at 60 frames/s
9.     requestAnimationFrame(mainloop);
10. }

Detecting axes (joystick) values

Example on JSBin

Code for detecting the axes' values:

Code extract!

1.  // detect axis (joystick states)
2.  function checkAxes(gamepad) {
3.     if(gamepad === undefined) return;
4.     if(!gamepad.connected) return;
5.  
6.     for (var i=0; i<gamepad.axes.length; i++) {
7.       var axisValue = gamepad.axes[i];
8.       // do something with the value
9.       ...
10.    }
11. }

Detecting the direction (left, right, up, down, diagonals) and angle of the left joystick

We could add an inputStates object similar to the one we used in the game framework, and check its values in the mainloop to decide whether to move the player up/down/left/right, including diagonals - or maybe we'd prefer to use the current angle of the joystick. Here is how we manage this:

JSBin example:

Source code extract:

Code extract!

1.  var inputStates = {};
2.  ...
3.  function mainloop() {
4.     // clear, draw objects, etc...
5.     // update gamepad status
6.     scangamepads();
7.    // Check gamepad button states
8.    checkButtons(gamepad);
9.    // Check joysticks states
10.   checkAxes(gamepad);
11. 
12.   // Move the player, taking into account
13.   // the gamepad left joystick state
14.   updatePlayerPosition();
15. 
16.   // We could use the same technique in
17.   // order to react when buttons are pressed
18.   //...
19. 
20.   // animate at 60 frames/s
21.   requestAnimationFrame(mainloop);
22. }
23.  
24. function updatePlayerPosition() {
25.    directionDiv.innerHTML += "";
26.    if(inputStates.left) {
27.       directionDiv.innerHTML = "Moving left";
28.    }
29.    if(inputStates.right) {
30.       directionDiv.innerHTML = "Moving right";
31.    }
32.    if(inputStates.up) {
33.       directionDiv.innerHTML = "Moving up";
34.    }
35.    if(inputStates.down) {
36.       directionDiv.innerHTML = "Moving down";
37.    }
38.    // Display the angle in degrees, in the HTML page
39.  angleDiv.innerHTML = Math.round((inputStates.angle*180/Math.PI));
40. }
41.  
42. // gamepad code below
43. // -------------------------
44. // detect axis (joystick states)
45. function checkAxes(gamepad) {
46.   if(gamepad === undefined) return;
47.   if(!gamepad.connected) return;
48. 
49.   ...
50. 
51.   // Set inputStates.left, right, up, down
52.  inputStates.left = inputStates.right = inputStates.up = inputStates.down = false;
53. 
54.   // all values between [-1 and 1]
55.   // Horizontal detection
56.   if(gamepad.axes[0] > 0.5) {
57.      inputStates.right=true;
58.      inputStates.left=false;
59.   } else if(gamepad.axes[0] < -0.5) {
60.      inputStates.left=true;
61.      inputStates.right=false;
62.   }
63. 
64.   // vertical detection
65.   if(gamepad.axes[1] > 0.5) {
66.     inputStates.down=true;
67.     inputStates.up=false;
68.   } else if(gamepad.axes[1] < -0.5) {
69.     inputStates.up=true;
70.     inputStates.down=false;
71.   }
72.  
73.   // compute the angle. gamepad.axes[1] is the
74.   // sinus of the angle (values between [-1, 1]),
75.   // gamepad.axes[0] is the cosinus of the angle.
76.   // we display the value in degree as in a regular
77.   // trigonometric circle, with the x axis to the right
78.   // and the y axis that goes up.
79.   // The angle = arcTan(sin/cos); We inverse the sign of
80.   // the sinus in order to have the angle in standard
81.   // x and y axis (y going up)
82.   inputStates.angle = Math.atan2(-gamepad.axes[1], gamepad.axes[0]);
83. }

Other gamepads and joysticks tested

Logitech Attack 3 Joystick on Linux

Hi (=dn reports): Have successfully installed a Logitech Attack 3 joy-stick on my Thinkpad running Linux Mint 17.1. It runs all of the code presented here correctly, reporting 11 buttons and 3 axes (the knurled rotating knob (closest to my pen) is described as a 'throttle' or 'accelerator')).

Traditionally Linux has been described as 'for work only' or 'no games', so it was a pleasant surprise to see how easy things were - no "driver" to install (it seems important to uninstall any existing connection between a device and the x-server), installed "joystick" testing and calibration tool, and the "jstest-gtk" configuration and testing tool; and that was 'it' - no actual configuration was necessary!

External resources

THE BEST resource: this paper from smashingmagazine.com tells you everything about the GamePad API. Very complete, explains how to set a dead zone, a keyboard fallback, etc.
Good article about using the gamepad API on the Mozilla Developer Network site
An interesting article on the gamepad support, published on the HTML5 Rocks Web site
gamepad.js is a Javascript library to enable the use of gamepads and joysticks in the browser. It smoothes over the differences between browsers, platforms, APIs, and a wide variety of gamepad/joystick devices.
Another library we used in our team for controlling a mobile robot (good support from the authors)
Gamepad Controls for HTML5 Games

2.3.7 Move the monster!

Hi, This time, I will show you in that video what we have done so far.

We started from one of the examples from the HTML5 Part 1 course, the one that just use the canvas to draw small monster.

We have a canvas here, and in a function called when the page is loaded in the window.onload callback, we get the canvas, we get the context of the canvas and we called the draw monster function.

Here we have got just a function that is called once after the page is loaded and this function just draws the monster using translate, stroke, fillRect and so on.

Then, in order to turn this into a small game framework and in order to draw the monster 60 times per seconds, we introduced what is called the black box model for creating JavaScript objects.

Instead of having functions, we have got objects.

We create an object that is the game framework:

game = new GF() with capital letters, this is called a constructor function.

In JavaScript, when you start a function with capital letters, it means that it is meant to be use with a new.

Then we call just some parts of the game framework that are exposed to an external user.

Here we can call game.start but we can not call anything. And in this way to design object, the internals of the game framework that are exposed are located in an object we return at the end of the object.

In the end of the game framework constructor function, we do return start, this is the name of the property we will able to use from the outside, and start here is the name of an internal function.

So, game.start will call this start function inside the game framework. In this function we do the initialization so we will create a div for displaying the number of frame per seconds, we get the canvas, the context, and we call requestAnimationFrame(mainLoop) in order to start the animation.

The mainLoop is a private function inside the game framework because it is not exposed, we do not return its name here so it is a sort of private function.

And what do we do in the mainLoop? We clear the canvas, we draw the monster and we call again requestAnimationFrame(mainLoop).

In addition, we also measure the frames per seconds taking into account the current time.

In order to measure the number of FPS, we pass the time and we compute deltas in this function here, that is also a private function that has been presented previously in the course.

This is a very small skeleton and then we can build on that by adding new methods, by adding new properties.

The properties are the local variables that are usable only inside the game framework. In the current lesson, we are adding user interaction like detecting the mouse buttons that are pressed, the mouse position or the different keys that can be pressed.

You can notice that the diagonal movements are very smooth because we can manage different key presses at the same time and also I can press a key and a button and the monster will move faster.

Multiple events are managed using a global status variable, that we called inputStates, and that is checked 60 times per seconds from inside the mainLoop.

The event listeners, the key event listeners and the mouse event listeners will just add properties to this variable.

Let me show you how it is done.

It is done in the start method that is called when you want to initialize the game framework, we declare the event listener and in case we have got a keydown and if this key is the left arrow for example, we set the left property to the inputStates to left.

And in the animation loop, we call a method called updateMonsterPosition that will check for this global inputStates and if the left key is pressed, we will display a message "left key pressed" and we will modify the speed of the monster.

And this variable ‘speedX’ is taken into account to increment the x coordinate.

And this is how we can get a smooth animation because this updateMonsterPosition is called 60 times per seconds. If I keep a key pressed, it is not important if the key is repeated, if I have got several keys pressed, because the status of the left key in the inputStates will be unchanged and my monster will keep moving on the left.

Take time to look at the code, read slowly the explanation in the page and I will meet you in the next video in which we will add enemies and obstacles and detect collisions.

Make the monster move using the arrow keys, and to increase its speed by pressing a mouse button

To conclude this topic about events, we will use the arrow keys to move our favorite monster up/down/left/right, and make it accelerate when we press a mouse button while it is moving. Notice that pressing two keys at the same time will make it move diagonally.

Check this online example at JSBin: we've changed very few lines of code from the previous evolution!

Add a JavaScript object to describe the monster

1.  // The monster!
2.  var monster = {
3.    x:10,
4.    y:10,
5.    speed:1
6.  };

Where monster.x and monster.y define the monster's current position and monster.speed corresponds to the number of pixels the monster will move between animation frames.

Note: this is not the best way to animate objects in a game; we will look at a far better solution - "time based animation" - in another lesson.

We modified the game loop as follows:

Code extract!

1.  var mainLoop = function(time){
2.    // Main function, called each frame
3.    measureFPS(time);
4.  
5.    // Clears the canvas
6.    clearCanvas();
7.  
8.    // Draws the monster
9.    drawMyMonster(monster.x, monster.y);
10. 
11.   // Checks inputs and moves the monster
12.   updateMonsterPosition();
13. 
14.   // Calls the animation loop every 1/60th of second
15.   requestAnimationFrame(mainLoop);
16. };

We moved all the parts that check the input states in the updateMonsterPosition() function:

Code extract!

1.  function updateMonsterPosition() {
2.    monster.speedX = monster.speedY = 0;
3.  
4.    // Checks inputStates
5.    if (inputStates.left) {
6.      ctx.fillText("left", 150, 20);
7.      monster.speedX = -monster.speed;
8.    }
9.    if (inputStates.up) {
10.     ctx.fillText("up", 150, 40);
11.     monster.speedY = -monster.speed;
12.   }
13.   if (inputStates.right) {
14.     ctx.fillText("right", 150, 60);
15.     monster.speedX = monster.speed;
16.   }
17.   if (inputStates.down) {
18.     ctx.fillText("down", 150, 80);
19.     monster.speedY = monster.speed;
20.   }
21.   if (inputStates.space) {
22.     ctx.fillText("space bar", 140, 100);
23.   }
24.   if (inputStates.mousePos) {
25.     ctx.fillText("x = " + inputStates.mousePos.x + " y = " +
26.                           inputStates.mousePos.y, 5, 150);
27.   }
28.   if (inputStates.mousedown) {
29.     ctx.fillText("mousedown b" + inputStates.mouseButton, 5, 180);
30.     monster.speed = 5;
31.   } else {
32.   // Mouse up
33.   monster.speed = 1;
34.   }
35. 
36.   monster.x += monster.speedX;
37.   monster.y += monster.speedY;
38. }

Explanations:

In this function, we added two properties to the monster object: speedX and speedY which will correspond to the number of pixels we will add to the x and y position of the monster at each new frame of animation.
We first set these to zero (line 2), then depending on the keyboard input states, we set them to a value equal to monster.speed or -monster.speed modified by the keys that are being pressed at the time (lines 4-20).
Finally, we add speedX and speedY pixels to the x and/or y position of the monster (lines 36 and 37).
When the function is called by the game loop, if speedX and/or speedY are non-zero they will change the x and y position of the monster in successive frames, making it move smoothly.
If a mouse button is pressed or released we set the monster.speed value to +5 or to +1. This will make the monster move faster when a mouse button is down, and return to its normal speed when no button is down.

Notice that two arrow keys and a mouse button can be pressed down at the same time. In this situation, the monster will take a diagonal direction and accelerate. This is why it is important to keep all the input states up-to-date, and not to handle single events individually.

Gamepad enhancements

Let's add the gamepad utility functions from the previous lesson (we tidied them a bit too, removing the code for displaying the progress bars, buttons, etc.), added a gamepad property to the game framework, and added one new call in the game loop for updating the gamepad status:

Check the result on JSBin:

The new updated mainloop:

Code extract!

1.  var mainLoop = function(time){
2.     //main function, called each frame
3.     measureFPS(time);
4.  
5.     // Clear the canvas
6.     clearCanvas();
7.  
8.     // gamepad
9.     updateGamePadStatus();
10. 
11.    // draw the monster
12.    drawMyMonster(monster.x, monster.y);
13. 
14.    // Check inputs and move the monster
15.    updateMonsterPosition();
16. 
17.    // Call the animation loop every 1/60th of second
18.    requestAnimationFrame(mainLoop);
19. };

And here is the updateGamePadStatus function (the inner function calls are to gamepad utility functions detailed in the previous lesson):

Code extract!

1.  function updateGamePadStatus() {
2.    // get new snapshot of the gamepad properties
3.    scangamepads();
4.    // Check gamepad button states
5.    checkButtons(gamepad);
6.    // Check joysticks
7.    checkAxes(gamepad);
8.  }

The checkAxes function updates the left, right, up, down properties of the inputStates object we previously used with key events. Therefore, without changing any code in the updatePlayerPosition function, the monster moves by joystick command!

2.4.1 Introduction

Hi! This time I will talk to you about what is called time-based animation.

Here, we have got a simple example of a bouncing rectangle and the animation is done in an animationLoop that is called 60 times per seconds using the requestAnimationFrame. We clear the rectangle that corresponds to the canvas area, and we add a fixed increment to the x position.

If x is just bounces on the side, we just reverse the speed.

The speed is a fixed value of 3px per frame. What is happening if we run this application on a low end-smartphone or a Raspberry Pi, a low end-computer with a not very powerful GPU?

We can simulate what would happen by just adding a big loop here to slow down artificially the animation.

This is what I have done here. Here I count up to 70 millions in the loop.

This takes time and slows down the animation.

The rectangle is just moving 3px every frame, but as the the frame rate drops down a lot, the actual speed on the screen of the rectangle is much slower than what I have got here.

And here I added also a loop so this is a normal speed.

What we can do in order not to have the speed going down?

We will compute the time between two consecutive frames.

So here I am using the date object from JavaScript to get the current time and I compute a delta that is a difference between the current time and the time at the previous animation.

So the delta is the number of milliseconds elapsed since the last animation.

And then we will compute the distance instead of using 3px per frame, the number of pixels we move the rectangle will increase if the time between frames increases.

So we have got a function that is explained in the course that will compute the distance taking into account the time elapsed since the last frame and the speed we want to achieve in pixels per seconds, not in pixels per frames, but in pixels per seconds.

This time, we are adding the increment but it is adjusted at each frame of animation.

Here is an example that I slow down and if I change the time that this will take you can see that it goes from smooth to a bit jerky… to really jerky, but the speed on the screen is the same.

It takes the same time to bounce from on side to another.

This is what is done in real games.

I wanted to show you also another thing, instead of using the date object from JavaScript that gives the time in milliseconds, but as we are animating at 60 times per seconds, having an high resolution timer that has a sub-millisecond accuracy is much better and was asked by game developers.

The callback function that is called when you use requestAnimationFrame can have an extra parameter that we have not used until now.

So you can add a parameter here that will be a high resolution time.

And then you compute the delta using this time that is automatically passed to you by the requestAnimationFrame API.

Here, I added this technique to the game framework.

So, I am in the mainLoop from the last example we used with the game framework for moving the small monster using the keys.

I artificially slowed down the animation. So if I do this, I have got 8 frames per seconds, and you can see that the monster moves, but it jumps with larger steps between two consecutive frames.

If I remove this artificial part of the code that slows down the animation, I go up to 60 FPS and I have got a smooth animation.

That means that this example will be usable on many different devices as the animation will be adapted depending on the power of the device.

So this is time-based animation.

Let's study an important technique known as "time-based animation", that is used by nearly all "real" video games.

This technique is useful when:

Your application runs on different devices, and where 60 frames/s are definitely not possible. More generally, you want your animated objects to move at the same speed on screen, regardless of the device that runs the game.

For example, imagine a game or an animation running on a smartphone and on a desktop computer with a powerful GPU. On the phone, you might achieve a maximum of 20 fps with no guarantee that this number will be constant; whereas on the desktop, you will reliably achieve 60 fps. If the application is a car racing game, for example, your car will take 30s to make a complete loop on the race track when running on a desktop, whilst on a smartphone it might take 5 minutes.

The way to address this is to run at a lower frame-rate on the phone. This will enable the car to race around the track in the same amount of (real) time as it does on a powerful desktop computer.

Solution: you need to compute the amount of time that has elapsed between the last frame that was drawn and the current one; and depending on this delta of time, adjust the distance the car must move across the screen. We will see several examples of this later.
You want to perform some animations only a few times per second. For example, in sprite-based animation (drawing different images as a character moves, for example), you will not change the images of the animation 60 times/s, but only ten times per second. Mario will walk on the screen in a 60 fps animation, but his posture will not change every 1/60th of second.
You may also want to accurately set the framerate, leaving some CPU time for other tasks. Many games consoles limit the frame-rate to 1/30th of a second, in order to allow time for other sorts of computations (physics engine, artificial intelligence, etc.)

How to measure time when we use requestAnimationFrame?

Let's take a simple example with a small rectangle that moves from left to right. At each animation loop, we erase the canvas content, calculate the rectangle's new position, draw the rectangle, and call the animation loop again. So you animate a shape as follows (note: steps 2 and 3 can be swapped):

erase the canvas,
draw the shapes,
move the shapes,
go to step 1.

When we use requestAnimationFrame for implementing such an animation, as we did in the previous lessons, the browser tries to keep the frame-rate at 60 fps, meaning that the ideal time between frames will be 1/60 second = 16.66 ms.

Example #1: no use of time-based animation

Online example at JSBin

Code extract!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.    <meta charset=utf-8 />
5.    <title>Small animation example</title>
6.    <script>
7.      var canvas, ctx;
8.      var width, height;
9.      var x, y;
10.     var speedX;
11. 
12.     // Called after the DOM is ready (page loaded)
13.     function init() {
14.        // init the different variables
15.        canvas = document.querySelector("#mycanvas");
16.        ctx = canvas.getContext('2d');
17.        width = canvas.width;
18.        height = canvas.height;
19. 
20.        x=10; y = 10;
21. 
22.        // Move 3 pixels left or right at each frame
23.        speedX = 3;
24. 
25.        // Start animation
26.        animationLoop();
27.     }
28. 
29.     function animationLoop() {
30.        // an animation involves: 1) clear canvas and 2) draw shapes,
31.        // 3) move shapes, 4) recall the loop with requestAnimationFrame
32. 
33.        // clear canvas
34.        ctx.clearRect(0, 0, width, height);
35. 
36.        ctx.strokeRect(x, y, 10, 10);
37. 
38.        // move rectangle
39.        x += speedX;
40. 
41.        // check collision on left or right
42.        if(((x+5) > width) || (x <= 0)) {
43.           // cancel move + inverse speed
44.           x -= speedX;
45.           speedX = -speedX;
46.        }
47. 
48.        // animate.
49.        requestAnimationFrame(animationLoop);
50.     }
51. </script>
52. </head>
53. 
54. <body onload="init();">
55. <canvas id="mycanvas" width="200" height="50" style="border: 2px solid black">
56.  </canvas>
57. </body>
58. </html>

If you try this example on a low-end smartphone (use this URL for the example in stand-alone mode) and if you run it at the same time on a desktop PC, it is obvious that the rectangle moves faster on the desktop computer screen than on your phone.

This is because the frame rate differs between the computer and the smartphone: perhaps 60 fps on the computer and 25 fps on the phone. As we only move the rectangle in the animationLoop, in one second the rectangle will be moved 25 times on the smartphone compared with 60 times on the computer! Since we move the rectangle the same number of pixels each time, the rectangle moves faster on the computer!

Example #2: simulating a low-end device

Here is the same example to which we have added a loop that wastes time right in the middle of the animation loop. It will artificially extend the time spent inside the animation loop, making the 1/60th of second ideal impossible to reach.

Try it on JsBin and notice that the square moves much slower on the screen. Indeed, its speed is a direct consequence of the extra time spent in the animation loop.

1.  function animationLoop() {
2.      ...
3.      for(var i = 0; i < 50000000; i++) {
4.         // slow down artificially the animation
5.      }
6.      ...
7.      requestAnimationFrame(animationLoop);
8.  }

2.4.2 Measuring time between frames

Let's find out how to measuring time between frames to achieve a constant speed on screen, even when the frame rate changes.

Method #1: using the JavaScript Date object

Let's modify the example from the previous lesson slightly by adding a time-based animation. Here we use the "standard JavaScript" way for measuring time, using JavaScript's Date object:

var time = new Date().getTime();

The getTime() method returns the number of milliseconds since midnight on January 1, 1970. This is the number of milliseconds that have elapsed during the Unix epoch (!).

There is an alternative. We could have called:

var time = Date.now();

So, if we measure the time at the beginning of each animation loop, and store it, we can then compute the delta of times elapsed between two consecutive loops.

We then apply some simple math to compute the number of pixels we need to move the shape to achieve a given speed (in pixels/s).

Example #1: using time based animation: the bouncing square

Online example at JSBin:

Source code from the example:

Code extract!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.    <meta charset=utf-8 />
5.    <title>Move rectangle using time based animation</title>
6.    <script>
7.      var canvas, ctx;
8.      var width, height;
9.      var x, y, incX; // incX is the distance from the previously > drawn
10.                     // rectangle to the new one
11.     var speedX; // speedX is the target speed of the rectangle, in > pixels/s
12. 
13.     // for time based animation
14.     var now, delta;
15.     var then = new Date().getTime();
16. 
17.     // Called after the DOM is ready (page loaded)
18.     function init() {
19.       // Init the different variables
20.       canvas = document.querySelector("#mycanvas");
21.       ctx = canvas.getContext('2d');
22.       width = canvas.width;
23.       height = canvas.height;
24. 
25.       x=10; y = 10;
26.       // Target speed in pixels/second, try with high values, 1000, > 2000...
27.       speedX = 200;
28. 
29.       // Start animation
30.      animationLoop();
31.    }
32. 
33.    function animationLoop() {
34.      // Measure time
35.      now = new Date().getTime();
36.  
37.      // How long between the current frame and the previous one?
38.      delta = now - then;
39.      //console.log(delta);
40.      // Compute the displacement in x (in pixels) in function of the > time elapsed and
41.      // in function of the wanted speed
42.      incX = calcDistanceToMove(delta, speedX);
43. 
44.      // an animation involves: 1) clear canvas and 2) draw shapes,
45.      // 3) move shapes, 4) recall the loop with  > requestAnimationFrame
46. 
47.      // clear canvas
48.      ctx.clearRect(0, 0, width, height);
49. 
50.      ctx.strokeRect(x, y, 10, 10);
51. 
52.      // move rectangle
53.      x += incX;
54. 
55.      // check collision on left or right
56.      if((x+10 >= width) || (x <= 0)) {
57.         // cancel move + inverse speed
58.         x -= incX;
59.         speedX = -speedX;
60.      }
61. 
62.      // Store time
63.      then = now;
64. 
65.      requestAnimationFrame(animationLoop);
66.  }
67. 
68. 
69. 
70.  // We want the rectangle to move at a speed given in pixels/second
71.  // (there are 60 frames in a second)
72.  // If we are really running at 60 frames/s, the delay between
73.  // frames should be 1/60
74.  // = 16.66 ms, so the number of pixels to move = (speed * > del)/1000.
75.  // If the delay is twice as
76.  // long, the formula works: let's move the rectangle for twice as > long!
77.  var calcDistanceToMove = function(delta, speed) {
78.      return (speed * delta) / 1000;
79.  }
80. 
81.  </script>
82. </head>
83. 
84. <body onload="init();">
85.  <canvas id="mycanvas" width="200" height="50" style="border: 2px solid > black"></canvas>
86. </body>
87. </html>

In this example, we only added a few lines of code for measuring the time and computing the time elapsed between two consecutive frames (see line 38). Normally, requestAnimationFrame(callback) tries to call the callback function every 16.66 ms (this corresponds to 60 frames/s)... but this is never exactly the case. If you do a console.log(delta)in the animation loop, you will see that even on a very powerful computer, the delta is "very close" to 16.6666 ms, but 99% of the time it will be slightly different.

The function calcDistanceToMove(delta, speed) takes two parameters: 1) the time elapsed in ms, and 2) the target speed in pixels/s.

Try this example on a smartphone, use this link to run the JSBin example in stand-alone mode. Normally you should see no difference in speed, but it may look a bit jerky on a low-end smartphone or on a slow computer. This is the correct behavior.

Or you can try the next example that simulates a complex animation loop that takes a long time to draw each frame...

Example #2: using a simulation that spends a lot of time in the animation loop, to compare with the previous example

[Try it on JsBin](https://jsbin.com/jeribi/edit):

We added a long loop in the middle of the animation loop. This time, the animation should be very jerky. However, notice that the apparent speed of the square is the same as in the previous example: the animation adapts itself!

Code extract!

1.  function animationLoop() {
2.     // Measure time
3.    now = new Date().getTime();
4.   
5.    // How long between the current frame and the previous one ?
6.    delta = now - then;
7.    //console.log(delta);
8.    // Compute the displacement in x (in pixels) in function of the time elapsed and
9.    // in function of the wanted speed
10.   incX = calcDistanceToMove(delta, speedX);
11. 
12.   // an animation is : 1) clear canvas and 2) draw shapes,
13.   // 3) move shapes, 4) recall the loop with requestAnimationFrame
14. 
15.   // clear canvas
16.   ctx.clearRect(0, 0, width, height);
17. 
18.   for(var i = 0; i < 50000000; i++) {
19.     // just to slow down the animation
20.   }
21. 
22.   ctx.strokeRect(x, y, 10, 10);
23. 
24.   // move rectangle
25.   x += incX;
26. 
27.   // check collision on left or right
28.   if((x+10 >= width) || (x <= 0)) {
29.    // cancel move + inverse speed
30.    x -= incX;
31.    speedX = -speedX;
32.   }
33. 
34.   // Store time
35.   then = now;
36. 
37.   requestAnimationFrame(animationLoop);
38. }

Method #2: using the new HTML5 high-resolution timer

Since the beginning of HTML5, game developers, musicians, and others have asked for a sub-millisecond timer to be able to avoid some glitches that occur with the regular JavaScript timer. This API is called the "High Resolution Time API".

This API is very simple to use - just do:

var time = performance.now();

... to get a sub-millisecond time-stamp. It is similar to Date.now() except that the accuracy is much higher and that the result is not exactly the same. The value returned is a floating point number, not an integer value!

From this article that explains the High Resolution Time API: "The only method exposed is now(), which returns a DOMHighResTimeStamp representing the current time in milliseconds. The timestamp is very accurate, with precision to a thousandth of a millisecond. Please note that while Date.now() returns the number of milliseconds elapsed since 1 January 1970 00:00:00 UTC, performance.now() returns the number of milliseconds, with microseconds in the fractional part, from performance.timing.navigationStart(), the start of navigation of the document, to the performance.now() call. Another important difference between Date.now() and performance.now() is that the latter is monotonically increasing, so the difference between two calls will never be negative."

To sum up:

performance.now() returns the time since the load of the document (it is called a DOMHighResTimeStamp), with a sub mill-second accuracy, as a floating point value, with very high accuracy.
Date.now() returns the number of mill-seconds since the Unix epoch, as an integer value.

Support for this API is quite good - see the compatibility table online.

Here is a version on JSBin of the previous example with the bouncing rectangle, that uses the high resolution timer.

Source code of the example:

Code extract!

1.  ...
2.  <script>
3.    ...
4.    var speedX; // speedX is the target speed of the rectangle in pixels/s
5. 
6.    // for time based animation
7.    var now, delta;
8.    // High resolution timer
9.    var then = performance.now();
10. 
11.    // Called after the DOM is ready (page loaded)
12.    function init() {
13.      ...
14.    }
15. 
16.    function animationLoop() {
17.      // Measure time, with high resolution timer
18.      now = performance.now();
19.  
20.      // How long between the current frame and the previous one?
21.     delta = now - then;
22.     //console.log(delta);
23.     // Compute the displacement in x (in pixels) in function
24.     // of the time elapsed and
25.     // in function of the wanted speed
26.     incX = calcDistanceToMove(delta, speedX);
27.     //console.log("dist = " + incX);
28.     // an animation involves: 1) clear canvas and 2) draw shapes,
29.     // 3) move shapes, 4) recall the loop with requestAnimationFrame
30. 
31.     // clear canvas
32.    ctx.clearRect(0, 0, width, height);
33. 
34.    ctx.strokeRect(x, y, 10, 10);
35. 
36.    // move rectangle
37.    x += incX;
38. 
39.    // check collision on left or right
40.    if((x+10 >= width) || (x <= 0)) {
41.       // cancel move + inverse speed
42.       x -= incX;
43.       speedX = -speedX;
44.    }
45. 
46.    // Store time
47.    then = now;
48. 
49.    // call the animation loop again
50.    requestAnimationFrame(animationLoop);
51.  }
52.  ...
53. 
54.  </script>

Only two lines have changed but the accuracy is much higher, if you uncomment the console.log(...) calls in the main loop. You will see the difference.

Method #3: using the optional timestamp parameter of the callback function of requestAnimationFrame

This is the recommended method!

There is an optional parameter that is passed to the callback function called by requestAnimationFrame: a timestamp!

The requestAnimationFrame API specification says that this timestamp corresponds to the time elapsed since the page has been loaded.

It is similar to the value sent by the high resolution timer using performance.now().

Here is a running example of the animated rectangle, that uses this timestamp parameter.

Online example at JSBin:

Source code of the example:

Source code!

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset=utf-8 />
<title>Time based animation using the parameter of the requestAnimationFrame callback</title>
  <script>
   var canvas, ctx;
   var width, height;
   var x, y, incX; // incX is the distance from the previously drawn rectangle
                   // to the new one
   var speedX;     // speedX is the target speed of the rectangle in pixels/s

   // for time based animation
   var now, delta=0;
   // High resolution timer
   var oldTime = 0;

   // Called after the DOM is ready (page loaded)
   function init() {
     // init the different variables
     canvas = document.querySelector("#mycanvas");
     ctx = canvas.getContext('2d');
     width = canvas.width;
     height = canvas.height;

     x=10; y = 10;
     // Target speed in pixels/second, try with high values, 1000, 2000...
     speedX = 200;

     // Start animation
     requestAnimationFrame(animationLoop);
   }

   function animationLoop(currentTime) {
     // How long between the current frame and the previous one?
     delta = currentTime - oldTime;

     // Compute the displacement in x (in pixels) in function of the time elapsed and
     // in function of the wanted speed
     incX = calcDistanceToMove(delta, speedX);

     // clear canvas
     ctx.clearRect(0, 0, width, height);

     ctx.strokeRect(x, y, 10, 10);

     // move rectangle
     x += incX;

     // check collision on left or right
     if(((x+10) > width) || (x < 0)) {
       // inverse speed
       x -= incX;
       speedX = -speedX;
     }

     // Store time
     oldTime = currentTime;

     // asks for next frame
     requestAnimationFrame(animationLoop);
   }

   var calcDistanceToMove = function(delta, speed) {
     return (speed * delta) / 1000;
   }

 </script>
</head>

<body onload="init();">
 <canvas id="mycanvas" width="200" height="50" style="border: 2px solid black"></canvas>
</body>
</html>

2.4.3 Setting the frame rate

Principle: even if the mainloop is called 60 times per second, ignore some frames in order to reach the desired frame rate.

It is also possible to set the frame rate using time based animation. We can set a global variable that corresponds to the desired frame rate and compare the elapsed time between two executions of the animation loop:

If the time elapsed is too short for the target frame rate: do nothing,
If the time elapsed exceeds the delay corresponding to the chosen frame rate: draw the frame and reset this time to zero.

Here is the online example at JSBin.

Try to change the parameter value of the call to:

<sup>setFrameRateInFramesPerSecond(5); // try other values!</sup>

Source code of the example:

Code extract!

<!DOCTYPE html>
<html lang="en">
  <head>
  <meta charset=utf-8 />
  <title>Set framerate using a high resolution timer</title>
  </head>
  <body>
  <p>This example measures and sums deltas of time between
     consecutive frames of animation. It includes
     a <code>setFrameRateInFramesPerSecond</code> function you can
     use to reduce the number of frames per second of the main
     animation.</p>
 
  <canvas id="myCanvas" width="700" height="350">
  </canvas>
  <script>
    var canvas = document.querySelector("#myCanvas");
    var ctx = canvas.getContext("2d");
    var width = canvas.width, height = canvas.height;
    var lastX = width * Math.random();
    var lastY = height * Math.random();
    var hue = 0;
 
    // Michel Buffa: set the target frame rate. TRY TO CHANGE THIS VALUE AND SEE
    // THE RESULT. Try 2 frames/s, 10 frames/s, 60 frames/s Normally there
    // should be a limit of 60 frames/s in the browser's implementations.
    setFrameRateInFramesPerSecond(60);
 
  // for time based animation. DelayInMS corresponds to the target framerate
  var now, delta, delayInMS, totalTimeSinceLastRedraw = 0;
 
  // High resolution timer
  var then = performance.now();
 
  // start the animation
  requestAnimationFrame(mainloop);
 
  function setFrameRateInFramesPerSecond(frameRate) {
    delayInMs = 1000 / frameRate;
  }
 
  // each function that is going to be run as an animation should end by
  // asking again for a new frame of animation
  function mainloop(time) {
    // Here we will only redraw something if the time we want between frames has
    // elapsed
    // Measure time with high resolution timer
    now = time;
 
    // How long between the current frame and the previous one?
    delta = now - then;
    // TRY TO UNCOMMENT THIS LINE AND LOOK AT THE CONSOLE
    // console.log("delay = " + delayInMs + " delta = " + delta + " total time = " +
    // totalTimeSinceLastRedraw);
 
    // If the total time since the last redraw is > delay corresponding to the wanted
    // framerate, then redraw, else add the delta time between the last call to line()
    // by requestAnimFrame to the total time..
    if (totalTimeSinceLastRedraw > delayInMs) {
      // if the time between the last frame and now is > delay then we
      // clear the canvas and redraw
 
      ctx.save();
 
      // Trick to make a blur effect: instead of clearing the canvas
      // we draw a rectangle with a transparent color. Changing the 0.1
      // for a smaller value will increase the blur...
      ctx.fillStyle = "rgba(0,0,0,0.1)";
      ctx.fillRect(0, 0, width, height);
 
      ctx.translate(width / 2, height / 2);
      ctx.scale(0.9, 0.9);
      ctx.translate(-width / 2, -height / 2);
 
      ctx.beginPath();
      ctx.lineWidth = 5 + Math.random() * 10;
      ctx.moveTo(lastX, lastY);
      lastX = width * Math.random();
      lastY = height * Math.random();
 
      ctx.bezierCurveTo(width * Math.random(),
                        height * Math.random(),
                        width * Math.random(),
                        height * Math.random(),
                        lastX, lastY);
 
      hue = hue + 10 * Math.random();
      ctx.strokeStyle = "hsl(" + hue + ", 50%, 50%)";
      ctx.shadowColor = "white";
      ctx.shadowBlur = 10;
      ctx.stroke();
 
      ctx.restore();
 
      // reset the total time since last redraw
      totalTimeSinceLastRedraw = 0;
  } else {
     // sum the total time since last redraw
     totalTimeSinceLastRedraw += delta;
  }
 
  // Store time
  then = now;
 
  // request new frame
  requestAnimationFrame(mainloop);
  }
  </script>
</body>
</html>

Same technique with the bouncing rectangle

See how we can set both the speed (in pixels/s) and the frame-rate using a high-resolution time with this modified version on JSBin of the example with the rectangle that also uses this technique.

Source code:

Source code!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.  <meta charset=utf-8 />
5.  <title>Bouncing rectangle with high resolution timer and adjustable frame rate</title>
6.   <script>
7.     var canvas, ctx;
8.     var width, height;
9.     var x, y, incX; // incX is the distance from the previously drawn rectangle
10.                    // to the new one
11.    var speedX; // speedX is the target speed of the rectangle in pixels/s
12. 
13.    // for time based animation, DelayInMS corresponds to the target frame rate
14.    var now, delta, delayInMS, totalTimeSinceLastRedraw=0;
15.    // High resolution timer
16.    var then = performance.now();
17. 
18.    // Michel Buffa: set the target frame rate. TRY TO CHANGE THIS VALUE AND SEE
19.    // THE RESULT. Try 2 frames/s, 10 frames/s, 60, 100 frames/s Normally there
20.    // should be a limit of 60 frames/s in the browser's implementations, but you can 
21.    // try higher values
22.    setFrameRateInFramesPerSecond(25);
23. 
24.    function setFrameRateInFramesPerSecond(framerate) {
25.      delayInMs = 1000 / framerate;
26.    }
27. 
28.    // Called after the DOM is ready (page loaded)
29.    function init() {
30.      // init the different variables
31.      canvas = document.querySelector("#mycanvas");
32.      ctx = canvas.getContext('2d');
33.      width = canvas.width;
34.      height = canvas.height;
35. 
36.      x=10; y = 10;
37.      // Target speed in pixels/second, try with high values, 1000, 2000...
38.      speedX = 2000;
39. 
40.      // Start animation
41.      requestAnimationFrame(animationLoop)
42.    }
43. 
44.    function animationLoop(time) {
45.      // Measure time with high resolution timer
46.      now = time;
47.  
48.      // How long between the current frame and the previous one?
49.      delta = now - then;
50. 
51.      if(totalTimeSinceLastRedraw > delayInMs) {
52.        // Compute the displacement in x (in pixels) in function of the time elapsed
53.        // since the last draw and
54.        // in function of the wanted speed. This time, instead of delta we
55.        // use totalTimeSinceLastRedraw as we're not always drawing at
56.        // each execution of mainloop
57.      incX = calcDistanceToMove(totalTimeSinceLastRedraw, speedX);
58. 
59.        // an animation involves: 1) clear canvas and 2) draw shapes,
60.        // 3) move shapes, 4) recall the loop with requestAnimationFrame
61. 
62.        // clear canvas
63.        ctx.clearRect(0, 0, width, height);
64. 
65.        ctx.strokeRect(x, y, 10, 10);
66. 
67.        // move rectangle
68.        x += incX;
69. 
70.       // check collision on left or right
71.       if((x+10 >= width) || (x <= 0)) {
72.         // cancel move + inverse speed
73.         x -= incX;
74.         speedX = -speedX;
75.       }
76.       // reset the total time since last redraw
77.       totalTimeSinceLastRedraw = delta;
78.    } else {
79.      // sum the total time since last redraw
80.      totalTimeSinceLastRedraw += delta;
81.    }
82.    // Store time
83.    then = now;
84.  
85. 
86.    // animate. 
87.    requestAnimationFrame(animationLoop);
88.  }
89. 
90.  var calcDistanceToMove = function(delta, speed) {
91.     return (speed * delta) / 1000;
92.  }
93. 
94.  </script>
95. </head>
96. 
97. <body onload="init();">
98.  <canvas id="mycanvas" width="200" height="50" style="border: 2px solid black"></canvas>
99. </body>
100. </html>

Can we use setInterval?

It's quite possible to use setInterval(function, interval) if you do not need an accurate scheduling.

To animate a monster at 60 fps but blinking his eyes once per second, you would use a mainloop with requestAnimationFrame and target a 60 fps animation, but you would also have a call to setInterval(changeEyeColor, 1000); and the changeEyeColor function will update a global variable, eyeColor, every second, which will be taken into account within the drawMonster function, called 60 times/s from the mainloop.

2.4.4 Adding time-based animation

To add time-based animation to our game engine, we will be using the technique discussed in the previous lesson. This technique is now widely supported by browsers, and adds time-based animation to our game framework, through the timestamp parameter passed to the callback function (mainLoop) by the call to requestAnimationFrame(mainLoop).

Here is an online example of the game framework at JSBin: this time, the monster has a speed in pixels/s and we use time-based animation. Try it and verify the smoothness of the animation; the FPS counter on a Mac Book Pro core i7 shows 60 fps.

Now try this slightly modified version in which we added a delay inside the animation loop. This should slow down the frame rate. On a Mac Book Pro + core i7, the frame-rate drops down to 37 fps. However, if you move the monster using the arrow keys, its speed on the screen is the same, excepting that it's not as smooth as in the previous version, which ran at 60 fps.

Here are the parts we changed:

Declaration of the monster object - now the speed is in pixels/s instead of in pixels per frame

1.  // The monster !
2.  var monster = {
3.     x:10,
4.     y:10,
5.     speed:100, // pixels/s this time !
6.  };

We added a timer(currentTime) function that returns the delta of the time elapsed since its last call

We refer to it from the game loop, to measure the time between frames. Notice that here we pass the delta as a parameter to the updateMonsterPosition call:

Code extract!

1.  function timer(currentTime) {
2.     var delta = currentTime - oldTime;
3.     oldTime = currentTime;
4.     return delta;
5.  }
6.  
7.  var mainLoop = function(time){
8.     //main function, called each frame
9.     measureFPS(time);
10. 
11.    // number of ms since last frame draw
12.    delta = timer(time);
13. 
14.    // Clear the canvas
15.    clearCanvas();
16. 
17.    // draw the monster
18.    drawMyMonster(monster.x, monster.y);
19. 
20.    // Check inputs and move the monster
21.    updateMonsterPosition(delta);
22. 
23.    // call the animation loop every 1/60th of second
24.    requestAnimationFrame(mainLoop);
25. };

Finally, we use the time-delta in the updateMonsterPosition(...) function

1.  function updateMonsterPosition(delta) {
2.     ...
3.     // Compute the incX and inY in pixels depending
4.     // on the time elapsed since last redraw
5.     monster.x += calcDistanceToMove(delta, monster.speedX);
6.     monster.y += calcDistanceToMove(delta, monster.speedY);
7.   }

2.5.1 Animating multiple objects

Hi, welcome! Let me show you how we can add many animated objects to the game framework.

You can imagine them as being the enemies the player should fight or whatever. For the sake of this example, we are using black balls, but you can imagine small images or small monsters or whatever.

Using here a constructor function is interesting because we can design a sort of class.

I mean a bit like Java classes or C# classes, if we make a comparison with other objected oriented languages. So we just say that the ball has a x position and a y position, an angle, a speed (the ‘v’ here if for he speed), and a radius.

We also said that each ball will be able to move and to be drawn on the screen.

It is a way to encapsulate in one single function the properties and the methods for manipulating the balls. And the advantage is that now we can create many balls using the new operator and passing different parameters.

Let's have a look at a function that will build a certain amount of balls with different parameters.

We called it createBalls: it takes as parameters a number of balls and will, in a loop, create new balls.

The new ball here will create a ball with a random x position, a random y position, a random angle between 0 and 2*PI, and a random speed and a given size. So I can change the size here, I can use another size so the reduce is a fixe parameter here.

Every ball is added to an array, so we have got a variable called ballArray that contains all the balls.

At the initialization, when the page is loaded we call this createBalls function that will fill the ball array. The mainLoop is called 60 times per seconds and goes along the whole ballArray and for each ball in the array, we will call ‘move’ that will change the x and y position depending on the angle and the speed of the ball and we will draw the balls.

In the middle we test if the ball is colliding with a side and we change the angle.

I can just call createBalls with a large number: here I created 100 balls, 1000 balls, and I can change some of their properties. This small example, that is 50 lines of code long, we can just take the function and add them inside the game framework.

So here, what we did is that we just added the ballArray variable inside the game framework, we called createBalls in the start function of the game framework, start function, I am inside here, so I create 160 balls or 1 ball so I can do whatever I like.

And we call the draw ball and move ball from inside the animation loop.

If you look at the animation loop, we clear the canvas, we draw the monster and we update the monster position to take into account the keys. If I press some keys the monster moves. So we call updateBalls that will move all the balls and draw all the balls.

What we have is that we got the last example with the moving monster plus a set of enemies that are animated.

In this section, we will see how we can animate and control not only the player but also other objects on the screen.

Let's study a simple example: animating a few balls and detecting collisions with the surrounding walls. For the sake of simplicity, we will not use time-based animation in the first examples.

Animating multiple balls which bounce off horizontal and vertical walls

Online example at JSBin:

In this example, we define a constructor function for creating balls. This is a way to design JavaScript "pseudo classes" as found in other object-oriented languages like Java, C# etc. It's useful when you plan to create many objects of the same class. Using this we could animate hundreds of balls on the screen.

Each ball has an x and y position, and in this example, instead of working with angles, we defined two "speeds" - horizontal and vertical speeds - in the form of the increments we will add to the x and y positions at each frame of animation. We also added a variable for adjusting the size of the balls: the radius.

Here is the constructor function for building balls:

Code extract!

1.  // Constructor function for balls
2.  function Ball(x, y, vx, vy, diameter) {
3.    // property of each ball: a x and y position, speeds, radius
4.    this.x = x;
5.    this.y = y;
6.    this.vx = vx;
7.    this.vy = vy;
8.    this.radius = diameter/2;
9.  
10.   // methods
11.   this.draw = function() {
12.     ctx.beginPath();
13.     ctx.arc(this.x, this.y, this.radius, 0, 2*Math.PI);
14.     ctx.fill();
15.   };
16. 
17.   this.move = function() {
18.     // add horizontal increment to the x pos
19.     // add vertical increment to the y pos
20.     this.x += this.vx;
21.     this.y += this.vy;
22.  };
23. }

Using a constructor function makes it easy to build new balls as follows:

1.  var b1 = new Ball(10, 10, 2, 2, 5); // x, y, vx, vy, radius
2.  var b1 = new Ball(100, 130, 4, 5, 5); 
3.  etc...

We defined two methods in the constructor function for moving the ball and for drawing the ball as a black filled circle. Here is the syntax for moving and drawing a ball:

1.  b1.draw();
2.  b1.move();

We will call these methods from inside the mainLoop, and as you'll see, we will create many balls. This object-oriented design makes it easier to handle large quantities.

Here is the rest of the code from this example:

Code extract!

1.  var canvas, ctx, width, height;
2.  
3.  // array of balls to animate
4.  var ballArray = [];
5.  
6.  function init() {
7.    canvas = document.querySelector("#myCanvas");
8.    ctx = canvas.getContext('2d');
9.    width = canvas.width;
10.   height = canvas.height;
11. 
12.   // try to change this number
13.   createBalls(16);
14. 
15.   requestAnimationFrame(mainLoop);
16. }
17. 
18. function createBalls(numberOfBalls) {
19.   for(var i=0; i < numberOfBalls; i++) {
20. 
21.     // Create a ball with random position and speed.
22.     // You can change the radius
23.     var ball = new Ball(width*Math.random(),
24.                         height*Math.random(),
25.                         (10*Math.random())-5,
26.                         (10*Math.random())-5,
27.                         30);
28. 
29.     // add the ball to the array
30.     ballArray[i] = ball;
31.    }
32. }
33. 
34. function mainLoop() {
35.   // clear the canvas
36.   ctx.clearRect(0, 0, width, height);
37. 
38.   // for each ball in the array
39.   for(var i=0; i < ballArray.length; i++) {
40.     var ball = ballArray[i]; 
41. 
42.     // 1) move the ball
43.     ball.move();
44. 
45.     // 2) test if the ball collides with a wall
46.     testCollisionWithWalls(ball);
47. 
48.     // 3) draw the ball
49.     ball.draw();
50.   }
51.   // ask for a new frame of animation at 60f/s
52.   window.requestAnimationFrame(mainLoop);
53. }
54. 
55. function testCollisionWithWalls(ball) {
56.   // left
57.   if (ball.x < ball.radius) { // x and y of the ball are at the center of the circle
58.     ball.x = ball.radius;     // if collision, we replace the ball at a position
59.     ball.vx *= -1;            // where it's exactly in contact with the left border
60.   }                           // and we reverse the horizontal speed
61.   // right
62.   if (ball.x > width - (ball.radius)) {
63.     ball.x = width - (ball.radius);
64.     ball.vx *= -1;
65.   }
66.   // up
67.   if (ball.y < ball.radius) {
68.     ball.y = ball.radius;
69.     ball.vy *= -1;
70.   }
71.   // down
72.   if (ball.y > height - (ball.radius)) {
73.     ball.y = height - (ball.radius);
74.     ball.vy *= -1;
75.   }
76. }

Notice that:

All the balls are stored in an array (line 4),
We wrote a createBalls(nb) function that creates a given number of balls (and stores them in the array) with random values for position and speed (lines 18-32)
In the mainLoop, we iterate on the array of balls and for each ball we: 1) move it, 2) test if it collides with the boundaries of the canvas (in the function testCollisionWithWalls), and 3) we draw the balls (lines 38-50). The order of these steps is not critical and may be changed.
The function that tests collisions is straightforward (lines 55-76). We did not use "if... else if" since a ball may sometimes touch two walls at once (in the corners). In that rare case, we need to invert both the horizontal and vertical speeds. When a ball collides with a wall, we need to replace it in a position where it is no longer against the wall (otherwise it will collide again during the next animation loop execution).

Similar example but with the ball direction as an angle, and a single velocity variable

Try this example at JSBin: it behaves in the same way as the previous example.

Note that we just changed the way we designed the balls and computed the angles after they rebound from the walls. The changes are highlighted in bold:

Code extract!

1.  var canvas, ctx, width, height;
2.  
3.  // Array of balls to animate
4.  var ballArray = [];
5.  
6.  function init() {
7.    ...
8.  }
9.  
10. function createBalls(numberOfBalls) {
11.   for(var i=0; i < numberOfBalls; i++) {
12. 
13.     // Create a ball with random position and speed.
14.     // You can change the radius
15.     var ball = new Ball(width*Math.random(),
16.                         height*Math.random(),
17.               (2*Math.PI)*Math.random(), // angle
18.               (10*Math.random())-5,      // speed
19.                         30);
20. 
21.    // We add it in an array
22.    ballArray[i] = ball;
23.   }
24. }
25. 
26. function mainLoop() {
27.   ...
28. }
29. 
30. function testCollisionWithWalls(ball) {
31.   // left
32.   if (ball.x < ball.radius) {
33.      ball.x = ball.radius;
34.      ball.angle = -ball.angle + Math.PI;
35.   }
36.   // right
37.   if (ball.x > width - (ball.radius)) {
38.     ball.x = width - (ball.radius);
39.     ball.angle = -ball.angle + Math.PI;
40.   }
41.   // up
42.   if (ball.y < ball.radius) {
43.     ball.y = ball.radius;
44.     ball.angle = -ball.angle;
45.   }
46.   // down
47.   if (ball.y > height - (ball.radius)) {
48.     ball.y = height - (ball.radius);
49.     ball.angle =-ball.angle;
50.   }
51. }
52. 
53. // constructor function for balls
54. function Ball(x, y, angle, v, diameter) {
55.   this.x = x;
56.   this.y = y;
57.   this.angle = angle;
58.   this.v = v;
59.   this.radius = diameter/2;
60. 
61.   this.draw = function() {
62.    ...
63.   };
64. 
65.   this.move = function() {
66.    // add horizontal increment to the x pos
67.    // add vertical increment to the y pos
68. 
69.    this.x += this.v * Math.cos(this.angle);
70.    this.y += this.v * Math.sin(this.angle);
71.   };
72. }

Using angles or horizontal and vertical increments is equivalent. However, one method might be preferable to the other: for example, to control an object that follows the mouse, or that tracks another object in order to attack it, angles would be more practical input to the computations required.

2.5.2 Adding balls to the game framework

This time, let's extract the source code used to create the balls, and include it in our game framework. We are also going to use time-based animation. The distance that the player and each ball should move is computed and may vary between animation frames, depending on the time-delta since the previous frame.

Online example at JSBin.

Try to move the monster with arrow keys and use the mouse button while moving to change the monster's speed. Look at the source code and change the parameters controlling the creation of the balls: number, speed, radius, etc. Also, try changing the monster's default speed. See the results.

For this version, we copied and pasted some code from the previous example and we also modified the mainLoop to make it more readable. In a next lesson, we will split the game engine into different files and clean the code-base to make it more manageable. But for the moment, jsbin.com is a good playground to try-out and test things...

The new mainLoop:

Code extract!

1.  var mainLoop = function(time){
2.    //main function, called each frame
3.    measureFPS(time);
4.  
5.    // number of ms since last frame draw
6.    delta = timer(time);
7.  
8.    // Clear the canvas
9.    clearCanvas();
10. 
11.   // Draw the monster
12.   drawMyMonster(monster.x, monster.y);
13. 
14.   // Check inputs and move the monster
15.   updateMonsterPosition(delta);
16. 
17.   // Update and draw balls
18.   updateBalls(delta);
19. 
20.   // Call the animation loop every 1/60th of second
21.   requestAnimationFrame(mainLoop);
22. };

As you can see, we draw the player/monster, we update its position; and we call an updateBalls function to do the same for the balls: draw and update their position.

Code extract!

1.  function updateMonsterPosition(delta) {
2.    monster.speedX = monster.speedY = 0;
3.    // check inputStates
4.    if (inputStates.left) {
5.      monster.speedX = -monster.speed;
6.    }
7.    if (inputStates.up) {
8.      monster.speedY = -monster.speed;
9.    }
10.   ...
11. 
12.   // Compute the incX and incY in pixels depending
13.   // on the time elapsed since last redraw
14.   monster.x += calcDistanceToMove(delta, monster.speedX);
15.   monster.y += calcDistanceToMove(delta, monster.speedY);
16. }
17. 
18. function updateBalls(delta) {
19.   // for each ball in the array
20.   for(var i=0; i < ballArray.length; i++) {
21.    var ball = ballArray[i];
22. 
23.    // 1) move the ball
24.    ball.move();
25. 
26.    // 2) test if the ball collides with a wall
27.    testCollisionWithWalls(ball);
28. 
29.    // 3) draw the ball
30.    ball.draw();
31. }
32. }

Now, in order to turn this into a game, we need to create some interactions between the player (the monster) and the obstacles/enemies (balls, walls)... It's time to take a look at collision detection.

2.5.3 Collision detection

In this chapter, we explore some techniques for detecting collisions between objects. This includes moving and static objects. We first present three "classic" collision tests, and follow them with brief sketches of more complex algorithms.

Circle collision test

Collision between circles is easy. Imagine there are two circles:

Circle c1 with center (x1,y1) and radius r1;
Circle c2 with center (x2,y2) and radius r2.

Imagine there is a line running between those two center points. The distances from the center points to the edge of each circle is, by definition, equal to their respective radii. So:

if the edges of the circles touch, the distance between the centers is r1+r2;
any greater distance and the circles don't touch or collide; whereas
any less and they do collide or overlay.

In other words: if the distance between the center points is less than the sum of the radii, then the circles collide.

Let's implement this as a JavaScript function step-by-step:

Code extract!

1.  function circleCollideNonOptimised(x1, y1, r1, x2, y2, r2) {
2.  var dx = x1 - x2;
3.  var dy = y1 - y2;
4.  var distance = Math.sqrt(dx * dx + dy * dy);
457. 
5.  return (distance < r1 + r2);
6.  }

This could be optimized a little averting the need to compute a square root:

(x2-x1)^2 + (y1-y2)^2 <= (r1+r2)^2

Which yields:

1.  function circleCollide(x1, y1, r1, x2, y2, r2) {
2.  var dx = x1 - x2;
3.  var dy = y1 - y2;
4.  return ((dx * dx + dy * dy) < (r1 + r2)*(r1+r2));
5.  }

This technique is attractive because a "bounding circle" can often be used with graphic objects of other shapes, providing they are not too elongated horizontally or vertically.

Let's test this idea

Try this example at JSBin: move the monster with the arrow keys and use the mouse to move "the player": a small circle. Try to make collisions between the monster and the circle you control.

This online example uses the game framework (without time-based animation in this one). We just added a "player" (for the moment, a circle that follows the mouse cursor), and a "monster". We created two JavaScript objects for describing the monster and the player, and these objects both have a boundingCircleRadius property:

JavaScript code extract!

// The monster!
var monster = {
  x:80,
  y:80,
  width: 100,
  height : 100,
  speed:1,
  boundingCircleRadius: 70
};

var player = {
  x:0,
  y:0,
  boundingCircleRadius: 20
};

The collision test occurs in the main loop:

JavaScript code extract!

var mainLoop = function(time){
//main function, called each frame
measureFPS(time);
// Clear the canvas
clearCanvas();
// Draw the monster
  drawMyMonster();
// Check inputs and move the monster
updateMonsterPosition();
updatePlayer();
checkCollisions();
// Call the animation loop every 1/60th of second
requestAnimationFrame(mainLoop);
};
function updatePlayer() {
// The player is just a circle drawn at the mouse position
// Just to test circle/circle collision.
if(inputStates.mousePos) {            // Move the player and draw it as a circle
  player.x = inputStates.mousePos.x;  // when the mouse moves
  player.y = inputStates.mousePos.y;
  ctx.beginPath();
  ctx.arc(player.x, player.y, player.boundingCircleRadius, 0, 2*Math.PI);
  ctx.stroke();
  }
}
function checkCollisions() {
if(circleCollide(player.x, player.y, player.boundingCircleRadius,
   monster.x, monster.y, monster.boundingCircleRadius)) {  
   // Draw everything in red
   ctx.fillText("Collision", 150, 20);
   ctx.strokeStyle = ctx.fillStyle = 'red';
  } else {  
// Draw in black
  ctx.fillText("No collision", 150, 20);

  ctx.strokeStyle = ctx.fillStyle = 'black';
  }
}
function circleCollide(x1, y1, r1, x2, y2, r2) {
   var dx = x1 - x2;
   var dy = y1 - y2;
   return ((dx * dx + dy * dy) < (r1 + r2)*(r1+r2));
}

[Advanced technique] Use several bounding circles for complex shapes, recompute bounding circles when the shape changes over time (animated objects)

This is an advanced technique: you can use a list of bounding circles or better still, a hierarchy of bounding circles in order to reduce the number of tests. The image below of an "arm" can be associated with a hierarchy of bounding circles. First, test against the "big one" on the left that contains the whole arm, then if there is a collision, test for the two sub-circles, etc... this recursive algorithm will not be covered in this course, but it's a classic optimization.

In 3D, you can use spheres instead of circles:

The famous game Gran Turismo 4 on the PlayStation 2 uses bounding spheres for detecting collisions between cars:

Rectangle (aligned along X and Y axis) detection test

Let's look at a simple illustration:

From this:

To detect a collision between two aligned rectangles, we project the horizontal and vertical axis of the rectangles over the X and Y axis. If both projections overlap, there is a collision!

[<span id="_Toc98696616" class="anchor"></span>Try this online demonstration of rectangle - rectangle detection](https://silentmatt.com/rectangle-intersection/)

1 - Only horizontal axis projections overlap: no collision between rectangles

2 - Only vertical axis projections overlap: no collision between rectangles

3 - Horizontal and vertical axis projections overlap: collision detected!

Here is a JavaScript implementation of a rectangle - rectangle (aligned) collision test:

JavaScript code extract!

// Collisions between aligned rectangles
function rectsOverlap(x1, y1, w1, h1, x2, y2, w2, h2) {
  
  if ((x1 > (x2 + w2)) || ((x1 + w1) < x2))
    return false; // No horizontal axis projection overlap

  if ((y1 > (y2 + h2)) || ((y1 + h1) < y2))
    return false; // No vertical axis projection overlap

  return true;    // If previous tests failed, then both axis projections
                  // overlap and the rectangles intersect
}

Let's test this method

Try this example at JSBin: move the monster with the arrow keys and use the mouse to move "the player": this time a small rectangle. Try to make collisions between the monster and the circle you control. Notice that this time the collision detection is more accurate and can work with elongated shapes.

Here is what we modified:

JavaScript code extract!

1.  ...
2.  // The monster!
3.  var monster = {
4.  x: 80,
5.  y: 80,
6.  width: 100,
7.  height: 100,
8.  speed: 1,
9.  boundingCircleRadius: 70
10. };
11. 
12. var player = {
13. x: 0,
14. y: 0,
15. boundingCircleRadius: 20
16. };
17. ...
18. 
19. function updatePlayer() {
20. // The player is just a square drawn at the mouse position
21. // Just to test rectangle/rectangle collisions.
22. 
23. if (inputStates.mousePos) {
24. player.x = inputStates.mousePos.x;
25. player.y = inputStates.mousePos.y;
26. 
535. 1.  // draws a rectangle centered on the mouse position
     2.  // we draw it as a square.
     3.  // We remove size/2 to the x and y position at drawing time in
     4.  // order to recenter the rectangle on the mouse pos (normally
     5.  // the 0, 0 of a rectangle is at its top left corner)
     6.  var size = player.boundingCircleRadius;
     7.  ctx.fillRect(player.x - size / 2, player.y - size / 2, size, size);
27. }
28. }
536. 
29. function checkCollisions() {
30. // Bounding rect position and size for the player. We need to
   translate
31. // it to half the player's size
32. var playerSize = player.boundingCircleRadius;
33. var playerXBoundingRect = player.x - playerSize / 2;
34. var playerYBoundingRect = player.y - playerSize / 2;
35. // Same with the monster bounding rect
36. var monsterXBoundingRect = monster.x - monster.width / 2;
37. var monsterYBoundingRect = monster.y - monster.height / 2;
537. 
38. if (rectsOverlap(playerXBoundingRect, playerYBoundingRect,
    playerSize, playerSize,
    monsterXBoundingRect, monsterYBoundingRect,
    monster.width, monster.height)) {
    <!-- -->
    ctx.fillText("Collision", 150, 20);
    ctx.strokeStyle = ctx.fillStyle = 'red';
39. } else {
    ctx.fillText("No collision", 150, 20);
    ctx.strokeStyle = ctx.fillStyle = 'black';
40. }
41. }
538. 
42. // Collisions between aligned rectangles
43. function rectsOverlap(x1, y1, w1, h1, x2, y2, w2, h2) {
539. 
44. if ((x1 > (x2 + w2)) || ((x1 + w1) < x2))
    return false; // No horizontal axis projection overlap
540. 
45. if ((y1 > (y2 + h2)) || ((y1 + h1) < y2))
    return false; // No vertical axis projection overlap
541. 
46. return true; // If previous tests failed, then both axis
   projections
    // overlap and the rectangles intersect
47. }

Many real games use aligned rectangle collision tests

Testing "circle-circle" or "rectangle-rectangle collisions is cheap in terms of computation. "Rectangle-rectangle" collisions are used in many 2D games, such as Dodonpachi (one of the most famous and enjoyable shoot'em'ups ever made - you can play it using the MAME arcade game emulator):

You could also try the free Genetos shoot'em up game (Windows only) that retraces the history of the genre over its different levels (download here). Press the G key to see the bounding rectangles used for collision test. Here is a screenshot:

These games run at 60 fps and can have hundreds of bullets moving at the same time. Collisions have to be tested: did the player's bullets hit an enemy, AND did an enemy bullet (for one of the many enemies) hit the player? These examples demonstrate the efficiency of such collision test techniques.

Other collision tests

In this section, we only give sketches and examples of more sophisticated collision tests. For further explanation, please follow the links provided.

Aligned rectangle-circle

There are only two cases when a circle intersects with a rectangle:

Either the circle's center lies inside the rectangle, or
One of the edges of the rectangle intersects the circle.

We propose this function (implemented after reading this Thread at StackOverflow):

JavaScript code extract!

1.  // Collisions between rectangle and circle
2.  function circRectsOverlap(x0, y0, w0, h0, cx, cy, r) {
3.     var testX=cx;
4.     var testY=cy;
5.  
6.     if (testX < x0) testX=x0;
7.     if (testX > (x0+w0)) testX=(x0+w0);
8.     if (testY < y0) testY=y0;
9.     if (testY > (y0+h0)) testY=(y0+h0);
10. 
11.    return (((cx-testX)*(cx-testX)+(cy-testY)*(cy-testY))< r*r);
12. }

Try this function in this example on JSBin.

[ADVANCED] Collision between balls (pool like)

Math and physics: please read this external resource (for math), a great article that explains the physics of a pool game.
Example of colliding balls at JSBin (author: M.Buffa), and also try this example that does the same with a blurring effect

The principle behind collision resolution for pool balls is as follows. You have a situation where two balls are colliding, and you know their velocities (step 1 in the diagram below). You separate out each ball’s velocity (the solid blue and green arrows in step 1, below) into two perpendicular components: the "normal" component heading towards the other ball (the dotted blue and green arrows in step 2) and the "tangential" component that is perpendicular to the other ball (the dashed blue and green arrows in step 2). We use "normal" for the first component as its direction is along the line that links the centers of the balls, and this line is perpendicular to the collision plane (the plane that is tangent to the two balls at collision point).

The solution for computing the resulting velocities is to swap the components between the two balls (as we move from step 2 to step 3), then finally recombine the velocities for each ball to achieve the result (step 4):

The above picture has been borrowed from this interesting article about how to implement in C# pool like collision detection.

Of course, we will only compute these steps if the balls collide, and for that test we will have used the basic circle collision test outlined earlier.

To illustrate the algorithm, here is an example at JSBin that displays the different vectors in real time, with only two balls. The math for the collision test have also been expanded in the source code to make computations clearer. Note that this is not for beginners: advanced math and physics are involved!

To go further... video game physics!

For the ones who are not afraid by some math and physics and would like to learn how to do collision detection in a more realistic way (using physics modeling), we recommend this tutorial, that is the first of a three-part series about video game physics.

2.5.4 Adding collision detection to the game framework

Our previous lesson enabled us to animate balls in the game framework ([this example](https://jsbin.com/tehuve/edit)).

Now we can add the functionality presented in the last lesson, to perform collision tests between a circle and a rectangle. It will be called 60 times/s when we update the position of the balls. If there is a collision between a ball (circle) and the monster (rectangle), we set the ball color to red.

Try the example at JsBin!

Source code extract:

JavaScript code extract!

1.  function updateBalls(delta) {
2.    // for each ball in the array
3.    for(var i=0; i < ballArray.length; i++) {
4.      var ball = ballArray[i];
5.  
6.      // 1) move the ball
7.      ball.move();
8.  
9.      // 2) test if the ball collides with a wall
10.     testCollisionWithWalls(ball);
11.  
12.     // 3) Test if the monster collides
13.     if(circRectsOverlap(monster.x, monster.y,
14.                         monster.width, monster.height,
15.                         ball.x, ball.y, ball.radius)) {
16. 
17.     //change the color of the ball
18.     ball.color = 'red';
19.     }
20. 
21.     // 3) draw the ball
22.     ball.draw();
23.   }
24. }

The only additions are: lines 13-19 in the updateBalls function, and the circRectsOverlap function!

2.6.1 Introduction

In this lesson, we learn how to animate images - which are known as "sprites". This technique uses components from a collection of animation frames. By drawing different component images, rapidly, one-after-the-other, we obtain an animation effect.

Here is an example of a spritesheet, where each line animates a woman walking in a particular direction:

The first line corresponds to the direction we called "south", the second "south west", the third "west", etc. The 8 lines cover movement in all eight cardinal directions.

Each line is composed of 13 small images which together comprise an "animated" sprite. If we draw each of the 13 animations of the first line, in turn; we will see a woman who seems to move towards the screen. And if we draw each sprite a little closer to the bottom of the screen, we obtain a woman who appears to approach the bottom of the screen, swinging her arms and legs, as she walks!

Try it yourself: here is a quick and dirty example to try at JSBin working with the above sprite sheet. Use the arrow keys and take a look! We accentuated the movement by changing the scale of the sprite as the woman moves up (further from us) or down (closer to us).

We have not yet investigated how this works, nor have we built it into the small game engine we started to build in earlier chapters. First, let's explain how to use "sprites" in JavaScript and canvas.

2.6.2 Different sorts of sprite sheets

There are different sorts of sprite sheets. See some examples below.

Multiple postures on a single sprite sheet

A sprite sheet with different "sprite" sets that correspond to different "postures": this is the case for the walking woman we just saw in the previous lesson. This sprite sheet contains 8 different sets of sprites, or postures, each corresponding to a direction. In this example, each posture comprises exactly 13 sprites, aligned in a single row across the sprite sheet.

One posture per sprite sheet

Some sprite sheets have a single sprite set, spreading over multiple lines; like this walking robot:

This is an example that you will see a lot around the Internet, in many sprite sheets. For the full animation of the robot, we will need multiple sprite sheets: one for each posture.

As another example, here is the "jumping robot" sprite sheet:

Whereas the walking robot posture is made of 16 sprites, the jumping robot needs 26!

Hybrid sprite sheets

You will also find sprite sheets that contain completely different sets of sprites (this one comes from the famous Gridrunner IOS game by Jeff Minter):

So, when we think about writing a "sprite engine", we need to consider how to support different layouts of sprite sheet.

2.6.3 Sprite extraction and animation

Principle

Before doing anything interesting with the sprites, we need to:

Load the sprite sheet(s),
Extract the different postures and store them in an array of sprites,
Choose the appropriate one, and draw it within the animation loop, taking into account elapsed time. We cannot draw a different image of the woman walking at 60 updates per second. We will have to create a realistic "delay" between each change of sprite image.

In this lesson, let's construct an interactive tool to present the principles of sprite extraction and animation.

Example #1

In this example, we'll move the slider to extract the sprite indicated by the slider value. See the red rectangle? This is the sprite image currently selected! When you move the slider, the corresponding sprite is drawn in the small canvas. As you move the slider from one to the next, see how the animation is created?

Try it at JSBin:

HTML code:

HTML code extract!

1.  <html lang="en">
2.   <head>
3.   <title>Extract and draw sprite</title>
4.   <style>
5.     canvas {
6.        border: 1px solid black;
7.      }
8.   </style>
9.   </head>
10.  <body>
11.    Sprite width: 48, height: 92, rows: 8, sprites per
    > posture: 13<p>
12.    <label for="x">x: <input id="x" type="number" min=0><br/>
13.    <label for="y">y: <input id="y" type="number" min=0><br/>
14.    <label for="width">width: <input
   > id="width" type="number" min=0><br/>
15.    <label for="height">height: <input
   > id="height" type="number" min=0><p>
16. 
17. Select current sprite: <input type=range
    > id="spriteSelect" value=0> <output id="spriteNumber">
18. 
19.  <p/>
20.    <canvas id="canvas" width="48" height="92" />
21.  </p>
22.    <canvas id="spritesheet"></canvas>
23.  </body>
24. </html>

Notice that we use an <input type="range"> to select the current sprite, and we have two canvases: a small one for displaying the currently-selected sprite, and a larger one that contains the sprite sheet and in which we draw a red square to highlight the selected sprite.

Here's an extract from the JavaScript. You don't have to understand all the details, just look at the part in bold which extracts the individual sprites:

JavaScript source code extract!

1.  var SPRITE_WIDTH = 48;   // Characteristics of the sprites and
    > spritesheet
2.  var SPRITE_HEIGHT = 92;
3.  var NB_ROWS = 8;
4.  var NB_FRAMES_PER_POSTURE = 13;
5.   
6.  // the different input and output fields
7.  var xField, yField, wField, hField, spriteSelect, spriteNumber;
8.  // The two canvases and respective contexts
9.  var canvas, canvasSpriteSheet, ctx1, ctx2;
10.  
11. window.onload = function() {
12.     canvas = document.getElementById("canvas");
13.     ctx1 = canvas.getContext("2d");
14.     canvasSpriteSheet = document.getElementById("spritesheet");
15.     ctx2 = canvasSpriteSheet.getContext("2d");
16. 
17.     xField = document.querySelector("#x");
18.     yField = document.querySelector("#y");
19.     wField = document.querySelector("#width");
20.     hField = document.querySelector("#height");
21.     spriteSelect = document.querySelector("#spriteSelect");
22.     spriteNumber = document.querySelector("#spriteNumber");
23.     
24.     // Update values of the input fields in the page
25.     wField.value = SPRITE_WIDTH;
26.     hField.value = SPRITE_HEIGHT;
27.     xField.value = 0;
28.     yField.value = 0;
29.     // Set attributes for the slider depending on the number of
   > sprites on the 
30.     // sprite sheet
31.     spriteSelect.min = 0;
32.     spriteSelect.max=NB_ROWS*NB_FRAMES_PER_POSTURE - 1;
33.     // By default the slider is disabled until the sprite sheet is
   > fully loaded
34.     spriteSelect.disabled = true;
35.     spriteNumber.innerHTML=0;
36. 
37.     // Load the spritesheet
38.     spritesheet = new Image();
39.     spritesheet.src="https://i.imgur.com/3VesWqx.webp";
40. 
41.     // Called when the spritesheet has been loaded
42.     spritesheet.onload = function() {
43.       // enable slider
44.       spriteSelect.disabled = false;
45. 
46.       // Resize big canvas to the size of the sprite sheet image
47.       canvasSpriteSheet.width = spritesheet.width;
48.       canvasSpriteSheet.height = spritesheet.height;
49. 
50.      // Draw the whole spritesheet
51.      ctx2.drawImage(spritesheet, 0, 0);
52.      // Draw the first sprite in the big canvas, corresponding to
   > sprite 0
53.      // wireframe rectangle in the sprite sheet
54.    >  drawWireFrameRect(ctx2, 0 , 0, SPRITE_WIDTH, SPRITE_HEIGHT, 'red', 3);
55.    >      // small canvas, draw sub image corresponding to sprite 0
56.      ctx1.drawImage(spritesheet, 0, 0, SPRITE_WIDTH, SPRITE_HEIGHT,
57.                                  0, 0, SPRITE_WIDTH, SPRITE_HEIGHT);
58.    };
59. 
60.    // input listener on the slider
61.    spriteSelect.oninput = function(evt) {
62.      // Current sprite number from 0 to NB_FRAMES_PER_POSTURE *
    > NB_ROWS
63.      var index = spriteSelect.value;
64. 
65.      // Computation of the x and y position that corresponds to
    > the sprite
66.      // number index as selected by the slider
67.      var x = index * SPRITE_WIDTH % spritesheet.width;
68.    >  var y = Math.floor(index / NB_FRAMES_PER_POSTURE) * SPRITE_HEIGHT;
69. 
70.      // Update fields
71.      xField.value = x;
72.      yField.value = y;
73. 
74.      // Clear big canvas, draw wireframe rect at x, y, redraw
    > stylesheet
75.    >  ctx2.clearRect(0, 0, canvasSpriteSheet.width, canvasSpriteSheet.height);
76.      ctx2.drawImage(spritesheet, 0, 0);
77.    >  drawWireFrameRect(ctx2, x , y, SPRITE_WIDTH, SPRITE_HEIGHT, 'red', 3);
78.  
79.      // Draw the current sprite in the small canvas
80.      ctx1.clearRect(0, 0, SPRITE_WIDTH, SPRITE_HEIGHT);
81.      ctx1.drawImage(spritesheet, x, y, SPRITE_WIDTH, SPRITE_HEIGHT,
82.                                  0, 0, SPRITE_WIDTH, SPRITE_HEIGHT);
83.  
84.      // Update output elem on the right of the slider
85.      spriteNumber.innerHTML = index;
86.   };
87. };
88.  
89. function drawWireFrameRect(ctx, x, y, w, h, color, lineWidth) {
90.   ctx.save();
91.   ctx.strokeStyle = color;
92.   ctx.lineWidth = lineWidth;
93.   ctx.strokeRect(x , y, w, h);
94.   ctx.restore();
95. }
96.

Explanations:

Lines 1-4: characteristics of the sprite sheet. How many rows, i.e., how many sprites per row, etc.
Lines 11-39: initializations that run just after the page has been loaded. We first get the canvas and contexts. Then we set the minimum and maximum values of the slider (an <input type=range>) at lines 31-32, and disable it at line 34 (we cannot slide it before the sprite sheet image has been loaded). We display the current sprite number 0 in the field to the right of the slider (line 35). Finally, in lines 37-39, we load the sprite sheet image.
Lines 42-58: this callback is run once the sprite sheet image has been loaded. We enable the slider, set the big canvas to the size of the loaded image, and then draw it (line 51). We also draw the first sprite from the sprite sheet in the small canvas and draw a red wireframe rectangle around the first sprite in the sprite sheet (lines 52-58).
Lines 61-87: the input listener callback, called each time the slider moves. Lines 65-68 are the most important ones here: we compute the x and y position of the sprite selected with the slider. We take into account the number of sprites per posture, the number of rows, and the dimensions of each sprite. Then, as in the previous step, we draw the current sprite in the small canvas and highlight the current sprite with a red rectangle in the sprite sheet.
The code is generic enough to work with different kinds of sprite sheets. Adjust the global parameters in bold at lines 1-5 and try the extractor.

Example #2

This is the same application with another sprite sheet.
We just changed these parameter values: try the same code but with another sprite sheet (the one with the robot) - see on JSBin.

Now it's time to see how we can make a small sprite animation framework!

2.6.4 A small sprite animation framework

Now that we have presented the principle of sprite extraction (sprites as sub-images of a single composite image), let's write a small sprite animation framework.

Here is how you would create and animate a sprite:

JavaScript source code extract!

1.  var robot;
2.   
3.  window.onload = function() {
4.     canvas = document.getElementById("canvas");
5.     ctx = canvas.getContext("2d");
6.  
7.    // Load the spritesheet
8.    spritesheet = new Image();
9.    spritesheet.src = SPRITESHEET_URL;
10. 
11.   // Called when the spritesheet has been loaded
12.   spritesheet.onload = function() {
13.     ...
14.     robot = new Sprite();
15.     // 1 is the posture number in the sprite sheet. We have
16.     // only one with the robot.
17.     robot.extractSprites(spritesheet, NB_POSTURES, 1,
18.                          NB_FRAMES_PER_POSTURE,
19.                          SPRITE_WIDTH, SPRITE_HEIGHT);
20.     robot.setNbImagesPerSecond(20);
21. 
22.     requestAnimationFrame(mainloop);
23.  }; // onload
24. };
25.  
26. function mainloop() {
27.   // Clear the canvas
28.   ctx.clearRect(0, 0, canvas.width, canvas.height);
29.   // draw sprite at 0, 0 in the small canvas
30.   robot.draw(ctx, 0, 0, 1);
31. 
32.   requestAnimationFrame(mainloop);
33. }

Try the example on JSBin that uses this framework first! Experiment by editing >line 20: robot.setNbImagesPerSecond(20); changing the value of the parameter and observing the result.

The SpriteImage object and sprite models

In this small framework we use "SpriteImage ", a JS object we build to represent one sprite image. Its properties are: the global sprite sheet to which it belongs, its position in the sprite sheet, and its size. It also has a draw method for drawing the sprite image at an xPos, yPos position, and at anappropriate size.

Function SpriteImage code!

1.  function SpriteImage(img, x, y, width, height) {
2.     this.img = img; // the whole image that contains all sprites
3.     this.x = x;     // x, y position of the sprite image in the whole
    > image
4.     this.y = y;
5.     this.width = width; // width and height of the sprite image
6.     this.height = height;
7.    
8.     this.draw = function(ctx, xPos, yPos, scale) {
9.        ctx.drawImage(this.img,
10.         this.x, this.y, // x, y, width and height of img to extract
11.         this.width, this.height,
12.         xPos, yPos, // x, y, width and height of img to draw
13.         this.width*scale, this.height*scale);
14.  };
15. }

We define the Sprite model. This is the one we used to create the small robot in the previous example.

A Sprite is defined by an array of SpriteImage objects.
It has a method for extracting all SpriteImages from a given sprite sheet and filling the above array.
It has a draw method which will draw the current SpriteImage. A Sprite is an animated object, therefore, calling draw multiple times will involve an automatic change of the current SpriteImage being drawn.
The number of different images to be drawn per second is a parameter of the sprite.

Here is the code of the Sprite model:

Function Sprite source code!

1.  function Sprite() {
2.    this.spriteArray = [];
3.    this.currentFrame = 0;
4.    this.delayBetweenFrames = 10;
5.  
6.    this.extractSprites = function(spritesheet,
7.                                   nbPostures, postureToExtract,
8.                                   nbFramesPerPosture,
9.                                   spriteWidth, spriteHeight) {
10.     // number of sprites per row in the spritesheet
11.  
   >   var nbSpritesPerRow = Math.floor(spritesheet.width / spriteWidth);
12. 
13.     // Extract each sprite
14.     var startIndex = (postureToExtract -1) * nbFramesPerPosture;
15.     var endIndex = startIndex + nbFramesPerPosture;
16.     for(var index = startIndex; index < maxIndex; index++) {
17.     // Computation of the x and y position that corresponds to the
    > sprite
18.       // index
19.       // x is the rest of index/nbSpritesPerRow * width of a sprite
20.       var x = (index % nbSpritesPerRow) * spriteWidth;
21.       // y is the divisor of index by nbSpritesPerRow * height of a
    > sprite
22.       var y = Math.floor(index / nbSpritesPerRow) * spriteHeight;
23. 
24.       // build a spriteImage object
25.    
    >   var s = new SpriteImage(spritesheet, x, y, spriteWidth, spriteHeight);
26. 
27.       this.spriteArray.push(s);
28.     }
29.   };
30. 
31.   this.then = performance.now();
32.   this.totalTimeSinceLastRedraw = 0;
33. 
34.   this.draw = function(ctx, x, y) {
35.     // Use time based animation to draw only a few images per second
36.     var now = performance.now();
37.     var delta = now - this.then;
38. 
39.     // Draw currentSpriteImage
40.     var currentSpriteImage = this.spriteArray[this.currentFrame];
41.     // x, y, scale. 1 = size unchanged
42.     currentSpriteImage.draw(ctx, x, y, 1);
43. 
44.     // if the delay between images is elapsed, go to the next one
45.     if (this.totalTimeSinceLastRedraw > this.delayBetweenFrames) {
46.     // Go to the next sprite image
47.       this.currentFrame++;
48.       this.currentFrame %= this.spriteArray.length;
49. 
50.     // reset the total time since last image has been drawn
51.       this.totalTimeSinceLastRedraw = 0;
52.     } else {
53.       // sum the total time since last redraw
54.       this. totalTimeSinceLastRedraw += delta;
55.     }
56. 
57.     this.then = now;
58.   };
59. 
60.   this.setNbImagesPerSecond = function(nb) {
61.     // delay in ms between images
62.     this.delayBetweenFrames = 1000 / nb;
63.   };
64. }

Same example but with the walking woman sprite sheet

Try this JsBin

This time, we have changed the parameters of the sprites and sprite sheet. Now you can select the index of the posture to extract: the woman sprite sheet has 8 different postures, so you can call:

JavaScript source code!

1.  womanDown.extractSprites(spritesheet, NB_POSTURES, 1,
2.                           NB_FRAMES_PER_POSTURE,
3.                           SPRITE_WIDTH, SPRITE_HEIGHT);
4.   
5.  womanDiagonalBottomLeft.extractSprites(spritesheet, NB_POSTURES, 2,
6.                          NB_FRAMES_PER_POSTURE,
7.                          SPRITE_WIDTH, SPRITE_HEIGHT);
8.   
9.  womanLeft.extractSprites(spritesheet, NB_POSTURES, 3,
10.                          NB_FRAMES_PER_POSTURE,
11.                          SPRITE_WIDTH, SPRITE_HEIGHT);
12. // etc...

Moving the sprites, stopping the sprites

Example at JsBin

As usual, we used key listeners, an inputStates global object, and this time we created 8 woman sprites, one for each direction.

Notice that we added a drawStopped method in the Sprite model in order to stop animating the woman when no key is pressed for moving her.

External resource

Game development tutorial (5 May 2020): [Draw images and sprite animations] (https://spicyyoghurt.com/tutorials/html5-javascript-game-development/images-and-sprite-animations)

2.6.5 Adding sprites to the game framework

Let us use the animated woman example and take the sprite utility functions and some predefined values, such as the sprite sheet URL, the size of the sprites, the number of postures, etc., and add it all to one of the examples that used the game framework (the last one from the time based animation lesson (to keep things simple, we did not use the ones with gamepad, etc)).

First, try this example at JsBin

How to add sprites to the game framework...

We declare a woman object, similar to the monster object, with x, y, speed, width properties. We add a direction property that corresponds to a posture's index (so direction = 2 corresponds to the sprite animation for the woman moving to the left, whereas direction = 6 corresponds to the posture of the woman moving to the right...)
We add the Sprite and SpriteImage objects to the game framework,
We write a loadAssets(callback) function which: a) loads the sprite sheet, b) extracts all the woman sprites and builds the womanSprites array, and c) calls the callback function passed as a parameter once finished,
We call the loadAssets function from the game framework start function, and we start the animation loop only when the loadAssets function has completed loading and extracting the sprites. In a real game the loadAssets function would also load the sounds, and perhaps other sprite sheets or resources etc. In this function, you could use the BufferLoader utility for loading multiple resources asynchronously, as discussed during Module 1.

Source code extract:

JavaScript source code extract!

1.  // Inits
2.  window.onload = function init() {
3.  var game = new GF();
4.  game.start();
5.  };
6.   
7.  // GAME FRAMEWORK STARTS HERE
8.  var GF = function(){
9.     ...
10.    // Woman object and sprites
11.    // sprite index corresponding to posture
12.    var WOMAN_DIR_RIGHT = 6;
13.    var WOMAN_DIR_LEFT = 2;
14.    var woman = {
15.       x:100,
16.       y:200,
17.       width:48,
18.       speed:100, // pixels/s this time!
19.       direction: WOMAN_DIR_RIGHT
20.    };
21.  
22.    var womanSprites = [];
23.  
24.    var mainLoop = function(time){
25.      ...
26.      // Draw a woman moving left and right
27.     womanSprites[woman.direction].draw(ctx, woman.x, woman.y);
28.     updateWomanPosition(delta);
29.     ...
30.   };
31. 
32.   function updateWomanPosition(delta) {
33.     // check collision on left or right
34.     if(((woman.x+woman.width) > canvas.width) || (woman.x < 0)) {
35.       // inverse speed
36.       woman.speed = -woman.speed;
37.     }
38. 
39.     // change sprite direction
40.     if(woman.speed >= 0) {
41.       woman.direction = WOMAN_DIR_RIGHT;
42.     } else {
43.       woman.direction = WOMAN_DIR_LEFT;
44.     }
45.     woman.x += calcDistanceToMove(delta, woman.speed);
46.   }
47.  
48.   /*---------------------------------------*/
49.   /* SPRITE UTILITY FUNCTIONS              */
50.   /*---------------------------------------*/
51.   function SpriteImage(img, x, y, width, height) {
52.     ...
53.     this.draw = function(ctx, xPos, yPos, scale) {...};
54.   }
55.  
56.   function Sprite() {
57.     ...
58.     this.extractSprites = function(...) {...};
59.     this.drawStopped = function(ctx, x, y) {...};
60.     this.draw = function(ctx, x, y) {...};
61.     this.setNbImagesPerSecond = function(nb) {...};
62.   }
63.   /*---------------------------------------*/
64.   /* EN OF SPRITE UTILITY FUNCTIONS        */
65.   /*---------------------------------------*/
66. 
67. var loadAssets = function(callback) {
68.     var SPRITESHEET_URL = "https://i.imgur.com/3VesWqx.webp";
69.     var SPRITE_WIDTH = 48;
70.     var SPRITE_HEIGHT = 92;
71.     var NB_POSTURES=8;
72.     var NB_FRAMES_PER_POSTURE = 13;
73. 
74.     // load the spritesheet
75.     var spritesheet = new Image();
76.     spritesheet.src = SPRITESHEET_URL;
77. 
78.     // Called when the spritesheet has been loaded
79.     spritesheet.onload = function() {
80.       // Create woman sprites
81.       for(var i = 0; i < NB_POSTURES; i++) {
82.         var sprite = new Sprite();
83. 
84.         sprite.extractSprites(spritesheet, NB_POSTURES, (i+1),
85.                               NB_FRAMES_PER_POSTURE,
86.                               SPRITE_WIDTH, SPRITE_HEIGHT);
87.         sprite.setNbImagesPerSecond(20);
88.         womanSprites[i] = sprite;
89.       }
90.       // call the callback function passed as a parameter,
91.       // we're done with loading assets and building the sprites
92.       callback();
93.     };
94.   };
95. 
96.   var start = function(){
97.     ...
98.     // Load sounds and images, then when this is done, start the  mainLoop
99.     loadAssets(function() {
100.        // We enter here only when all assets have been loaded
101.        requestAnimationFrame(mainLoop);
102.     });
103.   };
104.   ...
105.  };
106.

2.7.1 Menus, high score tables, etc.

With our game framework handling the basics, we can make things more exciting by causing something to happen when a collision occurs - maybe the player 'dies', balls are removed, or we should add to your score? Usually, we add extra properties to the player and enemy objects. For example, a boolean dead property that will record if an object is dead or alive: if a ball is marked "dead", do not draw it! If all balls are dead: go to the next level with more balls, faster balls, etc.

Let's try adding a dead property to balls and consult it before drawing them. We could also test to see if all the balls are dead, in which case we recreate them and add one more ball. Let's update the score whenever the monster eats a ball. And finally, we should add a test in the createBalls function to ensure that no balls are created on top of the monster.

Try this jsBin!

Source code extract:

JavaScript function updateBalls source code extract!

1.  function updateBalls(delta) {
2.     // for each ball in the array
3.     var allBallDead = true;
4.  
5.     for(var i=0; i < ballArray.length; i++) {
6.       var ball = ballArray[i];
7.  
8.       if(ball.dead) continue; // do nothing if the ball is dead
9.  
10.      // if we are here: the ball is not dead
11.      allBallDead = false;
12. 
13.      // 1) move the ball
14.      ball.move();
15. 
16.      // 2) test if the ball collides with a wall
17.      testCollisionWithWalls(ball);
18.  
19.      // Test if the monster collides
20.      if(circRectsOverlap(monster.x, monster.y,
21.                          monster.width, monster.height,
22.                          ball.x, ball.y, ball.radius)) {
23. 
24.         //change the color of the ball
25.         ball.color = 'red';
26.         ball.dead = true;
27.         // Here, a sound effect would greatly improve
28.         // the experience!
29. 
30.         currentScore+= 1;
31.      }
32. 
33.      // 3) draw the ball
34.      ball.draw();
35.   }

JavaScript code extract!

1. if(allBallDead) {
2.   // reset all balls, create more balls each time
3.   // as a way of increasing the difficulty
4.   // in a real game: change the level, play nice music!
5.   nbBalls++;
6.   createBalls(nbBalls);
7.   }
8. }

Game states:menus, high score tables, etc.

In this example, let's make use of a global gameState variable for managing the life-cycle of the game. Usually, there is a main menu with a "start new game" option, then we play the game, and if we 'die' we suffer a game-over screen, etc...

Ah!... you think that the game has been too easy? Let's reverse the game: now you must survive without being touched by a ball!!!

Also, every five seconds a next level will start: the set of balls is re-created, and each level has two more than before. How long can you survive?

Try this JsBin, then look at the source code. Start from the mainloop!

currentGameState = gameStates.gameOver:

currentGameState = gameStates.gamerunning:

Game state management in the JavaScript code:

JavaScript code extract!

1.  ...
2.  // game states
3.  var gameStates = {
4.     mainMenu: 0,
5.     gameRunning: 1,
6.     gameOver: 2
7.  };
8.   
9.  var currentGameState = gameStates.gameRunning;
10. var currentLevel = 1;
11. var TIME_BETWEEN_LEVELS = 5000; // 5 seconds
12. var currentLevelTime = TIME_BETWEEN_LEVELS;
13. ...
14. var mainLoop = function (time) {
15.    ...
16.    // number of ms since last frame draw
17.    delta = timer(time);
18.  
19.    // Clear the canvas
20.    clearCanvas();
21.  
22.    // monster.dead is set to true in updateBalls when there
23.    // is a collision
24.    if (monster.dead) {
25.       currentGameState = gameStates.gameOver;
26.    }
27.  
28.    switch (currentGameState) {
29.       case gameStates.gameRunning:
30.         // draw the monster
31.         drawMyMonster(monster.x, monster.y);
32.  
33.         // Check inputs and move the monster
34.         updateMonsterPosition(delta);
35.  
36.         // update and draw balls
37.         updateBalls(delta);
38.  
39.         // display Score
40.         displayScore();
41.  
42.         // decrease currentLevelTime. Survive 5s per level
43.         // When < 0 go to next level
44.        currentLevelTime -= delta;
45.  
46.        if (currentLevelTime < 0) {
47.           goToNextLevel();
48.        }
49.        break;
50.     case gameStates.mainMenu:
51.        // TO DO! We could have a main menu with high scores etc.
52.        break;
53.     case gameStates.gameOver:
54.        ctx.fillText("GAME OVER", 50, 100);
55.        ctx.fillText("Press SPACE to start again", 50, 150);
56.        ctx.fillText("Move with arrow keys", 50, 200);
57.        ctx.fillText("Survive 5 seconds for next level", 50, 250);
58.  
59.        if (inputStates.space) {
60.           startNewGame();
61.        }
62.        break;
63.     }
64.      ...
65.    };
66. ...

And below are the functions for starting a new level, starting a new game, and the updateBalls function that determines when a player loses and changes the current game-state to GameOver:

JavaScript extract code!

1.  function startNewGame() {
2.      monster.dead = false;
3.      currentLevelTime = 5000;
4.      currentLevel = 1;
5.      nbBalls = 5;
6.      createBalls(nbBalls);
7.      currentGameState = gameStates.gameRunning;
8.  }
9.   
10. function goToNextLevel() {
11.     // reset time available for next level
12.     // 5 seconds in this example
13.    currentLevelTime = 5000;
14.    currentLevel++;
15.    // Add two balls per level
16.    nbBalls += 2;
17.    createBalls(nbBalls);
18. }
19.  
20. function updateBalls(delta) {
21.     // Move and draw each ball, test collisions,
22.     for (var i = 0; i < ballArray.length; i++) {
23.        ...
24.        // Test if the monster collides
25.        if (circRectsOverlap(monster.x, monster.y,
26.                             monster.width, monster.height,
27.                             ball.x, ball.y, ball.radius)) {
28.  
29.            //change the color of the ball
30.            ball.color = 'red';
31.            monster.dead = true;
32.            // Here, a sound effect greatly improves
33.            // the experience!
34.            plopSound.play();
35.        }
36.  
37.        // 3) draw the ball
38.        ball.draw();
39.     }
40. }

2.7.2 Splitting the game into several JS files

JSBin is a great tool for sharing code, for experimenting, etc. But as soon as the size of your project increases, you'll find that the tool is not suited for developing large systems.

In order to keep working on this game framework, we recommend that you modularize the project and split the JavaScript code into several JavaScript files:

Review the different functions and isolate those that have no dependence on the framework. Obviously, the sprite utility functions, the collision detection functions, and the ball constructor function can be separated from the game framework, and could easily be reused in other projects. Key and mouse listeners also can be isolated, gamepad code too...
Look at what you could change to reduce dependencies: add a parameter in order to make a function independent from global variables, for example.
In the end, try to limit the game.js file to the core of the game framework (init function, mainloop, game states, score, levels), and separate the rest into functional groupings, eg utils.js, sprites.js, collision.js, listeners.js, etc.

Let's do this together!

Start with a simple structure

First, create a game.html file that contains the actual HTML code:

game.html:

game.html extract!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.    <meta charset="utf-8">
5.    <title>Nearly a real game</title>
6.    <!-- External JS libs -->
7.    <script src="https://cdnjs.cloudflare.com/ajax/libs/howler/1.1.25/howler.min.js"></script>
8.    <!-- CSS files for your game -->
9.    <link rel="stylesheet" href="css/game.css">
10.   <!-- Include here all game JS files-->
11.   <script src="js/game.js"></script>
12. </head>
13. <body>
14.   <canvas id="myCanvas" width="400" height="400"></canvas>
15. </body>
16. </html>

Here is the game.css file (very simple):

1.  canvas {
2.     border: 1px solid black;
3.  }

Let's take the JavaScript code from the last JSBin example, save it to a file called game.js, and locate it in a subdirectory js under the directory where the game.html file is located. Similarly, we'll keep the CSS file in a css subdirectory:

Try the game: open the game.html file in your browser. If the game does not work, open devtools, look at the console, fix the errors, try again, etc. You may have to do this several times when you split your files and encounter errors.

Isolate the ball function constructor

Put the Ball constructor function in a js/ball.js file, include it in the game.html file, and try the game: oops, it doesn't work! Let's open the console:

Ball.js:

Function Ball extract!

1.  // constructor function for balls
2.  function Ball(x, y, angle, v, diameter) {
3.     ...
4.      this.draw = function () {
5.         ctx.save();
6.         ...
7.      };
8.      this.move = function () {
9.        ...
10.       this.x += calcDistanceToMove(delta, incX);
11.       this.y += calcDistanceToMove(delta, incY);
12.     };
13. }

Isolate the time based animation functions into a separate file

Hmmm... the calcDistanceToMove function is used here, but is defined in the game.js file, inside the GF object and will certainly raise an error... Also, the ctx variable should be added as a parameter to the draw method, otherwise it won't be recognized...

Just for fun, let's try the game without fixing this, and look at the devtools console:

Aha! The calcDistanceToMove function is indeed used by the Ball constructor in ball.js at line 27 (it moves the ball using time-based animation). If you look carefully, you will see that it's also used for moving the monster, etc. In fact, there are parts in the game framework related to time-based animation. Let's move them all into a timeBasedAnim.js file!!

Fix: extract the utility functions related to time-based animation and add a ctx parameter to the draw method of ball.js. Don't forget to add it in game.js where ball.draw() is called. The call should be now ball.draw(ctx); instead of ball.draw() without any parameter.

timeBasedAnim.js:

JavaScript code extract!

1.  var delta, oldTime = 0;
2.   
3.  function timer(currentTime) {
4.     var delta = currentTime - oldTime;
5.     oldTime = currentTime;
6.     return delta;
7.  }
8.   
9.  var calcDistanceToMove = function (delta, speed) {
10.    //console.log("#delta = " + delta + " speed = " + speed);
11.    return (speed * delta) / 1000;
12. };

Isolate the part that counts the number of frames per second

We need to add a small initFPS function for creating the <div> that displays the FPS value... this function will be called from the GF.start() method. There was code in this start method that has been moved into the initFPS function we created and added into the fps.js file.

fps.js

fps.js extract!

1.  // vars for counting frames, used by the measureFPS function
2.  var frameCount = 0;
3.  var lastTime;
4.  var fpsContainer;
5.  var fps;
6.   
7.  var initFPSCounter = function() {
8.     // adds a div for displaying the fps value
9.     fpsContainer = document.createElement('div');
10.    document.body.appendChild(fpsContainer);
11. }
12. 
13. var measureFPS = function (newTime) {
14.  
15.    // test for the very first invocation
16.    if (lastTime === undefined) {
17.       lastTime = newTime;
18.       return;
19.    }
20.  
21.    //calculate the difference between last & current frame
22.    var diffTime = newTime - lastTime;
23.  
24.    if (diffTime >= 1000) {
25.       fps = frameCount;
26.       frameCount = 0;
27.       lastTime = newTime;
28.    }
29.  
30.    //and display it in an element we appended to the
31.    // document in the start() function
32.    fpsContainer.innerHTML = 'FPS: ' + fps;
33.    frameCount++;
34. };

At this stage, the structure looks like this:

Let's continue and isolate the event listeners

Now, consider the code that creates the listeners, can we move it from the GF.start() method into a listeners.js file? We'll have to pass the canvas as an extra parameter (to resolve a dependency) and we also move the getMousePos method into there.

listeners.js:

JavaScript code extract!

function addListeners(inputStates, canvas) {
   //add the listener to the main, window object, and update the states
   window.addEventListener('keydown', function (event) {
      if (event.keyCode === 37) {
         inputStates.left = true;
      } else if (event.keyCode === 38) {
         inputStates.up = true;
      } ...
   }, false);
 
   //if the key is released, change the states object
   window.addEventListener('keyup', function (event) {
      ...
   }, false);
 
   // Mouse event listeners
   canvas.addEventListener('mousemove', function (evt) {
      inputStates.mousePos = getMousePos(evt, canvas);
   }, false);
...
}
 
function getMousePos(evt, canvas) {
   ...
}

Isolate the collision tests

Following the same idea, let's put these into a collisions.js file:

// We can add the other collision functions seen in the
// course here...

// Collisions between rectangle and circle
function circRectsOverlap(x0, y0, w0, h0, cx, cy, r) {
   ...
}
 
function testCollisionWithWalls(ball, w, h) {
   ...
}

We added the width and height of the canvas as parameters to the testCollisionWithWalls function to resolve dependencies. The other collision functions (circle-circle and rectangle-rectangle) presented during the course, could be put into this file as well.

Final downloadable version and conclusion

After all that, we reach this tidy structure:

Final game.html file:

Final game game.html!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.  <meta charset="utf-8">
5.  <title>Nearly a real game</title>
6.  <link rel="stylesheet" href="css/game.css">
7.  <script src="https://cdnjs.cloudflare.com/ajax/libs/howler/1.1.25/howler.min.js"></script>
8.  <!-- Include here all JS files -->
9.  <script src="js/game.js"></script>
10. <script src="js/ball.js"></script>
11. <script src="js/timeBasedAnim.js"></script>
12. <script src="js/fps.js"></script>
13. <script src="js/listeners.js"></script>
14. <script src="js/collisions.js"></script>
15. </head>
16. <body>
17. <canvas id="myCanvas" width="400" height="400"></canvas>
18. </body>
19. </html>

We could go further by defining a monster.js file, turning all the code related to the monster/player into a well-formed object, with draw and move methods, etc. There are many potential improvements you could make. JavaScript experts are welcome to make a much fancier version of this little game :-)

Download the zip for this version, just open the game.html file in your browser!

Our intent this week was to show you the primary techniques/approaches for dealing with animation, interactions, collisions, managing with game states, etc.

The quizzes for this week are not so important. We're keen to see you write your own game! You are welcome to freely re-use the examples presented in the lessons and modify them, improve the code structure, playability, add sounds, better graphics, more levels, etc. We like to give points for style and flair, but most especially because we've been (pleasantly) surprised!

2.7.3 Discussion and projects

Here is the discussion forum for this part of the course. Please either post your comments/observations/questions or share your creations.

Suggested topics of discussion:

What additional content would you like to see in this part of the course?
Did you notice we gave you (without saying) a small particle engine during Week 1? This is very useful for simulating explosions or bullet impacts...

Optional projects: create your own game!

Try to make your own game, either by modifying/completing the given example (change the scenario, add levels, colors, sound effects, sprites, shots, etc.) or by writing your own using the methods explained in the course.
To inspire you, here are some examples written by Michel's students. Some of them are based on the framework presented in the course.

The Wanted: escape the Police and steal as many gold coins as you can! This game did not really follow the game framework presented in the course, but the final game is fun and is an interesting variation on examples presented in the course.

Duck Hunt (by Benjamin Reale and Anthony Caron). Get the whole game at this GitHub link + just grab the sprite sheet and sounds in this archive and try to make the game yourself (500k). This one started with the game framework presented in the course.

Shoot em up (by Max Chazarra) (sources on the GitHub site). This one started with the game framework presented in the course:

Gameboy Tetris (written starting from the game framework presented in the course). Download the sources (author Maxime Demetrio)

A small "kill the ninjas" game (by Tristan Poilvet). This game used the game framework presented in the course as a starting point. Sources available here.

Module 3. HTML5 File Upload and Download

Hi. This week, we will talk about the new version of Ajax called XHR2 for XMLHttpRequest Level 2 that plays an important role in today's modern complex Web apps.

XHR2 introduces new capabilities like cross-origin requests, uploading progress events, and support for uploading/downloading binary data.

You will learn how to download and upload files using XHR2, monitoring the upload/download progress, etc.

The course will provide full examples of HTML5 forms that handle files with client and server side code provided.

Drag’n’drop will be covered as well, between elements inside an HTML document but also with files from and to the desktop.

The second part of the week will show you how to use IndexedDB, a powerful NoSQL database located in your browser, client side. Many examples will be provided that will help you create, research, update and delete data in that database.

We encourage you to use IndexedDB for saving and restoring the high scores

in the small game you have certainly wrote during week 2. Enjoy!

3.2.1 Ajax and XHR2

We present below a short history of Ajax: an introduction to XMLHttpRequest level 2 (XHR2).

Wikipedia definition: "Ajax, short for Asynchronous JavaScript and XML), is a group of interrelated Web development techniques used on the client-side to create asynchronous Web applications. With Ajax, Web applications can send data to and retrieve from a server asynchronously (in the background) without interfering with the display and behavior of the existing page. Data can be retrieved using the XMLHttpRequest object. Despite the name, the use of XML is not required (JSON is often used), and the requests do not need to be asynchronous."

Ajax appeared around 2005 with Google Maps, and is now widely used. We are not going to teach you Ajax programming, but instead focus on the relationships between "the new version of Ajax", known as XHR2 (for XmlHttpRequest level 2) and the File API (seen in the W3Cx HTML5 Coding Essentials and Best Practices MOOC). Also, you will discover that the HTML5 <progress> element is of great use for monitoring the progress of file uploads (or downloads).

We recommend reading this article from HTML5Rocks.com that presents the main features of XHR2.

Briefly, these improvements include:

New, easier to use syntax,
In-browser encoding/decoding of binary files,
Progress monitoring of uploads and downloads.

The following sections of this course present a few examples of file downloads/uploads together with the file API and show how to monitor progress.

The current support of XHR2 is excellent: see related CanIUse's browser compatibility table.

3.2.2 Ajax/XHR2 and binary files

Hi everyone! In this lesson, I will present you how we can manage file uploads and file downloads with nice visualization of the progression.

In this example, I'm downloading directly binary files into the browser and as you will notice, we've got some progression bars that move in real time.

Earlier, before XHR2, the new version of Ajax, it was possible to do this but we had to ask at regular intervals of time the cerver. « Hey server! how many bytes did you receive from my Browser? » Now, monitoring the progression is made much easier because we've got progression callbacks that are called by the browser while it's uploading or downloading a file.

This small multitrack player I've already shown during the week one, about Web Audio, is downloading binary files, monitoring progress, and so on.

Let's start with a very simple example.

Here I will download a song that is located on one of my servers here...

mainline.i3s.unice.fr

When I will click on the download (button) and play the example song, it will start the download.

Let's look at the downloadSoundFile function, that is called when we click on the button.

How do we use XHR2?

We first start by creating an object that is an XML HTTP request.

We set the method: GET is for downloading a file, the URL, and the last parameter "true", leave it like that.

Then, as we are going to download a binary file, we set the response type to "arrayBuffer", and this is new. It was not possible with the previous version of Ajax, that was available in the browsers. HTTP is a text based transfer protocol.

The text encoded files needed to be decoded in the browser or in the server.

And when we manually, programmatically, in the JavaScript code, used Ajax, we had two decode this by hand.

Now, we ask the browser: "please, decode this for me" and give me directly, automatically, a binary file.

We prepared the request here, then we send the request: xhr.send() will send asynchronously the request.

That means that the server will handle this in the background.

This message "Ajax request sent...", will be displayed as soon as this instruction is executed.

And if the file we are going to download is big, is large, then it can take from dozens of seconds, or maybe one minute.

And when the file is arrived the callback, xhr.onload callback, will be called by the browser.

Then, we call the initSound() function that will decode in memory the mp3 file, and then enable the start and stop buttons, so that we can play the song using Web Audio. Before looking at the initSound (function) let's start it. I click…

You can see that the message "Ajax request sent" is displayed first, then "song downloaded".

This is on the onload callback, and then we call the initSound function that will decode the file and display the rest of the messages.

And now I can play the song.

In this example, we are using Web Audio: it's not streaming the sound.

It’s playing a sample directly in memory so… you can give a look at the code.

It uses what we saw during the first week.

The playSound function builds a small graph with the decoded buffer as a source and directly to the speakers (to the destination).

If you want to monitor progress during download (so, this is the same example), it's very easy.

Just use a progress HTML element with a value of zero, and it will grow depending on the number of bytes the browser will have sent to the remote server.

Or will have downloaded... in case of a download.

When we started the request, we added a xhr.onprogress callback here that gets in event sent by the browser, that has two properties.

One is called "total", that is a total number of bytes in the files we are downloading, and "loaded" is the number of bytes we have actually downloaded.

And we can set these two properties directly to the "value" property of the progress element and to the "max" property.

And if we try this (download the file), we can see the progress bar growing because this onprogress callback is called regularly by the browser every second or something like that.

It's very easy to monitor the progress.

You can put vertically this bar using CSS, change the style.

Refer to the HTML5 Part 1 course for that.

Ajax and binary files - downloading files and monitoring progress

HTTP is a text based protocol, so when you upload/download images, videos or any binary file, they must first be text encoded for transmission, then decoded on-the-fly upon receipt by the server or browser. For a long time, when using Ajax, these binary files had to be decoded "by hand", using JavaScript code. Not recommended!

We won't go into too much detail here, but all browsers (> 2012) support XHR2. XHR2 adds the option to directly download binary data. With XHR2, you can ask the browser to encode/decode the file you send/receive, natively. To do this, when you use XMLHttpRequest to send or receive a file, you must set the xhr.responseType as arrayBuffer.

Below is a function that loads a sound sample using XMLHttpRequest level 2.

Note: 1) the simple and concise syntax, and 2) the use of the new arrayBuffer type for the expected response (line 5):

1.  // Load a binary file from a URL as an ArrayBuffer.
2.  function loadSoundFile(url) {
3.  var xhr = new XMLHttpRequest();
4.  xhr.open('GET', url, true);
5.  xhr.responseType = 'arraybuffer';
6.  xhr.onload = function(e) {
7.  initSound(this.response); // this.response is an ArrayBuffer.
8.  };
9.  xhr.send();
10. }

Example: download a binary song file using XHR2 and responseType='arraybuffer', and play it using Web Audio

Try this example on JSBin:

In this example, instead of reading the file from disk, we download it using XHR2.

Complete source code:

Complete source code!

1.  <!DOCTYPE html>
2.  <html lang="en">
3.  <head>
4.    <title>XHR2 and binary files + Web Audio API</title>
5.  </head>
6.  <body>
7.    <p>Example of using XHR2 and <code>xhr.responseType =
        'arraybuffer';</code> to download a binary sound file
8.      and start playing it on user-click using the Web Audio API.</p>
9.  
10.   <p>
11.   <h2>Load file using Ajax/XHR2 and the arrayBuffer response
        type</h2>
12.   <button onclick="downloadSoundFile('https://myserver.com/song.mp3');">
13.     Download and play example song.
14.   </button>
15.   <button onclick="playSound()" disabled>Start</button>
16.   <button onclick="stopSound()" disabled>Stop</button>
17.   <script>
18.     // WebAudio context
19.     var context = new window.AudioContext();
20.     var source = null;
21.     var audioBuffer = null;
22.  
23.     function stopSound() {
24.       if (source) {
25.         source.stop();
26.       }
27.    }
28.  
29.     function playSound() {
30.       // Build a source node for the audio graph
31.       source = context.createBufferSource();
32.       source.buffer = audioBuffer;
33.       source.loop = false;
34.       // connect to the speakers
35.       source.connect(context.destination);
36.       source.start(0); // Play immediately.
37.     }
38.  
39.     function initSound(audioFile) {
40.       // The audio file may be an mp3 - we must decode it before
          playing it from memory
41.       context.decodeAudioData(audioFile, function(buffer) {
42.       console.log("Song decoded!");
43.       // audioBuffer the decoded audio file we're going to work with
44.       audioBuffer = buffer;
45. 
46.       // Enable all buttons once the audio file is
47.       // decoded
48.       var buttons = document.querySelectorAll('button');
49.       buttons[1].disabled = false; // play
50.       buttons[2].disabled = false; // stop
51.       alert("Binary file has been loaded and decoded, use play /
            stop buttons!")
52.       }, function(e) {
53.        console.log('Error decoding file', e);
54.       });
55.     }
56.  
57.     // Load a binary file from a URL as an ArrayBuffer.
58.     function downloadSoundFile(url) {
59.       var xhr = new XMLHttpRequest();
60.       xhr.open('GET', url, true);
61.  
62.       xhr.responseType = 'arraybuffer'; // THIS IS NEW WITH HTML5!
63.       xhr.onload = function(e) {
64.         console.log("Song downloaded, decoding...");
65.         initSound(this.response); // this.response is an ArrayBuffer.
66.       };
67.     xhr.onerror = function(e) {
68.       console.log("error downloading file");
69.     }
70.  
71.     xhr.send();
72.       console.log("Ajax request sent... wait until it downloads
          completely");
73.     }
74.   </script>
75. </body>
76. </html>

Explanations

:

Line 12: a click on this button will call the downloadSoundFile function, passing it the URL of a sample mp3 file.
Lines 58-73: this function sends the Ajax request, and when the file has arrived, the xhr.onload callback is called (line 63).
Lines 39-55: The initSound function decodes the mp3 into memory using the WebAudio API, and enables the play and stop buttons.
When the play button is enabled and clicked (line 15) it calls the playSound function. This builds a minimal Web Audio graph with a BufferSource node that contains the decoded sound (lines 31-32), connects it to the speakers (line 35), and then plays it.

Monitoring uploads or downloads using a progress event

1 - Declare a progress event handler

XHR2 now provides progress event attributes for monitoring data transfers. Previous implementations of XmlHttpRequest didn't tell us anything about how much data has been sent or received. The ProgressEvent interface adds 7 events relating to uploading or downloading files.

attribute	type	Explanation
onloadstart	loadstart	When the request starts.
onprogress	progress	While loading and sending data.
onabort	abort	When the request has been aborted, either by invoking the abort() method or navigating away from the page.
onerror	error	When the request has failed.
onload	load	When the request has successfully completed.
ontimeout	timeout	When the author specified timeout has passed before the request could complete.
onloadend	loadend	When the request has completed, regardless of whether or not it was successful.

The syntax for declaring progress event handlers is slightly different depending on the type of operation: a download (using the GET HTTP method), or an upload (using POST).

Syntax for download:

Download syntax for declaring progress!

1.  var xhr = new XMLHttpRequest();
2.  xhr.open('GET', url, true);
3.  ...
4.  xhr.onprogress = function(e) {
5.  // do something
6.  }
580. 
<!-- -->
7.  xhr.send();

Note that an alternative syntax such as xhr.addEventListener('progress', callback, false) also works.

Syntax for upload:

Upload syntax for declaring progress!

1.  var xhr = new XMLHttpRequest();
2.  xhr.open('POST', url, true);
3.  ...
4.  xhr.upload.onprogress = function(e) {
5.  // do something
6.  }
7.   
8.  xhr.send();

Notice that the only difference is the "upload" added after the name of the request object: with GET we use xhr.onprogress and with POST we use xhr.upload.onprogress.

Note that an alternative syntax such as xhr.upload.addEventListener('progress', callback, false) also works.

2 - Get progress values (how many bytes have been downloaded) and the total file size

The event e passed to the onprogress callback has two pertinent properties:

loaded which corresponds to the number of bytes that have been downloaded or uploaded by the browser, so far, and
total which contains the file's size (in bytes).

Combining these with a <progress> element, makes it very easy to render an animated progress bar. Here is a source code extract that does this for a download operation:

HTML

<progress id="downloadProgress" value=0><progress>

Script:

JavaScript code!

1.  // progress element
2.  var progress = document.querySelector('#downloadProgress');
3.   
4.  function downloadSoundFile(url) {
5.    var xhr = new XMLHttpRequest();
6.    xhr.open('GET', url, true);
7.   
8.    ...
9.    xhr.onprogress = function(e) {
10.     progress

bbauska/W3Cx-4of5

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages