For guidance and examples on how to use the Amazon Chime SDK for JavaScript in your application, see API overview for more information. - Give feedback on this guide
+For guidance and examples on how to use the Amazon Chime SDK for JavaScript in your application, see API overview for more information.
+Amazon Chime SDK for JavaScript allows builders to choose a downlink policy on each device when joining a meeting. Downlink policies control the way how a recipient subscribes to the remote video sources. This document explains how to use the VideoPriorityBasedPolicy introduced in version 2.8, how it differs from the existing policies and how builders can take advantage of it to meet their use cases.
Amazon Chime SDK for JavaScript defines VideoPriorityBasedPolicy as the central class providing the ability to request remote video sources to receive and set their respective priorities. The policy ensures bandwidth is reserved for video sources with higher priorities. This policy can be used in conjunction with clients sending simulcast renditions of sources, as well as single stream or mixed environments.
All new users should enable server side network adaptation features for vastly improved behavior. When server side network adapatation is used, all browsers are supported. Otherwise since previously the policy uses WebRTC's available downlink bandwidth estimation, which is not supported Firefox, it should not be used there.
-The existing AllHighestVideoBandwidthPolicy and AdaptiveProbePolicy both subscribe to a fixed number of available remote video sources. Builders listen to videoTileDidUpdate when a new remote video source becomes available and call pauseVideoTile if they don’t want it to be seen, otherwise video sources will be consumed automatically. VideoPriorityBasedPolicy relies on builders subscribing to AudioVideoObserver.remoteVideoSourcesDidChange to receive updates on available remote sources, and then calling chooseRemoteVideoSources to set which video sources to receive and their related preferences. Note that when using VideoPriorityBasePolicy, if builders do not call chooseRemoteVideoSources, no videos will be subscribed to. videoTileDidUpdate will then be called if we are able to successfully subscribe to the stream.
Under constrained networks where simulcast is in use, VideoPriorityBasedPolicy may lower the resolution of remote video sources, starting with the lowest priority sources. All video sources are separeted into multiple groups by different priorities. If all video sources within same priority group are at the lowest resolution possible, or simulcast is not being used, the policy may further pause video tiles until the network has recovered. Same operations will be repeated group by group, from priority lowest to highest.
Amazon Chime SDK for JavaScript allows builders to choose a downlink policy on each device when joining a meeting. Downlink policies control the way how a recipient subscribes to the remote video sources. This document explains how to use the VideoPriorityBasedPolicy introduced in version 2.8, how it differs from the default behavior, and how builders can take advantage of it to meet their use cases.
Amazon Chime SDK for JavaScript defines VideoPriorityBasedPolicy as the central class providing the ability to request remote video sources to receive and set their respective priorities. The policy ensures bandwidth is reserved for video sources with higher priorities. This policy can be used in conjunction with clients sending simulcast renditions of sources, as well as single stream or mixed environments. All browsers are supported.
The existing AllHighestVideoBandwidthPolicy automatically subscribes to a fixed number of available remote video sources. VideoPriorityBasedPolicy relies on builders subscribing to AudioVideoObserver.remoteVideoSourcesDidChange to receive updates on available remote sources, and then calling chooseRemoteVideoSources to set which video sources to receive and their related preferences. Note that when using VideoPriorityBasePolicy, if builders do not call chooseRemoteVideoSources, no videos will be subscribed to. videoTileDidUpdate will then be called if we are able to successfully subscribe to the stream.
It is recommended to enable simulcast or SVC to fully take advantage of the network adaptation capabilities, and to avoid the need to fully pause videos unless under extreme network conditions. When simulcast is in use, VideoPriorityBasedPolicy may lower the resolution of remote video sources in response to network constraints, starting with the lowest priority sources. All video sources are separated into multiple groups by different priorities. If all video sources within same priority group are at the lowest resolution possible, or simulcast is not being used, the policy may further pause video tiles until the network has recovered. Same operations will be repeated group by group, from priority lowest to highest.
If you want to keep the last frame of the video stream when paused due to insufficient bandwidth, please set MeetingSessionConfiguration.keepLastFrameWhenPaused to true.
A typical workflow to use this policy would be:
meetingSession.configuration, construct and set a VideoPriorityBasedPolicy object as the videoDownlinkBandwidthPolicymeetingSession.configuration, construct and set a VideoPriorityBasedPolicy object as thevideoDownlinkBandwidthPolicyAudioVideoObserver.remoteVideoSourcesDidChange to receive updates on available sources.VideoPreference for each video stream and then call chooseRemoteVideoSourcesAudioVideoObserver.remoteVideoSourcesDidChange or other application events (like switching the current page of videos).The default preference set the same priority 1 for all attendees. The target size will be:
-The default preference set the same priority 1 for all attendees.
The main API being used is:
-chooseRemoteVideoSources(sources: VideoPreference[]): void
+ chooseRemoteVideoSources(sources: VideoPreference[]): void
- An array of VideoPreference is passed in to specify preference for each remote video.
- Be aware of the following when calling this function:
+ An array of VideoPreference is passed in to specify preference for each remote video. When calling this function:
- You must pass a complete array of VideoPreference objects each time you call chooseRemoteVideoSources. Each call overrides the behavior defined by any previous call. The AudioVideoController will unsubscribe from any existing video streams associated with attendee IDs that are not present in the new array. Do not modify the provided array after calling this function; for efficiency, the policy object does not clone the array.
- Including a remote video stream in this list does not guarantee that the video will be received. If there is insufficient bandwidth then the lowest priority video(s) will be paused and the client will be notified.
- - A video tile will be created for all requested video streams. If there is insufficient bandwidth to receive the video then the policy will pause it and provide notification with a reason. This is a difference in behavior from the current
AdaptiveProbePolicy which removes the tile for this condition.
+ - A video tile will be created for all requested video streams. If there is insufficient bandwidth to receive the video then the policy will pause it and provide notification with a reason.
- Content share is treated as another video stream**:** Builders will need to specify content video and its priority as well. It will not be auto subscribed to anymore when using priority based downlink policy.
- Any newly added remote videos (e.g. a new
VideoPreference was added to the list) to the list will result in a videoTileDidUpdate event, and any removed remote videos (e.g. a VideoPreference was removed from the list, or an attendee left the meeting) will trigger a videoTileWasRemoved event.
- Any preferences entries associated with attendee IDs that are paused by the application or no longer available in the call will be ignored until unpaused.
- The policy may limit how often it updates the list of actual streams it subscribes to.
- (Note that the exact internal behavior of this policy may slightly change in future releases)
Receiving notifications that a remote video was paused due to bandwidth constraint
If a remote video source is paused due to insufficient bandwidth, then the application will be notified through tileWillBePausedByDownlinkPolicy by VideoDownlinkObserver and the VideoTileState will be set to paused.
-
- Enabling Server Side Network Adaption Features
-
- VideoPriorityBasedPolicy has support for enabling additional features on the backend related to network adaptation. These features lead to the following improvements relative to the default policy:
-
- - Faster network constraint response time, e.g. compared to default it will downgrade and pause videos faster to prioritize audio.
- - Avoidance of video upgrades that cause network overuse. The server side adaptation uses pacing and probing to determine headroom before upgrading videos.
- - Stable video quality selections under constraint due to a backend calculated bandwidth estimation utilizing transport wide congestion control.
- - Faster recovery time due to use of pacing and probing on backend.
-
- These features are currently opt-in but may become defaults in the future. To enable these features, set the following on VideoPriorityBasedPolicyConfig before injecting:
- const config = new VideoPriorityBasedPolicyConfig;
-config.serverSideNetworkAdaption =
- ServerSideNetworkAdaption.BandwidthProbingAndRemoteVideoQualityAdaption;
-let priorityBasedDownlinkPolicy =
- new VideoPriorityBasedPolicy(this.meetingLogger, config);
-
Builder Code Sample
Below, we are going to show one potential use case, “featured video”, that can be built on top of this policy.
A video tile can be featured by clicking a “Pin” button on it and its priority will be set to 1 whereas the rest unpinned tiles are set to 2 by default. The pinned tile will be displayed in a larger size and will not be paused/downgraded (if simulcast uplink policy being enabled on the remote side) until all of the rest are when encountering network constraint.
First create an instance of VideoPriorityBasedPolicy that you will share with the meeting session. Then create a meeting session configuration and set videoDownlinkBandwidthPolicy to that created policy.
- // `meetingLogger: Logger` created elsewhere
+ // `meetingLogger: Logger` created elsewhere
const priorityBasedDownlinkPolicy = new VideoPriorityBasedPolicy(meetingLogger);
// `configuration: MeetingSessionConfiguration` created elsewhere
configuration.videoDownlinkBandwidthPolicy = priorityBasedDownlinkPolicy;
Once connected to the meeting, we can an observer on remoteVideoSourcesDidChange and keep track of if which clients in the roster list are publishing video.
- // Assumes we are storing roster in a member variable like `roster: any = {};`
+ // Assumes we are storing roster in a member variable like `roster: any = {};`
remoteVideoSourcesDidChange(videoSources: VideoSource[]) {
// Reset current hasVideo flag in roster list
for (const attendeeId in this.roster) {
@@ -200,7 +174,7 @@ Builder Code Sample
}
We can then add a pin button to our application, and wire it up to change the priority of the respective video. Update the tile button click callback to set or unset the pin status.
- pinButtonElement.addEventListener('click', () => {
+ pinButtonElement.addEventListener('click', () => {
// Get the attendeeID associated with this tile
const attendeeId = tileState.boundAttendeeId;
// Change pin state
@@ -213,8 +187,8 @@ Builder Code Sample
});
We can then write updateDownlinkPreference which is shared by the above two snippets. This logic simply sets featured tiles to priority 1 and all others to priority 2. This will ensure featured tiles are requested first at the highest resolution allowed by bandwidth and other participants are requested as bandwidth allows. To store the VideoPreference objects we use VideoPreferences, a set like data structure that allows efficient modification of an existing list while only copying when necessary.
- updateDownlinkPreference(): void {
- const videoPreferences: MutableVideoPreferences = VideoPreferences.prepare();
+ updateDownlinkPreference(): void {
+ const videoPreferences = VideoPreferences.prepare();
for (const attendeeId in this.roster) {
if (this.roster[attendeeId].hasVideo) {
// Make pinned videos of equal high importance and the rest as secondary
diff --git a/docs/modules/qualitybandwidth_connectivity.html b/docs/modules/qualitybandwidth_connectivity.html
index 84a53cdd7..992ecc00d 100644
--- a/docs/modules/qualitybandwidth_connectivity.html
+++ b/docs/modules/qualitybandwidth_connectivity.html
@@ -96,7 +96,6 @@ Quality, Bandwidth, and Connectivity
- Adjust local video quality
- Turning off local video
- Configure a video uplink policy
- - Pause remote attendee video
- Use active speaker detection
- Configure a video downlink policy
@@ -305,10 +304,6 @@ Turning off local video
Configure a video uplink policy
The SDK by default uses the NScaleVideoUplinkBandwidthPolicy which monitors number of participants in the meeting and automatically scales down the maxBandwidthKbps as the number of remote video tiles increases. This can be customized by implementing a VideoUplinkBandwidth Policy and setting it in the MeetingSessionConfiguration class.
-
- Pause remote attendee video
-
- When more video is being received than available estimated downlink bandwidth can support, the event videoNotReceivingEnoughData can is triggered with a list of attendee IDs and the bandwidth being consumed due to them. You can use this information to selectively pause attendees that are sending the highest bitrate video streams using pauseVideoTile. When a video tile is paused, the action only affects your client. It does not pause the video for other attendees.
Use active speaker detection
@@ -316,14 +311,13 @@ Use active speaker detection
Configure a video downlink policy
- By default the SDK uses the AllHighestVideoBandwidthPolicy, but VideoAdaptiveProbePolicy is used if simulcast is enabled. AllHighestVideoBandwidthPolicy subscribes to the highest quality video of all participants and VideoAdaptiveProbePolicy tries to subscribe to videos based on the browser downlink estimate, dropping videos in case of not enough bandwidth.
+ By default the SDK uses the AllHighestVideoBandwidthPolicy which will not adapt to local receive network conditions and always subscribes to the highest quality video of all participant, but VideoPriorityBasedPolicy will downgrade or pause videos in response to network constraints. Additional configuration of that policy is required to select which remote videos to subscribe to, see the guide for more information. Downgrades are only possible if simulcast or SVC is enabled.
You can customize the default downlink policy by setting the VideoDownlinkBandwidthPolicy in MeetingSessionConfiguration class.
- Browser clients currently only send one stream resolution. You would only need to use this function if you were also using the Amazon Chime SDK for iOS or the Amazon Chime SDK for Android.
Use video simulcast
- Simulcast is a standardized technique in WebRTC to enhance video quality in a multi-party call. Amazon Chime SDK for JavaScript provides a video simulcast solution for Chromium-based browsers. To use simulcast, please refer to the documentation here.
- Give feedback on this guide
+ Simulcast is a standardized technique in WebRTC to enhance video quality in a multi-party call. Amazon Chime SDK for JavaScript provides a video simulcast solution for Chromium-based browsers. To use simulcast, please refer to the documentation here.
+
In multi-party video calls, attendees can enable the simulcast feature to enhance the overall video quality. Simulcast is a standardized technique where the video publishers create multiple renditions, or layers, of the same video source and video subscribers have the flexibility to choose the rendition that best fits their needs based on such factors as available bandwidth, compute and screen size. The uplink policy controls the configuration of the renditions through camera capture and encoding parameters. The simulcast-enabled uplink policy is SimulcastUplinkPolicy.
Simulcast is currently disabled by default. To enable it MeetingSessionConfiguration.enableSimulcastForUnifiedPlanChromiumBasedBrowsers must be set. Currently, only Chrome 76 and above is supported.
-The VideoAdaptiveProbePolicy downlink policy adaptively subscribes to the best simulcast layer and is automatically selected if [MeetingSessionConfiguration.enableSimulcastForUnifiedPlanChromiumBasedBrowsers is set to true.
-If you want more fine-grained control of which simulcast layer to subscribe, please use VideoPriorityBasedPolicy. More details about priority-based downlink policy can be +
When enabling simulcast, you should use VideoPriorityBasedPolicy to allow switching between layers in response to application use-cases or network adaptation. More details about priority-based downlink policy can be found here.
The table entries represent the maximum configuration. When CPU and bandwidth consumption is overused, WebRTC will dynamically adjust bitrate, disable a layer or scale down resolution. The SimulcastUplinkPolicy has a monitoring mechanism which tracks the sending status and automatically adjusts without need for application intervention.
Note that simulcast is disabled when there are only 2 or fewer attendees. This is because WebRTC has additional functionality to request lower bitrates from the remote end, and we will forward these requests if there are no competing receivers (i.e. if the receiving estimates it has 200kbps downlink bandwidth available, this will be sent and relayed in a message to the sending client). Therefore there is no need for simulcast based adaption.
- -The VideoAdaptiveProbePolicy adds functionality to take advantage of video simulcast. The goal of this policy is to leave the complexity of managing the downlink bandwidth and decision of which streams to request inside the SDK and remove that burden from the application. The policy monitors numerous pieces of information and uses that to determine which streams to subscribe to. The data considered are:
-The policy works by listening to any changes in the information listed above. Anytime there is a change it re-evaluates the best mix of remote videos to fit within the target downlink bandwidth. The policy is currently configured to try to request as many remote clients as possible, then increase resolution where possible. It prioritizes screen share video above people video. If there is insufficient bandwidth to request all far participants then only a certain number will be seen.
-Applications will receive notifications of remote video being added or removed through the videoTileDidUpdate and videoTileWasRemoved callbacks. You can use encodingSimulcastLayersDidChange to know when the selected upstream simulcast resolutions change.
-If the application pauses a remote video by calling pauseVideoTile, then this policy will not include that remote video in the list of available streams and therefore will not automatically subscribe or unsubscribe from it while it is in the paused state.
-When using the policy it is recommended that applications do not take action on the downlink monitoring metrics from metricsDidReceive since it may conflict with decisions being made within the policy.
-The VideoAdaptiveProbePolicy can be used with or without simulcast. To enable it without simulcast set the MeetingSessionConfiguration.videoDownlinkBandwidthPolicy to VideoAdaptiveProbePolicy. If simulcast is not active, then the policy will only be able to add or remove remote videos.
configuration.enableSimulcastForUnifiedPlanChromiumBasedBrowsers = true;
-//Specify the apdative probe downlink policy
-configuration.videoDownlinkBandwidthPolicy = new VideoAdaptiveProbePolicy(logger);
+// Specify the apdative probe downlink policy
+configuration.videoDownlinkBandwidthPolicy = new VideoPriorityBasedPolicy(logger);
Now create a meeting session with the simulcast enabled meeting session configuration.
// In the examples below, you will use this meetingSession object.
@@ -214,7 +198,7 @@ Creating a simulcast enabled meeting
deviceController
);
This meetingSession is now simulcast enabled and will have the videoUplinkBandwidthPolicy set to DefaultSimulcastUplinkPolicy and videoDownlinkBandwidthPolicy set to VideoAdaptiveProbePolicy. Due to these policies, the local and remote video resolutions may change. The video resolution depends on the available simulcast streams. The available simlucast streams are dependent on the number of attendees and the current bandwidth estimations. Check "Simulcast resolutions and behavior" section in this guide for more information.
This meetingSession is now simulcast enabled and will have the videoUplinkBandwidthPolicy set to DefaultSimulcastUplinkPolicy. Due to these policies, the local and remote video resolutions may change. The video resolution depends on the available simulcast streams. The available simlucast streams are dependent on the number of attendees and the current bandwidth estimations. Check "Simulcast resolutions and behavior" section in this guide for more information.
VideoAdaptiveProbePolicy wraps VideoPriorityBasedPolicy with customized behavior to automatically - assign a high preference to content share.
+ assign a high preference to content share. This is a legacy policy used by default when simulcast is enabled, but + all customers should be usingVideoPriorityBasedPolicyinstead, since this policy does not work with + server side network adaptation.You can imitate this classes behavior using the
+VideoPriorityBasedPolicywith the following snippet