Live Stream on Win10 & Edge/IE11 starts without sound

[oomkat]

What version of Hls.js are you using?

0.12.4

What browser and OS are you using?

Win10 & Edge or IE11

Test stream:

https://hls-js.netlify.com/demo/?src=https%3A%2F%2Fmcdn.daserste.de%2Fdaserste%2Fde%2Fmaster.m3u8&demoConfig=eyJlbmFibGVTdHJlYW1pbmciOnRydWUsImF1dG9SZWNvdmVyRXJyb3IiOnRydWUsImVuYWJsZVdvcmtlciI6dHJ1ZSwiZHVtcGZNUDQiOmZhbHNlLCJsZXZlbENhcHBpbmciOi0xLCJsaW1pdE1ldHJpY3MiOi0xLCJ3aWRldmluZUxpY2Vuc2VVcmwiOiIifQ==

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client on https://hls-js.netlify.com/demo and not just on my page

• The issue occurs in the latest client on https://hls-js-latest.netlify.com/demo and not just on my page

• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Please provide clear steps to reproduce your problem
   Open the demo page on a Win10 system in Edge or IE11

2. If the bug is intermittent, give a rough frequency
   Happens every time

Expected behavior

Livestream plays with video and audio

Actual behavior

Livestream plays without audio hearable
But:

• If you seek a bit backward sound appears to be hearable
• If you plug in a headset (if none is already plugged in. In this case just unplug it) and the sound will be hearable

[tchakabam]

We have already reported this here: #2286
I think this is a duplicate :) Good to see we are not the only one to observe this. Not sure if this isn't actually an OS/browser related bug.

[tchakabam]

If you plug in a headset (if none is already plugged in. In this case just unplug it) and the sound will be hearable

Hmm, that sounds like something very OS related. Hls.js doesn't do anything at the level of handling various output device types like headset or other audio destinations. Did you raise a ticket with Microsoft?

[oomkat]

Thanks for the Feedback @tchakabam!
I am not sure this is really OS related because theoplayer (https://demo.theoplayer.com/test-your-stream-with-statistics) and VideoJS (https://videojs.github.io/videojs-contrib-hls/) and even G&L Streamtester (http://playback.gl-systemhaus.de/player/streamtester/?gl--src=https%3A%2F%2Fmcdn.daserste.de%2Fdaserste%2Fde%2Fmaster.m3u8&gl--type=application%2Fx-mpegURL&gl--player=gundl&gl--livestream=on) - which is based on an older version of hls.js - work without any problems on the same machine in Edge.

[tchakabam]

@oomkat Ok understood. That's very interesting then! Let's investigate further. Can you backtrace to which version of Hls.js you can make it work, so we could figure what was the breaking change (i.e which one is actually used in G&L Streamtester?)

That would be the easiest approach to quickly come to a guess where to look for the problem I would say.

It most likely has something to do with MediaSource ingested fMP4 data-muxing, that's really the only plausible explanation if it comes to audio being played or not to me.

However it is funny that the headphone thing makes it work ... Curious to find out!

[oomkat]

G&L Streamtester is based on version 10.1

[tchakabam]

presuming you mean v0.10.1. A possible starting point to investigate further when the breaking change came. A lot of stuff came with v0.11.x concerning things that might have caused this to break in MS browsers. Thanks for the hint :)

[gogonowski]

Live AAC Audio-Only ES and fMP4 HLS Streams that exhibit the same No Audio issue in MS Edge:
https://db2.indexcom.com/playertest/hlsjs4
There are 4 streams in the pull-down. They are titled accordingly. The live streams do not produce audio in MS Edge, however the file streams do. This might be an AAC encoder mode issue, that hls.js should be able to handle. We will check this soon. These streams play in Chrome and Firefox. These streams also work in Edge natively, if you enter the stream URL (not hls.js Player URL) directly in Edge. This is indeed a very strange issue, since it only affects MS Edge.
Thanks in advance for looking.

Also, off topic for this post, and should probably be another, but these four streams also contain fully HLS-compliant id3v2.4 metadata. We would like to see support for fMP4 emsg id3v2.4 metadata in hls.js. Please contact me for those details. A few years ago we supplied the details for ES (Elementary Stream) id3v2.4 metadata which hls.js already supports. This metadata can all be verified by playing the HLS streams in iTunes.

[johnBartos]

@gogonowski That sounds like a problem with MSE in edge - when given fmp4 media, Hls.js passes them through to MSE with little modification. Can you test your stream in other players to see if they can play them?

We would like to see support for fMP4 emsg id3v2.4 metadata in hls.js.
I have this planned, but only after 1.0.0 is released (which will be in the fall).

[mangui]

the only tweak we make on the incoming fmp4 is adjusting fragment start DTS so that we control the append offset.
https://github.com/video-dev/hls.js/blob/master/src/demux/mp4demuxer.js#L349

this adjustment was introduced in https://github.com/video-dev/hls.js/releases/tag/v0.7.10
through c3ceee6#diff-d25242b8b1f3eba1408b83df5655c15a
to fix #1177

it would be interesting to compare 0.7.9 and 0.7.10 to see if it is coming from this adjustment.

[johnBartos]

@mangui Could be - but if we were writing the timestamp incorrectly it'd still buffer, but at the wrong time. The stream would stall because the A/V buffers don't overlap. Maybe Edge drops the samples when we append but sounds more likely that we're not even declaring an audio SB.

Nevermind, I forgot we only declare 1 SB for fmp4 streams, but if we messed up the timestamps I don't think that it would even play.

Going to grab an Edge laptop and give it a go.

[michaelcunningham19]

Chiming in here to call out that I've seen this behaviour before a while ago (and forgot about it unfortunately)

Can confirm that our solution was to perform a small seek operation immediately after playback begins

[michaelcunningham19]

Can confirm that our solution was to perform a small seek operation immediately after playback begins

Some more details:

• the seek operation was simply a seek-to-live-edge operation, as we were starting at the live edge anyways, it made sense as it was less-janky that way
• we encountered an unrelated need for using the liveBackBufferLength functionality introduced in 0.12.x, and discovered after the upgrade process (0.11.0 -> 0.12.2) that this audio-specific issue was resolved without the seek-to-live workaround.

[johnBartos]

The behavior I'm seeing is that time is advancing, but no audio plays. I tested the stream in my 1.0.0 branch and the audio plays fine; I did not modify the offsetStartDTS function so I don't believe that's the problem. It's creating the SB with the same codec too. Hard to tell what the problem is at a glance.

[gogonowski]

Thanks for the quick response on this.
And yes, nudging the play head starts the audio.
The interesting thing is why this only affects the live streams and not the file streams in MS Edge only. They were created with two different AAC Encoders.
We are going to test different AAC Encoder parameters to see if this is the difference.
We will report the findings.

[tchakabam]

@mangui @johnBartos The stream we see the issue with uses MPEG-TS segments. And concerning changes causing the issue, keep in mind this isn't observable in version as new as v0.10.1 !

We are working on a fix atm to look at the root cause most probably related to the diff between v0.10.1 and latest.

A workaround is indeed to nudge the playhead, but we would like to understand the actual change causing Edge to drop all audio frames apparently.

[tchakabam]

The issue is getting fixed by reverting this chunk ok a PR that was supposed to fix PTS compensation: https://github.com/video-dev/hls.js/pull/2030/files#diff-92ec9c117a188d62ae4becd0540933d6L1370

EDIT: This was part of the v0.12.0 release.

@johnBartos @michaelcunningham19 Could you still remember why this was removed? Wasn't it out of scope of the initial PR intention (i.e not related to any remuxing and timestamp application to samples) but rather a consequence that Gap-controller should take care of this?

EDIT: AFAIR this was removed because this kind of playhead nudging is supposed to be done by Gap-controller.

What I found out here is that the stream where this is failing (see above, to access, please use Germany located proxy to access, for example via a VPN provider) is one where the segments buffered first by Hls.js have their audio segments having their first PTS approx 10-20 ms (min/max variable) after the first PTS of the video stream.

It seems that Edge shows a bug in this situation where video frames are available in the buffers timelines before audio. It will then play the video frames and even when audio frames become available at the played position it seems to just drop or ignore them. It's unclear wether they get decoded, but at least they don't get rendered/played-out.

To be clear, this is a bug in Edge obviously (or usually other browsers will just not start at all from the requested position or skip the audio gap on their own, but dropping audio all together seems defective to me). But this issue will not show if we handle the audio gap on our side by nudging the playead to the first A/V syncable position (and Edge allows us to do that via its buffered time-ranges output). Which is something that Gap-controller should do.

We also see that Edge would advertise buffered time-ranges in a way that take respect of the A/V syncability (frames overlaying in fact). So the first buffered range corresponds to the audio part buffered. When we check that with the chunk of code above, we just step forward right to where audio is actually buffered again.

This kind of playhead nudging should be done by gap-controller. So something is wrong there I would assume, as I had indicated in a previous still WIP PR of mine. I would try to further resolve it there.

I wouldn't know how else than with playhead nudging this could be resolved. We can not cut away the video frames since we would have to do that up to the next key-frame (IDR sample) eventually. And that would affect seeking accuracy / random access as well as lead to loading data on startup which would not get effectively buffered and played out.

Let me know your thoughts, or wether you think we missed something here! Thanks!

[tchakabam]

@OrenMe I am working on a nicer fix in case you are interested ^ I.e reverting the above chunk might already help you, or actually add that to Kaltura instead of a magic number ;)

[tchakabam]

@robwalch @itsjamie Please add this PR (#2359) fixing the issue here to v0.13 milestone (this issue-ticket has already been added to the milestone).

[robwalch]

I'm testing this on Edge Windows 10 and the stream doesn't even start. It also does not appear to be fixed by #2359.

What I am seeing is that something just causes the parent manifest to be reloaded (maybe a retry but I haven't identified an error) so all we see if buffering and eventually Edge runs out of memory.

@oomkat When I request the child manifests in Chrome, they return a proper playlist, but when they are requested in Edge or I curl -L them, the response is always the content of the master manifest. Do you have a valid test stream or a way to get around this behavior?

[oomkat]

@robwalch
I can not reproduce the issue. curl -L on child manifests are correctly returning the child manifests.