Block background throttling of video processing #2747
Conversation
|
Thank you for the development work. We have also encountered a similar issue in our service, as described by hunnorth. Furthermore, we have verified that the problem is resolved when launching the demo application on the "video-ml/release" branch. Could you please provide an estimate for when this release is expected? We would greatly appreciate it if you could release it as soon as possible. |
hensmi-amazon
force-pushed
the
video-ml/release
branch
from
a7b598b
to
447abf6
Compare
| this.pipe = new DefaultVideoFrameProcessorPipeline( | ||
| this.logger, | ||
| this.processors, | ||
| new DefaultVideoFrameProcessorTimer() |
There was a problem hiding this comment.
This explicit passing is not needed as you create one by default when instantiating.
| } | ||
| } | ||
|
|
||
| destroy(): void { |
There was a problem hiding this comment.
Not necessary but the assigned callback could also be set back to undefined.
| /** | ||
| * [[VideoFrameProcessorTimer]] defines a timer used in the context of video frame processing. | ||
| */ | ||
| export default interface VideoFrameProcessorTimer { |
There was a problem hiding this comment.
Nice! Should we make start and destroy promise based? We can make this interface destroyable by extending existing Destroyable interface:
If any async processing goes in them in future, the consuming code will not be able to use await on them and thus async programming may become harder.
There was a problem hiding this comment.
good point, will change! I wish js had a better way of transitioning from sync -> async APIs in a backwards compatible way.
|
@kenpp unfortunately even after this is merged it may be until December until we do another release. |
Description of changes:
Frames per second of video stream was being processed on the main thread using setTimeout. However, when a meeting window gets moved into the background, the main thread timer used in setTimeout gets throttled and the fps drops dramatically. To overcome this issue, this PR will move the setTimeout call into a worker thread that does not get throttled.
Testing:
Can these tested using a demo application? Please provide reproducible step-by-step instructions.
Yes, start a meeting with at least one other attendee. Enable a video filtered and then hide the window (switch tabs). The other attendee should then observe no fps degradation in filter-enabled attendee.
Checklist:
Have you successfully run
npm run build:releaselocally?yes
Do you add, modify, or delete public API definitions? If yes, has that been reviewed and approved?
no
Do you change the wire protocol, e.g. the request method? If yes, has that been reviewed and approved?
no
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.