Record canvas snapshots N times per second #859

Juice10 · 2022-03-10T20:22:24Z

Problem & solution summary

Sometimes recording a canvas via regular means (recordCanvas: true) isn't possible or is to data intensive.
When regular canvas recording isn't feasible it's nice to be able to record full snapshots of the canvas on a period basis.

Configuration

rrwebRecord({
  emit,
  // records at a maximum of 10 frames per second
  // actual snapshot record count is probably lower for large canvases
  // as we wait till a snapshot is finished before we make a new one
  recordCanvas: 10,
});

Problem explained

Recording snapshots is especially useful in the event that an rrweb recording gets initialized after a canvas was added to the dom and was populated as there is no way to read the initial state of these canvas events.

Another useful case is if a canvas gets repainted every requestAnimationFrame with the same (large) sources but doesn't actually change that much.
For example if you were to call:

const fn = ()=>{
  canvasContext.drawImage(HUGE_IMAGE_SOURCE, 0,0);
  requestAnimationFrame(fn);
};
fn()

Every invocation of drawImage would get recorded, the HUGE_IMAGE_SOURCE would get base64 encoded and saved, rapidly ballooning the size of recorded events. Utilizing the current recording mechanism I was able to generate about 300mb worth of json events in just 10-20 seconds of recording.

Performance

This feature uses web workers to minimize the impact of recording on the performance of the website being recorded.
If the computer experiences heavy load the web workers will slow down, the recorded frame rate will go down, but no frames will get dropped (because of the recording).

Browser support

This feature uses OffscreenCanvas and isn't supported by every browser yet. According to caniuse this OffscreenCanvas is currently supported by 77% of browsers with webgl(2) support hidden behind feature flags in Firefox and pretty far along in Safari. There is currently no polyfill for browsers that don't support OffscreenCanvas so any canvas snapshotting attempted in those browsers will fail gracefully.

Other benefits of this PR

The web worker is currently used to do the heavy operation of converting a Canvas element to a base64 encoded string.
We are currently using canvas.toDataURL and wrapping it in a setTimeout in the hopes of not locking the main thread. Building on the work in this PR, we can move this to a web worker where supported.

Serializes & deserializes 2d canvas arguments (fixes #851)

Makes recompiling+debugging a lot faster

Replaces @rollup/plugin-typescript for rollup-plugin-typescript2 as the former is incompatible with rollup-plugin-web-worker-loader

We don't have any code for it yet

Fixes https://linguinecode.com/post/import-export-appear-at-the-top-level

YunFeng0817

I'm not familiar with webGL so there are some parts I don't understand. But I built a demo to test this PR and it does solve the problem.

Juice10 · 2022-03-11T15:13:28Z

@Mark-Fenng Thanks Mark!

DeanGracey · 2022-04-07T12:32:14Z

Hi @Juice10 Thanks for doing this amazing work! Is there a reason this hasn't been merged?

Juice10 · 2022-04-07T18:54:15Z

Thanks @DeanGracey! It's currently waiting on one more rrweb core team member review and then it should be good to go.
There are a number of other big pull requests in the pipeline which also take some time to review

DeanGracey · 2022-04-08T07:17:10Z

Amazing, thanks @Juice10. We're currently thinking of patching rr-web in our system to get this functionality in. Do you have a rough idea when this would be released? If not too long we would prefer to wait for the release rather than patching

Yuyz0112 · 2022-03-11T10:19:41Z

docs/recipes/canvas.zh_CN.md

@@ -12,6 +12,15 @@ rrweb.record({
 });
 ```

+或者启用每秒 15 帧的 Canvas 图像快照记录：


I'm sorry if it doesn't make any sense 😅 I figured I'd use Google translate to put in something and then you, @Mark-Fenng or another community member could put in a suggestion for something that is correct.

Yuyz0112 · 2022-04-09T06:18:13Z

docs/recipes/canvas.md

+```js
+rrweb.record({
+  emit(event) {},
+  recordCanvas: 15,


does it make sense to move the fps option to our sampling object?

Oh yes that totally makes sense, thanks for the suggestion @Yuyz0112
I think I have some time this Monday to move it over!

3e6a901 moves the config over to the sampling config. I think this makes much more sense, thanks for the suggestion @Yuyz0112!

Based on: #859 (comment)

ryanZiegler · 2022-04-18T03:33:23Z

@Juice10
When will this feature be merged into master? I can't wait to experience this feature.

Yuyz0112 · 2022-04-18T05:25:07Z

@ryanZiegler done

ryanZiegler · 2022-04-22T07:08:46Z

@Yuyz0112 Thank you very much for your quick response 👍 👍 👍 ! Hope NPM can also update this version synchronously.

* Only record canvas when recordCanvas is true * All should be compiled first Makes recompiling+debugging a lot faster * Add support for compiling web workes Replaces @rollup/plugin-typescript for rollup-plugin-typescript2 as the former is incompatible with rollup-plugin-web-worker-loader * Update yarn.lock * Upgrade to typescript 4.5.5 * add support for replay of ImageBitmap in 2d canvas * Snapshot canvases in a web-worker on FPS basis * Fix performance of canvas recording and playback * Wait for all images to be preloaded before checking results * flatten base64 strings, as encoding isn't consistent * Cleanup * Add serializing to 2d canvases as well * Disable blob serialize test We don't have any code for it yet * Upgrade @rollup/plugin-commonjs to 21.0.2 Fixes https://linguinecode.com/post/import-export-appear-at-the-top-level * Move canvas recording options to `sampling` Based on: rrweb-io/rrweb#859 (comment)

Juice10 added 12 commits February 3, 2022 15:23

Only record canvas when recordCanvas is true

4a09c4c

All should be compiled first

76150a3

Makes recompiling+debugging a lot faster

Add support for compiling web workes

63d1287

Replaces @rollup/plugin-typescript for rollup-plugin-typescript2 as the former is incompatible with rollup-plugin-web-worker-loader

Update yarn.lock

e579c72

Upgrade to typescript 4.5.5

5fb6cfa

add support for replay of ImageBitmap in 2d canvas

15418b8

Snapshot canvases in a web-worker on FPS basis

58070d7

Fix performance of canvas recording and playback

dfd7b4b

Wait for all images to be preloaded before checking results

258e64b

flatten base64 strings, as encoding isn't consistent

21f373a

Merge branch 'master' of git://github.com/rrweb-io/rrweb into canvas-fps

34d4108

Cleanup

b8a529b

Juice10 changed the title ~~Record canvas snapshots N times per second~~ [draft] Record canvas snapshots N times per second Mar 10, 2022

Juice10 added 3 commits March 10, 2022 22:11

Add serializing to 2d canvases as well

0534796

Disable blob serialize test

ded7cc0

We don't have any code for it yet

Upgrade @rollup/plugin-commonjs to 21.0.2

4b5e8a8

Fixes https://linguinecode.com/post/import-export-appear-at-the-top-level

Juice10 changed the title ~~[draft] Record canvas snapshots N times per second~~ Record canvas snapshots N times per second Mar 10, 2022

This was referenced Mar 10, 2022

canvas回放时img空白 #851

Closed

回放pdf时报错如下图 #857

Open

YunFeng0817 approved these changes Mar 11, 2022

View reviewed changes

Merge branch 'master' into canvas-fps

0d23849

Yuyz0112 reviewed Apr 9, 2022

View reviewed changes

Juice10 added 2 commits April 11, 2022 16:19

Merge branch 'master' into canvas-fps

6225f3f

Move canvas recording options to sampling

3e6a901

Based on: #859 (comment)

Yuyz0112 merged commit e238462 into master Apr 18, 2022

YunFeng0817 deleted the canvas-fps branch July 11, 2022 06:22

Juice10 added the 2.0 label Dec 1, 2022

This was referenced Jul 6, 2023

add sentry v1 tests getsentry/rrweb#93

Closed

feat mask all text pr getsentry/rrweb#94

Closed

feat add custom attribute transform getsentry/rrweb#96

Closed

billyvg mentioned this pull request Jul 17, 2023

feat pass element to maskTextFn getsentry/rrweb#99

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record canvas snapshots N times per second #859

Record canvas snapshots N times per second #859

Juice10 commented Mar 10, 2022 •

edited

Loading

YunFeng0817 left a comment

Juice10 commented Mar 11, 2022

DeanGracey commented Apr 7, 2022

Juice10 commented Apr 7, 2022

DeanGracey commented Apr 8, 2022

Yuyz0112 Mar 11, 2022

Juice10 Apr 9, 2022

Yuyz0112 Apr 9, 2022

Juice10 Apr 9, 2022

Juice10 Apr 11, 2022

ryanZiegler commented Apr 18, 2022

Yuyz0112 commented Apr 18, 2022

ryanZiegler commented Apr 22, 2022

Record canvas snapshots N times per second #859

Record canvas snapshots N times per second #859

Conversation

Juice10 commented Mar 10, 2022 • edited Loading

Problem & solution summary

Configuration

Problem explained

Performance

Browser support

Other benefits of this PR

YunFeng0817 left a comment

Choose a reason for hiding this comment

Juice10 commented Mar 11, 2022

DeanGracey commented Apr 7, 2022

Juice10 commented Apr 7, 2022

DeanGracey commented Apr 8, 2022

Yuyz0112 Mar 11, 2022

Choose a reason for hiding this comment

Juice10 Apr 9, 2022

Choose a reason for hiding this comment

Yuyz0112 Apr 9, 2022

Choose a reason for hiding this comment

Juice10 Apr 9, 2022

Choose a reason for hiding this comment

Juice10 Apr 11, 2022

Choose a reason for hiding this comment

ryanZiegler commented Apr 18, 2022

Yuyz0112 commented Apr 18, 2022

ryanZiegler commented Apr 22, 2022

Juice10 commented Mar 10, 2022 •

edited

Loading