Suspense fuzz tester

[acdlite]

The fuzzer works by generating a random tree of React elements. The tree two types of custom components:

• A Text component suspends rendering on initial mount for a fuzzy duration of time. It may update a fuzzy number of times; each update supsends for a fuzzy duration of time.
• A Container component wraps some children. It may remount its children a fuzzy number of times, by updating its key.

The tree may also include nested Suspense components.

After this tree is generated, the tester sets a flag to temporarily disable Text components from suspending. The tree is rendered synchronously. The output of this render is the expected output.

Then the tester flips the flag back to enable suspending. It renders the tree again. This time the Text components will suspend for the amount of time configured by the props. The tester waits until everything has resolved. The resolved output is then compared to the expected output generated in the previous step.

Finally, we render once more, but this time in concurrent mode. Once again, the resolved output is compared to the expected output.

I tested by commenting out various parts of the Suspense implementation to see if broke in the expected way. I also confirmed that it would have caught #14133, a recent bug related to deletions.

The final commit also includes a fix for a bug the tester found. I can split this into a separate PR.

[sizebot]

ReactDOM: size: 🔺+0.1%, gzip: 🔺+0.2%

Details of bundled changes.

Comparing: f9e9913...a8e9909

react-dom

File	Filesize Diff	Gzip Diff	Prev Size	Current Size	Prev Gzip	Current Gzip	ENV
react-dom.development.js	+0.1%	+0.1%	708.59 KB	709.42 KB	163.82 KB	163.99 KB	UMD_DEV
react-dom.production.min.js	🔺+0.1%	🔺+0.2%	97.54 KB	97.67 KB	31.74 KB	31.79 KB	UMD_PROD
react-dom.development.js	+0.1%	+0.1%	703.9 KB	704.73 KB	162.45 KB	162.61 KB	NODE_DEV
react-dom.production.min.js	🔺+0.1%	🔺+0.1%	97.53 KB	97.66 KB	31.28 KB	31.32 KB	NODE_PROD
ReactDOM-dev.js	+0.1%	+0.1%	724.57 KB	725.36 KB	163.57 KB	163.72 KB	FB_WWW_DEV
ReactDOM-prod.js	🔺+0.1%	🔺+0.2%	309.31 KB	309.68 KB	56.97 KB	57.07 KB	FB_WWW_PROD
react-dom.profiling.min.js	+0.1%	+0.1%	100.53 KB	100.67 KB	31.89 KB	31.93 KB	NODE_PROFILING
ReactDOM-profiling.js	+0.1%	+0.1%	316.17 KB	316.57 KB	58.4 KB	58.46 KB	FB_WWW_PROFILING
react-dom.profiling.min.js	+0.1%	+0.1%	100.44 KB	100.58 KB	32.45 KB	32.47 KB	UMD_PROFILING

react-art

File	Filesize Diff	Gzip Diff	Prev Size	Current Size	Prev Gzip	Current Gzip	ENV
react-art.development.js	+0.2%	+0.1%	495.73 KB	496.56 KB	109.44 KB	109.6 KB	UMD_DEV
react-art.production.min.js	🔺+0.1%	🔺+0.2%	89.7 KB	89.84 KB	27.56 KB	27.61 KB	UMD_PROD
react-art.development.js	+0.2%	+0.2%	427.5 KB	428.34 KB	92.4 KB	92.56 KB	NODE_DEV
react-art.production.min.js	🔺+0.2%	🔺+0.2%	54.69 KB	54.82 KB	16.87 KB	16.91 KB	NODE_PROD
ReactART-dev.js	+0.2%	+0.2%	434.59 KB	435.37 KB	91.32 KB	91.46 KB	FB_WWW_DEV
ReactART-prod.js	🔺+0.1%	🔺+0.1%	183.23 KB	183.45 KB	31.31 KB	31.36 KB	FB_WWW_PROD

react-test-renderer

File	Filesize Diff	Gzip Diff	Prev Size	Current Size	Prev Gzip	Current Gzip	ENV
react-test-renderer.development.js	+0.2%	+0.2%	440.17 KB	441 KB	95.1 KB	95.26 KB	UMD_DEV
react-test-renderer.production.min.js	🔺+0.2%	0.0%	55.91 KB	56.04 KB	17.19 KB	17.2 KB	UMD_PROD
react-test-renderer.development.js	+0.2%	+0.2%	435.38 KB	436.22 KB	93.95 KB	94.11 KB	NODE_DEV
react-test-renderer.production.min.js	🔺+0.2%	🔺+0.2%	55.58 KB	55.71 KB	17.02 KB	17.05 KB	NODE_PROD
ReactTestRenderer-dev.js	+0.2%	+0.2%	442.65 KB	443.44 KB	93.16 KB	93.31 KB	FB_WWW_DEV

react-reconciler

File	Filesize Diff	Gzip Diff	Prev Size	Current Size	Prev Gzip	Current Gzip	ENV
react-reconciler.development.js	+0.2%	+0.2%	425.35 KB	426.18 KB	90.89 KB	91.06 KB	NODE_DEV
react-reconciler.production.min.js	🔺+0.2%	🔺+0.2%	55.82 KB	55.95 KB	16.74 KB	16.78 KB	NODE_PROD
react-reconciler-persistent.development.js	+0.2%	+0.2%	423.8 KB	424.63 KB	90.27 KB	90.43 KB	NODE_DEV
react-reconciler-persistent.production.min.js	🔺+0.2%	🔺+0.2%	55.83 KB	55.96 KB	16.74 KB	16.78 KB	NODE_PROD

react-native-renderer

File	Filesize Diff	Gzip Diff	Prev Size	Current Size	Prev Gzip	Current Gzip	ENV
ReactNativeRenderer-dev.js	+0.1%	+0.2%	559.78 KB	560.56 KB	122 KB	122.19 KB	RN_FB_DEV
ReactNativeRenderer-prod.js	🔺+0.1%	🔺+0.1%	238.56 KB	238.88 KB	41.99 KB	42.05 KB	RN_FB_PROD
ReactNativeRenderer-dev.js	+0.1%	+0.2%	559.48 KB	560.26 KB	121.9 KB	122.09 KB	RN_OSS_DEV
ReactNativeRenderer-prod.js	🔺+0.1%	🔺+0.2%	224 KB	224.32 KB	38.9 KB	38.97 KB	RN_OSS_PROD
ReactFabric-dev.js	+0.1%	+0.2%	550.09 KB	550.88 KB	119.5 KB	119.7 KB	RN_FB_DEV
ReactFabric-prod.js	0.0%	🔺+0.1%	218.21 KB	218.31 KB	37.61 KB	37.63 KB	RN_FB_PROD
ReactFabric-dev.js	+0.1%	+0.2%	550.13 KB	550.91 KB	119.52 KB	119.72 KB	RN_OSS_DEV
ReactFabric-prod.js	0.0%	🔺+0.1%	218.25 KB	218.35 KB	37.63 KB	37.65 KB	RN_OSS_PROD
ReactNativeRenderer-profiling.js	+0.2%	+0.1%	229.64 KB	230.03 KB	40.26 KB	40.32 KB	RN_OSS_PROFILING
ReactFabric-profiling.js	+0.1%	+0.1%	222.98 KB	223.12 KB	39.01 KB	39.06 KB	RN_OSS_PROFILING
ReactNativeRenderer-profiling.js	+0.2%	+0.1%	244.44 KB	244.84 KB	43.34 KB	43.4 KB	RN_FB_PROFILING
ReactFabric-profiling.js	+0.1%	+0.1%	222.94 KB	223.08 KB	38.99 KB	39.04 KB	RN_FB_PROFILING

scheduler

File	Filesize Diff	Gzip Diff	Prev Size	Current Size	Prev Gzip	Current Gzip	ENV
scheduler.development.js	n/a	n/a	0 B	19.17 KB	0 B	5.74 KB	UMD_DEV
scheduler.production.min.js	n/a	n/a	0 B	3.16 KB	0 B	1.53 KB	UMD_PROD

Generated by 🚫 dangerJS

[sophiebits]

Does this catch the bugs fixed by #14083 too?

[gaearon]

Then the tester flips the flag back to enable suspending. It renders the tree again. This time the Text components will suspend for the amount of time configured by the props. The tester waits until everything has resolved. The resolved output is then compared to the expected output generated in the previous step.

Technically this doesn't test that updates are handled consistently, does it? Would it be more solid if we instead incremented every value in the cache before suspending, and the test verified that all rendered output is original output + 1 after it settles?

[acdlite]

@gaearon Could you rephrase? Are you concerned that the cache isn't being cleared in between renders? Each one uses a separate cache.

[acdlite]

Here's an example test case. In this test, the Text component suspends for 2 seconds on initial mount. 100 ms after that, it updates itself by incrementing a counter in local state. That update will suspend for 200 ms.

The Container component will also schedule an update remount once after 250 ms, which will trigger the Text mount and updates again (since it's a fresh component).

react/packages/react-reconciler/src/__tests__/ReactSuspenseFuzz-test.internal.js

Lines 346 to 354 in 01c1093

	testResolvedOutput(
	<Container updates={[{remountAfter: 150}]}>
	<Text
	text="Hi"
	initialDelay={2000}
	updates={[{beginAfter: 100, suspendFor: 200}]}
	/>
	</Container>,
	);

The runner will keep advancing time until everything settles, both promises and updates.

The final output of this test is Hi:1 (because it updated 1 time).

[acdlite]

@sophiebits Yep!

[dubzzz]

I know that this PR is not recent at all and I felt on the test while browsing the source code of React.

Out of curiosity: Why don't you used property based testing libraries to help you to generate those random tests?
They can be really helpful for those kind of tests in general as they offer what they call shrinking capabilities - in other words whenever a test fails they try to produce a failing test with inputs simpler to understand (smaller most of the time).

You may have a look to fast-check, jsverify or testcheck-js for more details.

[gaearon]

Wanna send a PR converting it? :-)

For me personally learning the API has been a barrier. I couldn’t actually figure out how to generate recursive things with testcheck-js.

[dubzzz]

Yes, I will give it a try but with fast-check as it handles recursive types (the lib is used by jest and jasmine)

[dubzzz]

@gaearon Here is a first draft of the test migrated to property based testing. If you believe it can be useful, I can rework and clean a bit the code before opening a PR.

See dubzzz@0b1df02

Side note: For the moment I removed the variable numberOfElements in order to let the framework generate random trees of any sizes.

[gaearon]

Looks interesting. Send a PR and let’s discuss?

[dubzzz]

I just opened the PR to move fuzz tests of ReactSuspense to property based testing. I will also migrate ReactNewContext-test.js (another fuzz-based test)