Running query tests in browser #8584

arindam1993 · 2019-08-02T00:00:15Z

This PR adds a bunch of new tooling to allow the query tests to be run in the browser.
New tools added:

testem: this is a test-runner that that handles
- launching browsers
- serving test files to launched browsers, and receiving test results back via websocket
- persisting test results to disk once tests have finished in xunit format
tape: browser compatible version of tap, used to write the tests.
browserify: tape is not browser compatible ootb, so we need to build it ourselves, and rollup does not handle tape's commonjs style well and fails at trying to inject node globals and builtins. So we build it with browserify.

Architecture:

1.Loading fixture data to the browser

This is done via a preprocessor that generates fixtures.json file within test/integration/dist that contains all the data within the directory inlined within it, this file can then be imported into the test root file and the rollup will inline everything so all the data is available synchronously.

2.Bundling resources for the browser

There are 3 bundles that get built in parallel before the test suite starts
-tape_config.js: Done via browserify, and exposed as a global tape.

mapboxgl: Done via rollup, exposed as a global mapboxgl.
query-browser.js: this is the entrypoint of the test suite, built via rollupas an iife so that tests start executing as soon as the script is loaded.

3. Test running

testem provides hooks to run custom code before and after tests. The before_tests hook is used to run the build, and start an http server thats used to serve up certain resources such as spritesheets and fonts( this is the same as the existing implementation).
testem then launches its own server with a static html page which server up the built javascript resources. An extra script injected by testem onto the page captures the tap formatted console.log messages and relays them over websocket to be interpreted on the server to detect when the test suite has finished.
It then converts the gathered tap formatted report and convertes it to xunit format for integration with CircleCI's test reporting.

4.Test reporting

testem finishes running tests and persists the test report in xunit format to disk.
I've intentionally left a failing query test so we can see how it shows up in circle.

Guide:

Adds a bunch of npm scripts:
yarn run test-query-browser runs all the tests in headless chrome and exits
yarn run watch-query starts build watchers and launches a chrome window to run tests so you can work iterate on tests.
yarn run start-query-debug starts both the test server and the debug server while sharing the saem build-watcher.

briefly describe the changes in this PR

…est-browserify # Conflicts: # yarn.lock

…est-browserify # Conflicts: # package.json # yarn.lock

…ectly.

…est-browserify # Conflicts: # test/integration/lib/server.js

arindam1993 · 2019-08-30T00:43:33Z

Would be good to add flow typing to the new test lib scripts.

Unfortunately testem does not play nice with esm, and flow for commonjs modules is not great.

I found a couple issues with running the testem server locally and trying to build new query tests or modify code to fix test output. Looks like you are missing watchers for the test fixture and gl-js targets.

Addtionally, if a browser stays open to the testem page, it holds on to the page (and reruns it) even if the testem server is re-launched in the terminal. This gives a false sense of whether the old/new code is being run.

I added the watchers you can now do.
yarn run watch-query: runs only the query server in watch mode
yarn run watch-query-debug: runs both the query and the debug server working off the same gl-js build.

kkaefer · 2019-08-15T15:31:25Z

testem.js

@@ -0,0 +1,66 @@
+/* eslint-disable import/no-commonjs */


Does this file need to be in the root directory? If not, let's move it to somewhere in the /test/* directory hierarchy to avoid clutter.

kkaefer · 2019-09-05T10:05:29Z

.circleci/config.yml

-            - v2-yarn-{{ checksum "yarn.lock" }}
-      - run: yarn
+            - v3-yarn-{{ checksum "yarn.lock" }}
+      - run: sudo npm install -g yarn & yarn install


Why sudo? Docker builds typically run as the root user anyway.

…est-browserify # Conflicts: # .circleci/config.yml # yarn.lock

asheemmamoowala

This looks nearly ready. A few items to wrap up before merging:

Update test-query and related package.json scripts to use this version of the test instead of the node based tests
Update integration tests readme with instructions to run the node-based tests, and to run/debug the testem based tests.

asheemmamoowala · 2019-09-20T21:58:26Z

.circleci/config.yml

@@ -100,7 +100,7 @@ jobs:
      - restore_cache:
          keys:
            - v3-yarn-{{ checksum "yarn.lock" }}
-      - run: sudo npm install -g yarn & yarn install
+      - run: npm install -g yarn & yarn install


needs a && to serialize the steps. A Single & will fork and try to run it in the background

- update debug pages to have a more reliable part to `access_token_generated.js` - Update npm run scripts for query tests to use browser version by default.

arindam1993 · 2019-09-24T00:15:43Z

This looks nearly ready. A few items to wrap up before merging:

Update test-query and related package.json scripts to use this version of the test instead of the node based tests

Update integration tests readme with instructions to run the node-based tests, and to run/debug the testem based tests.

Done 👍 , I also made it so that it starts the testem server when running start/start-debug both the debug pages on :9966/debug and the test page on :7357 run off the same bundle and build watcher.

… process to prevent terminal from getting messed up.

asheemmamoowala

LGTM!

…est-browserify # Conflicts: # package.json # yarn.lock

…est-browserify # Conflicts: # yarn.lock

Arindam Bose added 30 commits July 22, 2019 17:31

WIP investigating migrating to tape

a6f18fc

staging work with using rollup to build tests for the browser

f286372

working prototype of tests in browser with browserify

2143ed6

remove beforeEach and afterEach temporarily

4cf2312

stage

6d3df51

refactor tape build process, working with rollupnow :party:

42013e3

revert change to util/test

62923a7

working build script for bundling fixtures into browser bundle

a6ae97e

working style json preprocessing

f54c2fc

first working protoype of in-browser query tests

1b19da4

add __ENVIRONMENT__ flag and add request proxies

8705ef6

working query tests!!

6f88052

further improments

c6a5b9a

attempting switch to non esm node

e02bc60

migrate to using hook in testem

f5b5995

switch to using testem hooks

95ca5c9

fix most linking errors

6619e8b

Merge branch 'master' of github.com:mapbox/mapbox-gl-js into devops/t…

c2c0404

…est-browserify # Conflicts: # yarn.lock

fix error in suite_implementation

dd71bb1

change environment variable style

964a698

fix failing query tests

36a0042

first attempt at running query tests in-browser on circle-ci

925848c

switch to curl instead of wget

c47b4ac

Using apt-get -f instead

bf03603

attempt auto install of dependencies

0077659

switch to using public node image

3f4f8e4

de-derp

de1cde3

uncache node_modules

580a0cd

try node 10.16

b3c79a0

remove xvfb

cf81bcd

Arindam Bose added 8 commits August 26, 2019 13:11

Merge branch 'master' of github.com:mapbox/mapbox-gl-js into devops/t…

7b4ec5b

…est-browserify # Conflicts: # package.json # yarn.lock

Remove unecessary mapbox-gl-test.js build, using gl-dev build dir…

1415b4c

…ectly.

add build watchers for convenient local debugging

7352d3b

fix circle integration

a41873c

Merge branch 'master' of github.com:mapbox/mapbox-gl-js into devops/t…

758822a

…est-browserify # Conflicts: # test/integration/lib/server.js

Fix linting

ebcc44b

Revert whitespace change in rollup_plugins.js

a70055b

Remove unused env-var flow definition

3d49d95

kkaefer mentioned this pull request Sep 2, 2019

Use randomly generated task IDs to avoid collisions when receiving messages from multiple workers #8708

Merged

Arindam Bose added 2 commits September 4, 2019 17:45

Add prettified HTML reporter for tap output

62a5a2a

revert intentionally broken test

657f211

kkaefer reviewed Sep 5, 2019

View reviewed changes

Arindam Bose added 4 commits September 5, 2019 19:45

Add persisting of testem test-page to disk for reporting on CI

f926ef2

Lock circleci to use debian-stretch

614e795

address CR comments: move testem config file and update circle yml

688f177

Merge branch 'master' of github.com:mapbox/mapbox-gl-js into devops/t…

fc0feec

…est-browserify # Conflicts: # .circleci/config.yml # yarn.lock

asheemmamoowala reviewed Sep 20, 2019

View reviewed changes

Arindam Bose added 3 commits September 23, 2019 14:30

Add env var to disable build notifications

d1acb15

- Move tap-parser to devDependencies

f3e2574

- update debug pages to have a more reliable part to `access_token_generated.js` - Update npm run scripts for query tests to use browser version by default.

Add documentation to the readme

a631a9c

fix aggresive fixture watcher, and silence warnings from rollup build…

53f87a6

… process to prevent terminal from getting messed up.

asheemmamoowala approved these changes Sep 27, 2019

View reviewed changes

Arindam Bose added 2 commits October 1, 2019 13:14

Merge branch 'master' of github.com:mapbox/mapbox-gl-js into devops/t…

bc7e46b

…est-browserify # Conflicts: # package.json # yarn.lock

Merge branch 'master' of github.com:mapbox/mapbox-gl-js into devops/t…

521363d

…est-browserify # Conflicts: # yarn.lock

arindam1993 merged commit d6513e2 into master Oct 10, 2019

arindam1993 deleted the devops/test-browserify branch October 10, 2019 18:59

ryanhamley mentioned this pull request Mar 10, 2020

Master ticket: Make all tests runnable in the browser #9401

Open

4 tasks

asheemmamoowala mentioned this pull request Jan 28, 2021

Create an in-browser implementation of the integration test harness #4190

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running query tests in browser #8584

Running query tests in browser #8584

arindam1993 commented Aug 2, 2019 •

edited

Loading

arindam1993 commented Aug 30, 2019

kkaefer Aug 15, 2019

kkaefer Sep 5, 2019

asheemmamoowala left a comment

asheemmamoowala Sep 20, 2019

arindam1993 commented Sep 24, 2019

asheemmamoowala left a comment

Running query tests in browser #8584

Running query tests in browser #8584

Conversation

arindam1993 commented Aug 2, 2019 • edited Loading

Architecture:

1.Loading fixture data to the browser

2.Bundling resources for the browser

3. Test running

4.Test reporting

Guide:

arindam1993 commented Aug 30, 2019

kkaefer Aug 15, 2019

Choose a reason for hiding this comment

kkaefer Sep 5, 2019

Choose a reason for hiding this comment

asheemmamoowala left a comment

Choose a reason for hiding this comment

asheemmamoowala Sep 20, 2019

Choose a reason for hiding this comment

arindam1993 commented Sep 24, 2019

asheemmamoowala left a comment

Choose a reason for hiding this comment

arindam1993 commented Aug 2, 2019 •

edited

Loading