The HighlighEditor breaks on scrolling in the viewer

[Snuffleupagus]

Attach (recommended) or Link to PDF file here: N/A

Configuration:

• Web browser and its version: N/A
• Operating system and its version: N/A
• PDF.js version: master
• Is a browser extension: N/A

Steps to reproduce the problem:

1. Open http://localhost:8888/web/viewer.html
2. Enable the HighlighEditor.
3. Select some text on the first page, I used "Trace-based Just-in-Time Type Specialization for Dynamic Languages".
4. Scroll down to the end of the document.

What is the expected behavior? (add screenshot)
Everything works, without any Errors being thrown.

What went wrong? (add screenshot)
Editing breaks, and the following Errors are logged.

renderView: "TypeError: can't access property "remove", this[#mapping].get(...) is undefined" pdf_rendering_queue.js:204:21
Uncaught TypeError: can't access property "remove", this[#mapping].get(...) is undefined
    remove http://localhost:8888/src/display/draw_layer.js:188
    #cleanDrawLayer http://localhost:8888/src/display/editor/highlight.js:267
    setParent http://localhost:8888/src/display/editor/highlight.js:256
    destroy http://localhost:8888/src/display/editor/annotation_editor_layer.js:839
    cancel http://localhost:8888/web/annotation_editor_layer_builder.js:126
    cancelRendering http://localhost:8888/web/pdf_page_view.js:735
    reset http://localhost:8888/web/pdf_page_view.js:495
    destroy http://localhost:8888/web/pdf_page_view.js:311
    #destroyFirstView http://localhost:8888/web/pdf_viewer.js:192
    resize http://localhost:8888/web/pdf_viewer.js:177
    update http://localhost:8888/web/pdf_viewer.js:1595
    _scrollUpdate http://localhost:8888/web/pdf_viewer.js:1169
    viewAreaElementScrolled http://localhost:8888/web/ui_utils.js:179
    debounceScroll http://localhost:8888/web/ui_utils.js:164
10 draw_layer.js:188:19

Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension): N/A

/cc @calixteman