Hat snapshots (#318)

* Initial attempt to at snapshots

* Switch to unnamed snapshots

* A kind of workign attempt

* A kind of workign attempt

* Use command server signal API

* More error robustness

* Working version

* Have navigation map return snapshot

* Attempt at big refactor

* Fixes to get it running

* Rename

* Bind function

* Remove unnecessary field

* Add docstring

* snapshot => prePhraseSnapshot

* Clean yaml

* Do disposal in hat allocator

* navigationMap => hatTokenMap

* Add tests

* Make isTesting into function

* Set testing env var

* Try to change env var

* Initial cleanup work for edits outside viewport

* More cleanup

* Fix yarn lockfile

* Fix yarn

* refactoring

* Fixes; add tests

* File rename

* Finish merging

* Some cleanup

* Create command runner class

* Working backwards compatible command runner

* More backward compatibility fixes

* Rejects stale snapshots

* Add link

* A bunch of refactoring

* Test fixes

* Revert change

* Improved canonicalization

* Add comment

* Fix ci

* Rollback decoration test change

* Attempt to fix decorations

* Normalize hat enablement during testing

* Fix recorded tests

* Fix tests

* Cleanup test recording

* Add docs

* Fix creating nested recorded test directories

* Cleanup test case bulk transformer

* More transform script fixes

* More transform stuff

* Upgrade a test

CONTRIBUTING.md

@@ -14,15 +14,15 @@ yarn run test
 
 ### Adding tests
 
-See [test-case-recorder.md](docs/test-case-recorder.md).
+See [test-case-recorder.md](docs/contributing/test-case-recorder.md).
 
 ### Adding a new programming language
 
-See [docs](docs/adding-a-new-language.md).
+See [docs](docs/contributing/adding-a-new-language.md).
 
 ### Adding syntactic scope types to an existing language
 
-See [parse-tree-patterns.md](docs/parse-tree-patterns.md).
+See [parse-tree-patterns.md](docs/contributing/parse-tree-patterns.md).
 
 ### Changing SVGs
 

docs/architecture/hat-snapshots.md

@@ -0,0 +1,7 @@
+# Hat snapshots
+
+In order to allow long chained command phrases, we take a snapshot of the hat token map at the start of a phrase and continue to use this map during the course of the entire phrase. This way you can be sure that any commands issued during the course of a single phrase that refer to a decorated token will continue to refer to the same logical token no matter what happens in the document during phrase execution. Note that the ranges of tokens will be kept current as the document changes so that they refer to the same logical range, but the same logical token will keep the same key in the hat token map over the course of a phrase.
+
+To make this work, first the voice engine touches a file within the signals subdirectory of the command server communication directory after the phrase has been parsed but right before execution begins. Then cursorless will check the version of the signal file via the command server signal API. If the signal has been emitted since the last time cursorless took a snapshot of the hat token map, it will take a new snapshot and continue to use that snapshot of the hats until the next time the signal is emitted. Note that the signal transmission is asynchronous so cursorless just needs to make sure to check the version of the signal before it either updates or reads the map.
+
+![flow diagram](hat-token-map-snapshots.png)

docs/adding-a-new-language.md → docs/contributing/adding-a-new-language.md

@@ -11,7 +11,7 @@ for how to add support for a new parser
 
 ## 2. Define parse tree patterns in Cursorless
 
-The parse trees exposed by tree-sitter are often pretty close to what we're
+The parse trees exposed by tree-sitter are often pretty close to what we``'re
 looking for, but we often need to look for specific patterns within the parse
 tree to get the scopes that the user expects. Fortunately, we have a
 domain-specific language that makes these definitions fairly compact.

docs/test-case-recorder.md → docs/contributing/test-case-recorder.md

@@ -56,9 +56,18 @@ and can be run in vscode or via yarn in terminal.
 
 ## Changing recorded test cases in bulk
 
-1. Change the `FIXTURE_TRANSFORMATION` function at the top of
-   [`transformRecordedTests.ts`](../src/scripts/transformRecordedTests.ts) to
-   perform the transformation you'd like
-2. Run `yarn run compile && node ./out/scripts/transformRecordedTests.js`
+### Autoformatting
 
-You might find the `transformPrimitiveTargets` function useful for this purpose.
+To clean up the formatting of all of the yaml test cases, run `yarn run compile && node ./out/scripts/transformRecordedTests/index.js`
+
+### Upgrading fixtures
+
+To upgrade all the test fixtures to the latest command version, run the command `yarn run compile && node ./out/scripts/transformRecordedTests/index.js upgrade`. This command should be idempotent.
+
+### Custom transformation
+
+1. Add a new transformation to the `src/scripts/transformRecordedTests/transformations` directory. Look at the existing transformations in that directory for inspiration.
+1. Change the value at the `custom` key in `AVAILABLE_TRANSFORMATIONS` at the top of
+   [`transformRecordedTests/index.ts`](../src/scripts/transformRecordedTests/index.ts) to
+   point to your new transformation
+1. Run `yarn run compile && node ./out/scripts/transformRecordedTests/index.js custom`

package.json

@@ -15,7 +15,7 @@
 		"url": "https://github.com/pokey/cursorless-vscode.git"
 	},
 	"engines": {
-		"vscode": "^1.53.0"
+		"vscode": "^1.61.0"
 	},
 	"extensionKind": [
 		"workspace"
@@ -427,15 +427,15 @@
 		"watch": "tsc -watch -p ./",
 		"pretest": "yarn run compile && yarn run lint && yarn run esbuild",
 		"lint": "eslint src --ext ts",
-		"test": "node ./out/test/runTest.js"
+		"test": "env CURSORLESS_TEST=true node ./out/test/runTest.js"
 	},
 	"devDependencies": {
 		"@types/glob": "^7.1.3",
 		"@types/js-yaml": "^4.0.2",
 		"@types/mocha": "^8.0.4",
 		"@types/node": "^16.11.3",
 		"@types/sinon": "^10.0.2",
-		"@types/vscode": "^1.53.0",
+		"@types/vscode": "^1.61.0",
 		"@typescript-eslint/eslint-plugin": "^4.9.0",
 		"@typescript-eslint/parser": "^4.9.0",
 		"esbuild": "^0.11.12",

src/core/Decorations.ts

@@ -11,7 +11,7 @@ import {
 } from "./constants";
 import { readFileSync } from "fs";
 import FontMeasurements from "./FontMeasurements";
-import { sortBy } from "lodash";
+import { pull, sortBy } from "lodash";
 import getHatThemeColors from "./getHatThemeColors";
 import {
   IndividualHatAdjustmentMap,
@@ -20,6 +20,7 @@ import {
   DEFAULT_VERTICAL_OFFSET_EM,
 } from "./shapeAdjustments";
 import { Graph } from "../typings/Types";
+import isTesting from "../testUtil/isTesting";
 
 export type DecorationMap = {
   [k in HatStyleName]?: vscode.TextEditorDecorationType;
@@ -30,24 +31,66 @@ export interface NamedDecoration {
   decoration: vscode.TextEditorDecorationType;
 }
 
+type DecorationChangeListener = () => void;
+
 export default class Decorations {
   decorations!: NamedDecoration[];
   decorationMap!: DecorationMap;
-  hatStyleMap!: Record<HatStyleName, HatStyle>;
+  private hatStyleMap!: Record<HatStyleName, HatStyle>;
   hatStyleNames!: HatStyleName[];
+  private decorationChangeListeners: DecorationChangeListener[] = [];
+  private disposables: vscode.Disposable[] = [];
 
   constructor(private graph: Graph) {
-    this.constructDecorations(graph.fontMeasurements);
     graph.extensionContext.subscriptions.push(this);
+
+    this.recomputeDecorationStyles = this.recomputeDecorationStyles.bind(this);
+
+    this.disposables.push(
+      vscode.commands.registerCommand(
+        "cursorless.recomputeDecorationStyles",
+        () => {
+          graph.fontMeasurements.clearCache();
+          this.recomputeDecorationStyles();
+        }
+      ),
+
+      vscode.workspace.onDidChangeConfiguration(this.recomputeDecorationStyles)
+    );
+  }
+
+  async init() {
+    await this.graph.fontMeasurements.calculate();
+    this.constructDecorations(this.graph.fontMeasurements);
   }
 
-  destroyDecorations() {
+  /**
+   * Register to be notified when decoration styles are updated, for example if
+   * the user enables a new hat style
+   * @param listener A function to be called when decoration styles are updated
+   * @returns A function that can be called to unsubscribe from notifications
+   */
+  registerDecorationChangeListener(listener: DecorationChangeListener) {
+    this.decorationChangeListeners.push(listener);
+
+    return () => {
+      pull(this.decorationChangeListeners, listener);
+    };
+  }
+
+  private destroyDecorations() {
     this.decorations.forEach(({ decoration }) => {
       decoration.dispose();
     });
   }
 
-  constructDecorations(fontMeasurements: FontMeasurements) {
+  private async recomputeDecorationStyles() {
+    this.destroyDecorations();
+    await this.graph.fontMeasurements.calculate();
+    this.constructDecorations(this.graph.fontMeasurements);
+  }
+
+  private constructDecorations(fontMeasurements: FontMeasurements) {
     this.constructHatStyleMap();
 
     const userSizeAdjustment = vscode.workspace
@@ -125,6 +168,8 @@ export default class Decorations {
     this.decorationMap = Object.fromEntries(
       this.decorations.map(({ name, decoration }) => [name, decoration])
     );
+
+    this.decorationChangeListeners.forEach((listener) => listener());
   }
 
   private constructHatStyleMap() {
@@ -146,9 +191,10 @@ export default class Decorations {
     shapePenalties.default = 0;
     colorPenalties.default = 0;
 
-    const activeHatColors = HAT_COLORS.filter(
-      (color) => colorEnablement[color]
-    );
+    // So that unit tests don't fail locally if you have some colors disabled
+    const activeHatColors = isTesting()
+      ? HAT_COLORS
+      : HAT_COLORS.filter((color) => colorEnablement[color]);
     const activeNonDefaultHatShapes = HAT_NON_DEFAULT_SHAPES.filter(
       (shape) => shapeEnablement[shape]
     );
@@ -279,5 +325,6 @@ export default class Decorations {
 
   dispose() {
     this.destroyDecorations();
+    this.disposables.forEach(({ dispose }) => dispose());
   }
 }

src/core/HatAllocator.ts

@@ -0,0 +1,95 @@
+import * as vscode from "vscode";
+import { addDecorationsToEditors } from "../util/addDecorationsToEditor";
+import { DECORATION_DEBOUNCE_DELAY } from "../core/constants";
+import { Graph } from "../typings/Types";
+import { Disposable } from "vscode";
+import { IndividualHatMap } from "./IndividualHatMap";
+
+interface Context {
+  getActiveMap(): Promise<IndividualHatMap>;
+}
+
+export class HatAllocator {
+  private timeoutHandle: NodeJS.Timeout | null = null;
+  private isActive: boolean;
+  private disposables: Disposable[] = [];
+  private disposalFunctions: (() => void)[] = [];
+
+  constructor(private graph: Graph, private context: Context) {
+    graph.extensionContext.subscriptions.push(this);
+
+    this.isActive = vscode.workspace
+      .getConfiguration("cursorless")
+      .get<boolean>("showOnStart")!;
+
+    this.addDecorationsDebounced = this.addDecorationsDebounced.bind(this);
+    this.toggleDecorations = this.toggleDecorations.bind(this);
+    this.clearEditorDecorations = this.clearEditorDecorations.bind(this);
+
+    this.disposalFunctions.push(
+      graph.decorations.registerDecorationChangeListener(
+        this.addDecorationsDebounced
+      )
+    );
+
+    this.disposables.push(
+      vscode.commands.registerCommand(
+        "cursorless.toggleDecorations",
+        this.toggleDecorations
+      ),
+
+      vscode.window.onDidChangeTextEditorVisibleRanges(
+        this.addDecorationsDebounced
+      ),
+      vscode.window.onDidChangeActiveTextEditor(this.addDecorationsDebounced),
+      vscode.window.onDidChangeVisibleTextEditors(this.addDecorationsDebounced),
+      vscode.window.onDidChangeTextEditorSelection(
+        this.addDecorationsDebounced
+      ),
+      vscode.workspace.onDidChangeTextDocument(this.addDecorationsDebounced)
+    );
+  }
+
+  private clearEditorDecorations(editor: vscode.TextEditor) {
+    this.graph.decorations.decorations.forEach(({ decoration }) => {
+      editor.setDecorations(decoration, []);
+    });
+  }
+
+  async addDecorations() {
+    const activeMap = await this.context.getActiveMap();
+
+    if (this.isActive) {
+      addDecorationsToEditors(activeMap, this.graph.decorations);
+    } else {
+      vscode.window.visibleTextEditors.forEach(this.clearEditorDecorations);
+      activeMap.clear();
+    }
+  }
+
+  addDecorationsDebounced() {
+    if (this.timeoutHandle != null) {
+      clearTimeout(this.timeoutHandle);
+    }
+
+    this.timeoutHandle = setTimeout(() => {
+      this.addDecorations();
+
+      this.timeoutHandle = null;
+    }, DECORATION_DEBOUNCE_DELAY);
+  }
+
+  private toggleDecorations() {
+    this.isActive = !this.isActive;
+    this.addDecorationsDebounced();
+  }
+
+  dispose() {
+    this.disposables.forEach(({ dispose }) => dispose());
+    this.disposalFunctions.forEach((dispose) => dispose());
+
+    if (this.timeoutHandle != null) {
+      clearTimeout(this.timeoutHandle);
+    }
+  }
+}