Support "Feature Preview" rollout of settings/defaults.

[JacksonKearl]

A few times now we've done "Phased Rollouts" of new feature (welcome, native notebooks, etc.), default setting values. It'd be nice to have some sort of standardized way both for

• VS Code core to have phased roll-out of a change in default setting values (including new feature flags)
• Users to have a way to view a list of the "experimental" changes currently available/enabled, ideally with a link to a tracking issue on GH

Perhaps an @experiments tag in settings editor could help here? Or something new, maybe taking inspi from GitHub's "Feature Preview" modal.

cc @roblourens @digitarald @luabud

[sbatten]

Some thoughts on this from the experiment service (now called workbench assignment service) side...

@JacksonKearl had already made all the technical changes to override experiment values (explicitly on or off).

The configuration service cannot rely on the assignment because it needs to rely on settings for overrides, so the settings editor should do the merging logic. Right now, overrides are hidden settings with fixed names based on the treatment variables. I would expect this could be done by adding explicit settings that would allow you to describe the experiment. The settings editor would then need to handle the different states (am i opted in or out by choice or by the treatment group).

[digitarald]

Love this for many reasons, as it naturally allows the rollout both via outreach leading to manual opt-in and experiments leading to opting in a random set of users.

To support this for extensions, I'd love to see this '@experiment' tag supported in settings contributed by package.json as well.

[sandy081]

@JacksonKearl @sbatten Can you please shed some light here - what is expected from configuration land?

[sandy081]

@digitarald @rzhao271

Added support for overriding default values of experimental settings through experiments service. This works in the following way:

• A setting can be tagged experimental using experimental tag
• Workbench gets default values for these settings through experiments service (workbench assignment service) by passing the key config.${setting}. Eg: config.workbench.welcomePage.preferReducedMotion
• Workbench overrides these defaults values for these settings.

@JacksonKearl - You can adopt your setting workbench.welcomePage.preferReducedMotion by adding experimental tag and workbench will update the default value.

[sandy081]

Let me know if there is any thing needs to be done here. Otherwise we are good to close it.

[sandy081]

@JacksonKearl / @digitarald Can you please verify this?

[luabud]

Thanks a lot for working on this, @sandy081! I verified it works 🥳 I added "tags": ["experimental"] to the python.formatting.provider setting in the Python extension. Then I set up the experiment in the control tower for my machine only with config.python.formatting.provider set to black, and it correctly enabled black as my default formatter:

It also shows correctly when auto completing the setting:

[digitarald]

Also 👍 from me. I verified with @sbatten who added the experimental flag to workbench.experimental.layoutControl.enabled.