Shell commands in launch.json args are escaped

[llabusch93]

We have written the needed data into your clipboard because it was too large to send. Please paste.
Issue Type: Bug

Hello,

I worked with this launch configuration for ages now:

{
            "name": "Odoo 13.0 EE",
            "type": "python",
            "request": "launch",
            "program": "/usr/local/bin/odoo",
            "console": "integratedTerminal",
            "justMyCode": false,
            "args": [
                // "shell",
                "--database=dev_ee",
                "--log-level=test",
                "--config=${workspaceFolder}/.devcontainer/odoo.conf",
                "--addons-path=`/usr/bin/gen_addons_path -d ${workspaceFolder}`",
                "--dev=qweb,xml",
                "--load=base,web,base_test_db",
                "--init=lboucher_portal_order_widget",
                // "--test-enable",
                // "--test-tags=/",
                // "--stop-after-init",
                // "--load-language=en_US,de_DE,fr_CA",
                // "--language=fr_CA",
                // "--i18n-export=${workspaceFolder}/fr_CA.po",
                // "--modules=/"
            ]
        }

The special part is this line in the args:

"--addons-path=`/usr/bin/gen_addons_path -d ${workspaceFolder}`",

It calls the gen_addons_path script, what returns it then. This worked all the time, but as I wanted to start working today, it gives me this:
bash: /usr/bin/gen_addons_path -d /workspace: No such file or directory

Maybe it's because of the escape characters in the call itself:

/usr/bin/env /usr/bin/python3 /home/odoo/.vscode-server/extensions/ms-python.python-2022.6.2/pythonFiles/lib/python/debugpy/launcher 37455 -- /usr/local/bin/odoo --database=dev_ee --log-level=test --config=/workspace/.devcontainer/odoo.conf --addons-path=`/usr/bin/gen_addons_path\ -d\ /workspace` --dev=qweb,xml --load=base,web,base_test_db --init=lboucher_portal_order_widget

If I remove the arguments for this script, the call seems to work, but then the script fails, as it needs the arguments obviously...

"--addons-path=`/usr/bin/gen_addons_path`",

gen_addons_path: error: the following arguments are required: -d/--dirs

So what's the issue here, did you guys update something on the vs-code-server? If so, please revert it...

VS Code version: Code 1.67.2 (c3511e6, 2022-05-17T18:23:40.286Z)
OS version: Linux x64 5.15.40-xanmod1-tt
Restricted Mode: No
Remote OS version: Linux x64 5.15.40-xanmod1-tt

System Info

Item	Value
CPUs	Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz (16 x 2400)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off opengl: enabled_on rasterization: disabled_software raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: disabled_software video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	1, 1, 1
Memory (System)	30.99GB (8.54GB free)
Process Argv	--unity-launch --crash-reporter-id 557eb4c0-99af-49aa-bb58-b678e307c4c2
Screen Reader	no
VM	0%
DESKTOP_SESSION	pop
XDG_CURRENT_DESKTOP	Unity
XDG_SESSION_DESKTOP	pop
XDG_SESSION_TYPE	x11

Item	Value
Remote	Dev Container: Odoo 13.0 Development Container
OS	Linux x64 5.15.40-xanmod1-tt
CPUs	Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz (16 x 4534)
Memory (System)	30.99GB (8.56GB free)
VM	0%

Extensions (30)

Extension	Author (truncated)	Version
better-comments	aar	3.0.0
eml	lei	0.4.0
dotenv	mik	1.0.1
jupyter-keymap	ms-	1.0.0
remote-containers	ms-	0.234.0
remote-ssh	ms-	0.80.0
remote-ssh-edit	ms-	0.80.0
remote-wsl	ms-	0.66.3
vscode-remote-extensionpack	ms-	0.21.0
material-icon-theme	PKi	4.17.0
better-comments	aar	3.0.0
gitlens	eam	12.0.6
vscode-html-css	ecm	1.12.2
todo-tree	Gru	0.0.215
beautify	Hoo	1.5.0
OdooSnippets	jig	1.5.0
restructuredtext	lex	189.0.0
document	min	2.1.21
python	ms-	2022.6.2
vscode-pylance	ms-	2022.5.2
jupyter	ms-	2022.4.1021342353
jupyter-keymap	ms-	1.0.0
jupyter-renderers	ms-	1.0.6
indent-rainbow	ode	8.3.1
vscode-thunder-client	ran	1.16.3
vscode-xml	red	0.20.0
code-spell-checker	str	2.2.0
simple-rst	tro	1.5.2
vscodeintellicode	Vis	1.2.21
markdown-all-in-one	yzh	3.4.3

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyl392:30443607
pythontb:30283811
pythonptprofiler:30281270
vsdfh931:30280409
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805cf:30301675
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30449410
pythonvsnew555:30457759
vscscmwlcmt:30465135
cppdebug:30492333
vsclangdc:30486549

[roblourens]

I will copy my answer from #149391 (comment) and we can continue the discussion here

The history is this - we were not fully escaping args, making it impossible to pass some special characters to a program as args. #145265. I think we never intended to enable arbitrary shell scripts in launch.json args, but it worked because arg escaping was incomplete. Now we escape all special characters in args for the shell. My view is that args should be arguments passed to a program, not a shell script which only works in a particular shell.

Since lots of people were using < and > for input and output redirection, our compromise was not escaping those characters when they are alone in an arg. But other characters will be escaped so they can be passed literally to a program.

There might be another way you can get the same transformation like by doing it in your .py file. That would be platform-independent too. Or, maybe you can start a shell script that runs your code with these options and use an "attach" config to attach to it.

[roblourens]

This does show that I am not escaping backticks, which I guess is a big omission

[llabusch93]

This does show that I am not escaping backticks, which I guess is a big omission

Please just leave the backticks then! I think anyway a lot of people to use output of other scripts as arguments in the launch configuration, so it would be more appretierend to just have a configuration option to escape special chars if one wants that...

[roblourens]

Examples from other issues:

• Shell wildcards
• Other stdout redirection (&>)

[darcyrush]

@roblourens I'm not sure if this is the exact same issue, but seems related and is definitely non obvious. In code version 1.60.0, I can pass in an RSA key like so;

"args": [
  "--option-one=test"
   "--option-two=-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvmcQzmJeN9+Td84ExBNl\nSItvghT....   ....9b9\n5h0DGNShR0vKa2qGcTIIwTMCAwEAAQ==\n-----END PUBLIC KEY-----",
]

Notice the \n newlines and no escaping whatsoever. In 1.60.0 the program will run as;

$ ts-node -T main.js --option-one=test "--option-two=-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvmcQzmJeN9+Td84ExBNl
SItvghT....   ....9b9
5h0DGNShR0vKa2qGcTIIwTMCAwEAAQ==
-----END PUBLIC KEY-----",

While in 1.67.0 it will run as;

$ ts-node -T main.js --option-one=test --option-two=-----BEGIN\ PUBLIC\ KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvmcQzmJeN9+Td84ExBNl
SItvghT....   ....9b9
5h0DGNShR0vKa2qGcTIIwTMCAwEAAQ==
-----END\ PUBLIC\ KEY-----

Notice that in the older release double quotes are used for the entire argument (including the option name itself - but this doesn't matter). A bash specific 'one-line' implementation could be to use $'...\n...' but I believe the correct POSIX implementation is what 1.60.0 did, which is to use double quotes and output the RSA parts on actual separate lines.

Since the escape logic is hidden, I cannot use either approach since the dollar and single quotes are escaped, and I cannot escape double quotes myself since the escaped double quote is output in terminal.

I also tried the 'hidden' flag but it didn't work

"argsExpansion": "none"

[weinand]

@roblourens there is already a DAP feature request to support passing arguments unescaped to the shell.

[weinand]

@roblourens supporting unescaped command line arguments generically in VS Code is not trivial because VS Code knows nothing about launch.json properties that represent those arguments. Properties like "args" are contributed by individual debugger extensions via schema contributions. So only the debug extension itself knows what launch.json properties needs to be passed to DAP's runInTerminal request.

The cleanest approach to address the "unescaped arguments" problem would be to:

• extend DAP's runInTerminal request by additional properties that allow to prevent the escaping and/or add another argument that allows to pass a complete command line (a DAP feature request already exists for this).
• ask debug extensions to introduce new debug configuration properties and make use of the new runInTerminal options.

The main disadvantage of the clean approach is that each debug extension has to to opt-into the feature by introducing and implementing new debug configuration properties. This takes time and can lead to inconsistencies in naming, since each extension is free to choose its own property names.

I wonder whether we could find a robust and elegant way to solve the problem in VS Code only.

E.g. something like this:

• VS Code introduces a new launch.json property "no-escape",
• VS Code's implementation of DAP's runInTerminal request has access to the "current debug configuration" and honours the value of no-escape before escaping the arguments.

One problem with that approach is that debug adapters are free to call runInTerminal at any time even for running internal programs where the no-escape flag must be ignored.

Any better ideas to solve the problem?

[roblourens]

The first solution is what I had in mind. Yes, every debug adapter has to add a flag independently, but if we get a fix in the python and node DA's, that solves most of the problem. And I think breaking other calls to runInTerminal like you describe would be a real risk.

[weinand]

@roblourens If we go that route, then we should probably give some advice what new properties to use and how to name them.
E.g. should a new boolean property "argsNoEscape" be used together with an existing "args" property array?
Or should a single string property "commandLine" be used, where all arguments and shell special characters are passed as a single command line?

[roblourens]

Maybe escapeArgsForShell (undefined defaults to true)

[roblourens]

@connor4312 suggested a string format for args vs a string array, similar to the commandLine property above. @Tyriar said

The string instead of string[] is already a concept we have in the product. I can't remember exactly where we use it but it is always Windows only to workaround edge cases in the escaping needed on Windows

vscode/src/vs/platform/terminal/common/terminal.ts

Lines 418 to 423 in 8feb40b

	/**
	* The CLI arguments to use with executable, a string[] is in argv format and will be escaped,
	* a string is in "CommandLine" pre-escaped format and will be used as is. The string option is
	* only supported on Windows and will throw an exception if used on macOS or Linux.
	*/
	args?: string[] | string;

I think it would be useful to be able to toggle this behavior without having to reformat your args, with a simple bool toggle in the launch config. On the other hand, it makes more sense to have a string where the format of the array doesn't actually matter, and "toggling" this probably won't actually happen that often. I'll try it and see how it feels.

@karthiknadig heads up, I will be filing a feature request on the python extension to adopt this for launch configs as well. vscode-python can implement it independently however it wants to, but might as well match js-debug for consistency. I'm wondering whether you have a preference on the shape in the launch config.

[karthiknadig]

@roblourens is this for run-in-terminal request?

/cc @int19h @rchiodo

[roblourens]

Yes

[weinand]

Yes, I was proposing the string (instead of string array) format for args in various feature requests. One advantage of a single string "command line" is the ability to use an interactive "input variable" for it (where the user can enter the full command line in a prompt).
But debug extensions would have to contribute the "string[] | string" because VS Code cannot provide this generically.

[roblourens]

Hm... if we reuse the args property for this, I wonder whether that makes the working of the args property confusing, or would lead to people setting it to a string when they should be using an array.

[roblourens]

Closing this since the DAP change is implemented in vscode, we can continue the discussion about launch config format in microsoft/vscode-js-debug#1335.

[hansmbakker]

@roblourens great that you are working on this! Seeing that the discussion is now moved to a javascript-specific repo - will this be implemented for all languages?

My use case is that I want to run a debugger session with the current date as a parameter to the program to be debugged. I can get the current date with date -I and would like to use the output of that command as parameter.

[weinand]

@hansmbakker VS Code's underlying support for this feature will work for all debuggers, but they have to adopt it in their implementation (which depends on the extension author).

Yes, it should be possible to use `date -I` (in backticks) for unix shells.