Invoke host-select via bash login shell

[MetRonnie]

This closes #2552

Changes the command that gets run in the background, as follows:

 ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 <hostname> \
-    env CYLC_VERSION=<version> rose host-select-client
+    env CYLC_VERSION=<version> CYLC_ENV_NAME=<$CYLC_ENV_NAME> bash -l -c 'rose host-select-client'

[oliver-sanders]

Looks good, just need to drag through the value of CYLC_ENV_NAME provided by the wrapper script.

[MetRonnie]

Might have to use exec, let me see

[MetRonnie]

@oliver-sanders Any idea why this fails for localhost? I tried using exec instead of shlex.quoteing the command but that didn't work either

$ rose host-select -vv localhost
[INFO] Rank method: load
[INFO] env CYLC_VERSION=8.0rc2.dev bash -l -c \'rose\ host-select-client\' <<'__STDIN__'
[INFO] 
[INFO] ***start**
[INFO] [["getloadavg"], ["cpu_count"]]
[INFO] **end**
[INFO] 
[INFO] __STDIN__
[WARN] <name-of-localhost>: (failed 127)
[FAIL] No hosts selected.

[oliver-sanders]

Not sure, worth checking whether stdin is being passed through properly.

[dpmatthews]

The escaping is the problem:

$ ssh $(hostname) env CYLC_ENV_NAME=cylc-8.0rc2.dev-1 bash -l -c \'rose\ version\'
rose 2.0rc2.dev
$ env CYLC_ENV_NAME=cylc-8.0rc2.dev-1 bash -l -c \'rose\ version\'
bash: rose version: command not found
$ env CYLC_ENV_NAME=cylc-8.0rc2.dev-1 bash -l -c 'rose version'
rose 2.0rc2.dev

[MetRonnie]

The escaping is only in the reporting/logging, it doesn't affect the call to subprocess.Popen(). You can see this by applying

diff --git a/metomi/rose/popen.py b/metomi/rose/popen.py
index 6a5249e5..074ade25 100644
--- a/metomi/rose/popen.py
+++ b/metomi/rose/popen.py
@@ -132,7 +132,7 @@ class RosePopener:
     def list_to_shell_str(cls, args):
         if not args:
             return ""
-        return " ".join([re.sub(r"([\"'\s])", r"\\\1", arg) for arg in args])
+        return " ".join(args)
 
     def __init__(self, event_handler=None):
         self.event_handler = event_handler

and it still fails in the same way

[MetRonnie]

Ah wait, I think I've got it

[MetRonnie]

I've figured out why the tests are failing this time round. It has brought up a question: rose-app.conf[poll]all-files and rose-app.conf[poll]any-files seem to accept globs but this is not documented in https://metomi.github.io/rose/2.0rc1/html/api/configuration/application.html#rose:conf:rose-app.conf[poll]all-files

I have updated the docs

[dpmatthews]

I've tested this but not reviewed the change

[MetRonnie]

MacOS test showing as failed but when you click on it the "re-run failed tests" step actually passed

[oliver-sanders]

MacOS test showing as failed but when you click on it the "re-run failed tests" step actually passed

For the Rose repo we re-run tests in serial purely for debugging purposes, they are expected to pass first time around in parallel.

(in Cylc the functional tests are sufficiently flaky we've had no choice but to rerun them but this is really bad and something we should dispense with one day)

[MetRonnie]

Actually the MacOS tests passed the first time around in the previous commit: https://github.com/metomi/rose/runs/5536961434?check_suite_focus=true

Most recent commit is only a change to documentation so shouldn't have affected it

[oliver-sanders]

(easier just to re-run)

(failed again, but only one test this time)

(kicked off a run against master to see if this issue has crept in https://github.com/metomi/rose/actions/runs/1986244844)

[oliver-sanders]

Looks good.

[oliver-sanders]

Tests passed first time on master https://github.com/metomi/rose/actions/runs/1986244844

Perhaps some change here made tests incompatible when running in parallel?

[MetRonnie]

Reran on master, failed: https://github.com/metomi/rose/actions/runs/1986244844

[oliver-sanders]

The error from the failed jobs is genuine and looks the same: #2558

[oliver-sanders]

Tested, everything worked as intended.

Had a quick look to see why we weren't quoting commands in the Popen thinggy, I think this is perfectly sensible.

[oliver-sanders]

Strangely t/rose-host-select/01-timeout.t failed, but only for Python 3.7 on Linux?

--- -	2022-03-16 15:26:42.653472890 +0000
+++ mock-ssh.out.sorted	2022-03-16 15:26:42.646377238 +0000
@@ -1,2 +1 @@
-sleepy1 bash -l -c 'rose host-select-client'
 sleepy2 bash -l -c 'rose host-select-client'

https://github.com/metomi/rose/runs/5572332850?check_suite_focus=true#step:12:238

[wxtim]

• Read code
• Functionally tested