Configure ironic to use Ipmitool retries

[maelk]

Without this configuration, the ipmitool timeout is 1 second. This is too short
for vbmc. This commit uses the ipmitool retry feature and extends the
timeout.

This PR also set console=ttyS0 in the IPA kernel parameter to gather IPA logs
in Metal3-dev-env

[maelk]

/test-integration

[maelk]

/test-integration

[dtantsur]

Without this configuration, the ipmitool timeout is 1 second.

Why do you think so? The -R and -N option specify the retry number and the delay between retries. The global ironic retries and timeout still apply (unless we have a bug, obviously).

[maelk]

if Ironic is handling the retires and timeouts, it sets the -R and -N options to 1 : https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/ipmitool.py#L483 . In that case the internal ipmitool timeout is 1 second. I agree that ironic would retry later on. But in metal3-dev-env this 1s timeout is too short on a lot of operations. So letting ipmitool handle the timeout and retries allow us to wait longer for the vbmc answer.

[maelk]

/test-integration

[dtantsur]

Mmm, I think I understand what you're getting at. The retransmission rate is too fast? Do you have a link to example where you hit this?

@bfournie FYI, it may be a problem with your patch.

[maelk]

The issue is rather that ipmitool does not wait long enough before giving up when running with -N 1. And when you configure the retries to be handled by ironic, the interval between retries is set properly, but the timeout is forced to be 1 , i.e. you send a request, fail after 1s and wait 4s before sending the next request. That's the problematic part for Metal3 Dev env

[maelk]

/assign @russellb
/cc @dtantsur
/cc @dhellmann

[maelk]

This PR fixes the error that is otherwise visible in ironic logs :

2020-07-08 14:44:57.013 119 DEBUG oslo_concurrency.processutils [req-bbc52505-10b7-42d0-9d29-1b1d7675eecc - - - - -] 'ipmitool -I lanplus -H 192.168.111.1 -L ADMINISTRATOR -p 6230 -U admin -R 1 -N 1 -f /tmp/tmpwchdt9mc power status' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:455�[00m
2020-07-08 14:44:57.015 119 WARNING ironic.drivers.modules.ipmitool [req-bbc52505-10b7-42d0-9d29-1b1d7675eecc - - - - -] IPMI Error encountered, retrying "ipmitool -I lanplus -H 192.168.111.1 -L ADMINISTRATOR -p 6230 -U admin -R 1 -N 1 -f /tmp/tmpwchdt9mc power status" for node 99970ccd-4deb-4d48-b624-77bc60b26bc0. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.111.1 -L ADMINISTRATOR -p 6230 -U admin -R 1 -N 1 -f /tmp/tmpwchdt9mc power status
Exit code: 1
Stdout: ''
Stderr: 'Unable to get Chassis Power Status\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.�[00m

Ironic usually retries but sometimes fails all the retries. By waiting a bit more, there is no need for retries. the ironic logs for an example of a failed CI run can be found here : https://jenkins.nordix.org/view/Metal3/job/airship_master_v1a3_integration_test_ubuntu/243/artifact/logs-jenkins-airship_master_v1a3_integration_test_ubuntu-243.tgz (in the docker folder)

[maelk]

For comparison purposes, with this PR : https://jenkins.nordix.org/job/airship_metal3io_ironic_image_v1a3_integration_test_ubuntu/47/artifact/logs-jenkins-airship_metal3io_ironic_image_v1a3_integration_test_ubuntu-47.tgz . All ipmitools related errors are gone.

[dhellmann]

The description makes sense. Let's see if these changes make CI more stable.

[metal3-io-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhellmann, maelk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dhellmann]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dhellmann]

/lgtm