Almost all the windows machines are DOWN

[UlisesGascon]

Overview

This incident started the last Friday, seems like all the testing windows machines are currently down, in the release ci some windows machines are running.

Slack discussion

Release machines

Test machines

Next steps

• Reboot all the machines

[UlisesGascon]

Seems like the root cause is a connectivity issue, I will check Azure and Rackspace

[UlisesGascon]

I rebooted the machines in Azure and Rackspace. Overall, around 20 machines are back. Most of the down machuines currently are from Rackspace

Relese

Testing

@nodejs/build: I will suggest to check more in detail the root cause and the logs from the machine to check by the connection with Jenkins is not properly stablished.

[rluvaton]

This may be related to recent Jenkins upgrade (Cc @targos )

[StefanStojanovic]

A quick update: I continued what @UlisesGascon started and went through machines and got all of them except a few back online. Looks like the worst has passed, but this should be investigated so we know the root cause.

[UlisesGascon]

Thanks a lot @StefanStojanovic!

[mhdawson]

Discussed in the Build WG meeting, agreed to close. Will re-open if we get another similar outage.