Controlling Clouds – Use Azure Batch to control Google Cloud(gcloud)

[itoleck]

Sometimes you want to monitor or control your cloud from another separate cloud for reliability or maybe you just like to use Microsoft Azure but have a few resources in Google Cloud to manage; either way, using the below steps you can utilize Azure Batch to manage your Google Cloud resources.

The Google SDK is used to manage Google Cloud from the command line. There are a few options for using gcloud SDK commands; gcloud.cmd, gcloud-ps.ps1 and gcloud.py. In this example we will use gcloud.cmd but with changes others can be used as well.

The first step in automating the gcloud batch commands is to have a machine with the gcloud SDK installed. In this example we are going to use Windows Server but you could also use Linux with some modifications.

Step 1: Create a Virtual Machine

Create a new Windows Server 2019 Datacenter virtual machine with a small disk to save on disk costs. There should not be much data usage on the operating system disk so a small disk will be plenty of room. This can be done using the Portal, Azure PowerShell, Azure(az) Command, REST or many other ways. We will use az.

There should be an available Resource Group and Virtual Network available in the Azure location you want to create the Azure Batch account in.

az vm create -g MyResourceGroup -n gcloudwintst --image MicrosoftWindowsServer:WindowsServer:2019-Datacenter-smalldisk:latest -l "South Central US" --size "Standard_F1" --admin-username gadmin --admin-password Azureisfun! --vnet-name MyVNet --subnet Default --storage-sku Standard_LRS

After this command completes you should have a virtual machine with a public IP address that can be connected to via RDP. If you do not need an external IP address you can delete it from the network adapter and connect to the internal IP address.

Login to the Virtual Machine and download the gcloud SDK using a PowerShell window.

(New-Object System.Net.WebClient).DownloadFile("https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-301.0.0-windows-x86_64-bundled-python.zip",".\gcloud.zip") Expand-Archive -Path .\gcloud.zip -DestinationPath c:\gcloud\ -Force c:\gcloud\google-cloud-sdk\install.bat --quiet

You will be asked some questions on if you want to sent info to Google and if you want to update the %PATH% variable.

y/n to send info to Google

y/n to update %PATH% This usually fails and needs to be checked and done manually if not updated

After the install is finished update the %PATH% variable manually.

setx /M PATH "$Env:PATH;C:\gcloud\google-cloud-sdk\bin"

Open new PS or CMD window and run a test gcloud command from another path to test that the %PATH% was updated. It is also good to update the Gcloud SDK at this point.

glcoud.cmd --version gcloud.cmd components update

The results of the gcloud version command should look similar to the below.

Step 2: Google Cloud

Connect to Google Cloud (https://console.cloud.google.com/) and download the service account’s .json key for the project you want to test management.

*** You will specify the .json file for the Azure Batch account later. This step is to test connectivity under your user account.

Go to IAM & Admin –> Service Accounts. Then create a new service account with the permissions needed for the commands you will run on Azure Batch.

When the account is created create a new key and select to download as .json.

Download the .json file and copy it to the Azure Batch template VM. After copying the .json key add the key to the gcloud profile and set the current project using the project name.

gcloud.cmd auth activate-service-account --key-file .\<projectkeyfile.json> gcloud.cmd config set project "<projectid>"

After the commands run successfully you can check the credentials and project are set.

gcloud.cmd auth list gcloud.cmd projects list

If you have virtual machines in your project you could list them now with gcloud.cmd.

gcloud.cmd compute instances list

If the gcloud commands work then you can finish installing and configuring the operating system; set time/zone, install tools, set security policies, etc.

Step 3: Snapshot VM

When the configuration of the VM is complete you will want to use this VM as a template for the Azure Batch pool nodes. This can be done by adding the snapshot to a Shared Image Gallery. By creating a snapshot and new Shared Image Gallery version, you can keep updating the source VM with new versions of the Gcloud SDK and OS updates.

The Azure Batch nodes can be scaled based on auto scale rules; https://docs.microsoft.com/en-us/azure/batch/batch-automatic-scaling or manually set to a specific number of nodes.

To make a snapshot of the VM go to the VM in the Azure Portal and click the “Disks” button in the menu. Click on the OS Disk name and then click “Create snapshot”.

Create a new Shared Image Gallery.

Step 4: Azure Batch Setup

In Azure Batch, create a new pool. Select Shared Image Gallery for the image type and select the image version created in previous steps. Select the OS distribution and SKU that most closely matches the image. Select a VM size for the node/s in the pool and set the networking options.

Create a new job and select the pool created in the last step.

To run the gcloud commands each command can be ran in a batch task. Credentials will be needed to successfully run the glcoud commands. The previous key file was saved under your user account but Azure batch runs under it’s own user so we have to save the key for the Azure Batch user account. This only needs to be done once or if the key or project changes.

Set the default project for the Azure Batch account.

That’s it, now you can run gcloud commands in Azure Batch and schedule jobs to manage Google Cloud from Azure Batch.

If you want to run more than one command at a time you can create a script on the batch template VM and create a new image, manually update the pool node or create and application package with the script inside that can multiple commands in 1 batch task.