get_properties: a property collector action

[jeking3]

The property collector is a more efficient way to retrieve properties from vSphere. If you use get_properties it can replace things like vm_guest_info_get and vm_runtime_info_get as it allows for getting arbitrary properties from types supported by CreateContainerView.

96% coverage on tests for get_properties (in the vSphere pack)

Example: Datacenter name and status

# st2 run vsphere.get_properties type=Datacenter property=name property=overallStatus
.
id: 5bb77101668fd4010c013067
status: succeeded
parameters:
  property:
  - name
  - overallStatus
  type: Datacenter
result:
  exit_code: 0
  result:
    datacenter-2:
      name: MyDatacenter
      overallStatus: gray
  stderr: ''
  stdout: ''

Example: Select properties from a couple VMs

# st2 run vsphere.get_properties type=VirtualMachine property=name property=guest.ipAddress property=config.hardware.memoryMB id=vm-45 id=vm-22
.
id: 5bb77182668fd4010c01306a
status: succeeded
parameters:
  id:
  - vm-45
  - vm-22
  property:
  - name
  - guest.ipAddress
  - config.hardware.memoryMB
  type: VirtualMachine
result:
  exit_code: 0
  result:
    vm-22:
      config.hardware.memoryMB: 10240
      guest.ipAddress: 10.99.0.4
      name: VMware vCenter Server Appliance
    vm-45:
      config.hardware.memoryMB: 4096
      guest.ipAddress: 10.99.20.100
      name: W2016 Test
  stderr: ''
  stdout: ''

Example: Retrieving an array property or a property that is a class

root@b9bab223e4fa:/# st2 run vsphere.get_properties type=VirtualMachine id=vm-22 property=guest.disk
.
id: 5bbe5064366fe70104cebe46
status: succeeded
parameters:
  id:
  - vm-22
  property:
  - guest.disk
  type: VirtualMachine
result:
  exit_code: 0
  result:
    vm-22:
      guest.disk:
      - capacity: 11355283456
        diskPath: /
        freeSpace: 4859494400
      - capacity: 125783040
        diskPath: /boot
        freeSpace: 97401856
      - capacity: 26279735296
        diskPath: /storage/core
        freeSpace: 26233651200
...
  stderr: ''
  stdout: ''

I have been able to use the results from these queries to publish variables in workflows. In an orquesta workflow for the second example above, one could extract the memory size (assuming an input called vm_id which is the MOID):

    next:
      - when: <% succeeded() %>
        publish:
          - vm_mem: <% result().result[ctx().vm_id]['config.hardware.memoryMB'] %>

[jeking3]

I think it may be necessary to leverage the serialize internals used in vm_guest_info_get (for example) to deal with serializing results that are complex types (pyVmomi). This will allow things like guest.disk or network (which is an array) to be retrieved and serialized properly as structured results.

[jeking3]

@LindsayHill This one is ready for review.

[jeking3]

I added code to strip out the useless dynamicType and dynamicProperty if they are None and [] respectively.

[jeking3]

So I am seeing this in some runs this morning:

Traceback (most recent call last):
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 320, in <module>
    obj.run()
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 179, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs.dev/vsphere/actions/get_properties.py", line 43, in run
    return self.collect(self.si_content, type, property, id, raw)
  File "/opt/stackstorm/packs.dev/vsphere/actions/get_properties.py", line 97, in collect
    return self.transform(ids, rawdata) if not raw else rawdata
  File "/opt/stackstorm/packs.dev/vsphere/actions/get_properties.py", line 131, in transform
    ps[unquote(prop.name)] = self.jsonify_vsphere_obj(prop.val)
  File "/opt/stackstorm/packs.dev/vsphere/actions/get_properties.py", line 123, in jsonify_vsphere_obj
    return json.loads(PyVmomiObjectJSONEncoder().encode(obj))
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/opt/stackstorm/packs.dev/vsphere/actions/get_properties.py", line 113, in default
    tmp = copy.deepcopy(obj.__dict__)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 351, in _reconstruct
    item = deepcopy(item, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 298, in _deepcopy_inst
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
    y = callable(*args)
  File "/opt/stackstorm/virtualenvs/vsphere/lib/python2.7/copy_reg.py", line 93, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

I'd wait on merging this until I figure it out.

[jeking3]

This will be changing, as I am pushing a JSON encoder down into pyVmomi.

[jeking3]

This is blocked waiting for:

1. This project to support pyVmomi > 5.5
2. Add VmomiSupport.VmomiJSONEncoder for encoding pyVmomi objects as JSON vmware/pyvmomi#732 to be merged upstream (pyVmomi>6.7.1)

[jeking3]

pyVmomi was updated - I'll finish this off...

[blag]

@jeking3 Any update to this? I'm liking this PR.

[jeking3]

I wish I had better news but sadly we're not using SS and I can't allocate the time to bring it back up and implement this, sorry.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}