ESX back-end support #30

grleblanc · 2017-02-16T20:29:19Z

I love this! I tend to use my own ESX server for malware analysis. I am thinking about integrating ESX into malboxes with the Vagrant ESX plugin.

Any interest?

Svieg · 2017-02-16T20:37:27Z

That would be really cool! If I were you, I would check for Packer templates as it is the main tool we wrap around. You could check this for example: https://github.com/tsugliani/packer-vsphere-templates. Don't hesitate if you have any questions!

grleblanc · 2017-02-23T01:39:00Z

So I pretty much have it working. I'm trying to go through and figure out the past way to do this, if we want to merge this in with the current malboxes code.

I currently only implemented it on the win10_64_analyst.json profile, it requires pretty much a complete change. Also, the Autounattended.xml need a small change in the DiskConfiguration section. Lastly I had to change box_win.rb file, but that shouldn't be a big deal because we can add logic in there to do the vmware piece of it only if the provider is vmware-iso.

Here's my proposal:

Make a change to the config-example.js file to offer up a "provider" (doesn't have to use that name) section
If it's set to "esx" - we can ensure they have all of the ESX server host information in there.
Then we'd have to edit malboxes.py to ensure the right profiles and Autounattended.xml are called.

Does that make sense? Would love any sort of discussion on possible other ideas or options. In reality malboxes should work with VirtualBox, VMWare Workstation, ESX - which are probably the 3 most popular.

Svieg · 2017-02-23T02:55:29Z

I agree with the additionnal provider option in config.js ! Can you make your fork available, I don't see on your profile? Everything sounds good for me.

grleblanc · 2017-02-23T03:04:20Z

Really? It should be here: https://github.com/gleblanc1783/malboxes

I will commit my changes once I am done testing tonight. Then will work on making the other changes to the other profiles and any edits to malboxes.py.

Svieg · 2017-02-23T03:06:15Z

Yeah, my bad, I should buy glasses I think eheh. Cool! I look forward to review that!

obilodeau · 2017-02-23T14:44:16Z

Your proposal sounds great @gleblanc1783!

I would also be interested in hearing performance numbers. How fast is using ESX versus VirtualBox?

grleblanc · 2017-02-28T01:21:25Z

Okay a solid first pass at this is done. You can view the changes here, and I will have a PR submitted sometime this week after we review.

Sorry it took me so long, between the time it took to consistently try and create the windows images and working through some non-specific bugs and gotchas with Packer and Vagrant it's working now.

Description
As mentioned above, because of how you originally developed the program, not too many changes had to have been made.

1. Config changes

I added a "provision settings" area in the example-config.js where all of the required settings will be added

	//Provision settings
	//Which Hypervisor for privisoning and deployment? (Options are: "virtualbox" and "vsphere") Default is "virtualbox"
	"hypervisor": "vsphere",
	//If vsphere, the following configuration options are mandatory
	"remote_host": "",
	"remote_datastore": "",
	"remote_username": "",
	"remote_password": "",
	"vsphere_host": "",
	"vsphere_clone_from_vm" = "",
	"vsphere_name" = "malboxestest",
	"vsphere_user" = "",
	"vsphere_password" = "",
	"vsphere_insecure" = "true"

You can either choose vsphere or virtualbox for now and I think that leaves us with more room to expand on the number of hypervisors that are supported

2. New builder

I created "builder_vsphere_windows.json" as an addition to the virtualbox builder. This has all of the vSphere specific configuration options. The variables in here are taken from the config file
Note: Until I can find some other way to do this, I needed to create a new "installconfig" directory named "windows8srv-64" as vSphere doesn't have a Windows10 default os type (the recommended name is windows8srv-64.

3. Vagrant

I created an "analyst_vsphere.rb" file which has all of the required Vagrant vsphere information it it. It also takes in information from the config file

Requirements

OVFTool (https://www.vmware.com/support/developer/ovf/)
-This has one specific external plugin requirement for vSphere.
Note: If you have the paid/commercial version of vSphere, then you will need this plugin. If you only have the free version, you can still get this to work, just without utilizing the malboxes spin command (more information below)
To obtain the plugin simply run the following command
vagrant plugin install vagrant-vsphere
I also used this tutorial to help with the initial configuration of the "dummy" box (http://blog.broez.com/vagrant-vsphere-centos-7/)

Usage

Make any required changes to your config.js, filling in all of the required fields
Run malboxes build {profile}
If you're using the Vagrant plugin, now you would run malboxes spin {profile} {name}
3a. This will clone your current VM that was built on the vSphere server and name it the {name} value, and you should be all set to use it.
3b. Note: This currently is not set to delete the original VM (as designed), see TODO section for more information. This VM will the original one that will be cloned for that profile
Run vagrant up --provider=vsphere
Once done using the VM, run vagrant halt

Note: If you're not using paid/commercial version of vSphere, you can still use this. Just log into your ESXi server, and clone the VM yourself OR use the base image created by malboxes build

TODO

Implement Windows7x64
Add/Implement the other features of malboxes like registry, directory, document
Figure out what the, hopefully eventual, official name for Windows 10 will be as a guest-os

Please let me know if this works for you and any bugs/concerns!

obilodeau · 2017-03-23T23:56:46Z

Everything you say sounds fine. Send the pull-request! Let me know if you need a hand in doing that.

treed593 · 2017-05-31T18:07:08Z

Was this pulled in? @obilodeau @gleblanc1783

Svieg · 2017-05-31T18:11:40Z

No PR was made and no commits have been done after February 27th on his fork so I don't know what is the status of this.

obilodeau · 2017-05-31T18:55:09Z

Even if @gleblanc1783 is no longer interested, I am. I will get back to it and merge his stuff.

I now have access to an ESX server with enough resources to do some tests.

grleblanc · 2017-05-31T18:56:20Z

Sorry, all. I've been busy with work and sidetracked with fixing my ESX Server to test this again.

@obilodeau -- feel free to take my code and run with it. I was pretty close.

Will be following this thread to see how it makes out.

treed593 · 2017-06-01T02:38:46Z

I got it working today, for the most part. Couldn't get winrm to connect though

…

On May 31, 2017 2:56 PM, "Gregory LeBlanc" ***@***.***> wrote: Sorry, all. I've been busy with work and sidetracked with fixing my ESX Server to test this again. @obilodeau <https://github.com/obilodeau> -- feel free to take my code and run with it. I was pretty close. Will be following this thread to see how it makes out. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGffqsPoOX2U-PxygEwLE4YRZo4oL5nBks5r_bfVgaJpZM4MDgXD> .

Svieg · 2017-06-01T02:58:56Z

@treed593 have you tried increasing the timeout limit?

treed593 · 2017-06-01T03:01:13Z

Not yet, that's the next thing I plan on trying in the morning. It didn't look like the script to enable winrm had run either

…

On May 31, 2017 10:58 PM, "Hugo Genesse" ***@***.***> wrote: @treed593 <https://github.com/treed593> have you tried increasing the timeout limit <https://github.com/GoSecure/malboxes/blob/master/malboxes/profiles/snippets/builder_virtualbox_windows.json#L8> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGffqph78HnLC203mjfTx6BqfB5j49LXks5r_ijxgaJpZM4MDgXD> .

treed593 · 2017-06-01T15:32:00Z

I extended the timeout, so we will see but the VM still seems to just be sitting on the desktop with the Network "Do you want to allow sharing" window open @Svieg

Svieg · 2017-06-01T15:35:00Z

What version of windows are you trying to install? Might be something wrong with the Autounattend.xml. Have you made changes to the config or something like that?

treed593 · 2017-06-01T15:39:23Z

win10_64, I didn't but @gleblanc1783 may have in his fork. I will compare the one from malboxes/master to gleblanc1783/master

treed593 · 2017-06-01T17:36:11Z

Looks like the issue was a network issue with the ESX server. I am working through some other issues now with chocolatey

treed593 · 2017-06-01T22:39:57Z

I was able to get winrm and chocolatey to build correctly, the last issue I am facing is:

==> vmware-iso: Keeping virtual machine registered with ESX host (keep_registered = true)
Build 'vmware-iso' errored: lstat malboxes: no such file or directory

==> Some builds didn't complete successfully and had errors:
--> vmware-iso: lstat malboxes: no such file or directory

==> Builds finished but no artifacts were created.

Svieg · 2017-06-01T22:43:16Z

Sorry for not saying that earlier but can you run malboxes with the -d switch. You'll have a much more complete output. Never had that error personnally, so that's what I would try first.

treed593 · 2017-06-02T15:13:00Z

2017/06/02 11:11:14 ui error: Build 'vmware-iso' errored: lstat malboxes: no such file or directory
2017/06/02 11:11:14 Builds completed. Waiting on interrupt barrier...
2017/06/02 11:11:14 machine readable: error-count []string{"1"}
2017/06/02 11:11:14 ui error:
==> Some builds didn't complete successfully and had errors:
2017/06/02 11:11:14 machine readable: vmware-iso,error []string{"lstat malboxes: no such file or directory"}
2017/06/02 11:11:14 ui error: --> vmware-iso: lstat malboxes: no such file or directory
2017/06/02 11:11:14 ui:
==> Builds finished but no artifacts were created.
2017/06/02 11:11:14 waiting for all plugin processes to complete...
Build 'vmware-iso' errored: lstat malboxes: no such file or directory

==> Some builds didn't complete successfully and had errors:
--> vmware-iso: lstat malboxes: no such file or directory

==> Builds finished but no artifacts were created.
2017/06/02 11:11:14 /usr/bin/packer: plugin process exited
2017/06/02 11:11:14 /usr/bin/packer: plugin process exited
2017/06/02 11:11:14 /usr/bin/packer: plugin process exited

packer completed with return code: 1
Packer failed. Build failed. Exiting...

treed593 · 2017-06-05T16:39:03Z

Any ideas @Svieg

Svieg · 2017-06-05T17:42:48Z

Not really, it doesn't seem to find the malboxes file and and I don't why it is looking for it honestly as it the host script. I don't have access to an ESX server too. Maybe @obilodeau can save the day

treed593 · 2017-06-05T17:46:58Z

Yea it is strange, hope @obilodeau or @gleblanc1783 can help

obilodeau · 2017-06-05T17:49:35Z

I'm finishing another project and I have a presentation coming up this week, I don't h ave time to reproduce right now, unfortunately.

For some reason, the vmware-iso plugin doesn't have access to the malboxes files. This might be due to several reasons. I would search for packer related problems with missing files removing references to "malboxes" and try to find the root cause.

grleblanc · 2017-06-05T20:16:57Z

Did you pull my code from my repo? I think I got around this, trying to remember.

In your config file for packer did you change the format to "ovf"?

hashicorp/packer#3087

treed593 · 2017-06-05T20:19:24Z

Yea, I pulled from you and it was over, yes

…

On Jun 5, 2017 4:17 PM, "Gregory LeBlanc" ***@***.***> wrote: Did you pull my code from my repo? I think I got around this, trying to remember. In your config file for packer did you change the format to "ovf"? hashicorp/packer#3087 <hashicorp/packer#3087> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGffqtK1MMKmErCQbTRUhWVj-w4c4Hj7ks5sBGI7gaJpZM4MDgXD> .

obilodeau · 2017-07-26T01:32:13Z

@treed593, @gleblanc1783 I got working malboxes builds with the code in #68 (which includes @gleblanc1783's early work) that just landed in master and which will be part of the 0.3.0 release.

Go and give it a try and let me know how it goes for you.

obilodeau added the enhancement label Jul 11, 2017

obilodeau changed the title ~~ESX Plugin~~ ESX back-end support Jul 11, 2017

obilodeau modified the milestone: 0.3.0 - BlackHat USA 2017 edition Jul 13, 2017

This was referenced Jul 18, 2017

WIP: ESX support #67

Closed

ESX support #68

Merged

obilodeau closed this as completed Jul 26, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESX back-end support #30

ESX back-end support #30

grleblanc commented Feb 16, 2017

Svieg commented Feb 16, 2017

grleblanc commented Feb 23, 2017 •

edited

Loading

Svieg commented Feb 23, 2017

grleblanc commented Feb 23, 2017

Svieg commented Feb 23, 2017

obilodeau commented Feb 23, 2017

grleblanc commented Feb 28, 2017 •

edited

Loading

obilodeau commented Mar 23, 2017

treed593 commented May 31, 2017

Svieg commented May 31, 2017

obilodeau commented May 31, 2017

grleblanc commented May 31, 2017

treed593 commented Jun 1, 2017 via email

Svieg commented Jun 1, 2017

treed593 commented Jun 1, 2017 via email

treed593 commented Jun 1, 2017

Svieg commented Jun 1, 2017

treed593 commented Jun 1, 2017

treed593 commented Jun 1, 2017

treed593 commented Jun 1, 2017

Svieg commented Jun 1, 2017

treed593 commented Jun 2, 2017

treed593 commented Jun 5, 2017

Svieg commented Jun 5, 2017

treed593 commented Jun 5, 2017

obilodeau commented Jun 5, 2017

grleblanc commented Jun 5, 2017

treed593 commented Jun 5, 2017 via email

obilodeau commented Jul 26, 2017

ESX back-end support #30

ESX back-end support #30

Comments

grleblanc commented Feb 16, 2017

Svieg commented Feb 16, 2017

grleblanc commented Feb 23, 2017 • edited Loading

Svieg commented Feb 23, 2017

grleblanc commented Feb 23, 2017

Svieg commented Feb 23, 2017

obilodeau commented Feb 23, 2017

grleblanc commented Feb 28, 2017 • edited Loading

obilodeau commented Mar 23, 2017

treed593 commented May 31, 2017

Svieg commented May 31, 2017

obilodeau commented May 31, 2017

grleblanc commented May 31, 2017

treed593 commented Jun 1, 2017 via email

Svieg commented Jun 1, 2017

treed593 commented Jun 1, 2017 via email

treed593 commented Jun 1, 2017

Svieg commented Jun 1, 2017

treed593 commented Jun 1, 2017

treed593 commented Jun 1, 2017

treed593 commented Jun 1, 2017

Svieg commented Jun 1, 2017

treed593 commented Jun 2, 2017

==> Builds finished but no artifacts were created. 2017/06/02 11:11:14 /usr/bin/packer: plugin process exited 2017/06/02 11:11:14 /usr/bin/packer: plugin process exited 2017/06/02 11:11:14 /usr/bin/packer: plugin process exited

treed593 commented Jun 5, 2017

Svieg commented Jun 5, 2017

treed593 commented Jun 5, 2017

obilodeau commented Jun 5, 2017

grleblanc commented Jun 5, 2017

treed593 commented Jun 5, 2017 via email

obilodeau commented Jul 26, 2017

grleblanc commented Feb 23, 2017 •

edited

Loading

grleblanc commented Feb 28, 2017 •

edited

Loading

==> Builds finished but no artifacts were created.
2017/06/02 11:11:14 /usr/bin/packer: plugin process exited
2017/06/02 11:11:14 /usr/bin/packer: plugin process exited
2017/06/02 11:11:14 /usr/bin/packer: plugin process exited