AWS feature for Malboxes #115

Bromulux · 2019-02-20T20:29:41Z

This allows Malboxes to create and run VMs on the Amazon Elastic Compute Cloud (EC2).

Deprecated URL for Windows10_64 has been replaced by a working one.

The build command and the end message are now adapted for the cloud feature.

The AMI description is no longer the default one and now gives information about the packer template used.

The code related to the AWS feature for Windows 10 is now documented,

The AWS feature now supports Windows 7 64x

The AMI ID is now retrieved from AWS by the library Boto 3 instead of a post-processor. Plus, an error message was added if you try to build more than once the same template.

Boto3 uses the AWS credentials from the config.js file instead of the ones stored in environmental variables.

The region can now be set in config.js

WinRM is now working even if the network profile is configured as public

Boto3 is now in the requirements.txt

The instance type and the security group can now be set by the user in the config.js

As a workaround for the public network connectivity problem, a new rule has been created with a public network type.

Windows 7 32-bit can now be run on AWS

It is now explained how to use the AWS feature of Malboxes in the README.

obilodeau

Excellent work! 🤘

Some notes but nothing major. You can address most of these now I think. Tomorrow, together, we can look for the ESX use-case together since that one is badly documented and a little bit complex.

obilodeau · 2019-02-26T20:18:48Z

README.adoc

@@ -40,6 +41,12 @@ https://github.com/gosecure/malboxes

    apt install vagrant git python3-pip

+=== AWS


For the title, I would use something clearer like:

=== To Deploy to AWS (Optional)

Also, this section should be moved at the end of the Installation section.

obilodeau · 2019-02-26T20:21:12Z

README.adoc

@@ -136,6 +143,50 @@ For example:

    malboxes spin win7_32_analyst 20160519.cryptolocker.xyz

+=== AWS


also:

=== To Deploy to AWS (Optional)

obilodeau · 2019-02-26T20:30:15Z

README.adoc

+
+. Your instance also requires a link:https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#CreatingSecurityGroups[security group] with at least a rule allowing inbound connections for WinRM (Type: WinRM-HTTP, Protocol: TCP, Port Range: 5985, Source: host's public IP).
+
+. If the default config is used, change the hypervisor to aws and fill the mandatory options related. Otherwise, be sure to add all the options about AWS to your custom config.


You can link to config with the following AsciiDoc syntax: <<_configuration,The configuration section>>.

I would do:

If the <<_configuration,default config>> is used [...]

obilodeau · 2019-02-26T20:32:23Z

README.adoc

+
+==== RDP
+
+To connect to an instance on the cloud using RDP, run this command at the same location of your `+Vagrantfile+`: 


I don't think you need the + around Vagrantfile.

obilodeau · 2019-02-26T20:33:19Z

README.adoc

+
+For this to work, the instance will require a security group allowing RDP inbound connections (Type: RDP, Protocol: TCP, Port Range: 3389, Source: host's public IP).
+
+NOTE: You can safely ignore the following error because rsync is not yet implemented:  << No host IP was given to the Vagrant core NFS helper. This is an internal error that should be reported as a bug. >> 


Wrap the error in backticks (`) so that it looks like terminal output. You can remove the << and >> if you do this.

obilodeau · 2019-02-26T20:58:54Z

malboxes/malboxes.py

 def spin(parser, args):
    """
    Creates a Vagrantfile meant for user-interaction in the current directory.
    """
    if os.path.isfile('Vagrantfile'):
        print("Vagrantfile already exists. Please move it away. Exiting...")
-        sys.exit(5)
+        sys.exit(6)


why change this? if there is no explicit reason we shouldn't change because users might rely on it.

obilodeau · 2019-02-26T20:59:45Z

malboxes/scripts/windows/malware_analysis.ps1

@@ -5,4 +5,4 @@ Set-ItemProperty -Path "HKLM:\System\CurrentControlSet\Control\Terminal Server"
 netsh advfirewall firewall set rule group="remote desktop" new enable=Yes

 # IDA Remote Debugging
-netsh advfirewall firewall add rule name="IDA Remote Debugging" dir=in action=allow protocol=TCP localport=23946
+netsh advfirewall firewall add rule name="IDA Remote Debugging" dir=in action=allow protocol=TCP localport=23946


no-op changes should be removed to reduce unnecessary work on the reviewer's end

obilodeau · 2019-02-26T21:01:37Z

malboxes/templates/win10_32_analyst.json

@@ -22,8 +22,6 @@
 		]
 	}],

-	{% include 'snippets/postprocessor_vagrant.json' %},
-


why is this removed? This looks like a mistake.

Oh, it looks like it was there two times by mistake... Wouldn't removing this affect the ESX use-case?

obilodeau · 2019-02-26T21:03:58Z

malboxes/templates/win7_32_analyst.json

@@ -23,7 +23,11 @@
 		]
 	}],

-	{% include 'snippets/postprocessor_vagrant.json' %},
+	{% if hypervisor == 'virtualbox' %}
+		{% include 'snippets/postprocessor_vagrant.json' %},


Confirm that this doesn't impact ESX use-case

obilodeau · 2019-02-26T21:04:13Z

malboxes/templates/win7_64_analyst.json

@@ -23,7 +23,11 @@
 		]
 	}],

-	{% include 'snippets/postprocessor_vagrant.json' %},


same as above

General corrections of the README requested by obilodeau.

The imports now respect the PEP 8.

Constants were added to describe the exit status codes.

The function now follows the DRY principle and has a more precise name.

Each line of the code related to AWS has now 80 chars or less.

Details about the vagrant box were removed in the build message because the user does not need to know them.

Now there's two new lines between functions to respect the PEP 8.

The postprocessor_vagrant is now included for all the hypervisors except aws.

VAGRANT_FILE_ALREADY_EXISTS is now EXIT_VAGRANT_FILE_ALREADY_EXISTS

Finally, the postprocessor_vagrant.json was not required at all by the vsphere hypervisor.

Bromulux · 2019-02-28T16:33:51Z

The feature is ready for a second review.

obilodeau

LGTM, I will test the changes and docs by deploying a VM in the cloud before merging.

obilodeau · 2019-08-14T19:27:10Z

With #123 I am now able to try this PR into our CI system. Here's the result:

Fetching all templates...
Traceback (most recent call last):
  File "/var/jenkins_home/.local/bin/malboxes", line 9, in <module>
    load_entry_point('malboxes', 'console_scripts', 'malboxes')()
  File "/var/jenkins_home/workspace/malboxes-custom-branch-smoke-test/malboxes/__init__.py", line 20, in main
    from malboxes.malboxes import main
  File "/var/jenkins_home/workspace/malboxes-custom-branch-smoke-test/malboxes/malboxes.py", line 34, in <module>
    import boto3
ImportError: No module named 'boto3'

pip doesn't install boto3, I'm going to investigate as to why.

obilodeau · 2019-08-14T21:20:41Z

Modified build system and now this error is failing the build with 894741f

obilodeau · 2019-08-15T14:30:54Z

Smoke tests passed yesterday on this branch. I'll build a box today and merge if it's successful.

obilodeau · 2019-08-22T17:44:08Z

Several failed builds yesterday and today. Latest issue is:

* Post-processor failed: Failed to start import from s3://malboxes-images/packer-import-1566493516.ova: InvalidParameter: The service role <vmimport> does not exist or does not have sufficient permissions for the service to continue
	status code: 400, request id: 2e1c409b-4a5e-4b27-8387-e74123c3b640

I think we forgot to document the vmimport role requirement.

obilodeau · 2019-08-27T18:27:50Z

More tests around this yesterday and today. I think I've got our IT to get the AWS permissions sorted out.

Now there's rsync that should be disabled.

obilodeau · 2019-08-27T20:37:14Z

I was able to successfully deploy to AWS today. I've pushed a few fixes but this is good to go now!

Etienne Lacroix added 16 commits February 11, 2019 16:02

AWS Support: Packer portion

be5313d

AWS Support: Vagrant portion

0b4db0d

ISO download URL fixed for Windows10_64

a17f830

Deprecated URL for Windows10_64 has been replaced by a working one.

Build command for AWS

804f864

The build command and the end message are now adapted for the cloud feature.

Updated description of the AMI

079614a

The AMI description is no longer the default one and now gives information about the packer template used.

Documentation: AWS part

10ad46e

The code related to the AWS feature for Windows 10 is now documented,

AWS Support: Windows 7

bfb7fce

The AWS feature now supports Windows 7 64x

Manifest.json no longer needed

171e51b

The AMI ID is now retrieved from AWS by the library Boto 3 instead of a post-processor. Plus, an error message was added if you try to build more than once the same template.

Boto3 now gets AWS creds from config.js

59104ed

Boto3 uses the AWS credentials from the config.js file instead of the ones stored in environmental variables.

AWS region is now in config

e5091ac

The region can now be set in config.js

Fix WinRM inbound rules

e720a13

WinRM is now working even if the network profile is configured as public

Updated requirements for installation

a136aea

Boto3 is now in the requirements.txt

Instance type and group added to config

c5e29c8

The instance type and the security group can now be set by the user in the config.js

WinRM firewall rule added

e182f88

As a workaround for the public network connectivity problem, a new rule has been created with a public network type.

AWS support for win7 32-bit

3421419

Windows 7 32-bit can now be run on AWS

README and minor corrections

ffde781

It is now explained how to use the AWS feature of Malboxes in the README.

obilodeau suggested changes Feb 26, 2019

View reviewed changes

obilodeau mentioned this pull request Feb 26, 2019

Fix Windows 10 download links pointing to care.dlservice.microsoft.com #116

Merged

Etienne Lacroix added 10 commits February 27, 2019 11:04

Correction of README

e795a5f

General corrections of the README requested by obilodeau.

Imports’ order adjusted

a114e2a

The imports now respect the PEP 8.

Constants for exit status codes

fad539e

Constants were added to describe the exit status codes.

Refactoring of is_template_already_ami

d69d318

The function now follows the DRY principle and has a more precise name.

Length of lines is now below 80 chars

2f0ccab

Each line of the code related to AWS has now 80 chars or less.

AWS simpler build message

03d84a9

Details about the vagrant box were removed in the build message because the user does not need to know them.

New lines between functions

95a802d

Now there's two new lines between functions to respect the PEP 8.

Fix to not affect ESXI

13460aa

The postprocessor_vagrant is now included for all the hypervisors except aws.

Fix forgotten name for an exit status

f287550

VAGRANT_FILE_ALREADY_EXISTS is now EXIT_VAGRANT_FILE_ALREADY_EXISTS

ESXI correction for windows10 32bits

8077bee

Finally, the postprocessor_vagrant.json was not required at all by the vsphere hypervisor.

obilodeau approved these changes Aug 6, 2019

View reviewed changes

Added boto3 requirement to setup.py

fa71f33

doc: added a missing step in the guide

2911d15

obilodeau added 2 commits August 27, 2019 16:09

aws: prevent rsync failures on "vagrant up"

a40b989

docs: some rough notes on AWS support

aafec65

obilodeau merged commit 4ecd059 into GoSecure:master Aug 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS feature for Malboxes #115

AWS feature for Malboxes #115

Bromulux commented Feb 20, 2019

obilodeau left a comment

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

obilodeau Feb 26, 2019

Bromulux commented Feb 28, 2019

obilodeau left a comment

obilodeau commented Aug 14, 2019

obilodeau commented Aug 14, 2019

obilodeau commented Aug 15, 2019

obilodeau commented Aug 22, 2019

obilodeau commented Aug 27, 2019

obilodeau commented Aug 27, 2019

		@@ -40,6 +41,12 @@ https://github.com/gosecure/malboxes

		apt install vagrant git python3-pip

		=== AWS

		@@ -136,6 +143,50 @@ For example:

		malboxes spin win7_32_analyst 20160519.cryptolocker.xyz

		=== AWS


		. Your instance also requires a link:https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#CreatingSecurityGroups[security group] with at least a rule allowing inbound connections for WinRM (Type: WinRM-HTTP, Protocol: TCP, Port Range: 5985, Source: host's public IP).

		. If the default config is used, change the hypervisor to aws and fill the mandatory options related. Otherwise, be sure to add all the options about AWS to your custom config.


		==== RDP

		To connect to an instance on the cloud using RDP, run this command at the same location of your `+Vagrantfile+`:


		For this to work, the instance will require a security group allowing RDP inbound connections (Type: RDP, Protocol: TCP, Port Range: 3389, Source: host's public IP).

		NOTE: You can safely ignore the following error because rsync is not yet implemented: << No host IP was given to the Vagrant core NFS helper. This is an internal error that should be reported as a bug. >>

AWS feature for Malboxes #115

AWS feature for Malboxes #115

Conversation

Bromulux commented Feb 20, 2019

obilodeau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bromulux commented Feb 28, 2019

obilodeau left a comment

Choose a reason for hiding this comment

obilodeau commented Aug 14, 2019

obilodeau commented Aug 14, 2019

obilodeau commented Aug 15, 2019

obilodeau commented Aug 22, 2019

obilodeau commented Aug 27, 2019

obilodeau commented Aug 27, 2019