execnet error when using ploy configure (examples from quickstart, webserver tutorials)

[wolfgangpfnuer]

the below output was reproducable on a setup where the tutorial was followed during the training at europython.

venv was installed with python -m virtualenv
bin/activate was called
ploy ssh webserver / demo_jail both work without a problem

pkg install nginx worked without a problem inside the jail, so internet connection is not the problem.

pip freeze
Fabric==1.9.0
Jinja2==2.7.3
MarkupSafe==0.23
PyYAML==3.11
ansible==1.6.6
bsdploy==1.0.0
ecdsa==0.11
execnet==1.2.0
lazy==1.2
paramiko==1.14.0
ploy==1.0.0
ploy-ansible==1.0.0
ploy-ezjail==1.0.0
ploy-fabric==1.0.0
ploy-virtualbox==1.0.0
pycrypto==2.6.1
wsgiref==0.1.2

(lala)Retina Wolfgang 2012:ploy-quickstart wpfnuer$ ploy configure webserver
INFO: Using playbook at '/Users/wpfnuer/lala/ploy-quickstart/jailhost-webserver.yml'.

PLAY [jailhost-webserver] *****************************************************

GATHERING FACTS ***************************************************************
Write failed: Broken pipe
fatal: [jailhost-webserver] => Traceback (most recent call last):
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ansible/runner/init.py", line 536, in _executor
exec_rc = self._executor_internal(host, new_stdin)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ansible/runner/init.py", line 638, in _executor_internal
return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ansible/runner/init.py", line 832, in _executor_internal_inner
result = handler.run(conn, tmp, module_name, module_args, inject, complex_args)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ansible/runner/action_plugins/normal.py", line 57, in run
return self.runner._execute_module(conn, tmp, module_name, module_args, inject=inject, complex_args=complex_args)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ansible/runner/init.py", line 419, in _execute_module
tmp = self._make_tmp_path(conn)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ansible/runner/init.py", line 1046, in _make_tmp_path
result = self._low_level_exec_command(conn, cmd, None, sudoable=False)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ansible/runner/init.py", line 974, in _low_level_exec_command
in_data=in_data)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ploy_ansible/execnet_connection.py", line 90, in exec_command
rc, stdout, stderr = self.rpc.exec_command(remote_cmd)
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/ploy_ansible/execnet_connection.py", line 17, in call
self.channel.send((name, args, kw))
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/execnet/gateway_base.py", line 681, in send
self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
File "/Users/wpfnuer/lala/lib/python2.7/site-packages/execnet/gateway_base.py", line 950, in _send
raise IOError("cannot send (already closed?)")
IOError: cannot send (already closed?)

TASK: [install nginx] *********************************************************
FATAL: no hosts matched or all hosts have already failed -- aborting

[fschulze]

What OS was this on? Is that the default system Python? Does this still happen with the latest releases?

[fbarthez]

I'm getting a similar error on OS X 10.9.5. I cloned the repo, ran make, then created the ploy-quickstart directory one level up and ran "virtualenv ." followed by "source ../bsdploy/bin/activate".

(bsdploy)[fabian@toubab ~/Documents/gitolite/ploy-quickstart] % ploy configure demo_jail
INFO: Using playbook at '/Users/fabian/Documents/gitolite/ploy-quickstart/jailhost-demo_jail.yml'.

PLAY [jailhost-demo_jail] *****************************************************

GATHERING FACTS ***************************************************************
Write failed: Broken pipe
fatal: [jailhost-demo_jail] => Traceback (most recent call last):
File "/Users/fabian/Documents/gitolite/bsdploy/lib/python2.7/site-packages/ansible/runner/init.py", line 561, in _executor
exec_rc = self._executor_internal(host, new_stdin)
File "/Users/fabian/Documents/gitolite/bsdploy/lib/python2.7/site-packages/ansible/runner/init.py", line 666, in _executor_internal
return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
File "/Users/fabian/Documents/gitolite/bsdploy/lib/python2.7/site-packages/ansible/runner/init.py", line 884, in _executor_internal_inner
result = handler.run(conn, tmp, module_name, module_args, inject, complex_args)
File "/Users/fabian/Documents/gitolite/bsdploy/lib/python2.7/site-packages/ansible/runner/action_plugins/normal.py", line 57, in run
return self.runner._execute_module(conn, tmp, module_name, module_args, inject=inject, complex_args=complex_args)
File "/Users/fabian/Documents/gitolite/bsdploy/lib/python2.7/site-packages/ansible/runner/init.py", line 446, in _execute_module
tmp = self._make_tmp_path(conn)
File "/Users/fabian/Documents/gitolite/bsdploy/lib/python2.7/site-packages/ansible/runner/init.py", line 1086, in _make_tmp_path
result = self._low_level_exec_command(conn, cmd, None, sudoable=False)
File "/Users/fabian/Documents/gitolite/bsdploy/lib/python2.7/site-packages/ansible/runner/init.py", line 1025, in _low_level_exec_command
in_data=in_data)
File "/Users/fabian/Documents/gitolite/bsdploy/src/ploy_ansible/ploy_ansible/execnet_connection.py", line 98, in exec_command
rc, stdout, stderr = self.rpc.exec_command(remote_cmd)
File "/Users/fabian/Documents/gitolite/bsdploy/src/ploy_ansible/ploy_ansible/execnet_connection.py", line 18, in call
self.channel.send((name, args, kw))
File "/Users/fabian/Documents/gitolite/bsdploy/eggs/execnet-1.2.0-py2.7.egg/execnet/gateway_base.py", line 681, in send
self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
File "/Users/fabian/Documents/gitolite/bsdploy/eggs/execnet-1.2.0-py2.7.egg/execnet/gateway_base.py", line 950, in _send
raise IOError("cannot send (already closed?)")
IOError: cannot send (already closed?)

TASK: [install nginx] *********************************************************
FATAL: no hosts matched or all hosts have already failed -- aborting

[tomster]

i've just used a fresh checkout of this repo and could not reproduce the error.

@fbarthez could you post the output from bin/ploy -v?

[fbarthez]

(bsdploy)[fabian@toubab ~/Documents/gitolite/ploy-quickstart] % ploy -v
bsdploy 1.3-dev (/Users/fabian/Documents/gitolite/bsdploy)
ploy 1.0.3dev (/Users/fabian/Documents/gitolite/bsdploy/src/ploy)
ploy-ansible 1.2.1dev (/Users/fabian/Documents/gitolite/bsdploy/src/ploy_ansible)
ploy-ec2 1.1.1dev (/Users/fabian/Documents/gitolite/bsdploy/src/ploy_ec2)
ploy-ezjail 1.1.1dev (/Users/fabian/Documents/gitolite/bsdploy/src/ploy_ezjail)
ploy-fabric 1.1.1dev (/Users/fabian/Documents/gitolite/bsdploy/src/ploy_fabric)
ploy-virtualbox 1.1.0dev (/Users/fabian/Documents/gitolite/bsdploy/src/ploy_virtualbox)

[tomster]

@fbarthez damn, that's exactly what it should be...

this is really baffling (i remember the original issue with @wolfgangpfnuer quite well, even debugging it directly on his machine at the time didn't lead anywhere)

@fschulze AFAIR you added using execnet for ssh connections for performance reasons, right? what would be an easy way to (temporarily) deactivate it? perhaps it might work then or at least we might see more helpful error messages.

[fschulze]

Can you ssh in? If not, what error do you get there? If you can ssh in, is python installed? If so at which path?

[fbarthez]

(bsdploy)[fabian@toubab ~/Documents/gitolite/ploy-quickstart] % ploy ssh demo_jail
Last login: Sat Nov 15 15:40:52 2014 from 10.0.0.1
FreeBSD 10.0-RELEASE-p12 (GENERIC) #0: Tue Nov 4 05:07:17 UTC 2014

Gehe nicht über Los.
root@demo_jail:~ # which python
python: Command not found.

jailhost also has no python installed.

[fschulze]

Try python27 or python2.7 (not sure right now) as well. Ansible and our custom connection using execnet require python. I think we set it to /usr/local/bin/python27 by default in bsdploy. If it's not installed, it fails with this rather crappy error message.

[fbarthez]

(bsdploy)[fabian@toubab ~/Documents/gitolite/ploy-quickstart] % ploy ssh demo_jail
Last login: Sun Nov 16 11:23:41 2014 from 10.0.0.1
FreeBSD 10.0-RELEASE (GENERIC) #0 r260789: Thu Jan 16 22:34:59 UTC 2014

Gehe nicht über Los.
root@demo_jail:~ # which python27
python27: Command not found.
root@demo_jail:~ # which python2.7
/usr/local/bin/python2.7

I did

root@demo_jail:~ # ln -s /usr/local/bin/python2.7 /usr/local/bin/python27
root@demo_jail:~ # ln -s /usr/local/bin/python2.7 /usr/local/bin/python

but the error stays the same.

[tomster]

@wolfgangpfnuer are you still willing to give this another try with the currently released 2.0.0 version?

[tomster]

And dito for @fbarthez of course :-)

[fbarthez]

I've had a quick go, but I don't get very far. After make clean && git pull && bin/develop up -v, deleting all existing ploy-related VMs and starting with the quickstart guide from scratch:

(bsdploy)[fabian@toubab ~/Documents/gitolite/bsdploy/ploy-quickstart] % ploy start ploy-demo
INFO: Creating instance 'ploy-demo'
ERROR: The host only interface 'vboxnet0' has an IP '192.168.22.1' that doesn't match the config '192.168.56.1'.

This may be a remnant from my earlier experiments? The corresponding configuration has vanished from the etc/ploy.conf file as per the quickstart guide.

This is still on OSX 10.9.5.

[fschulze]

@fbarthez if you don't need that specific IP (192.168.22.x) for your VirtualBox host only interface, then just change it to '192.168.56.1' and you should be able to continue.

[fbarthez]

I've tried to follow the quickstart tutorial, which in the current version does not contain any explicit IP configuration. I tried adding the following to /etc/ploy.conf but w/o effect:

[vb-hostonlyif:vboxnet0]
ip = 192.168.56.1

The virtualbox UI shows only a single network adaptor, which is configured for NAT.

[fschulze]

Try using the VBoxManage command:

VBoxManage list hostonlyifs
VBoxManage hostonlyif remove vboxnet0

Once the one with IP 192.168.22.x is removed, try ploy again. It should then create a new one with the correct IP.

[fbarthez]

Thanks, that did help! I ended up with the same "Write failed: Broken pipe" when running ploy configure demo_jail, but this time it helped to create a symlink for python27:

ln -s /usr/local/bin/python2.7 /usr/local/bin/python27

[tomster]

@fbarthez i'm very curious, where /usr/local/bin/python27 is coming from.

it certainly isn't part of bsdploy.

is there perhaps a typo in your following the quickstart or perhaps a global ansible configuration on your system?

[fbarthez]

Well, if I remove the link, I'm back to "Write failed: Broken pipe" on calling "ploy configure demo_jail". The alternative path was suggested by @fschulze.

My global ansible config:

(bsdploy)[fabian@toubab ~/Documents/gitolite/bsdploy/ploy-quickstart] % cat ~/.ansible.cfg
[defaults]
transport = smart

[ssh_connection]
pipelining = True

[tomster]

@fbarthez well, @fschulze only said he wasn't sure which spelling is the correct one and the correct one is definitely python2.7.

i'm wondering who or what is actually requesting python2.7 and thus failing w/o the symlink you mention.

have you tried grepping forpython2.7 (you will get a lot of hits for the pkg name for python 2.7 but besides those, can you find anything inside your project folder or your ploy installation?

i'm really curious, what this is :)

[fbarthez]

python2.7 is the name of the executable getting installed, so I'm guessing you'd like to see references to python27:

(bsdploy)[fabian@toubab ~/Documents/gitolite/bsdploy/ploy-quickstart] % ack python27
(bsdploy)[fabian@toubab ~/Documents/gitolite/bsdploy/ploy-quickstart] % 

So no python27 in the quickstart config – which seems unsurprising given I'm lazy and have copied verbatim from the docs.

(bsdploy)[fabian@toubab ~/Documents/gitolite/bsdploy] % ack python27
bsdploy/bootstrap_utils.py
445:        packages.update(['python27'])

bsdploy/fabfile_daemonology.py
44:    bu.install_pkg('/', chroot=False, packages=['python27'])

bsdploy/fabfile_mfsbsd.py
127:    bootstrap_packages = ['python27']

bsdploy/startup-ansible-jail.sh
7:pkg install python27

bsdploy/tests/test_bootstrap_daemonology.py
33:        ('pkg install python27', {'shell': False}, '')]

bsdploy/tests/test_bootstrap_mfsbsd.py
142:        ('chroot /mnt pkg install python27', {'shell': False}, ''),

bsdploy/tests/test_bootstrap_utils.py
232:        ('wget -c -O "%(tempdir)s/downloads/packages/freebsd:10:x86:64/latest/All/python27-2.7.6_4.txz" "http://pkg.freebsd.org/freebsd:10:x86:64/latest/All/python27-2.7.6_4.txz"' % format_info, {}, ''),

docs/advanced/customizing-bootstrap.rst
59:    pkg install python27

docs/quickstart.rst
101:    python27-2.7.9                 Interpreted object-oriented programming language
154:    python27-2.7.9                 Interpreted object-oriented programming language

docs/setup/provisioning-ec2.rst
68:        firstboot_pkgs_list="python27"

lib/python2.7/site-packages/setuptools/msvc9_support.py
60:                    message + ' Get it from http://aka.ms/vcpython27'

lib/python2.7/site-packages/setuptools/tests/test_msvc9compiler.py
77:        directing the user to download vcpython27.
89:                assert 'aka.ms/vcpython27' in str(exc)

lib/python2.7/site-packages/zc/buildout/buildout.txt
1204:python27

lib/python2.7/site-packages/zc/buildout/configparser.test
320:  [s2: any([python2, python3, python24 , python25 , python26 , python27 , python30 , python31 , python32 , python33 , python34 , python35 , python36]) ] 

lib/python2.7/site-packages/zc.buildout-2.3.1-py2.7.egg-info/PKG-INFO
1323:        python27