Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose asyncio to python3 rplugins, removing python3.3 support #294

Merged
merged 1 commit into from
Mar 1, 2018

Conversation

bfredl
Copy link
Member

@bfredl bfredl commented Jan 9, 2018

Currently, handling events beyond nvim itself is very limited, the only way additional sources can be added is with separate threads invoking nvim.async_call. A problem is that the implemenation is dynamic between two event loops: asyncio and pyuv, so plugins can't use them.

The plan is to require asyncio on python3 so we can expose asyncio event loop to python3 rplugins. Python3.3 is hereby unsupported. Python2.7 rplugins is considered legacy: existing py2.7 rplugins remain supported for now, but new features might not be available (python2 via vim :python interface remain supported of course)

I know issues on windows has held up this change in the past (#243). We might need to change the asyncio backend on windows. IIRC there is even a libuv based backend, but I don't know it's status. I will look into this tomorrow.

I will make example and test this with subprocess communication, using msgpack to comminucate with a python helper process. I know deoplete wants to use the feature for multiprocess. Ping @Shougo

@Shougo
Copy link
Contributor

Shougo commented Jan 10, 2018

Thanks. I want to use the feature for deoplete.

@Shougo
Copy link
Contributor

Shougo commented Jan 23, 2018

I will make example and test this with subprocess communication, using msgpack to comminucate with a python helper process. I know deoplete wants to use the feature for multiprocess. Ping @Shougo

@bfredl Ping.

@bfredl
Copy link
Member Author

bfredl commented Jan 23, 2018

Sorry for the delay, this works so far:

import neovim
import asyncio
import sys
from functools import partial

class MySubprocess(asyncio.SubprocessProtocol):
    def __init__(self, plugin):
        self.plugin = plugin
        self.nvim = plugin.nvim

    def connection_made(self, transport):
        self.nvim.async_call(self.plugin._on_connection, transport)
        self._transport = transport
        if isinstance(transport, asyncio.SubprocessTransport):
            self._transport = transport.get_pipe_transport(0)

    def pipe_data_received(self, fd, data):
        self.nvim.async_call(self.plugin._on_output, fd, data)

    def process_exited(self):
        pass

@neovim.plugin
class TestPlugin(object):

    def __init__(self, nvim):
        self.nvim = nvim

    @neovim.function("SubTest", sync=True)
    def subtest(self, args):
        code = "print('test')"
        create = self.nvim.loop.subprocess_exec(partial(MySubprocess,self),
                                  sys.executable, '-c', code,
                                  stdin=None, stderr=None)
        hnd = self.nvim.loop.create_task(create)

    def _on_connection(self, transport):
        self.nvim.out_write("connected!\n")
        self.sub_stdin = transport.get_pipe_transport(0)
        #self.sub_stdin.write(b"spam")

    def _on_output(self, fd, data):
        self.nvim.out_write(repr(fd)+" "+repr(data)+"\n")

But it is not very convenient. We probably want nvim.subprocess(out_cb,err_cb) wrapper or at least result = nvim.yieldfrom(coro) so that coroutines can be used inside greenlet handlers (i e method calls from nvim)

@Shougo
Copy link
Contributor

Shougo commented Jan 23, 2018

Thank you!

I will check it.

@Shougo
Copy link
Contributor

Shougo commented Feb 15, 2018

It is conflicting.
@bfredl Can you fix the conflict?

@Shougo
Copy link
Contributor

Shougo commented Feb 28, 2018

@bfredl Ping.

@bfredl
Copy link
Member Author

bfredl commented Feb 28, 2018

Sorry for the delay. I will add some documentation also tomorrow maybe, then we can merge this. Wrappers can be a follow-up PR. I e things like #72 can be implement much simpler, no need to traverse layers, can just implement thin wrapper directly in nvim module.

@Shougo
Copy link
Contributor

Shougo commented Mar 1, 2018

Thanks. I will add support the feature in deoplete later.

@Shougo
Copy link
Contributor

Shougo commented Mar 1, 2018

@bfredl I want to write input to sub process in asyncio loop.
It is possible?

@bfredl bfredl force-pushed the expose_asyncio branch 2 times, most recently from d1c9c31 to 55f751d Compare March 1, 2018 17:32
@bfredl
Copy link
Member Author

bfredl commented Mar 1, 2018

@Shougo Yes, in the example above, in _on_commection, call transport.get_pipe_transport(0) to get the stdin pipe. On this object you can call .write(data).

Disallow python3.3
python2.7 is unchanged (pyuv is still preferred; loop not exposed)
@bfredl bfredl merged commit e192f03 into neovim:master Mar 1, 2018
@justinmk justinmk changed the title [WIP] Expose asyncio to python3 rplugins, removing python3.3 support Expose asyncio to python3 rplugins, removing python3.3 support Mar 1, 2018
bfredl added a commit that referenced this pull request Mar 1, 2018
In this release support of python3.3 is dropped. Henceforth we want python3
rplugins to be able to assume the usage of asyncio, so they can use the asyncio
event loop and libraries that build on it.

Furthermore, a close() method is added on nvim session objects. When used as
a library for externally connecting to a nvim instance (i e not rplugins),
it is recommended to call the close() method on the session object when it is
not needed anymore. Alternatively, sessions can be used as a context manager:

    with neovim.attach('socket', path=thepath) as nvim:
        # do stuff with nvim session in this block:
        print(nvim.funcs.getpid())
        print(nvim.current.line)

This will close the session automatically.

Changes since 0.2.1:
* 2689ddc add tests for plugin decorators #298
* 63f257f allow library users to properly cleanup the event loop #303
* 59c184f expose the asyncio event loop as nvim.loop (python 3.4+ only) #294
@Shougo
Copy link
Contributor

Shougo commented Mar 2, 2018

Thank you.

        create = self.nvim.loop.subprocess_exec(partial(MySubprocess,self),
                                  sys.executable, '-c', code,
                                  stdin=None, stderr=None)

I think if stdin is None, the input feature will be disabled.

@Shougo
Copy link
Contributor

Shougo commented Mar 5, 2018

@bfredl Ping.

@Shougo
Copy link
Contributor

Shougo commented Mar 5, 2018

And I think the feature is not supported in Windows.

@Shougo
Copy link
Contributor

Shougo commented Mar 6, 2018

#294 (comment)

The test is works for me.

Hm.
I will look into it later.

@Shougo
Copy link
Contributor

Shougo commented Mar 6, 2018

@bfredl No problem. deoplete asyncio works for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants