Dynamic completion candidates update? #471

mrbiggfoot · 2017-04-29T00:32:06Z

Different sources have different speeds. Some are very fast - like "buffer" and "around", some may take many seconds to gather candidates from, e.g. "tag" source. Can we show the completion candidates as they become avaliable from the source? I.e. when I start typing, I should instantly get completions from "buffer" and "around", and if I'm willing to wait for so many seconds - the candidates from "tag" would appear and the candidates list will be dynamically redrawn.

brunogsa · 2017-04-30T02:05:19Z

+1

Shougo · 2017-05-01T01:34:06Z

Yes, it is on the todo list.

mrbiggfoot · 2017-05-05T01:17:37Z

I implemented this deoplete source to help dealing with 'tags' slowness in the meantime:
https://github.com/mrbiggfoot/deoplete-filesrc

It is pretty dumb, but works for me. @Shougo if you want you can improve on it and put it into deoplete since it's generic enough.

Shougo · 2017-07-05T03:15:21Z

I will work for the issue after startup optimization.

Shougo · 2017-08-10T02:13:27Z

It should be ver.4.0.

Shougo · 2017-10-19T08:56:51Z

It is the next task after Vim 8 support.
Please wait.

balta2ar · 2017-11-11T09:34:05Z

@Shougo Hi, I'd like to clarify things about is_async flag of gather_candidates.
I have a completion source that is somewhat slow. It takes about 0.3s to generate results. If I type some input, these 0.3s accumulate after each typed letter and I have to wait until previous requests are processed (even though I don't need them already). But I want to cancel the async request if new input arrives.

I've read the docs here (https://github.com/Shougo/deoplete.nvim/blob/master/doc/deoplete.txt#L980) and https://github.com/zchee/deoplete-jedi/blob/master/rplugin/python3/deoplete/sources/deoplete_jedi.py#L168 source. To do what I want I need to implement some kind of internal worker thread. When gather_candidates is called, I submit my request to my internal queue, set is_async flag to True, and then return an empty list, right?

Next time when gather_candidates is called, I need to check 2 things:

If complete_str has changed, I should cancel my current request and summit a new one.
Otherwise, I should check whether the request has completed and if it has, set is_async to False, and return an actual list of candidates.

Also, even though the word async is extensively used here, it has nothing to do with async coroutines in Python 3.4..3.6, right?

Is my understanding correct? Please correct me if I'm wrong.

balta2ar · 2017-11-11T10:08:43Z

It looks like what I need is the same is_async flag but for on_post_filter function. The reason is that I do my custom filtering both in gather_candidates and in on_post_filter. I do not rely on denite's filtering because it didn't match my needs well. I produced best results if I filtered on my own.

Is it possible to make on_post_filter async too?

Is there another way? If I need to apply a custom filter and that filter is known to be slow, what do I do? I would like to be able to cancel filtering requests if the input (complete_str) changes.

balta2ar · 2017-11-11T12:11:13Z

Apparently VSCode has something very close to my needs, specifically Promises and Cancellation Tokens: https://code.visualstudio.com/docs/extensionAPI/patterns-and-principles#_cancellation-tokens
Could something similar be implemented in deoplete?

balta2ar · 2017-11-11T18:34:00Z

Could you please explain, how and when to use this flag? The documentation is a bit scarce...

is_volatile
		(Bool)				(Optional)
If it is True, the source depends on the user input.

Shougo · 2017-11-14T03:48:02Z

When gather_candidates is called, I submit my request to my internal queue, set is_async flag to True, and then return an empty list, right?

Yes.

Shougo · 2017-11-14T03:49:47Z

Also, even though the word async is extensively used here, it has nothing to do with async coroutines in Python 3.4..3.6, right?

You don't have to use async coroutines.

Shougo · 2017-11-14T03:52:35Z

Could you please explain, how and when to use this flag? The documentation is a bit scarce...

If the flag is enabled, deoplete does not use the internal cache.
The volatile source is called when the input is changed everytime.

Shougo · 2017-11-14T05:09:03Z

I do not rely on denite's filtering because it didn't match my needs well. I produced best results if I filtered on my own.

I don't understand what is the your filter.

Is it possible to make on_post_filter async too?

I don't understand it.
on_post_filter is called after gather_candidates().
So if gather_candidates() is async, on_post_filter is also async.

Shougo · 2017-11-14T05:24:24Z

I would like to be able to cancel filtering requests if the input (complete_str) changes.

It should be new feature.
Please create the new issue.

balta2ar · 2017-11-14T21:24:57Z

Also, even though the word async is extensively used here, it has nothing to do with async coroutines in Python 3.4..3.6, right?

You don't have to use async coroutines.

How are you going to implement dynamic update them? I'm curious. A callback to update the results?

I don't understand what is the your filter.

It's basically the same fuzzy filtering & ranking in one place but done using a Python C extension. Nothing fancy though.

Is it possible to make on_post_filter async too?

I don't understand it.
on_post_filter is called after gather_candidates().
So if gather_candidates() is async, on_post_filter is also async.

Hm, not exactly. Suppose that I'm going against the suggestions and I'm doing my own filtering in gather_candidates and in on_post_filter. And that filtering is time-consuming. I could offload the filtering into a background thread in gather_candidates and use is_async flag, but then without is_volatile flag the method on_post_filter is still called on every input change. I can't return partial results from on_post_filter, right?

In my case, to be able to cancel time-consuming filtering in the middle (for example, if the input has changed) and to not block deoplete, I had to do the following:

Do the filtering in a separate thread and use is_async flag in gather_candidates.
Make sure the filtering is granular and can be canceled by a flag. The flag is frequently checked by the thread.
Remove on_post_filter and do the filtering in gather_candidates.
Set is_volatile=True.

It should be new feature.
Please create the new issue.

I should probably think more about the use cases. Now it looks like just detecting a change in complete_str is enough. Do you have any use cases in mind?

Shougo · 2017-11-15T03:07:07Z

How are you going to implement dynamic update them?

The solution is multi-process. async coroutines does not save the calculation.

It's basically the same fuzzy filtering & ranking in one place but done using a Python C extension. Nothing fancy though.

Hm. I can add matcher_cpsm like denite.nvim.

Shougo · 2017-11-15T03:19:16Z

but then without is_volatile flag the method on_post_filter is still called on every input change.

Yes. It is the feature.

I can't return partial results from on_post_filter, right?

Right.

Shougo · 2017-11-15T03:22:15Z

The current deoplete behavior is hard to cancel the results for deoplete sources.
It should be fixed.

The input change should be detected by is_async flag.

balta2ar · 2017-11-15T08:30:01Z

Hm. I can add matcher_cpsm like denite.nvim.

That would be useful! Could you please add it? They say they're using multithreading to parallelise matching so it sounds promising. It could be faster than my current solution.

The current deoplete behavior is hard to cancel the results for deoplete sources.
It should be fixed.

On top of that any slow source can slow down the whole deoplete plugin. Having sources in separate processes could solve this issue probably. Another improvement could be to call all sources in parallel (this is what nvim-completion-manager claims to do).

Shougo · 2017-11-15T08:46:04Z

Another improvement could be to call all sources in parallel (this is what nvim-completion-manager claims to do).

I have the similar solutions. But it takes time to implement. Not easy.
I will improve source async feature before that.

Shougo · 2017-11-23T05:17:04Z

@balta2ar Please use is_async flag to check the input.
If the input is changed, it will be False.

Shougo · 2017-11-23T05:41:01Z

@balta2ar I have added matcher_cpsm in deoplete.

balta2ar · 2017-11-23T09:14:14Z

Please use is_async flag to check the input.
If the input is changed, it will be False

Sorry, I don't quite understand it, why do I need a flag to detect input change? The source plugin can remember last input string and compare with the new one, thus detecting the change by itself. Is there something else to that?

Also, using the same flag both as input and output for different purposes ("input changed" versus "call me again soon") could be very confusing...

Thank you for the cpsm, I'll try it!

Shougo · 2017-11-23T12:16:34Z

Sorry, I don't quite understand it, why do I need a flag to detect input change? The source plugin can remember last input string and compare with the new one, thus detecting the change by itself. Is there something else to that?

Yes. But all async source needs the feature.
So deoplete should support it.

Also, using the same flag both as input and output for different purposes ("input changed" versus "call me again soon") could be very confusing...

OK. I have changed the flag.
Please use is_refresh instead.

balta2ar · 2017-11-23T12:57:51Z

Yes. But all async source needs the feature.

Could you give an example, please? I have an async source and I remember the previous input and compare it with the current one, and it's working fine. I may be missing something.

Please use is_refresh instead

May I suggest input_changed flag name? I think it more clearly describes the meaning...

mrbiggfoot · 2018-02-07T00:14:57Z

Frankly, I don't know the exact server configuration, but it is definitely not windows. The above hardware specs is what I see inside the VM, i.e. the provisioned resources.

mrbiggfoot · 2018-02-07T00:24:02Z

Latest master (3dee4b1) with profile settings:
deoplete.log
attempted the same completion multiple times.

mrbiggfoot · 2018-02-07T00:33:03Z

FWIW - the behavior is no any different on the latest nightly build of neovim.

mrbiggfoot · 2018-02-07T00:37:14Z

Is there any log that I can turn on that would help to debug ":qa!" issue?

Shougo · 2018-02-07T01:41:36Z

I think in VM timer interrupt is very slow.
So deoplete parallel is very slow.

I cannot fix the problem.
But I will add the serial support in deoplete later.

#638

Shougo · 2018-02-07T01:53:38Z

Is there any log that I can turn on that would help to debug ":qa!" issue?

I think it is same problem with #637.
It depends on your configuration and the installed plugins.
Please test it in minimal init.vim.

Shougo · 2018-02-07T01:54:05Z

Please test the latest version of deoplete in non VM environment.
I think it is faster.

Shougo · 2018-02-07T02:56:17Z

Latest master (3dee4b1) with profile settings:
deoplete.log
attempted the same completion multiple times.

I have checked the log.
Switching overhead is 100ms ~ 200ms.
It is very high.

mrbiggfoot · 2018-02-07T02:58:42Z

Please test the latest version of deoplete in non VM environment.

Even if it is faster, it's of no use to me. Sorry, but my work environment is not going to change for some objective reasons. I think I will just stay on the older version for now, which mostly works for me. Thank you!

Shougo · 2018-02-07T03:02:29Z

Even if it is faster, it's of no use to me. Sorry, but my work environment is not going to change for some objective reasons.

I know it. But I want to know the slowness is whether VM reason.

Shougo · 2018-02-07T03:03:13Z

I think I will just stay on the older version for now, which mostly works for me. Thank you!

And I will implement searial completion feature instead.

mrbiggfoot · 2018-02-07T03:05:39Z

I know it. But I want to know the slowness is whether VM reason.

Still, I can't do a fair apples to apples comparison because I don't have a CentOS running on bare metal. The only unix that I have on bare metal is MacOS, which is pretty far from CentOS. So even if I get different results, it could be something else.

Shougo · 2018-02-07T03:13:22Z

Still, I can't do a fair apples to apples comparison because I don't have a CentOS running on bare metal. The only unix that I have on bare metal is MacOS, which is pretty far from CentOS. So even if I get different results, it could be something else.

Yes. But you know CentOS bare metal is different from CentOS VM.
The performance is different.

Shougo · 2018-02-07T05:15:05Z

@mrbiggfoot Please test below patch and profile.

diff --git a/rplugin/python3/deoplete/child.py b/rplugin/python3/deoplete/child.py
index 1e14e09..f171be9 100644
--- a/rplugin/python3/deoplete/child.py
+++ b/rplugin/python3/deoplete/child.py
@@ -79,6 +79,7 @@ class Child(logger.LoggingMixin):
                     self._on_event(args[0])
                 elif name == 'merge_results':
                     self._write(self._merge_results(args[0], queue_id))
+                self.debug('main_loop: end')

     def _write(self, expr):
         sys.stdout.buffer.write(self._packer.pack(expr))
@@ -141,6 +142,7 @@ class Child(logger.LoggingMixin):
                 self.debug('Loaded Filter: %s (%s)', f.name, path)

     def _merge_results(self, context, queue_id):
+        self.debug('merged_results: begin')
         results = self._gather_results(context)

         merged_results = []
@@ -162,6 +164,7 @@ class Child(logger.LoggingMixin):

         is_async = len([x for x in results if x['context']['is_async']]) > 0

+        self.debug('merged_results: end')
         return {
             'queue_id': queue_id,
             'is_async': is_async,

Shougo · 2018-02-07T05:23:13Z

And please test the patch.

diff --git a/rplugin/python3/deoplete/process.py b/rplugin/python3/deoplete/process.py
index d1ac7b3..f6e6c47 100644
--- a/rplugin/python3/deoplete/process.py
+++ b/rplugin/python3/deoplete/process.py
@@ -52,7 +52,10 @@ class Process(object):

     def enqueue_output(self):
         while self._proc:
-            b = self._proc.stdout.read(1)
+            b = self._proc.stdout.raw.read(102400)
+            if b == b'':
+                return
+
             self._unpacker.feed(b)
             for child_out in self._unpacker:
                 self._queue_out.put(child_out)

brunogsa · 2018-02-07T16:45:08Z

Hi @Shougo

I think I'm also experiencing some issues now, after updating to the last version.
My completion stop working after a while, so I have to restart my nvim.

Also, It feels like my system is slower.
Neovim sometimes stucks, so I have to press CTRL+C to kill the process.

I'm trying to debug, following the docs:

let g:deoplete#enable_profile = 1
call deoplete#enable_logging('DEBUG', 'deoplete.log')

But now I'm retrieving an error when opening nvim:

Error detected while processing /home/brunogsa/.config/nvim/init.vim:
line  434:
E117: Unknown function: deoplete#enable_logging
Press ENTER or type command to continue

I would like to provide more useful information, coud you point me out how can I help?

Thanks for the great work!

Shougo · 2018-02-08T01:51:14Z

@brunogsa Please test the latest version of deoplete.

mrbiggfoot · 2018-02-09T01:14:31Z

@Shougo now I don't see the difference in performance (using master at a24d95e), or it is very minor. Also, the ":qa!" problem is gone. Thank you!

Shougo · 2018-02-09T01:36:00Z

OK. Cherrs.

mrbiggfoot · 2018-02-09T02:39:29Z

Sorry for spamming here, but don't yet want to open an issue for what may well be my own stupidity... the filesource https://github.com/mrbiggfoot/deoplete-filesrc has stopped working. It is not related to this change - it does not work even if I roll it back. But it used to work some time ago. Were there any changes to the source API? I'm not seeing anything relevant in the doc... perhaps you can take a quick look and spot what is wrong immediately?
https://github.com/mrbiggfoot/deoplete-filesrc/blob/master/rplugin/python3/deoplete/sources/filesrc.py
(less than one screen of trivial code, and I checked the class separately and verified that the file path g:deoplete#filesrc#path is set in vim)

Shougo · 2018-02-09T05:23:41Z

OK. I will test it.
Please upload the sample file.

Shougo · 2018-02-09T05:26:32Z

I have tested it. And it works.

I think you must expand() g:deoplete#filesrc#path.
Like this.

let g:deoplete#filesrc#path = expand('~/src/neovim/completions')

Shougo · 2018-02-09T05:27:39Z

https://github.com/mrbiggfoot/deoplete-filesrc/blob/master/rplugin/python3/deoplete/sources/filesrc.py#L23

You can use deoplete.util.expand() for it.

mrbiggfoot · 2018-02-09T07:15:06Z

Thanks for looking into it @Shougo, but that was not it. I have an absolute path set in that variable. There's something more weird going on... if I open a file like "vim file.cc", my completions from the file work. But if I open that same file via fzf (which I use from vim via fzf.vim) after staring vim, the same completions don't work! The #path var is set properly in both cases. I will investigate more. You already spent a lot of time on this.

Shougo · 2018-02-09T07:27:21Z

https://github.com/mrbiggfoot/deoplete-filesrc/blob/master/rplugin/python3/deoplete/sources/filesrc.py#L36

close() is needed. You should use with statement instead.

https://github.com/mrbiggfoot/deoplete-filesrc/blob/master/rplugin/python3/deoplete/sources/filesrc.py#L37

You can use f.readlines() instead.

Shougo · 2018-02-09T07:29:47Z

The code is broken.

    def on_event(self, context):
        if not exists(self.__filepath):
            self.__cache = []
            return

        mtime = getmtime(self.__filepath)
        if mtime == self.__mtime:
            return

        self.__mtime = mtime
        f = open(self.__filepath, 'r')
        self.__cache = list(f.read().split())

Shougo · 2018-02-09T07:31:38Z

The cache is cleared if if mtime == self.__mtime in previous code.

So,

But if I open that same file via fzf (which I use from vim via fzf.vim) after staring vim, the same completions don't work!

Yes.

It is your source's bug.

mrbiggfoot · 2018-02-09T07:38:39Z

Thank you again @Shougo, you are right. Stupid mistakes in my code.

You can use f.readlines() instead.

Not quite, readlines() does not trim the newline chars.

Shougo added the enhancement label May 1, 2017

Shougo added this to the ver.4.0 milestone Aug 10, 2017

Shougo mentioned this issue Aug 27, 2017

Deoplete API should never hang #534

Closed

Shougo added the performance label Sep 29, 2017

Shougo mentioned this issue Feb 7, 2018

Excessive CPU usge #637

Closed

Shougo mentioned this issue Apr 14, 2018

Flickering with certain set of plugins (probably async) #726

Closed

Dynamic completion candidates update? #471

Dynamic completion candidates update? #471

Comments

mrbiggfoot commented Apr 29, 2017

brunogsa commented Apr 30, 2017

Shougo commented May 1, 2017

mrbiggfoot commented May 5, 2017

Shougo commented Jul 5, 2017

Shougo commented Aug 10, 2017

Shougo commented Oct 19, 2017

balta2ar commented Nov 11, 2017

balta2ar commented Nov 11, 2017 • edited Loading

balta2ar commented Nov 11, 2017 • edited Loading

balta2ar commented Nov 11, 2017

Shougo commented Nov 14, 2017

Shougo commented Nov 14, 2017

Shougo commented Nov 14, 2017 • edited Loading

Shougo commented Nov 14, 2017 • edited Loading

Shougo commented Nov 14, 2017

balta2ar commented Nov 14, 2017

Shougo commented Nov 15, 2017

Shougo commented Nov 15, 2017

Shougo commented Nov 15, 2017

balta2ar commented Nov 15, 2017

Shougo commented Nov 15, 2017

Shougo commented Nov 23, 2017

Shougo commented Nov 23, 2017

balta2ar commented Nov 23, 2017

Shougo commented Nov 23, 2017

balta2ar commented Nov 23, 2017

mrbiggfoot commented Feb 7, 2018

mrbiggfoot commented Feb 7, 2018 • edited Loading

mrbiggfoot commented Feb 7, 2018

mrbiggfoot commented Feb 7, 2018

Shougo commented Feb 7, 2018

Shougo commented Feb 7, 2018

Shougo commented Feb 7, 2018

Shougo commented Feb 7, 2018

mrbiggfoot commented Feb 7, 2018

Shougo commented Feb 7, 2018

Shougo commented Feb 7, 2018

mrbiggfoot commented Feb 7, 2018

Shougo commented Feb 7, 2018

Shougo commented Feb 7, 2018

Shougo commented Feb 7, 2018

brunogsa commented Feb 7, 2018 • edited Loading

Shougo commented Feb 8, 2018

mrbiggfoot commented Feb 9, 2018

Shougo commented Feb 9, 2018

mrbiggfoot commented Feb 9, 2018

Shougo commented Feb 9, 2018

Shougo commented Feb 9, 2018

Shougo commented Feb 9, 2018

mrbiggfoot commented Feb 9, 2018

Shougo commented Feb 9, 2018

Shougo commented Feb 9, 2018

Shougo commented Feb 9, 2018

mrbiggfoot commented Feb 9, 2018

balta2ar commented Nov 11, 2017 •

edited

Loading

balta2ar commented Nov 11, 2017 •

edited

Loading

Shougo commented Nov 14, 2017 •

edited

Loading

Shougo commented Nov 14, 2017 •

edited

Loading

mrbiggfoot commented Feb 7, 2018 •

edited

Loading

brunogsa commented Feb 7, 2018 •

edited

Loading